backend/docs/LEGAL_REFRESH.md

# Legal Data Refresh Workflow

Use this sequence whenever new DOCX/PDF files are imported outside the user-facing UI (e.g. nightly ETL or bulk manifests).

## Prerequisites

- Postgres + Redis running.
- Celery worker online (for interactive uploads) or `CELERY_TASK_ALWAYS_EAGER=true` for synchronous runs.
- Tesseract OCR installed (see `OCR_SETUP.md`).

## Manual Command Sequence

```
cd backend/hue_portal
source ../.venv/bin/activate

python manage.py load_legal_document --file "/path/to/docx" --code DOC-123
python ../scripts/generate_embeddings.py --model legal
python ../scripts/build_faiss_index.py --model legal
```

Notes:

- `load_legal_document` can be substituted with the manifest loader (`scripts/load_legal_documents.py`) if multiple files need ingestion.
- The embedding script logs processed sections; expect a SHA checksum for each chunk.
- FAISS builder writes artifacts under `backend/hue_portal/artifacts/faiss_indexes`.

## Automated Helper

`backend/scripts/refresh_legal_data.sh` wraps the three steps:

```
./backend/scripts/refresh_legal_data.sh \
  --file "/path/to/THONG-TU.docx" \
  --code TT-02
```

Flags:

- `--skip-ingest` to only regenerate embeddings/index (useful after editing chunking logic).
- `--python` to point at a specific interpreter (default `python3`).

## CI / Nightly Jobs

1. Sync new files into `tài nguyên/`.
2. Run the helper script for each file (or call the manifest loader first).
3. Archive FAISS artifacts (upload to object storage) so the chatbot containers can download them at boot.
4. Record build duration and artifact checksums for auditing.

## Verification Checklist

- `generate_embeddings` log ends with `Completed model=legal`.
- FAISS directory contains fresh timestamped `.faiss` + `.mappings.pkl`.
- Sample chatbot query (“Thông tư 02 ...”) returns snippets referencing the newly ingested document.