Legal and compliance RAG

Legal and compliance RAG are the strictest RAG use cases. Hallucinations aren't just embarrassing, they're legal exposure. Citations aren't optional, they're required by procedure. The bar for quality is higher, and the acceptable failure modes are narrower.

What makes it different

• Accuracy is regulatory, not just aesthetic
• Citations are required and must be verifiable
• Auditability: every answer's provenance must be traceable
• Confidentiality: most content is privileged or confidential
• Document complexity is high (contracts, statutes, regulations, case law)
• Users are sophisticated and will catch errors

The content types

• Contracts and amendments
• Regulations and statutes
• Case law and precedents
• Compliance policies and procedures
• Legal memos and opinions
• Discovery documents (email, contracts, communications)
• Regulatory filings

Each has structure and citation conventions specific to legal practice.

Parsing: high stakes

Legal documents are PDF-heavy and layout-dependent:

• Contract numbering matters (Section 2.1.3 vs 2.13)
• Tables in exhibits carry specific commitments
• Footnotes carry legal weight
• Defined terms are capitalized and must be traced
• Cross-references need resolution

Commercial parsers (Azure DI, Llamaparse, Mathpix) or LLM vision parsers are often worth the cost here. See parsing PDFs.

Chunking: preserve structure

Legal chunks should align with legal units:

• Contracts: by clause or section
• Statutes: by section with surrounding context
• Case law: by paragraph with case citation metadata

Chunk metadata must include:

• Document type
• Jurisdiction
• Date
• Parties (for contracts)
• Section identifier
• Citation format

Defined terms

Legal documents define terms early and use them throughout. When retrieving a chunk that uses a defined term, the definition should come along.

Two patterns:

• Extract definitions as separate chunks, always retrieve relevant definitions alongside main content
• Prepend key definitions to every chunk (denormalized but simpler)

Citation format

Legal citations follow strict formats:

• Case law: Smith v. Jones, 123 F.3d 456 (9th Cir. 2020)
• Statutes: 15 U.S.C. § 1681
• Regulations: 17 C.F.R. § 240.10b-5
• Contracts: Section 4.2(b)(iii) of the Agreement

Generated answers must cite in these formats, not in loose paraphrase. Post-processing or prompt engineering to enforce proper citation format.

Jurisdiction and date filtering

Legal advice depends heavily on:

• Which jurisdiction applies (federal, state, international)
• Which version of the law was in effect
• Which court's precedents apply

Metadata filtering is essential. A query about employment law in California shouldn't retrieve New York precedents or pre-2020 rulings that have been overturned.

Hallucination risk is existential

A hallucinated citation (fake case, fake statute section) is malpractice. Mitigations:

• Strict prompt: "Only cite sources that appear in the retrieved context. If you cannot cite, say so."
• Post-processing: verify every cited case/statute against the retrieved context
• Reject answers with unverifiable citations
• Use models with strong grounding behavior (Claude has been notably strong here)

The "I don't know" discipline

Legal RAG must be willing to say "I don't have information on that" rather than guess. Train users to expect this. A reliable "don't know" beats an unreliable answer every time.

Human-in-the-loop

Legal RAG is rarely fully autonomous. The pattern:

• AI retrieves and drafts
• Human lawyer reviews and refines
• Human is accountable for the final output

The value is saving lawyer hours, not replacing lawyers. Frame the UX around this.

Audit trail

Every answer must be reconstructible months or years later:

• What was the exact query?
• What documents were retrieved?
• Which chunks were used in the context?
• What version of the index and model generated the answer?
• Who received the answer?
• What did they do with it?

For regulated use (investment advice, compliance decisions), this audit trail is required by law.

Confidentiality

• Attorney-client privilege: retrieval must respect privilege boundaries
• Work-product doctrine: internal analyses are protected
• Client segregation: in law firms, one client's data must not leak to another
• Ethical walls: restrictions between practice groups must be enforced

Self-hosted is typical

Legal organizations usually require:

• All data in-VPC
• No third-party LLM API usage (or limited to vetted providers with BAAs)
• Enterprise deployments of models (Bedrock, Azure OpenAI, private Claude)
• Comprehensive audit logging

Eval is human-intensive

Automated evaluation can catch obvious errors, but legal correctness requires human judgment:

• Subject-matter experts review sample outputs
• Regular calibration between human and automated judges
• Case studies of errors to improve prompts and retrieval

The systems that work

Commercial legal AI (Harvey, CoCounsel, Relativity AI, Lexis+ AI) combines:

• Curated content (licensed case law, statutes)
• Strong retrieval with legal-specific metadata
• Citation verification pipelines
• Human review workflows
• Enterprise security and compliance

For internal legal RAG, borrow these patterns. Don't ship a naive vector RAG over legal documents and call it done, the quality bar is categorically higher.

Next: Multi-tenant RAG.