Insurance PDFs Are Hostile: An AI Extraction Pipeline in Three Acts

Act 1 — the bake-off (April 2025)

The first version was a script with a question: which extractor actually reads these documents? The commit history records the tournament honestly — "feat: finish gemini and parser implementation" (April 10), then "feat(llama): add new logic and pre processing stuff" (April 27). GPT-4, Gemini, and LlamaExtract from LlamaCloud went head to head, on top of a full OCR pre-processing stack: ocrmypdf, Tesseract, PyMuPDF, OpenCV, pdf2image — because half the battle is making a scanned PDF readable before any model sees it.

LlamaExtract won, and then the real work started: the extraction agent is named "emr-parser-v1.1.18" — at least eighteen schema iterations behind that version number. The target schema is a ~20-field Pydantic model: risk identifiers, a state table with nine numeric fields per row, class-code tables with ELRs and D-ratios, actual and expected losses split three ways, ARAP/FLARAP factors, and the page number where each section was found. In insurance, a number without its page reference is a number nobody will trust.

Act 2 — make it a product (April 30, 2025)

"feat(api): create the first stable version of the API." The script became the EMR PDF Processor API: FastAPI, dockerized, API-key auth, two endpoints — process the PDF, validate the JSON. Boring on purpose. A model experiment becomes usable the day it has an endpoint, auth, and a version number.

The eval corpus was the real asset

Alongside the API lived a second repository that mattered more: 179 real PDFs across 132 submissions, sorted into directories that name the enemy — text-based, images-based, weird-format, worst-case, errors. Each document got manually built ground truth (correct.json), and outputs were versioned per document: one file went through ten output revisions before it passed. Four scoring rounds ran between May 11 and June 3, one commit per document: "feat: add score for 1141".

The corpus also powered the sharpest question you can ask of an extraction system: are we better than the vendor we're replacing? A comparator diffed our output against the incumbent (Sensible) field by field. Across the eleven comparison runs committed to the repo, similarity ranged from 97.68% to 100% — 259 fields compared in the largest run, six differences counted. A small committed sample, not a production benchmark — but every difference was inspectable, field by field, which is exactly what you need in the meeting where the switch gets decided.

Act 3 — serverless (May–October 2025)

With extraction quality proven, the pipeline moved to AWS. The Step Functions state machine that orchestrates it has 82 states: input validation, a DynamoDB cache check keyed by the document's SHA-256 (the same PDF is never paid for twice), a connectivity check, a semaphore with retry/wait that limits concurrency against LlamaCloud, a PDF chunker for oversized documents, an async-polling extraction engine, per-chunk continuation, fallback extraction, and a merge processor that reassembles the result.

• Sixteen individual Lambdas were consolidated into eight optimized groups, each with its own GitHub Actions deploy pipeline — ten workflows in total.
• Event routing supports three flows: S3 → SNS → SQS → Lambda, S3 → SNS → Lambda with an RDS check on the file type, and direct S3 → Lambda. Same ingestor, three entry paths.
• The repo's git history is the same lineage as the April script — one unbroken line of evolution from bake-off to state machine, all mine.

What I would tell anyone doing this

1. Build the evaluation corpus before you build the pipeline. The 179 PDFs with ground truth outlived every architectural decision and validated all of them.
2. Name your worst cases. Directories called weird-format and worst-case do more for extraction quality than any prompt tweak, because they keep the ugly documents in front of you.
3. Cache by content hash. Documents get resubmitted constantly; a SHA-256 keyed cache is the cheapest optimization in the whole system.
4. Treat the incumbent vendor as your benchmark, field by field. "97.7% similar, here are the six differences" wins the replacement conversation; "our AI is better" does not.
5. Version your extraction schema like a product. emr-parser-v1.1.18 is eighteen documented lessons about how insurance documents lie to you.