Automated Profile Research

Crawl, extract, verify and deliver to human review

Brent Theisen

The problem, and the contract

  • Researchers keep ~20k treatment-center profiles accurate by hand
  • A wrong phone number or insurance list can send someone in crisis to the wrong place
Every value is a proposal with a stored snapshot and a verbatim quote.
Nothing reaches the content system until a reviewer accepts or edits it.
The goal is faster reviewers, not fewer.

Assumptions

  • Scale: ~20k profiles × ~30 fields × 1–3 sources
  • Sources: provider sites, state licensing lookups, SAMHSA. HTML only; no PDFs in v1
  • Content system: Postgres-backed, API-writable, no field history, so provenance lives with us
  • Constraint: reviewer hours, not LLM spend
  • Models: no vendor preference; one interface; per-run spend cap
  • Crawling: obey robots.txt, identify ourselves, rate-limit per domain

Data model = audit trail

EntityStores
run / batchScope, caps, prompt + model versions, cost, status
work_itemProfile × stage: state, attempts, last error, lease
source_documentURL, fetch time, content hash, raw + cleaned snapshot
extraction / citationValue, model, prompt version; verbatim quote + offsets
verification / proposalCheck results; what the reviewer sees
review_decisionAccept / edit / reject, reason code, reviewer
field_provenancePublished value → decision → citation
llm_callModel, tokens, cost, latency, trace ID

Append-only outputs. Any published value traces back to the exact bytes the model read.

1. Pipeline end to end

flowchart LR P[Plan run] --> D[Discover sources] D --> F[Fetch + snapshot] F --> E[Extract
quote required] E --> V[Verify
checks + risk tier] V --> Q[Review queue] Q --> R{Reviewer} R -- accept / edit --> C[Content system
+ provenance] R -- reject --> FB[(review_decision)] C --> FB FB -. feeds .-> E FB -. feeds .-> V

1. Key pipeline decisions

  • No open web search in v1. Only sources tied to the facility; a look-alike facility is the easiest way to corrupt a profile
  • Snapshot everything, keyed by content hash, so a citation always points at what the model saw
  • No quote, no proposal. Citations are a schema rule, not a UI feature
  • Deterministic checks first: quote appears verbatim, format, cross-source agreement. LLM support check only on critical fields
  • Risk tiers: phone, address, levels of care, insurance, licensing go to the top of the queue
  • The review UI is the product: value beside highlighted passage, grouped by profile, keyboard-driven

2. State tracking and recovery

stateDiagram-v2 direction LR [*] --> pending pending --> fetching fetching --> fetched fetched --> extracting extracting --> extracted extracted --> verifying verifying --> delivered delivered --> [*] fetching --> failed_retryable extracting --> failed_retryable verifying --> failed_retryable failed_retryable --> pending failed_retryable --> failed_terminal fetched --> skipped

Where a run stands is always one SQL query: counts by state, cost, failure rate.

2. How it survives failure

  • Output + state change in one transaction. An item is fully before or fully after a stage
  • Sweep re-queues expired leases and items stuck in a ready state. A run that dies at 70% resumes at 70%
  • Cache by content hash + prompt version. Unchanged page and prompt means no LLM call
  • Caps: reserve estimated cost before each call; release batches one at a time; pause the run when a batch's failure rate trips
  • Small work items: one profile × one stage, so retries are cheap and queues control concurrency

3. Reviewer feedback loop

Every decision links to its extraction, prompt, model and source. Edit and reject require a reason code.

  1. Suppression: a rejected value is never re-proposed from the same content
  2. Scoped notes: per profile or domain, injected into future prompts
  3. Examples: edits become per-field prompt examples
  4. Eval set: reviewed decisions gate every prompt or model change

3. From fill-only to updates

  • Update proposals show the current value and its provenance next to the new one
  • Higher bar: source must be newer than the current citation, or the current value has none
  • Rejecting an update records "confirmed on this date"; not re-proposed until the source changes
  • Watch for rubber-stamping: time per decision, acceptance rate per reviewer, seeded known-wrong values

4. Path toward more autonomy

  • v1 linear chain: fixed stages; LLM only inside extract and verify
  • v2 orchestration: a planner picks sources and fields from freshness and past acceptance
  • v3 per-profile agent: search, fetch, follow link, look up license, with step and budget caps
Autonomy changes how evidence is gathered, not who decides. Output is still a cited proposal.

Gate: shadow-run, promote field by field when it beats the current version on the eval set at equal or lower cost.

4. What makes me cautious

  • Prompt injection: pages are untrusted; the agent has no write tools; the quote check doesn't trust the model
  • Cost blowups: hard per-profile step and budget caps from day one
  • Reproducibility: every tool call and result is logged with the run
  • Evaluation: agent paths vary; the eval set must exist before v2
  • Reviewer trust: if volume outgrows accuracy, reviewers skim, and that's the safeguard we can't lose

5. Complete profile creation

  1. Entity resolution first: match against the catalog (including acquired sites); a likely match goes to a reviewer as "same facility?"
  2. Source discovery: official site, state license, SAMHSA
  3. Same pipeline: the draft profile is a bundle of proposals
  4. Review as a unit: can't publish until every critical field is accepted

Only two new parts: entity resolution and the required-fields gate.

Out of scope for v1

  • PDF sources
  • Non-English sites
  • Auto-acceptance of any kind, including "low-risk" fields

Questions