Automated Profile Research
Crawl, extract, verify and deliver to human review
Brent Theisen
The problem, and the contract
- Researchers keep ~20k treatment-center profiles accurate by hand
- A wrong phone number or insurance list can send someone in crisis to the wrong place
Every value is a proposal with a stored snapshot and a verbatim quote.
Nothing reaches the content system until a reviewer accepts or edits it.
The goal is faster reviewers, not fewer.
Assumptions
- Scale: ~20k profiles × ~30 fields × 1–3 sources
- Sources: provider sites, state licensing lookups, SAMHSA. HTML only; no PDFs in v1
- Content system: Postgres-backed, API-writable, no field history, so provenance lives with us
- Constraint: reviewer hours, not LLM spend
- Models: no vendor preference; one interface; per-run spend cap
- Crawling: obey robots.txt, identify ourselves, rate-limit per domain
Data model = audit trail
| Entity | Stores |
run / batch | Scope, caps, prompt + model versions, cost, status |
work_item | Profile × stage: state, attempts, last error, lease |
source_document | URL, fetch time, content hash, raw + cleaned snapshot |
extraction / citation | Value, model, prompt version; verbatim quote + offsets |
verification / proposal | Check results; what the reviewer sees |
review_decision | Accept / edit / reject, reason code, reviewer |
field_provenance | Published value → decision → citation |
llm_call | Model, tokens, cost, latency, trace ID |
Append-only outputs. Any published value traces back to the exact bytes the model read.
1. Pipeline end to end
flowchart LR
P[Plan run] --> D[Discover sources]
D --> F[Fetch + snapshot]
F --> E[Extract
quote required]
E --> V[Verify
checks + risk tier]
V --> Q[Review queue]
Q --> R{Reviewer}
R -- accept / edit --> C[Content system
+ provenance]
R -- reject --> FB[(review_decision)]
C --> FB
FB -. feeds .-> E
FB -. feeds .-> V
1. Key pipeline decisions
- No open web search in v1. Only sources tied to the facility; a look-alike facility is the easiest way to corrupt a profile
- Snapshot everything, keyed by content hash, so a citation always points at what the model saw
- No quote, no proposal. Citations are a schema rule, not a UI feature
- Deterministic checks first: quote appears verbatim, format, cross-source agreement. LLM support check only on critical fields
- Risk tiers: phone, address, levels of care, insurance, licensing go to the top of the queue
- The review UI is the product: value beside highlighted passage, grouped by profile, keyboard-driven
2. State tracking and recovery
stateDiagram-v2
direction LR
[*] --> pending
pending --> fetching
fetching --> fetched
fetched --> extracting
extracting --> extracted
extracted --> verifying
verifying --> delivered
delivered --> [*]
fetching --> failed_retryable
extracting --> failed_retryable
verifying --> failed_retryable
failed_retryable --> pending
failed_retryable --> failed_terminal
fetched --> skipped
Where a run stands is always one SQL query: counts by state, cost, failure rate.
2. How it survives failure
- Output + state change in one transaction. An item is fully before or fully after a stage
- Sweep re-queues expired leases and items stuck in a ready state. A run that dies at 70% resumes at 70%
- Cache by content hash + prompt version. Unchanged page and prompt means no LLM call
- Caps: reserve estimated cost before each call; release batches one at a time; pause the run when a batch's failure rate trips
- Small work items: one profile × one stage, so retries are cheap and queues control concurrency
3. Reviewer feedback loop
Every decision links to its extraction, prompt, model and source. Edit and reject require a reason code.
- Suppression: a rejected value is never re-proposed from the same content
- Scoped notes: per profile or domain, injected into future prompts
- Examples: edits become per-field prompt examples
- Eval set: reviewed decisions gate every prompt or model change
3. From fill-only to updates
- Update proposals show the current value and its provenance next to the new one
- Higher bar: source must be newer than the current citation, or the current value has none
- Rejecting an update records "confirmed on this date"; not re-proposed until the source changes
- Watch for rubber-stamping: time per decision, acceptance rate per reviewer, seeded known-wrong values
4. Path toward more autonomy
- v1 linear chain: fixed stages; LLM only inside extract and verify
- v2 orchestration: a planner picks sources and fields from freshness and past acceptance
- v3 per-profile agent: search, fetch, follow link, look up license, with step and budget caps
Autonomy changes how evidence is gathered, not who decides. Output is still a cited proposal.
Gate: shadow-run, promote field by field when it beats the current version on the eval set at equal or lower cost.
4. What makes me cautious
- Prompt injection: pages are untrusted; the agent has no write tools; the quote check doesn't trust the model
- Cost blowups: hard per-profile step and budget caps from day one
- Reproducibility: every tool call and result is logged with the run
- Evaluation: agent paths vary; the eval set must exist before v2
- Reviewer trust: if volume outgrows accuracy, reviewers skim, and that's the safeguard we can't lose
5. Complete profile creation
- Entity resolution first: match against the catalog (including acquired sites); a likely match goes to a reviewer as "same facility?"
- Source discovery: official site, state license, SAMHSA
- Same pipeline: the draft profile is a bundle of proposals
- Review as a unit: can't publish until every critical field is accepted
Only two new parts: entity resolution and the required-fields gate.
Out of scope for v1
- PDF sources
- Non-English sites
- Auto-acceptance of any kind, including "low-risk" fields
Questions