Case Study · Healthcare AI
I built a compliance intelligence system for clinical trials. Here's every hard decision I made.
centiTMF helps clinical teams know if they're inspection-ready before the auditors show up. Building it meant figuring out where AI should make decisions, where humans should, and what happens when the LLM confidently gets it wrong.
Written by Max
The problem
Clinical trials run on paperwork. Nobody knows if it's complete until it's too late.
Every clinical trial produces a Trial Master File: hundreds of documents across multiple sites that must be complete, consistent, and audit-ready at all times. In practice, compliance teams spend hours manually checking whether documents are filed, current, and correctly categorized. They usually find the gaps when an inspector is already asking questions.
The question I started with was simple: could an AI system continuously answer “if regulators reviewed this study tomorrow, where would we be exposed?”
What broke first
The LLM couldn't explain itself. In compliance, that's a dealbreaker.
My first instinct was to let the LLM do the heavy lifting: read the documents, assess compliance, generate risk flags. It worked impressively in demos. It failed in the ways that matter.
The failure
The model would flag a site as non-compliant but couldn't trace the flag back to a specific rule or a specific missing document. When I asked “why did you flag this?”, the answer was a plausible-sounding paragraph that was sometimes wrong. In a compliance context, an unexplainable flag is worse than no flag at all. It creates false urgency, erodes trust, and can't survive an auditor asking “show me how you determined this.”
This changed my entire architecture. I stopped asking “what can the LLM do?” and started asking “where does explainability matter more than intelligence?”
Key decision
I split the system into three trust tiers.
Instead of one AI system doing everything, I designed three layers with different levels of autonomy based on how much explainability each function requires.
The principle: the higher the compliance stakes, the less the LLM is allowed to decide. It can summarize and explain, but the underlying judgments are deterministic and traceable.
Hardest call
Audit Q&A: free generation vs. grounded answers
The most useful feature in centiTMF is the ability to ask natural-language questions about a study's compliance status. “Are we ready for an inspection at Site 012?” “What's our biggest exposure right now?” Building this meant choosing between two approaches.
The trade-off
How should the AI answer audit questions?
Rejected
Free LLM generation
Let the model reason over raw documents and generate answers directly. More flexible, more natural. But impossible to verify and prone to hallucinating compliance status.
Chosen
Grounded answers
LLM reads from the structured fact model, compliance flags, and deviation signals. It can synthesize and narrate, but every claim maps back to a named data point.
Why: A compliance team can't use an answer they can't defend. If an auditor asks “how did you determine Site 012 is at risk?”, the answer needs to trace back to specific flags and specific rules, not a language model's interpretation of a PDF. Bounding the LLM's input to structured data means the answers are as reliable as the fact model, and the fact model is deterministic.
Key decision
Classification: automate it, but let humans override it.
When a document is uploaded, the system classifies it automatically (consent form, monitoring report, 1572, delegation log, etc.). Early on I had to decide: should classification be fully automated, or should there be a human check?
The trade-off
Should document classification be fully automatic?
Rejected
Fully automatic
Faster ingestion, less manual work. But a misclassified document silently corrupts the compliance model downstream. One wrong label means a missing artifact that isn't actually missing.
Chosen
Auto-classify + manual override
AI classifies with a confidence score. Users review and can override before filing. The original AI classification is preserved as metadata, so you can audit the AI's accuracy over time.
Why: Classification errors compound. A misclassified document means the completeness engine thinks something is missing when it isn't, or worse, thinks something is present when it's not. The override mechanism costs a few seconds of human time per document but prevents cascading errors in the compliance model. Preserving the AI's original guess as detected_artifact_type creates a feedback loop for improving classification accuracy over time.
What I built
The system in its final form
centiTMF processes uploaded trial documents through a layered pipeline: classify, extract, structure into a fact model, evaluate against compliance rules, score risk, and surface results through a dashboard and natural-language audit interface.
0–100
Inspection readiness score with explainable penalty breakdown per site
10
Data-driven compliance rules evaluated against structured facts
100%
Every flag traceable to a named rule and a named fact
Retrospective
What I'd do differently
Start with the fact model, not the LLM
I spent early cycles trying to get the LLM to reason about compliance directly. If I started over, I'd build the structured fact model first and only add LLM capabilities once the deterministic layer was solid. The foundation matters more than the intelligence layer.
Build the override audit trail from day one
The classification override feature was a later addition. In hindsight, any system where humans correct AI outputs should have the audit trail designed in from the start, not bolted on. The data it generates — where the AI is wrong, how often, on which document types — is some of the most valuable data in the system.
Test with messier data earlier
The demo seed data is clean and well-structured. Real TMF documents are inconsistent, mislabeled, and sometimes contradictory. I'd pressure-test the classification and extraction pipeline with uglier data sooner to find the edge cases that matter.
Tech stack
How it's built
- Backend
- FastAPI, Python 3.11, SQLAlchemy 2.0 (async), Pydantic v2
- Frontend
- Next.js 14, TypeScript, Tailwind CSS (App Router)
- Database
- PostgreSQL 16 + pgvector (Supabase in production)
- Storage
- Cloudflare R2 (production), MinIO (local dev)
- AI
- GPT-4o for narrative, OpenAI embeddings, deterministic fallbacks throughout
- Infra
- Docker Compose (local), Vercel + Render (production)