The problem
To track obligations on a commercial real estate loan, someone had to read the whole closing binder and pick out every covenant and key date by hand. For one deal, that was a full day of an asset manager’s or lawyer’s time.
We first trained a classic ML classifier on our 30 active deals. It did not hold up. When LLMs arrived, we rebuilt extraction around them, and the internal tool became a product of its own.
Context
A small team with an outsourced engineering group. Many users were senior asset managers who lived in Excel and email. Every extracted field had to hold up in front of lawyers, and the company was preparing for SOC 2.
Key decisions
I threw out the onboarding I designed
Version one asked users to confirm how every document related to each loan tranche. Nobody did it; they went back to the binder. I fixed extraction quality first, then cut onboarding to a drag-and-drop upload where you only fix what looks wrong. Usage went up after that.
Every field points back to its source
Each extracted value links to the page it came from, and every assistant answer shows its citation. The person’s job changed from typing to checking.
A document timeline with no model call
I designed a timeline that strings the original agreement and every amendment together from dates already captured at onboarding. No extra model call, so nothing can be made up.
States for every obligation
Each tracked obligation is active, amended, replaced or expired, so an amendment never leaves a stale reminder behind.
Outcome
Customers included a top-tier Korean financial group, Columbia Property Trust, and the law firms Sheppard Mullin and Schlam Stone & Dolan. I was promoted to Senior PM within two years.
What I’d change
We thought the product was intuitive enough to skip a manual. For users in their sixties and seventies, “intuitive” was our bias. Next time I would ship a short guide and keep it current with AI.
I would also push earlier to meet documents where they already live, in shared drives. The team started that integration after I left.