AI Loan Servicing pilot scorecard
A buyer-owned decision sheet for one bounded servicing workflow. Define the baseline, action boundary, data, controls, failure paths, evidence, and acceptance criteria before the results arrive.
00
Write the pilot contract before you run the pilot
A useful pilot tests one operational hypothesis. It does not turn a polished demo into a general production claim.
Pilot name
Workflow in scope
Business owner
Technology owner
Risk or control owner
Analytics owner
Start and end dates
Agent and instruction version
Weight confirmation and rationale
Map one servicing pilot with Lokta
Use the AI servicing workflow worksheet to define one case's facts, permissions, rejection path and evidence before testing it with this scorecard.
01Sign the baseline before looking at results
A before and after comparison is only meaningful when the population, clock, denominator, exclusions, and source are unchanged.
Population
Product, borrower state, delinquency stage, channel, language, geography, exclusions, and sampling rule.
Current workflow
Trigger, queue, operator steps, systems touched, approval, exception, result, and elapsed-time clock.
Baseline measures
Define the operational measure, quality measure, control measure, source query, period, and known limitations.
Acceptance thresholds
Set a target and a hard stop for each measure before results are reviewed. Do not use an invented universal benchmark.
02Set the action boundary explicitly
Use this as the starting boundary for a servicing pilot. Replace every default with the lender-approved rule for the chosen workflow.
| Action class | Pilot default | Proof to retain | Buyer decision |
|---|---|---|---|
| Read account and workflow context | Permitted inside assigned tenant, role, product, and field permissions. | Access policy and data-access audit record. | Approve / Narrow / Prohibit |
| Draft a borrower or operator explanation | Permitted when the source facts are retrieved from approved records and the channel rules are met. | Draft, cited account facts, instruction version, review result, and final communication. | Approve / Narrow / Prohibit |
| Prioritise or route work | Permitted only within an approved queue policy and with an accountable queue owner. | Policy version, reason code, queue movement, and exception route. | Approve / Narrow / Prohibit |
| Propose a next action | Permitted within explicit policy bounds. A proposal is not an executed account change. | Proposal, policy check, confidence or uncertainty signal, and approver decision. | Approve / Narrow / Prohibit |
| Change schedule, charge, waiver, settlement, restructure, write-off, or loan state | Requires the lender-defined deterministic checks and maker-checker approval before execution. | Change request, before and after state, approvals, execution result, and reversal path. | Approve / Narrow / Prohibit |
| Move money, alter asset classification, change policy, or bypass legal escalation | Prohibited for an unapproved pilot agent. Any later scope requires separate design, authority, testing, and acceptance. | Blocked-action tests and an audit record showing the fail-closed result. | Approve / Narrow / Prohibit |
03Score evidence across twelve dimensions
These are recommended starting weights and total 100%. Before scoring, the buyer must confirm or change each weight and record the rationale while keeping the total at 100%. Score 0 for absent, 1 for described, 2 for demonstrated, and 3 for evidenced in the pilot. A hard gate passes only with score 3 plus named owner sign-off. Clear every hard gate before using the weighted total.
0 Absent1 Described2 Demonstrated3 Evidenced
| Dimension and acceptance gate | Weight | Owner | Evidence to attach | Score | Weighted contribution | Finding or condition |
|---|---|---|---|---|---|---|
| Baseline and success definitionThe pilot target uses the same population, clock, denominator, and exclusions as the baseline. The buyer writes the target before pilot results are reviewed. | 7% | Operations owner and Analytics owner | Signed baseline period, case population, current handling path, metric definitions, source query, exclusions, and known data-quality limits. | 0 / 1 / 2 / 3 | ||
| Workflow and decision boundaryEvery step states whether the agent reads, drafts, recommends, proposes, executes, or escalates. No action is implied by a generic automation label. | 9% | Operations owner and Product owner | One current-state workflow, one pilot workflow, named decision points, action catalogue, and system-of-record map. | 0 / 1 / 2 / 3 | ||
| Data inputs and provenanceA reviewer can trace every material output back to the account data and policy material used, including missing or stale inputs. | 8% | Technology owner and Data owner | Field inventory, source system, freshness, ownership, consent or permitted-use basis, transformation, fallback, and retrieval trace for each input. | 0 / 1 / 2 / 3 | ||
| Security, privacy, provider, retention, and data egressThe named owners approve the provider terms and complete data path. No unapproved data egress, training or reuse, retention, or provider access remains, and the applicable request, deletion, incident, and exit paths have been tested. | 10% | Security owner, Privacy owner, Legal owner, and Technology owner | End-to-end data-flow and provider map covering inputs, outputs, storage, logs, backups, model calls, training or reuse terms, sub-processors, processing regions, egress, encryption, access, incidents, retention, deletion, data-principal requests, and contract exit. | 0 / 1 / 2 / 3 | ||
| Permitted and prohibited actionsProhibited actions fail closed. Permitted actions stay inside explicit policy bounds that can be tested without relying on the prompt alone. | 11% | Product owner, Risk owner, and Operations owner | Approved action boundary by workflow, amount, product, borrower state, channel, role, and time window. | 0 / 1 / 2 / 3 | ||
| Maker-checker and authorityThe selected consequential actions cannot complete without the required authorised checker, and the agent cannot approve its own proposal. | 9% | Risk owner and Operations owner | Approval matrix, thresholds, proposer and checker identities, delegation, expiry, rejection, and override handling. | 0 / 1 / 2 / 3 | ||
| Exceptions and escalationEvery test exception reaches the correct queue with the account context, reason, evidence, and next allowed action intact. | 8% | Operations owner and Collections owner | Exception taxonomy, severity, routing, service clock, handoff packet, retry rule, dead-letter handling, and named queue owner. | 0 / 1 / 2 / 3 | ||
| Audit evidenceAn independent reviewer can reconstruct a sampled case from source input to final result without a vendor-created explanation. | 9% | Internal Audit and Technology owner | Prompt or instruction version, model or agent version, data accessed, policy checked, proposal, approval, action, result, correlation identifier, and timestamps. | 0 / 1 / 2 / 3 | ||
| Accuracy and verificationThe buyer-defined threshold passes for each hard-gate case type. A blended average cannot hide a severe error class. | 9% | Analytics owner and Operations quality owner | Representative labelled case set, sampling method, error taxonomy, severity, verifier, disagreement handling, and result by case type. | 0 / 1 / 2 / 3 | ||
| Rollback and kill switchAn authorised operator can stop the agent, preserve evidence, route open work safely, and restore the prior workflow within the agreed operating target. | 7% | Technology owner and Incident owner | Disable control, access authority, detection trigger, open-work treatment, rollback steps, communication path, and tested recovery record. | 0 / 1 / 2 / 3 | ||
| Integration and production ownershipEach critical interface has one accountable owner, a tested failure path, and a reconciliation control before production use. | 7% | Technology owner and Vendor owner | Interface catalogue, event and field owner, retries, idempotency, reconciliation, monitoring, support boundary, deployment path, and change control. | 0 / 1 / 2 / 3 | ||
| Acceptance decision and operating handoffNo hard-gate failure is averaged away. The production boundary matches what was tested and accepted. | 6% | Pilot sponsor and all control owners | Signed results, unresolved findings, conditions, rollout scope, monitoring plan, review date, and go, revise, or stop decision. | 0 / 1 / 2 / 3 | ||
| Weighted total after all hard gates pass | 100% | Formula: Σ((row score / 3) × row weight) | ||||
03AA 95% total can still mean revise
Synthetic scoring example using the twelve starting weights above. This is a separate illustrative pilot assessment, not a measured Lokta result or an outcome of the WF-001 worksheet case.
A payment-exception pilot has candidate ratings of 3 on ten dimensions. Audit evidence (weight 9%) and acceptance decision and operating handoff (weight 6%) are both rated 2. Assume the other hard gates have the required evidence and named owner sign-off.
| Rating group | Weight | Score | Contribution |
|---|---|---|---|
| Ten other dimensions | 85% | 3 of 3 | 85 points |
| Audit evidence | 9% | 2 of 3 | 6 points |
| Acceptance decision and operating handoff | 6% | 2 of 3 | 4 points |
| Candidate total | 100% | Hard gates fail | 95 of 100 |
The calculation is weight × score ÷ 3: 85 + 6 + 4 = 95. Both dimensions rated 2 are hard gates. A demonstration earns 2; a hard gate needs evidence in the pilot, score 3 and named owner sign-off. This total cannot support a go decision or override either failed gate.
Add the missing evidence fields, rerun the rejection case, verify that no action occurred, and have the responsible owners review the record and operating handoff before rescoring. Retain the first result alongside the retest. Operations reconciles the earlier attempt with finance support; approval cannot bypass the unresolved check. The original proposal stays rejected. Any correction after factual resolution is a new linked proposal with its own checks, authority and verified result.
04Build the verification set around failure consequence
A blended accuracy percentage hides the cases that matter. Record results by case type and error severity.
| Case class | Cases | Target | Hard stop | Result | Evidence owner |
|---|---|---|---|---|---|
| Normal workflow cases | |||||
| Missing, stale, or conflicting data | |||||
| Policy edge and approval cases | |||||
| Duplicate or out-of-order events | |||||
| Integration failure and recovery | |||||
| Low-confidence or ambiguous requests | |||||
| Prohibited-action attempts | |||||
| Complaint, hardship, or sensitive handling in scope |
Record false statements, wrong-account context, policy breaches, unapproved state changes, missed escalations, duplicate actions, and missing audit evidence as separate error classes. The buyer sets the severity and tolerance for each class before the run.
05Test the rollback and kill switch
A written fallback is incomplete until an authorised operator has exercised it.
- Detect: Name the event, threshold, or control failure that pauses the pilot.
- Authorise: Name who can disable the agent and who must be informed.
- Stop: Prove new work stops while source systems and the deterministic book continue safely.
- Preserve: Retain queued work, prompts, data, decisions, approvals, results, and correlation identifiers.
- Route: Move open cases to the approved human queue without losing their context or service clock.
- Recover: Restore the prior workflow, reconcile any executed actions, and record the decision to restart or end the pilot.
Kill switch tested on
Authorised operator
Observed recovery result
06
Record one decision and one production boundary
The pilot result applies only to the workflow, population, integrations, permissions, versions, and controls that were tested.
Go
Every hard gate passes. Conditions, production scope, monitoring, ownership, review date, and rollback remain explicit.
Revise
The hypothesis remains useful and the gaps can be corrected and retested. Keep affected execution stopped while the control, data, evidence or handoff is revised. A failed hard gate prevents a go decision.
Stop
End or redesign the pilot when a hard-gate gap cannot be resolved within the accepted scope, evidence cannot be obtained, failure consequence is too high, or the workflow does not justify its operating cost.
DecisionGo / Revise / Stop
Conditions and unresolved findings
Production population and permissions
Monitoring owner and next review date
Business, risk, technology, and analytics sign-off
08
Frequently asked questions
Practical answers about selecting a workflow, constructing the case set, using hard gates, and interpreting a pilot result.
Which workflow should an AI Loan Servicing pilot start with?
Choose one bounded, frequent workflow with a clear system of record, visible human baseline, recoverable errors, and an owner who can review cases. Avoid combining borrower communication, payment posting, collections treatment, and account changes into one first pilot.
How many cases should the pilot test?
There is no universal number. Build the set from the workflow population and its failure modes. Include normal cases, missing and stale data, duplicate events, policy edges, vulnerable-customer handling where relevant, integration failure, low-confidence output, and each prohibited action.
Can a high average score offset a missing audit trail or kill switch?
No. Every dimension marked Hard gate must reach score 3 with named owner sign-off before a go decision. In the synthetic example, ten dimensions contribute 85 points, audit evidence contributes 6 and acceptance decision and operating handoff contributes 4. The arithmetic is 95 of 100, but the two hard gates rated 2 still fail. The decision is revise and retest, with no production approval. These are illustrative ratings, not measured pilot results.
Does a successful pilot prove a customer outcome?
It proves only what the signed design and evidence show for that workflow, population, period, and operating boundary. Do not turn a pilot result into a general deployment or portfolio-outcome claim without separate evidence and approval.