Building a quality operating system for a 500+ agent healthcare operation

Transformation approach

• 01Diagnose — built a DSAT attribution model to distinguish agent error from process, policy and tooling issues.
• 02Design — created a 100-point QA framework with zero-tolerance patient-safety triggers and risk-based severity tiers.
• 03Implement — established QA and coaching governance, escalation thresholds, sampling rules and weekly operational reporting.
• 04Measure — introduced post-coaching verification and root-cause tracking to determine whether interventions actually changed behaviour.

The governance principles underneath the framework

• 01Patient safety overrides performance metrics.
• 02Quality is observable and measurable.
• 03Feedback and coaching are Team-Lead-owned responsibilities.
• 04QA reviews and verifies; it does not coach.
• 05Systemic issues require training, not repeated coaching.
• 06Management decisions are based on documented trends, not anecdotes.

Designing the 100-point, 5-dimension rubric

Rather than treating all quality failures equally, I weighted the framework according to customer, compliance and operational risk.

Every reviewed ticket is scored across five weighted dimensions, with defined criteria at four performance levels (0 / 25 / 50 / 100) for each sub-category.

Certain failures — giving medical or dosage advice, missing an AE/PQC escalation, mishandling PII — trigger an automatic zero regardless of how the rest of the interaction scored, so patient-safety risk can never be averaged away by an otherwise-good conversation.

Key diagnostic insight — not every poor outcome is a frontline problem

I introduced a validation gate that separated agent error from policy, process and tooling failures before those outcomes entered performance scoring. That prevented the QA system from penalising agents for problems they could not control, while creating a separate route for systemic issues to reach policy owners.

Not every DSAT is the agent's fault, and treating them all as if they were erodes trust in the QA process. I built a domain-based attribution model that routes each DSAT to its real cause before it ever touches an agent's record. Only DSATs classified as agent error after this validation gate reach scoring or escalation logic; everything else is logged for systemic trend analysis.

Each agent-error domain carries example-backed sub-drivers so classification stays consistent across analysts rather than becoming a judgment call. A separate, multi-select layer of compliance flags (verification/HIPAA, consent, PII handling, AE/PQC documentation, closing protocol) is tracked independently of the DSAT driver — ticked even on calls the customer was satisfied with — so a compliance gap can never hide behind a good CSAT.

Diagnosing causes, not symptoms

Ownership gaps — agents closing tickets without investigating, or not wanting to dig further — were the single largest driver of failures, ahead of both knowledge and comprehension issues. That reframed coaching priorities: more training wouldn't have fixed the biggest problem. Accountability coaching would.

Thresholds grounded in real volume

The thresholds aren't arbitrary. Agents typically handle 150–200 tickets a week, so 1–2 agent-error DSATs (roughly 1%) is treated as normal variation, 3 or more (roughly 2%+) as elevated risk, and 4–6 (3–4%) as sustained failure requiring intervention — fair to agents working real volume, while still protecting patients.

Separating QA from coaching — and verifying it worked

QA reviews interactions, identifies defects, flags when coaching is required, and verifies outcomes afterward — it never delivers coaching directly. That's Team-Lead-owned: review the findings, deliver standards-based feedback, confirm the agent understood it, set expectations, and log the session. Uncompleted coaching is logged as not done; there's no partial credit for intent.

Verification is what most QA systems skip. After coaching, QA runs a 2-week follow-up window to check one thing only: did behaviour actually change on subsequent tickets. An apology in the moment isn't evidence. If the same defect appears in three or more agents, or spans teams, it stops being an individual coaching problem and escalates to systemic training, measured by defect reduction rather than attendance.

A tiered performance-escalation model

Built a tiered performance-escalation model linking repeated defects to documented root-cause analysis, coaching, improvement plans and formal consequences — with reset rules for sustained improvement, so flags clear after four consecutive weeks at target and the line is visible well before anyone reaches it.

Rolling out the Resolution Checklist and the Roundtable report

Two of the most-used tools I introduced were deliberately simple. The Resolution Checklist targets the framework's biggest DSAT driver — Resolution & Outcome — with a concrete pre-close checklist, turning “did you actually finish the job” from a judgment call into something checkable before a ticket leaves the queue.

The Roundtable document is the weekly operational report structuring the cross-team review: QA score trends, DSAT pattern classifications, confirmed critical events, and coaching/training status in one recurring format, so leadership sees the same shape of data every week. Both are still in active use.

Running the review operation at scale

Sampling scales review intensity to risk: new agents get 10–15 tickets reviewed per week, established agents 5–7, and high-risk workflows are sampled weekly regardless of tenure. Since launch that has produced over 2,500 DSAT reviews and 215 CSAT reviews across 9 sites — Lagos, Cape Town (x2), Johannesburg and Missouri — spanning intake, refills, checkout, shipping and specialty workflows like the Medicare Bridge Program.