Diagnosing and reducing recurring customer dissatisfaction

Building the QA framework, rubric and calibration process

I built the QA rubric and rating system from scratch, scoring every reviewed ticket across six dimensions — Accuracy of Response, Due Diligence, Brand Customization, Personalization, Thoroughness and Empathy — averaged into a single composite Klaus score.

To diagnose why a score moved, I maintained two companion taxonomies: a DSAT taxonomy categorising every 1–3-star rating (Incomplete Resolution, OS Policy, Issue Recognition, Lack of Empathy, User Error, Delayed Response, System Error) and a CSAT taxonomy doing the same for 4–5-star ratings. Together they turned “quality went down” into “quality went down because of X, in Y% of tickets.”

Calibration ran weekly or bi-weekly: a small set of representative tickets scored independently, then discussed until the group reached consensus, with deviations and rubric ambiguities logged to feed the next revision. Alignment with the client's own QA went from 16% at the first session to 50% by the second, and weekly calibration later pushed policy-ticket rating accuracy to a sustained 98–100%.

Training and onboarding 100+ agents

I trained the first batch of 40+ agents using a classroom-style model where every ticket was treated as a case study: debated by the group, then approved by a QA or coach. New agents spent their first 3–4 weeks on email-only queues so they could focus on issue recognition and product knowledge before being measured on speed, and worked through knowledge base articles with written essay assignments to build real comprehension.

Data showed that agents who tested strongest on logical and rapid problem-solving — two of seven screening parameters — were the best predictors of on-the-job success, and that agents with a naturally skeptical disposition outperformed natural optimists on issue recognition. Once onboarding coaches reviewed roughly 75% of a new agent's responses before send, new-agent CSAT in the first two weeks jumped to 97–100%, up from 91–95%, with no negative impact on the wider team.

The experiment library

• 01Stroop Experiment — diagnostic and conditioning method to build cognitive flexibility and root-cause issue recognition.
• 02Peer Reviews — weekly cross-agent ticket review against the QA rubric to surface blind spots.
• 03Workgroups — groups of 5 agents with a peer moderator plus a dedicated QA/TL for real-time shift support.
• 04Service Mechanics Blueprint — visual decision trees for common issue types.
• 05Habit Forming Principles — a cue-routine-reward loop for brand voice and conversation mechanics.
• 06Improv Games — role-play exercises run in daily stand-ups to build empathy and connection.
• 07Floating Notes — shortcut-based glossary of on-brand personalisation language.
• 08Hall of Fame — recognition track spotlighting the behaviours the rubric rewards.

Personality-based recruiting and project fit

Instead of screening candidates against one generic bar, I used Big Five assessment data to predetermine the personality profile a given project or bucket actually needed — a high-touch Concierge role calls for a different trait mix than a high-volume T1 queue — then screened and placed candidates against that specific profile. Placement decisions were grounded in measured traits, not gut feel.

Capacity projection and workforce management

Each month I pulled four weeks of ticket data from Intercom across every bucket, then broke it down by hour and weekday with pivot tables to find the actual demand curve across chat, email and Facebook channels.

That curve fed a headcount model across a three-shift structure (roughly 7am–4pm, 3pm–midnight, 11pm–8am) with explicit assumptions for shrinkage (20%) and productivity (80%), so numbers reflected real available capacity. I built the monthly shift schedule by hand against those targets while balancing leave, swaps and overtime — and documented the whole process plus its governance rules into a reusable build guide.

Leadership development

I created and launched two parallel mentorship tracks — one for Team Leads, one for QAs/Coaches — where each group designed its own approach rather than following a prescribed format. A mid-program survey showed mentees energised about learning from mentors and each other, with the gap between sessions flagged as too long. I separately surveyed the teams that mentees themselves managed, to see whether mentorship was changing how they led; results were largely positive, with delegation and time management the clearest areas still to develop. Both rounds of feedback fed a redesign and a reusable playbook.