Let’s take a hypothetical case study — a 5-month banking + payments modernization called PayStream Legacy → Cloud: a 15-member team, an 11-sprint cadence, a legacy COBOL/Oracle core, a cloud-native rebuild on ISO 20022, and an AI QE pod. The dashboards below are the ones I would use to govern it — not an operational diary — each one answering one question a Test Manager is asked in a steering review.
If a steering-committee member reads nothing else on this page, this is the slide they get. Business problem, current health, my Test Manager read, decisions I need, and where the evidence lives.
The regulator has mandated compression of the overnight reconciliation window by Q2 2027, and the legacy COBOL/Oracle payment core is on vendor runout with no defect-fix path. The programme replaces it with a cloud-native payments platform (Java + Postgres + Kafka, ISO 20022) and moves ops to Salesforce Service Cloud, delivered as pilot on 2027-01-24 and GA on 2027-02-21 — 11 two-week sprints end-to-end.
Scope: 48 stories across 4 epics, 2 sprints closed, 9 to go. Primary exposure: UAT environment instability — 2 outages last sprint, ~1.5 days of lost test execution.
Delivery against baseline is on track. The risk I’m carrying is downstream regression confidence, driven by environment stability — not velocity, not defect density. Today’s evidence supports a conditional go on the Sprint 8 regression gate, upgradable to a full go if D1 below lands by end of Sprint 3.
A dashboard reports; a Test Manager decides. Three scenarios that would land on this programme — each one with the signal, my analysis, the options I’d put to leadership, my recommendation, and the owner it goes to. All hypothetical.
2 UAT outages in Sprint 2 · ~1.5 days of test execution lost · pass-rate dropping on the shared environment. Environment-health card is Red.
Infrastructure, not defect density. The Sprint 8 regression suite needs a stable UAT by Sprint 7 to produce defensible exit evidence. Shared environment contention is the root cause.
Decision owner: Head of Platform Engineering + Programme Director.
Deadline: End of Sprint 3 (2026-11-01).
Escalation trigger: UAT stability not demonstrated by Sprint 5.
Option A. $18K/qtr is below the cost of a 1-sprint GA slip. If declined, I will formally lower the Sprint 8 exit statement to "conditional go, risk owned by Platform Engineering" and re-plan the regression sequence to protect the top-5 payment flows at the expense of the long tail.
In Sprint 8, 3 unresolved net-position mismatches above the 0.01% tolerance band. Reconciliation tile shows variance Red for 2 consecutive runs.
Two are transformation-rule gaps the team can close in Sprint 9. One is a central-bank schema disagreement (ownership outside the team) — it will not close on our calendar.
Option B. Hard-scope the pilot cohort. Full pilot (Option C) leaks risk I don’t own; holding (Option A) blocks on an external item we can’t accelerate. Decision owner: Programme Director + Compliance.
The risk-based-selection agent cuts the CI gate from 42 min to 24 min on an evaluation set of 180 historical PRs containing 50 known real defects (plus 130 clean PRs). Measured recall on the real-defect denominator: 47 of 50 = 94% (3 missed real defects, all non-P1).
A 6% miss at PR level is tolerable when the full suite still runs nightly. At the release gate the same miss rate is not acceptable — one leaked P1 at pilot could cost the GA date.
Option A + all 4 guardrails. Measurement: monthly precision/recall vs the versioned eval set, published on the AI QE Agents dashboard. Owner: SDET Lead + Dev Lead. Rollback triggers: severity-weighted miss score > 6 on the eval set, OR > 2 real defects leaked in a rolling 30-day window, OR a single P1 leak. The AI selection runs at PR level only; the release gate always runs the full suite.
Every dashboard on this page is a slice of one hypothetical delivery. Imagining a single, end-to-end programme is what forces each metric to earn its place — because an unanswered question somewhere else is a defect waiting to be shipped.
Legacy state. A regional bank runs its core payments ledger on a 22-year-old COBOL + Oracle system. Operations run cases from an in-house ticketing tool. Settlement is batch, overnight; reconciliation to the central bank is partially manual. Regulator has flagged the reconciliation window as a systemic risk and set a 12-month deadline.
Target state. A cloud-native payments platform (Java + Postgres + Kafka) speaking ISO 20022, with Salesforce Service Cloud as the agent desktop for operations. Settlement moves from batch to near-real-time with intra-day reconciliation. AI augments triage, data validation and release decisions — as explicit, auditable agents, not black-box automation.
Why now. Regulatory deadline, 25% cost base locked in legacy mainframe licensing, and no vendor-side defect fix path for the current switch.
A banking modernization touches every surface a Senior Test Manager owns in a steering review: data integrity (migration + reconciliation), functional risk (payment correctness, rollback), operational risk (Salesforce case flow + agent desktop), release risk (pilot → GA ramp) and programme risk (velocity, defect aging, environment health). The same twelve dashboards you need here are the twelve you need on any modernization — so building them against this story keeps them honest.
The 5-month milestone (Sprint 1 kickoff 2026-09-21 → GA 2027-02-21) decomposes into 4 epics and roughly 48 stories. Below is the top-level epic table and a representative story cut from each epic — the shape I expect the backlog to take, not the full backlog.
| Epic ID | Name & objective | Stories | Lead role |
|---|---|---|---|
| EPIC-DM | Data Migration. Move customer master + 11yr ledger from Oracle to Postgres with field-level mapping, chunked cutover and point-in-time rollback. | ~14 stories | Dev Lead + QA (1) |
| EPIC-PR | Payment Rail. ISO 20022 message handling, near-real-time settlement, intra-day reconciliation to the central bank. | ~16 stories | Dev Lead + QA (1) |
| EPIC-SF | Salesforce Integration. Service Cloud as agent desktop; case routing, SLAs, case-to-payment traceability; replaces in-house ticketing. | ~10 stories | Dev + QA (1) |
| EPIC-QE | QE AI Pod. Triage agent, risk-based test selection, reconciliation diff narrator, synthetic data generator. Underpins every other epic. | ~8 stories | SDET Lead (me) + DevOps |
| Story | Title | Owner | Sprint |
|---|---|---|---|
| DM-07 | Reconcile customer master row counts between Oracle and Postgres per batch, with field-level diff on primary + 6 risk columns. | QA + 2 Devs | Demo235–236 |
| DM-11 | Rollback runbook: point-in-time restore of Postgres ledger with reconciliation checksum verification before go-live. | DevOps + Dev Lead | Demo240 |
| PR-05 | Reject pacs.008 messages missing BIC/IBAN with structured error codes; contract-tested against the central-bank schema. | QA + 2 Devs | Demo234 |
| PR-14 | Intra-day reconciliation job: produce a signed CSV of net positions, send to the regulator sandbox, surface mismatches above a 0.01% tolerance. | QA + 2 Devs | Demo238–239 |
| SF-03 | Case routing by payment status: "Failed Settlement" routes to a tier-2 queue with payment ID attached; SLA 2h. | QA + 1 Dev | Demo236 |
| SF-09 | Agent desktop surfaces payment timeline (Kafka events) alongside case; feature-flagged for canary cohort. | QA + 1 Dev | Demo241 |
| QE-02 | Triage agent clusters pipeline failures by stack-trace signature; proposes "flake / env / real defect" with cited evidence; human approves. | SDET Lead + DevOps | Demo233 |
| QE-06 | Risk-based test selection: given a diff + 90-day defect history, propose a <30-min smoke set with coverage justification. | SDET Lead | Demo238 |
Headcount budgeted up front, with named role on each epic. A QA sits inside each delivery epic rather than behind a QA wall; the fourth QA capacity is my own hands-on SDET time on the AI pod.
Owns steering communications, budget, regulatory liaison. Reads the Release Readiness, Risk Register and Pilot Rollout dashboards weekly.
Facilitates ceremonies for all 4 epics; owns the velocity + defect-aging trends on the Governance dashboard.
Architecture, cutover plan, rollback sign-off. Primary engineering voice on steering; co-owns the Risk Register.
2 on EPIC-DM, 3 on EPIC-PR (biggest), 2 on EPIC-SF, 1 fractional on EPIC-QE. Daily standups per epic; cross-epic sync at Thursday-afternoon design review.
1 per delivery epic: Data Migration QA, Payments QA, Salesforce QA. I keep the fourth "QA seat" by doing the AI-pod SDET work myself.
CI/CD, environment management, cutover automation, rollback drill. Owns the environment-health signal on the Governance dashboard.
Eleven two-week sprints, named Sprint Demo232 through Sprint Demo242. Pilot (canary) kickoff: 2027-01-24 — Sprint Demo241. GA: 2027-02-21 — Sprint Demo242. Sprints 1 and 2 have closed; Sprint 3 (Demo234) is in flight. Each sprint has a theme so the dashboards above have an easy “what should look different this sprint?” check.
What the 3 QA + 1 SDET Lead would run, mapped to the dashboards that would show the evidence. "Done" isn’t a test type — it’s a dashboard reading "green for N consecutive sprints".
Source profiling, record-count reconciliation per table, field-level diff on primary + risk columns, transformation-rule assertions.
Net-position reconciliation between core ledger, payment switch and central-bank sandbox; signed CSV; tolerance 0.01%.
Point-in-time restore of Postgres ledger, cross-check against legacy Oracle snapshot, dual-run cutback runbook.
ISO 20022 schema contract tests; Salesforce case-flow BDD; payment correctness including reject paths.
Baseline at 1x expected volume in Sprint Demo236 (Sprint 5); 2x by Demo238 (Sprint 7); 3x by Demo240 (Sprint 9) so the pre-pilot sign-off pack is complete. Chaos: broker loss, dead-letter spike, DB failover.
Threat model refresh; SAST/DAST in CI; audit of PII on new paths; penetration test pre-pilot.
Feature-flagged ramp 5% → 25% → 50% → 100% over Sprint Demo242; auto-rollback on error-budget burn.
Pilot survey weekly, NPS from agents (Salesforce desktop), operations stand-up readout; telemetry on top-5 agent flows.
Reconciliation. The intra-day job writes a signed CSV. The Reconciliation dashboard shows net position by rail, variance against tolerance, and the top-5 unresolved mismatches with the AI diff narrator’s explanation attached. Steering sees one number — "variance within tolerance for 14 consecutive runs" — not the CSV.
Rollback. We don’t promise a flawless cutover; we promise a credible cutback. The Pilot Rollout dashboard shows the rollback RTO/RPO measured in the Sprint Demo240 drill, the time it took, which runbook step was slowest, and the owner for the next drill. Steering sees "drill completed within RTO, 1 action" — and asks about the one action.
Pilot & feedback. Pilot ramp percent, error-budget burn, agent NPS and the top-5 feedback themes are the same card, so there’s no "green on numbers, red on sentiment" blind spot.
Test metrics alone don’t ship a banking release. These three views cover the non-test levers a Test Manager owns the conversation on: dependencies on the critical path, the sign-off layers between “QA green” and “Go-Live approved”, and the recovery options when a gate slips.
A dependency becomes a defect the moment it’s late. Each one has an owner outside the delivery team and a sprint we need it by.
“All tests pass” is not a release decision for a payment platform. Five layers, each with its own evidence pack and its own sign-off.
If a gate slips, the last conversation to have is “we missed”. The first one is “here are the levers”.
Six AI agents, each with a clear job, a human in the loop, and a dashboard they report to. "AI in QE" without a dashboard is a demo; with one, it’s a decision.
Clusters pipeline failures by stack-trace signature; proposes flake / env / real defect with cited evidence; a human approves before close.
Dashboards: Governance, Defect Analytics, Automation Effort.
Given a diff + 90-day defect history, proposes a <30-min smoke set with a coverage justification. Full suite still runs nightly.
Dashboards: Automation Effort, ROI, Governance.
Takes a numeric variance from the reconciliation job and produces a 3-line explanation with pointers to the responsible payment IDs and rules.
Dashboards: Reconciliation, Risk Register.
Generates edge-case payments (currency corner cases, reject codes, out-of-order message pairs) with provenance for every row.
Dashboards: Automation Effort, Defect Analytics.
Clusters application logs across the message path; drafts a root-cause narrative for the on-call to confirm.
Dashboards: Defect Analytics, Pilot Rollout.
Reads a BRD / user story, drafts Gherkin scenarios with acceptance criteria and risk notes; QA edits before accepting.
Dashboards: Test Strategy, Test Plan.
See the Agents page for the architecture of these agents (prompts, guardrails, evaluation set, HITL rules).
Positioning: the AI-assisted suite is an additional, faster PR feedback mechanism — not a replacement for the full-suite release gate. The release decision never rests on the AI selection alone.
What a credible "AI in QE" story looks like at Test Manager level: a versioned evaluation set with a defined denominator, baseline vs AI-assisted, severity-weighted misses, confidence/uncertainty on each decision, human override rate, and the disable/rollback mechanism — all before the saving is claimed.
| Dimension | Baseline (full suite) | AI-assisted selection |
|---|---|---|
| Evaluation set (versioned v3) | 180 historical PRs: 50 real defects + 130 clean | Same 180 PRs |
| Recall denominator | 50 known real defects | 50 known real defects |
| Gate time (median) | 42 min | 24 min |
| True positives (real defects caught) | 50 / 50 = 100% | 47 / 50 = 94% |
| False negatives (missed real defects) | 0 | 3 (all non-P1; 2 P2, 1 P3) |
| Severity-weighted miss score | 0 | 5 (P1=5pts, P2=2, P3=1; threshold ≤6) |
| False positives (flagged clean PRs) | 12 / 130 | 10 / 130 |
| Decision confidence | n/a | Low-confidence (<0.75) PRs auto-escalate to full suite |
| Human override rate | n/a | 8% of agent decisions overridden; each with a reason |
| Disable / rollback | n/a | Single CI switch; full suite re-enables within 5 min |
| Audit trail | CI logs | Agent decision + rationale + override logged per PR |
| Release gate | Full suite, every release | Full suite, every release (AI selection does NOT replace it) |
All figures are illustrative. The real decision — see Scenario 3 above — is to enable this agent at PR level only, keep the full suite at the release gate, track these numbers monthly on the AI QE Agents dashboard, and auto-rollback if the severity-weighted miss score exceeds 6 or more than 2 real defects leak in a rolling 30-day window. Any "hours saved" number presented to steering is estimated until measured over 2 sprints in production.
Each tile answers one question a Test Manager is asked. Tiles labelled Live sample open a working HTML page on this site (interactive demo). Tiles labelled Field spec jump to a design specification further down — what the dashboard would show, which KPIs it reports (with denominator, time window, source of truth and owner), and where AI plugs in. No guessing required which link opens a page and which opens a spec.
The release dashboard the way I present status to a steering committee — milestone roadmap, velocity, grooming & test metrics, defect aging, resource breakdown, risk register.
Test levels, in/out of scope, entry/exit criteria per sprint, test environments, data strategy, tool stack and the four acceptance gates.
Manual-hours avoided, release cycle time cut, defects caught pre-prod, and the net cost / benefit of the AI pod — with the formula visible.
New automation vs maintenance split per sprint, flake rate, pass-rate stability, mean time to repair a broken test, scripts per QA.
The stakeholder-first test plan for the data-migration workstream — scope, approach, environments, entry/exit, dependencies, risks and sign-off.
One RAG per gate (functional, data, performance, security, operations), outstanding risks, rollback readiness, pilot cohort state. Updated two days before every steering committee.
Defect aging matrix, injection vs detection phase, defect density by module, reopened-defect ratio, trend by severity.
Top-15 risks, probability × impact, owner, mitigation state, latest-mentioned date. Fed by the AI diff narrator for data risks and by SRE for perf/security.
Row counts and field-level diffs source → target per table; net-position variance per rail with tolerance band; signed artifacts to the regulator sandbox.
Ramp % (5/25/50/100), error-budget burn, cohort composition, agent NPS, rollback RTO/RPO measured in last drill, top-5 feedback themes.
Latency p95/p99 at 1x/2x/3x volume; broker-loss and DB-failover chaos results; SAST/DAST findings; pen-test state; audit trail completeness.
Per-agent precision/recall on their eval set, HITL override rate, hours saved vs baseline, incidents attributable to agent decisions, cost per 1k decisions.
For the tiles that don’t yet have a live sample page, this is the field-level specification — the one question it answers, the KPIs it tracks, and where AI plugs in.
Answers: How are we testing this programme, and why?
The strategy card has one tab per layer (unit, API/contract, UI, data, perf, security, chaos) with a per-sprint entry/exit, tool, environment and owner. The acceptance gates (functional, data integrity, performance, operational readiness) are visible with target thresholds, not vague adjectives.
Answers: What did QE automation save this programme?
Open formulas, not black boxes. Each metric shows "(cost today) vs (cost without the automation / without the AI agent)" using sprint-level data. The AI pod has its own cost line — the ROI number is net of agent spend.
Answers: Where is automation effort landing, and is it paying off?
Keeps the team honest about new vs maintenance. A sprint where 70%+ of QA time went to maintenance is a flag, not a surprise. Every KPI shows its denominator and scope beside the number — cumulative regression automation % and sprint-level automation progress use different populations and must never share a cell.
Answers: Can we ship? What’s open?
Four cards — Functional, Data, Performance, Operations — each with a RAG, an owner, a cite-your-evidence link, and a "what would make this red" sentence. Updated T-2 days before every steering committee. The dashboard makes the distinction between a provisional Test Manager recommendation and a formal release authorization unmistakable — a conditional go is not a go.
Answers: Where do defects come from, and where are they stuck?
Every metric here carries a definition, denominator, time window, source of truth, target and owner. A metric without those cannot be defended in a steering committee.
Answers: What could stop us, and who owns it?
Answers: Did the right data land, and does it still balance?
Answers: How is the canary going, and can we roll back?
Answers: Will it stay up and stay safe at pilot scale?
Answers: Are the agents earning their seat?
Audience × cadence × the moment they’re opened. If a dashboard doesn’t have a weekly moment, it dies — so each one is anchored to a specific ceremony.
| Dashboard | Primary audience | Cadence | Key moment it’s opened |
|---|---|---|---|
| Test Governance Metrics | Steering + PM | Weekly | Steering committee readout; sprint review. |
| Test Strategy | QA + Dev Lead | Per sprint | Sprint planning entry / exit gate. |
| ROI Calculator | Steering + PM | Monthly | Programme financial review. |
| Automation Effort | SDET Lead + QA | Weekly | Retro, dependency planning. |
| Test Plan | PM + Steering | Milestone | Gate reviews; plan vs actuals. |
| Release Readiness | Steering + Ops | T-2 before Steering | Go / No-Go call. |
| Defect Analytics | QA + Dev Lead | Daily / weekly | Triage; sprint review. |
| Risk Register | PM + Steering | Weekly | Risk workshop; steering review. |
| Reconciliation | Data QA + Regulator liaison | Hourly / daily | Intra-day job; regulator submission. |
| Pilot Rollout | Steering + Ops + PM | Daily (during pilot) | Canary ramp decisions. |
| Non-functional | SRE + Security + QA | Per load test | Pre-pilot sign-off. |
| AI Agents — Status | SDET Lead + Dev Lead | Weekly | Agent eval review; rollback decisions. |
The Test Governance Metrics page is a complete HTML sample you can lift into Confluence, GitLab wiki or SharePoint — the layout, not the numbers. The Test Plan is a stakeholder-first plan companion for the data-migration workstream.
If a dashboard above sounds useful but doesn’t have a live sample yet, that just means I haven’t published the template — the field spec is the honest, implementable version. Reach out if you want one adapted.
Contact →