How I got here in 90 minutes
Full transparency: I'm not a testing-tools PM (yet), and I walked in knowing the category as a user, not an owner. So step one was leveling myself up, fast, and I'm bringing you along on that journey rather than hiding it.
1 · Read the six signals, clustered them
2 · Built myself a crash course on flake with AI → the deck that taught me
3 · Sent research agents to primary sources, sourced-only, no guessing
4 · Formed the POV below + a concept mock
5 · Ran two independent fact-check agents against my own doc, they caught an error, I corrected it (details at the end)
Step 2 is linked on purpose: the crash-course deck is the artifact of me learning this space in real time this afternoon, basics of E2E testing → why flake corrodes trust → what Cypress ships today → the competitive picture. It's what informed the ideation below, including the Spring Cleaning concept.
Reading the six signals
Five of the six describe the same journey breaking at different steps. One (F) seems like it may need additional consideration and discussion.
ARe-run failed tests onlyWants to retry the 3 failures from last night without re-running 97 passes. A workflow response to failures they don't yet trust.
BTriage flake by causeEnvironment vs data vs Cypress error: "may or may not be a priority to fix, or maybe can't be fixed." Wants flake classified, not just counted.
CDIY quarantineHand-rolls quarantine with tags + a side pipeline, and the trend data punishes the test while it's being rehabbed. Builds the missing feature himself, badly, because he has to.
DFlaky label hid a real regressionDev merged over a "known flaky" failure; next day the same failure was real and shipped to production. Asks: does anyone notice when a flaky test's behavior changes?
ESelective silenceKnown external flake (a login page another team owns). Wants to filter that noise, without silencing everything.
FParallelization & orchestrationSharding, subset runs, per-shard env configs. Real churn language ("why we started looking elsewhere"), but a different problem: CI orchestration flexibility, not flake trust. Named, and set aside.
The through-line (A–E): once a test is marked flaky, Cypress treats that as a terminal state. Customers are asking for what comes next: triage → quarantine → rehabilitate → re-trust, and an alarm when "flaky" stops being flaky. D is the one that costs real money: the flake label actively suppressed a true failure.
What the customer sees today
Current-state walkthrough, all images from Cypress's own docs (each links to its source page). This is a genuinely strong detection and reporting surface. Note what's absent: any state a flaky test can move through.
Detection: flake is flagged per run, and filterable
Analytics: rate over time, severity buckets, per-test detail
Depth exists: attempts, artifacts, history, replay
Alerting: flake announces itself, once
cypress/flake commit status on GitLab/Bitbucket, teams can configure these to block merges when flake is detected in a run. Powerful gate, but it fires on detection only: nothing follows the test afterward, and nothing distinguishes "same old flake" from "this one just changed." Source
What today's model amounts to (flake defined as "passes and fails across retries without any code change"):
- Detect — retries required, runs recorded to Cloud, Team plan+.
- Report — badges, analytics, severity buckets, per-test history.
- Announce — Slack/Teams alerts, PR status checks.
- Debug — Test Replay on the exact CI attempt.
What a test cannot do is change state:
- No quarantine or mute — "quarantine" appears zero times in the flake-management, retries, and experiments docs.
- No rehabilitation — no stability tracking toward earning trust back; the label is forever.
- No behavior-change alarm — nothing distinguishes "same old flake" from "this just became a real failure."
- And customers have asked — cypress #32178 requested selective flake silencing (signal E's exact case), closed June 2026 as "not planned" with a note that Product tracks the feedback internally. Demand exists; no public roadmap commitment covers it.
Q1The core problem
Cypress tells you a test is flaky, then abandons you. Flake is a permanent, binary label with no lifecycle behind it, so teams stop trusting the signal, and eventually stop trusting the suite.
There are two distinct problems in the signal:
- Flake lifecycle management (A, B, C, D, E) — the one I'd prioritize. Every one of these customers is inventing a workflow Cypress doesn't provide: rerun-failed-only (A), cause triage (B), tag-and-side-pipeline quarantine (C), manual behavior watching (D), selective silencing (E). C is literally describing a quarantine feature spec from the outside.
- Parallel orchestration flexibility (F). Sharding, subset runs, per-shard configs. Serious, churn-flavored language, but a different system (run orchestration, not flake trust). I'd hand it its own investigation rather than dilute this one.
Why lifecycle is most important: the trust death-spiral
D is the catastrophic case and the reason this ranks first. The sequence: test flakes → gets labeled → label teaches the team to ignore it → a real regression arrives wearing the flaky label → it ships. The docs themselves note a test can flake while showing zero final failures, so nothing forces a second look. The flake feature, working as designed, suppressed a true positive. Once that happens, teams distrust every green run, which devalues the exact thing Cypress Cloud sells: a signal you can act on. B, C, and E are the same trust problem at earlier stages; A is its workflow cousin.
Q2What I'd confirm before recommending anything
| Question | Why it matters | How I'd get it |
|---|---|---|
| How widespread is DIY quarantine (C's tag-and-side-pipeline pattern)? | If many teams hand-roll it, the feature is validated demand, not a one-off. C's workaround also corrupts their trend data, so adoption pain is measurable. | Support ticket + community mining; public GitHub asks are already evidence (#32178, closed Jun 2026 "not planned"; #22662, open). |
| How often do D-type incidents happen (regression hidden by a flaky label)? | It's my "most important" argument; one anecdote isn't a trend. If measurable, it's also the business case. | Cypress's own Data Extract API (`flaky-test-details` is near-real-time) joined against later real failures of the same tests; customer interviews. |
| What share of customers even have retries on (the prerequisite for all flake features)? | Flake detection only works with retries enabled and runs recorded (docs, default is 0). If adoption is low, the funnel starts earlier than lifecycle. | Product analytics on config telemetry; adoption of the experimental retry strategies. |
| Who exactly are these six voices (plan tier, scale, churned vs active)? | F uses churn language; if F-type enterprises are the revenue risk, priority could flip toward orchestration. Flake analytics is Team-plan+, so lifecycle work mostly serves paying tiers. | Tie feedback to accounts; churn-reason coding; segment revenue. |
| Can error signatures actually distinguish "same old flake" from "new behavior"? | The behavior-change alarm (Q3) depends on it. Attempt-level errors and artifacts exist per the retries docs; question is signal quality at scale. | Prototype on Data Extract error data; measure false-positive rate. |
Q3The one thing: a flake lifecycle, with quarantine at its core
Give a flaky test somewhere to go. A managed state machine: Flagged → Quarantined (still running, out of the pass/fail path) → Rehabilitating (stability streak tracked) → Restored, with one alarm that cuts across it: "this flaky test's failure just changed character."
Direction, engineer-legible, deliberately not over-specified:
- Quarantine as a first-class state (per test, permissioned, time-boxed). The test keeps running and recording, but stops gating CI and stops polluting suite health metrics, fixing C's "trend punishes the test while I fix it" and E's "filter the noise I can't act on." Time-boxing prevents C's other failure mode: "skipped and never thought of again."
- Rehabilitation, not amnesty: track consecutive stable runs in quarantine and prompt restoration. (BrowserStack's mute has exactly this auto-unmute concept, docs; Datadog runs a full quarantine/disable/attempt-to-fix lifecycle, docs. Playwright has no quarantine, closest is skip/fixme, so this is also differentiation against the runner Cypress loses mindshare to.)
- Behavior-change alarm (the D fix): Cypress already stores per-attempt errors, artifacts, and history (retries docs), signatures we'd derive from them. When a known-flaky or quarantined test starts failing with a new error signature, or its flake-vs-fail ratio shifts, alert through the existing Slack/Teams/PR-check rails (already built). "Flaky" stops meaning "ignore forever."
- Why Cypress can win here: the raw material already exists: attempt artifacts, error signatures, Test Replay, previous-runs timeline, near-real-time flake export. This is composition and product judgment, not a data moonshot.
- "Spring cleaning" — the cheap first slice. The lifecycle handles the flow; this handles the stock: a periodic, grouped review of every test currently wearing the flaky label, with recommendations to clear it, based on consecutive stable runs, not elapsed time ("passed its last N runs" is evidence; "hasn't failed in 90 days" might just mean it barely ran). Ships as read-only recommendations first, no state machine needed, entirely on data Cypress already collects (flake rate over time, run history). Directly answers D's "is it just marked flaky forever?" and C's "skipped and never thought of again," and it doubles as the validation probe: how many stale labels exist tells you how big the lifecycle problem really is. Concept mock →
How I'd validate before building big: design-partner with 3–5 Team/Enterprise accounts that show C-style workarounds; ship quarantine behind a flag; success = reduction in DIY tag-quarantine patterns, restored-test rate, and (the real prize) caught behavior-changes that would previously have merged. Guardrail: quarantine volume itself, if everything gets quarantined, we built a mute button, not a lifecycle.
Deliberately not designed in detail: exact UI and auto-quarantine policy thresholds (Datadog-style rules could be v2). But to make Spring Cleaning concrete, here's a directional concept mock.
Concept mock · Spring Cleaning dashboard
A directional sketch, not a spec. Built to show what "grouped recommendations, evidence-based, batch-actionable" could feel like inside Cypress Cloud's existing design language.
Why this sits in the paid plan, and why that matters: flake detection, analytics, and alerting are already Team-plan features (docs). Spring Cleaning and the lifecycle land exactly where Cypress monetizes: they deepen the reason to pay for Cloud on top of a free runner, in the precise spot where free Playwright has no answer. The open-source runner earns the love; trust management like this is what the Cloud subscription is for. Retention and upgrade driver, not a new SKU.
Q4Where I wouldn't trust my own output
- Six anonymous quotes, unknown weights. I treated them as representative; if they're cherry-picked or all from one segment, my ranking wobbles. Changes my mind: ticket-volume and account data showing F-type orchestration pain dominating churn, then F leads, not lifecycle.
- D is one anecdote carrying a lot of argument. It's vivid, and vivid can out-argue frequent. Changes my mind: Data Extract analysis showing flaky-masked regressions are vanishingly rare, then B/C/E-style triage-and-quiet workflow wins over the alarm.
- The alarm only works if "new failure" is actually detectable. My behavior-change alarm assumes we can reliably tell a test's usual flake apart from a genuinely new kind of failure by comparing errors. If real-world flake errors are too messy for that comparison, the alarm would fire on noise, and a false alarm engineers learn to ignore makes the trust problem worse, not better. Changes my mind: prototyping it on real flake data and finding too many false alarms.
- Competitive framing is docs-based, not usage-based. I verified Datadog/BrowserStack ship lifecycle features, not whether their customers love them. Changes my mind: evidence those features go unused, which would suggest the demand is louder than the need.
- AI assembled my evidence base under time pressure. I directed it to primary sources and every claim above carries its link, but a misread doc is possible, which is exactly why each one is verifiable in one click.
How I used AI in this exercise (90 minutes)
Per the brief, the interesting part isn't that I used it:
- Parallel research agents, sourced-only rules. Two agents ran simultaneously: one pulled current-state UI images from Cypress docs (with source page per image), one verified product facts and the competitive landscape, under an explicit "no guessing, flag anything unverifiable" instruction. They returned in ~3 minutes with citations; notably they flagged what they couldn't verify (e.g., don't claim quarantine for Develocity, Launchable unchecked) rather than filling gaps.
- The judgment stayed human. Which of the six signals cluster, which problem ranks first, why D is the spine, what not to build yet, that's the part I'd defend in this room, and it's mine.
- Strong vs weak output, in practice: The strongest find was something anyone can check: Cypress's docs never mention quarantine, and a customer filed a public request for it that was closed as "not planned." The weakest output was what the AI "remembered" about competitors without a source, so the rule was simple: no source, no claim. Those got cut.
Detection docs: flaky-test-management
Retries: test-retries
Experimental strategies: experiments
Silencing ask (closed "not planned" Jun 2026): cypress#32178
Datadog lifecycle: flaky_management
BrowserStack mute: mute-tests
Playwright: retries · annotations