Flake 101 · A teaching deck for Kevin
From "what even is this product" to a POV you can defend
Seven short lessons. Basics of the product area → why flake is a real business problem → what Cypress does today → the gap → your recommendation, including your spring-cleaning idea.
Navigate: → next · ← back · click the dashed boxes to test yourself before revealing answers.
The one-sentence destination: Cypress is great at detecting flaky tests, but a flaky test has nowhere to go afterward, no lifecycle, and that erodes the very trust the product sells.
Lesson 1 · The product area
What Cypress is, in plain terms
Teams that build web apps write end-to-end (E2E) tests: scripts that drive a real browser like a user would, click login, fill the form, check the dashboard loaded. They catch what unit tests can't: the whole system working together.
The Cypress App (open source)The test runner developers love. Free. Runs tests locally or in CI. This earns the mindshare.
Cypress Cloud (paid)The business. Records runs from CI, stores history, artifacts, analytics, flake detection, Test Replay. Team/Enterprise plans gate the good stuff.
Tests run two ways: on a developer's laptop while coding, and, the one that matters here, in CI (continuous integration): automatically, on every pull request or on a nightly schedule, across many machines in parallel.
Dev pushes code
→
CI spins up machines
→
Runs the test suite
→
Pass = merge/ship
·
Fail = blocked
The test suite is a gatekeeper. Its entire value is that a green result means "safe to ship" and a red result means "stop." Keep that word, trust, in your pocket. The whole exercise is about it.
Lesson 2 · The villain
What a flaky test is, and why it's poison
A flaky test fails sometimes and passes sometimes with no code change. Cypress's own definition: "a flaky test passes and fails across retries without any code change" (docs).
Where flake comes from
Timing / race conditionsThe test clicks before the page finished loading. Passes on a fast machine, fails on a slow one.
EnvironmentCI machine hiccups, network blips, a third-party login page you don't own (that was Feedback E's exact case).
Test dataShared state, leftover records, tests that depend on each other's leftovers.
Real bug, intermittentThe scary one: an actual race condition in the product that only bites sometimes. Looks identical to the others.
Why it's expensive
Every red build stops a team. If red sometimes means "nothing's wrong, run it again," engineers start treating red as noise. That's the boy-who-cried-wolf effect: the suite still runs, but nobody believes it. The gatekeeper is still standing there, everyone just walks around it.
Check yourself: why is a flaky test worse than a consistently failing test?
Click to reveal
A consistent failure gets fixed, it's loud and reproducible. A flaky test teaches people to ignore failures. It doesn't just waste time; it corrodes the signal for every other test. The damage is to trust, not just to minutes.
Lesson 3 · Current state
What Cypress does about flake today
The machinery, in the order it fires. (Prerequisite: retries must be on, default is off, and runs must be recorded to Cloud. Flake features are Team plan+.)
Test fails
→
Auto-retry (configurable per project/suite/test)
→
Passes on retry = flagged FLAKY
→
Badge + analytics + alert
DetectFail-then-pass in one run = flaky. Flagged in the run header and Latest Runs; filterable.
ReportAnalytics: flake rate over time, severity buckets (Low >0–10% · Med >10–50% · High >50%), per-test panel with errors + history.
AnnounceSlack/Teams alerts on every flaky run; GitHub status checks + PR comments; GitLab/Bitbucket cypress/flake checks that can block merges.
DebugTest Replay: step through the exact CI attempt (DOM, network, console), compare a passing vs failing attempt.
The flake analytics view: counts over time, severity groups, per-test log. Source
Check yourself: a test fails twice, passes on the third attempt. What does the run summary show, and why is that sneaky?
Click to reveal
The run is green, final status passing, flagged flaky. The docs note a test can show zero final failures while still flaking. Sneaky because nothing forces a human to look: the build shipped, the flake quietly accumulated.
Lesson 4 · The gap
Detection is a dead end: the label has no lifecycle
Once a test is marked flaky, that's it. It's a permanent, binary sticker. There is no next step in the product.
| Customers asked for (the six signals) | Exists in Cypress today? |
|---|
| A · Re-run only last night's 3 failures | Partial at best, workflow friction |
| B · Triage flake by cause (env vs data vs test bug) | No, artifacts exist, classification doesn't |
| C · Quarantine a test, rehab it, restore it | No. "Quarantine" appears zero times in the docs. Customer C built it himself with tags and a side pipeline, and the trend data punished him for it. There's also GitHub issue #32178, a public feature request asking to silence flaky-test notifications for specific tests, closed June 2026 as "not planned" with a note that Product is tracking the feedback internally. Translation: customers asked, and nothing public is committed. (It's an issue/request, not a pull request, no code was written.) |
| D · Tell me when a "known flaky" test changes behavior | No. Alerts fire when flake appears, never when an existing flaky test starts failing differently. |
| E · Selectively silence flake I can't act on | No, it's all-or-nothing |
| F · Parallel sharding config flexibility | Different problem (orchestration). Name it, set it aside. |
The D story is the spine. Dev sees a failure → "oh, that one's flaky" → merges → the same failure was a real regression → production. The flake feature, working exactly as designed, suppressed a true positive. That's the trust death-spiral completing its loop.
Check yourself: why is "the label has no lifecycle" a better problem statement than "Cypress needs a quarantine feature"?
Click to reveal
Quarantine is a solution; the lifecycle gap is the problem. Framing it as the problem explains all five signals at once (A's rerun, B's triage, C's quarantine, D's alarm, E's silencing), and leaves room to sequence solutions instead of jumping to one feature. PM discipline: name the problem, not your favorite fix.
Lesson 5 · The market
Everyone detects. The leaders manage.
| Tool | Detects flake | Lifecycle (quarantine/mute/rehab) | Source |
|---|
| Cypress Cloud | Yes, best-in-class reporting | None | docs |
| Playwright (the OSS rival) | Yes (retry-based, same idea) | None, closest is skip/fixme annotations | docs · annotations |
| Datadog Flaky Tests Mgmt | Yes | Full lifecycle: Quarantine (runs in background, doesn't gate CI) · Disable · Attempt-to-Fix · auto-policies | docs |
| BrowserStack Test Observability | Yes | Mute with auto-unmute after consecutive stable runs, note: that's rehabilitation | docs |
Two reads on this table: (1) The lifecycle concept is validated, observability leaders already ship it. (2) Playwright doesn't have it either, so this is differentiation against the tool eating Cypress's open-source mindshare, in the exact layer (Cloud) where Cypress makes money.
Check yourself: why does Cypress have an unfair advantage building this vs, say, Datadog?
Click to reveal
Cypress owns the richest raw material at the test level: per-attempt errors, screenshots/video, Test Replay, previous-runs timeline, and a flake export API (Enterprise plan). Datadog sees CI results; Cypress can see why the attempt failed. Lifecycle here is composition of existing assets, not new data collection.
Lesson 6 · Your POV
Give a flaky test somewhere to go
A managed lifecycle, one alarm, and a broom.
Quarantine, in plain English: a sick-bay for a misbehaving test. It keeps running on every build so you still collect evidence, but its failures stop blocking merges and stop dragging down your suite's health stats while someone fixes it. The alternative people use today, skipping the test, turns it off entirely: no data, no pressure to fix it, forgotten forever. Quarantine is "benched but still practicing," skip is "cut from the team."
Flagged
→
Quarantined
still runs, stops gating CI, stops polluting metrics, time-boxed
→
Rehabilitating
stability streak tracked
→
Restored
The alarm (fixes D)When a known-flaky test fails with a new error signature, or its flake-vs-fail ratio shifts, alert through the Slack/PR rails that already exist. "Flaky" stops meaning "ignore forever."
Spring cleaning (your idea, the cheap first slice)Periodic grouped review of every test wearing the label, with clear-it recommendations based on consecutive stable runs, not elapsed time ("hasn't failed in 90 days" might mean it barely ran). Read-only, ships fast, and the count of stale labels it finds is the validation data for the bigger investment.
GuardrailIf everything ends up quarantined, you built a mute button, not a lifecycle. Watch quarantine volume and restored-test rate.
How you'd validate before building big
Design-partner with 3–5 Team/Enterprise accounts showing C-style DIY quarantine; ship behind a flag; success = fewer workarounds, healthy restore rate, and caught behavior-changes that would previously have merged.
Check yourself: Mark asks "why not just build better auto-retry so flake never surfaces?" What's your answer?
Click to reveal
Retries are suppression, not information. More retries make the run greener while the underlying flake (and any intermittent real bug hiding inside it) accumulates silently, that's how D happened. The goal isn't to hide flake; it's to manage it transparently so the signal stays trustworthy. Retry tuning (the experimental strategies) is a useful lever inside the lifecycle, not a substitute for it.
Lesson 7 · Say it out loud
The 60-second version for the review
"Five of these six signals are the same problem at different stages: Cypress detects flake brilliantly and then abandons you. The label is permanent and binary, there's no triage, no quarantine, no rehabilitation, no alarm when 'flaky' starts hiding something real. D is the proof of cost: the label suppressed a true regression into production.
So the one thing I'd do is give a flaky test somewhere to go: quarantine that keeps running but stops gating CI, rehabilitation tracked by stable runs, and a behavior-change alarm on the alerting rails that already exist. I'd start with a spring-cleaning pass, grouped recommendations to clear stale labels based on consecutive stable runs, because it's read-only, ships fast, and tells us how big the problem really is.
F, the parallelization feedback, is real and churn-flavored, but it's an orchestration problem, not a trust problem. I'd give it its own investigation rather than dilute this one."
Likely pushback → your counters
| "Isn't this just Datadog's feature?" | Concept's validated there, but Cypress has attempt-level artifacts and Test Replay they don't. And Playwright has nothing, this is Cloud differentiation. |
| "Why rank D over F? F mentioned churn." | Honest answer: with only six anonymous signals I can't be sure, it's the first thing I'd confirm with account/churn data. I ranked lifecycle first because it breaks the product's core promise (trustworthy signal) and five signals point at it. |
| "Where would you not trust yourself?" | Six quotes with unknown weights; D is one vivid anecdote; the alarm assumes error signatures separate cleanly, a noisy prototype would kill it. Each has a named check that would change my mind. |
Full sourced version with every image and citation: exercise-flake-pov.html (same folder). Everything in this deck traces to those links.