Flaky network tests: tracing, retries, and an honest diagnosis
A retry can confirm flakiness but does not fix it. Preserve the first failure as a separate result with network events and page state.
Key points in one minute
- Flaky is an investigation signal, not a clean pass.
- A retry should start in an isolated worker with controlled state.
- A successful retry does not erase evidence from the first failure.
- Retry irreversible steps only with idempotency and reconciliation.
Evidence from the first failure
Keep status, token-free URL, step, duration, and trace. Do not overwrite them with the successful retry's data.
Link the first failure to run ID, test ID, route ID, and a safe lease identifier. Preserve request sequence, statuses, and timing before starting a retry. Traces, screenshots, and console output may contain personal data or tokens, so filter them before upload and apply bounded retention. Record a successful retry alongside, never over, the original artifact.
Bounded retries
Use a small retry count and report flaky separately. Unlimited retries increase load and conceal regressions.
One or two retries with identical configuration and a controlled delay are usually enough. Count passed, failed, and flaky separately and track instability against history. Do not retry payments or other irreversible steps without an idempotency key and post-check. A flaky-rate increase above baseline should block release or create a mandatory investigation task.
State isolation
Playwright restarts a worker after failure. Test data and external operations must tolerate a new process and not depend on an incomplete step.
Each retry receives a fresh browser context and clean test data while retaining the parent correlation ID. Create external objects idempotently or assign a unique name and verify cleanup. If a previous step has an unknown outcome, the new process must reconcile actual state before replay; otherwise the test itself can create a duplicate.
Root-cause classification
Separate client, proxy, mobile route, DNS, TLS, and destination failures. Fix or alert only the confirmed layer.
Use a short decision tree: does the failure reproduce without the proxy, on one path, across all destinations, or on one endpoint only? Compare DNS, connect, TLS, status, and content. Assign a cause only after independent confirmation; until then use unknown rather than blaming the carrier or application. The report should include evidence and the next safe diagnostic step.
Practical checklist
- 1Preserve the first trace.
- 2Limit retries.
- 3Report flaky separately.
- 4Track flaky rate separately by route.
- 5Add a decision tree to the report template.
Sources and documentation
This material is based on primary, official, and technical sources. The article is an original summary written for this knowledge base.