Multi-location monitoring without false fleet-wide incidents
One failed route does not prove a service outage. Automated response needs independent signals and protection against switching an entire fleet at once.
Key points in one minute
- Decisions should distinguish one proxy, one location, and the whole service.
- Fleet-wide actions require quorum and concurrency limits.
- A quorum requires genuinely independent paths and probes.
- A fleet-wide failure pauses automation rather than accelerating replacements.
Independent probes
Use at least two authorized endpoints and, for critical decisions, a second network location. Correlated failures increase confidence.
Independence means distinct network paths and diagnostic targets, not two requests from one process. Assign a probe ID and maintain one timeline with known clock-synchronization tolerance. For a user flow, verify status and a safe content marker; a successful TCP connect proves only port reachability, not service health.
Failure classification
Classify DNS, TCP, TLS, HTTP status, response content, and timeout separately to identify the failing layer.
Define states such as healthy, degraded, local_path_failure, destination_failure, and unknown. Transition only after a defined number of consecutive signals within a window while retaining original DNS, TCP, TLS, and HTTP outcomes. Do not merge a timeout with an incorrect response body; they require different owners, alerts, and recovery actions.
Fleet-wide circuit breaker
If a large share of one location fails quickly, pause automatic replacements, continue diagnostics, and alert an operator.
The circuit breaker should reflect customer and location size. For example, if a meaningful share of the pool fails within a short window, pause automatic replacement and permit only a small canary. Derive exact percentages from history and capacity, but define concurrency limits in advance. Manual release requires confirmation from a second location and a recorded reason.
Gradual recovery
After recovery is confirmed, enable a small canary set and then expand in batches. Record successful and failed checks.
Begin recovery with a small canary batch. For example, 1–5% of the affected pool can be a starting point, but it is not a universal rule; size it from capacity, failure cost, and customer risk. After a full validation cycle, observe a stability window before increasing the batch. Track recurrent-failure rate, recovery time, and replacements per lease. If health worsens, stop and restore the last safe policy rather than launching a fleet-wide reversal.
Practical checklist
- 1Use two independent endpoints.
- 2Add a fleet-failure threshold.
- 3Recover in batches through a canary.
- 4Document states and transition conditions.
- 5Define a canary size and observation window.
Sources and documentation
This material is based on primary, official, and technical sources. The article is an original summary written for this knowledge base.