ChaosLab

Overview

ChaosLab runs guarded failure experiments against the Spring PetClinic microservices on Docker Compose. It maps how the services depend on each other, predicts which user actions a given failure should break, injects one bounded fault into an allowlisted service, and reverts it automatically. It then compares what it predicted with what users actually received, and it separates "the request succeeded" from "the data was complete". Results are published as a read-only dashboard and a saved dependency map, so both can be explored without starting Docker.

Problem

A healthy status code does not mean a system is healthy. PetClinic's gateway has fallbacks, so a failed backend can still produce a 200 response that is quietly missing data. Plain error-rate monitoring would not notice. I wanted to see that gap with a real experiment, learn which signals expose it, and build the experiment so a fault could never be left running.

Constraints

• Local sandbox only — Docker Compose on one machine, not a cloud or shared environment
• Faults may only target allowlisted local services, one fault at a time
• A healthy baseline is required before any fault is injected
• The Spring PetClinic services are the upstream sample application; ChaosLab adds the experiment tooling around them
• Zipkin had no recent traces when the dependency map was generated, so some routes are labelled unknown or backed by source code rather than measured
• The AI planner and separate analysis agent described in the repository are planned, not implemented; the operator chooses the fault today
• Built as a two-day prototype

Approach

The Python controller runs each experiment in a fixed order: check a healthy steady state, inject one fault, watch users every second, revert, then measure recovery. A watchdog aborts on error rate, p95 latency, or throughput thresholds, a kill switch reverts active faults on demand, and a dead-man's switch reverts them even if the controller dies. Locust supplies steady simulated traffic, and an owner-data probe checks that the owner-details response still contains the expected visits. A dependency map built from Docker Compose, Docker, Eureka, Prometheus, and Locust observations (plus Zipkin traces where available) feeds a qualitative blast-radius prediction, which each run is later compared against.

Key Decisions

1. Decision 01

   Check data correctness, not only HTTP status

   Reasoning

   The gateway's fallback returns 200 when the Visits service is down, so status codes and error rates alone looked fine. The owner-details probe compares the response with a known baseline of six visits, which is what exposed the missing data.

   Alternatives considered

   • Error rate and latency only (misses silent degradation)
   • Manual inspection of responses (not repeatable)

2. Decision 02

   Make every fault bounded and self-reverting

   Reasoning

   An experiment tool that can leave a service broken is a hazard. The allowlist, watchdog, kill switch, and dead-man's switch each cover a different way the run could go wrong, including the controller itself failing.

   Alternatives considered

   • Manual cleanup after each run (easy to forget)
   • A single timeout (does not cover controller failure)

3. Decision 03

   Predict impact first, then compare with what was observed

   Reasoning

   Writing down which user actions should break before injecting the fault turns a demo into a test of the dependency model, and makes wrong predictions visible. Actions with too few finished requests are left unobserved rather than counted as contradicted.

   Alternatives considered

   • Inject faults and report whatever happens (no check on the model)

4. Decision 04

   Label evidence as measured, inferred, or stale

   Reasoning

   The saved map and the recorded run were captured at different times, and Zipkin had no recent traces. Showing source timestamps and manual corrections beside each claim keeps the output honest about what is measured and what is inferred.

   Alternatives considered

   • Present one clean topology with no uncertainty shown

5. Decision 05

   Publish a read-only recording instead of a live dashboard

   Reasoning

   A saved export can be opened by anyone without Docker, and it never injects faults or polls live services. The dashboard marks that it is a recording and states the demo date.

   Alternatives considered

   • Host a live sandbox (cost and safety exposure for a public site)

Tech Stack

• Python
• Spring Boot
• Docker Compose
• Locust
• Prometheus
• Grafana
• Zipkin

Result & Impact

• Recorded Run Visits service crash, 2026-10-10
• Gateway Response HTTP 200 with all six baseline visits missing
• Time to Revert Watchdog reverted the fault after 9.3 seconds
• Recovery Not confirmed — two baseline visits still missing at the last probe
• Fault Types Crash, freeze, network delay, packet loss, CPU starvation, traffic spike

The recorded run shows the point of the project: the gateway answered 200 while the owner's visit data was gone, and the watchdog reverted the fault after 9.3 seconds because throughput dropped below its threshold. The last recovery probe still lacked two baseline visits, so the run does not claim full recovery. Later experiments write a canary visit before injection and can report a service that recovered with lost data, but the committed recording predates that change and keeps its original outcome. This is a local prototype, not a validated or production fault-injection service.

Learnings

1. A 200 response is not evidence of a correct response — correctness needs its own probe with a known baseline
2. Safety controls should each cover a different failure, including the failure of the tool that is injecting faults
3. Predictions are only useful if they are written down before the experiment and compared with it afterwards
4. Labelling evidence as measured, inferred, or stale is part of the result, not decoration around it
5. A recording that says what it does not prove is more useful than a polished demo that hides it