Fault Injection (Chaos Testing)

A saved response defines what a simulated endpoint returns. A fault profile defines how that response misbehaves: slower, with a random error status, with a dropped connection, or with a cut-short body. Fault injection lets you test that your client retries, times out and recovers — without touching a real, flaky server.

Fault vs. functional error — a functional error is a normal response you configured (a 404 with a JSON body, or a SOAP fault): it is always returned when its condition matches. A fault is injected degradation: it is probabilistic and transport-level (latency, abort, truncation), and it can be armed and disarmed at run time. The two never replace each other.

Quickest scenario: verify a retry loop in one test case

1. Open the endpoint's Edit Response window and, in the Fault injection section of the left pane (below Condition), check Error reply — the rate seeds itself to 100 — and pick HTTP 503 Service Unavailable in the failure-code picker.
2. In your test case add, in order:
   • Send Request → the healthy call succeeds.
   • Service Simulation - Fault Injection step → Action: Arm, same server/operation/response, Failure code 503.
   • Send Request → assert the call failed (503 / retry triggered / timeout).
   • Service Simulation - Fault Injection step → Action: Disarm.
   • Send Request → the healthy call succeeds again.
3. Run the case. The simulation server itself stays up the whole time — only the response degrades, exactly like a real backend hiccup.

When to use

• Your client has retry / backoff / circuit-breaker logic and you need a server that fails on demand.
• You want to prove a timeout fires: arm Latency above the client's timeout.
• You simulate partial failures: Truncate cuts the body (or the gRPC stream) before completion.

Configuring a fault on a saved response (manual / ad-hoc)

In the Edit Response window, the FAULT INJECTION section sits in the left pane below CONDITION — the two sections share the pane through a draggable divider (each keeps its header visible at the minimum), and the section's header badge shows how many fault mechanisms are active. It works for HTTP, SOAP and gRPC responses alike. Timing lives here too: Delay is the response's fixed wait (applied to every request), and Jitter is a checkbox that adds a random extra 0–N ms per hit on top of it. The three delivery outcomes — error reply, abort connection, truncate body — are a radio group, because only one of them can reach the wire per request. Selecting an outcome with a still-zero value seeds a sensible starting point (100 % error rate, 300 ms jitter, 50 % truncation); clicking the selected radio again clears it (no fault).

Row	Meaning
Delay [s]	The response's fixed wait before replying (was the separate Response Delay field). Applied to every request; not fault-gated.
Jitter ✔ [ms]	Random extra delay in [0, value] added on top of the delay, per request (per frame on streaming gRPC). Composes with any selected outcome.
Error reply ◉ [%, → code]	Probability (0–100) that a request is answered with the failure code instead of the configured status. 0 = the code is never used, 100 = every request (the deterministic setting).
Failure-code picker	Pick-list with free typing; the box shows the bare code, the list the full names — HTTP 400/401/403/404/408/409/422/429/500/501/502/503/504, or the gRPC status codes 1-16 (listed first on a gRPC operation). Empty or 0 is answered as 500.
Abort connection ◉	Drop the socket (HTTP/2 stream for gRPC) without completing the response — the client sees a connection error, not a status code.
Truncate body ◉ [%]	Deliver only this share of the response (1–99) and then close early: the client sees a body shorter than announced, or a stream cut mid-flight.

A plain-language preview line at the bottom of the pane (Effective behavior: …) restates the combined effect and names ignored settings.

Save semantics

The jitter checkbox and the outcome radio are the fault switches, so there is no separate master toggle and nothing can be configured invisibly: the fault profile is armed exactly when the jitter box or one radio is set. The Delay row is baseline behavior — it applies whether or not a fault is configured. Selecting another radio switches the outcome; clicking the selected one again clears it. Saving stores the profile with the response, so it travels with the project and with .mockconfig.json exports.

How the settings combine

The editor offers extra latency plus one delivery outcome at a time (they are mutually exclusive there), but imported profiles — from *.faults.json or hand-edited mockconfigs — may still combine outcomes. The engine applies them in one fixed order either way, and the editor prints that order in a plain-language line at the bottom of the pane (Effective behavior: ...), which also names the settings that will be ignored. Saving such a combined profile from the editor persists only the outcome the radio shows (the one that reaches the wire).

Order	Setting	What actually happens
1	Delay + Jitter	The reply waits the fixed delay plus a random [0, value] ms of jitter. Both are applied even when the connection is aborted, so a client still sees the wait it would see against a slow backend.
2	Error rate (%) + Failure code	The error rate decides whether the response is degraded; the failure code is returned only on the requests where the roll triggers. 0 = the code is never used, 100 = every request, which is the deterministic setting.
3	Truncate body (%)	Only this share of the body (or of the stream frames) is written before the connection closes.
4	Abort connection	The connection is dropped before any response is written, so the failure code and the truncation have no effect at all — the editor marks them ignored while this is on.

Two consequences are worth remembering, because neither is visible from the labels alone:

• A failure code alone does nothing. Failure code = 503 with Error rate = 0 never returns 503.
• Abort beats everything except latency. Latency is applied before the drop; the failure code and the truncation never reach the client.
• A truncated reply never completes. The body is cut and the connection closes early, so the client's transport raises an error instead of delivering a reply. The bytes written before the cut reach the client: the send view opens an Error tab with the transport's explanation, and the response viewer shows the partial body in the Response tab when some of it made it through. Turn the truncation off to watch the error-rate mix.

The Fault Injection test step

Category Service Simulation, display name Service Simulation - Fault Injection. It arms or disarms a fault at run time — the same profile the editor writes, addressed by (server, operation id, response name).

Step properties

Property	Required	Description
Simulation Server	yes	The simulated server hosting the operation (ComboBox of available servers).
Operation Id	yes	The operation's id, e.g. GET /orders/{id}.
Response name	no	Name of the saved response; leave empty for the default response.
Action	yes	Arm or Disarm.
Extra latency (ms)	no	Same meaning as the editor's latency field.
Error rate (%)	no	Same as the editor.
Failure code (HTTP/gRPC)	no	Same as the editor: a pick-list with free typing, gRPC codes first on a gRPC server.
Abort connection	no	Same as the editor.
Truncate body (%)	no	Same as the editor.

The step fails (and reports a diagnostic) when no server or operation is selected.

Deterministic faults (target one condition)

A fault is stored per saved response, and a condition decides which response is served. Put those two facts together and you get a deterministic, condition-targeted fault:

1. Add a conditional response to the operation (for example a condition on header X-Chaos = 1) with the body and status you want to degrade.
2. In that response's Fault injection section select Error reply, keep the rate at 100 and pick the failure code — or select Abort connection for a hard network failure.
3. Save.

Only the requests that match the condition are degraded, every time, with no randomness, and the healthy path keeps its own fault-free response. This is the same thing the Fault Injection test step does at run time; the difference is only what selects the degraded response — a request predicate instead of a test step.

Want	Setting
Always fail with a status	Error rate 100 + failure code
Always time out	Delay well above the client timeout (deterministic — every request exceeds it)
Always look like a network failure	Abort connection
Fail roughly one request in three	Error rate 33
Cut a body short	Truncate body 1-99 (the announced Content-Length stays the full one, so the client sees a premature close)

Jitter is the one setting that is not exact: the engine picks a random extra delay inside [0, value]. Use Delay when you need a guaranteed timeout; add jitter when the client should see mostly slow replies.

gRPC status codes

On a gRPC operation the Failure code picker lists the gRPC status codes instead of the HTTP ones (the list is ordered by the endpoint protocol; both families stay selectable).

Code	Status	Code	Status
1	CANCELLED	9	FAILED_PRECONDITION
2	UNKNOWN	10	ABORTED
3	INVALID_ARGUMENT	11	OUT_OF_RANGE
4	DEADLINE_EXCEEDED	12	UNIMPLEMENTED
5	NOT_FOUND	13	INTERNAL
6	ALREADY_EXISTS	14	UNAVAILABLE
7	PERMISSION_DENIED	15	DATA_LOSS
8	RESOURCE_EXHAUSTED	16	UNAUTHENTICATED

Precedence rules (how a fault is chosen)

For each request the simulation server picks the effective fault in this order:

1. Runtime-armed profile — a fault set by this step (or the daemon's fault endpoint) wins over everything.
2. Persisted profile — the Fault saved on the response (from the editor).
3. None — the response is returned exactly as configured.

A disabled profile ("Enabled": false — possible only in imported or hand-edited *.faults.json / mockconfig files; the editor can no longer produce one) is treated as none.

Runs hosted by the Aethrix daemon

When the run process is launched by the Aethrix daemon (AETHRIX_SIM_ENDPOINT is set), the step routes through POST /v1/simulations/faults on the daemon, and the fault applies to the daemon's simulation server — the run child does not host a competing server. Semantics are identical to the in-process path.

Exporting fault profiles as a profile file

Fault profiles travel two ways:

1. With the mock configuration — aeth sim export / aeth sim import (and the Studio wizard) carry each response's fault along with it, because the profile is part of the response.
2. Standalone profile — for sharing one chaos setup across services without copying whole responses:

# Export every fault configured on a server's responses
aeth sim fault-export checkout.faults.json --server "HTTP Server" --description "checkout chaos"

# Apply it later (matches by operation id + response name)
aeth sim fault-import checkout.faults.json

Entries that match no response are reported as No Change — they never fail the import. The file is plain JSON (version, exportedAt, description, faults[]), keyed by (server, operation, response).

Behavior by protocol

Setting	HTTP / SOAP	gRPC
Extra latency	added before the response is written	added before the response and to each stream frame delay
Error rate + Failure code	replaces the HTTP status, body not sent	answer carries grpc-status: <code> and no message
Abort connection	socket dropped, no response completed	HTTP/2 stream dropped before the grpc-status trailer
Truncate	fewer bytes than the announced Content-Length, then close	fewer stream frames than configured, then cut

What fault injection does not do

• No faults below the HTTP/2 layer (packet loss, DNS) — this is a simulator, not a network proxy.
• No message-queue fault injection (Kafka/AMQP) — queue protocols are separate.
• It never changes the configured functional error responses: a SOAP fault configured on a response is still returned as content when matched; the fault profile only perturbs transport behavior on top.

Troubleshooting

• The fault has no effect. In the editor, select an outcome radio (or tick jitter); for the test step, use Action: Arm. A profile imported with Enabled: false stays inert until enabled.
• The step fails immediately. Server and Operation Id are both required; the diagnostic says so.
• The fault survived a disarm elsewhere. Precedence is runtime-armed → persisted → none; if the response itself carries a persisted fault, disarm only removes the runtime layer — clear the editor's fault section to remove the persisted one.