Lab L09 · Java + Go

Implementing a retry and circuit breaker executor

Build an executor that protects calls to a flaky dependency in two ways. Retries: if an attempt fails with an exception you declare retryable, wait (backoff) and try again, up to attempts times. Circuit breaker: after threshold consecutive failed attempts, stop calling the dependency for cooldown, failing fast with OpenCircuit; after the cooldown, let exactly one probe call through to decide whether to close the circuit again.

The task

The payments service calls a card processor. A timeout is worth retrying; “card declined” is not. If the processor goes down, every request retrying several times multiplies the load on a struggling dependency and ties up threads waiting on it. The breaker notices the run of failures and fails calls immediately for a while, then tests the water with a single probe. Clock, sleep and cancellation are injected so the tests are deterministic.

The three breaker states. Retries happen inside one execute call, but each attempt is reported to the breaker, so retries can open the circuit.

What you implement

public final class RetryBreaker {
    public static final class OpenCircuit extends Exception {}
    @FunctionalInterface public interface Sleeper { void sleep(long nanos) throws InterruptedException; }

    public RetryBreaker(int threshold, long cooldownNanos, LongSupplier clock);

    public <V> V execute(int attempts,                       // max attempts for this call, > 0
                         Callable<V> task,                   // the dependency call
                         Predicate<Exception> retryable,     // which failures may be retried
                         IntToLongFunction backoff,          // attempt number -> delay in nanos
                         BooleanSupplier cancelled,          // checked before each attempt
                         Sleeper sleeper) throws Exception;
}

Rules

CLOSED (normal): every attempt, including retries, is an observation. A success resets the consecutive-failure count; reaching threshold failures opens the circuit until now + cooldown.
OPEN: execute throws OpenCircuit without calling task.
HALF-OPEN: the first attempt after the cooldown becomes the single probe; other callers still get OpenCircuit while it runs. Probe success → CLOSED with count 0; probe failure → OPEN for another cooldown.
Retry loop: on failure, if this was not the last attempt and retryable.test(e), call sleeper.sleep(backoff.applyAsLong(attempt)) and try again; otherwise rethrow the original exception. Negative delay → IllegalArgumentException.
cancelled true before an attempt → CancellationException. An InterruptedException from the task restores the interrupt flag and is rethrown, never retried.
Stale outcomes are ignored: an attempt that started before the breaker changed state (keep a generation counter) must not affect the new state.

Worked example

threshold = 2, cooldown = 10. Call 1 (1 attempt) fails → 1 failure. Call 2 fails → OPEN until t+10.
At t+5: execute → OpenCircuit; the task is not invoked.
At t+10: the next call is the probe; it succeeds → CLOSED. Had it failed → OPEN until t+20.
Retry within one call: attempts = 3, task throws TimeoutException (retryable) then succeeds → sleeps backoff(1) once, returns the value.

Files and commands

Read the tests first, then replace each TODO in the starter until they pass. Java starter: RetryBreaker.java · tests: RetryBreakerTest.java. Go starter: executor.go · tests: executor_test.go.

cd java
./gradlew test --tests 'labs.resilience.*'

cd go
go test -race -count=1 ./resilience

Reveal the reference solutions

Java: RetryBreaker.java
Go: executor.go

When to use

Use this lab when a repeatable dependency operation needs bounded retries and local failure containment. Configure a positive consecutive-failure threshold and cooldown. Each dependency attempt counts toward the breaker; successes reset consecutive failures. Crossing the threshold opens the circuit. Calls before cooldown expiry fail without invoking the dependency. At expiry, exactly one caller may probe; others remain rejected until it finishes.

Retries require an explicit eligibility predicate and a maximum attempt count. This component cannot establish that a timed-out remote mutation did not commit; use an idempotent operation protocol before retrying consequential effects. The breaker is process-local and retains no durable state. A fleet of processes has a fleet of independent breakers.

Steps

Separate circuit admission from execution. Acquire a permit under the breaker lock, call the dependency outside it, and record the outcome afterward. Holding the lock during remote work would serialize healthy requests and prevent the breaker from changing state promptly.
Fence late outcomes. Attach a generation to each permit. When the circuit opens or a probe closes it, advance the generation. Ignore outcomes from an older generation so a slow successful call cannot close a circuit that newer failures have already opened.
Limit half-open concurrency. Change from open to probing atomically when admitting the first eligible probe. A successful probe closes and resets the circuit; a failed probe opens it for another cooldown. An uncooperative probe can keep the circuit in probing state indefinitely, so it needs an effective dependency deadline.
Bound retry cost. Inject a nonnegative backoff duration for each failed attempt. The caller may add jitter and a cap in that policy. Go waits with a cancellable timer and checks the context before each attempt. Java receives cancellation and sleep policies explicitly; its sleeper and task must honor interruption or deadlines. Neither example imposes a hard runtime limit on arbitrary task code.
Classify failures deliberately. Every observed task failure counts toward the breaker, including cancellation during a task. Decide whether that policy fits the dependency; user-request cancellation often should not represent backend unavailability. Java does not retry interruption and restores interrupt status; Go cancellation prevents another attempt. Recovery from runtime-fatal errors is not promised.
Test time and generations. Use a fake clock to open after two failed attempts, verify no extra task invocation occurs, advance exactly to cooldown, and complete a successful probe. In Go, retain an old permit, open with another failed call, finish the old permit successfully, and verify the circuit remains open. Also verify that a second probe is rejected.

Checklist

Recall and variation

Rebuild probe admission before reading Go or Java. Variation: an old successful attempt finishes after another call opens the circuit. Acceptance: generation fencing ignores the old outcome, and only one half-open probe is admitted at the cooldown boundary.

Retry eligibility excludes unsafe duplicate effects
Attempt count and backoff policy are explicit and bounded
Late outcomes cannot alter a newer breaker generation
Only one half-open probe may execute
Cancellation and effective IO deadlines belong to task and owner policies

Signals

Working: the open circuit suppresses dependency calls and an old success cannot reset a newer failure episode.

Going wrong: every error is retried, breaker rejection consumes retry budget in a busy loop, or a jitter policy returns negative delay. Retries can still amplify load across layers; assign one owner for the overall budget rather than multiplying nested policies.

Implementing a Per-Client Rate Limiter, because admission rate limits and failure containment solve different problems.