Lab L11 · Java + Go

Implementing an outbox and idempotent consumer

Build both halves of the transactional outbox pattern on SQLite. The producer saves a business command and the event announcing it in one transaction. A relay API reads unsent events in batches and marks each one sent after delivery. The consumer applies each event's effect at most once by recording the event id in an inbox table in the same transaction as the effect, so a redelivered event is detected and skipped.

The task

When an order is placed, the order service must publish OrderPlaced. Writing to the database and then publishing to a broker can crash in between (event lost); publishing first can announce an order that then fails to commit (phantom event). The outbox fixes the producer side: the event row is committed with the order, and a relay publishes it later. The relay may crash after delivering but before marking it sent, so it will send that event again — delivery is at-least-once. The consumer's inbox makes the effect happen once anyway.

Two local transactions and an at-least-once hop between them. The outbox guarantees an event exists if and only if the command committed; the inbox turns duplicate deliveries into no-ops.

What you implement

public final class OutboxStore {
    public record Event(String id, long amount) {}
    public OutboxStore(Path sqliteFile) throws SQLException;     // creates tables if missing

    // producer side (tables: commands, outgoing(sent))
    public boolean produce(Event e) throws SQLException;         // false if this id was already produced
    public List<Event> pending(int limit) throws SQLException;   // unsent events, ordered by id, at most limit
    public void ack(String id) throws SQLException;              // mark one event as sent

    // consumer side (tables: inbox, effects)
    public boolean consume(Event e) throws SQLException;         // false if this id was already consumed
    public long effectCount() throws SQLException;
}

Rules

produce: in one transaction, insert the event into commands and into outgoing (unsent). Returns true. The same id with the same amount again → false, nothing changes. The same id with a different amount → IllegalArgumentException, rolled back.
pending(limit): unsent events ordered by id, at most limit (limit <= 0 → IllegalArgumentException). Events stay pending until ack, so an un-acked event is returned again.
consume: the same pattern on inbox + effects. A redelivered event (same id and amount) → false, no second effect row. Same id, different amount → IllegalArgumentException.
Empty id or amount <= 0 → IllegalArgumentException.
Producer and consumer normally use separate database files (two services). No transaction spans the network hop between them.

Worked example

Producer: produce(e1), produce(e2) → true, true. pending(10) → [e1, e2].
Relay delivers e1: consume(e1) → true. The relay crashes before ack.
Restarted relay: pending(10) → [e1, e2] again. consume(e1) → false (duplicate), consume(e2) → true; ack(e1), ack(e2).
pending(10) → []. Consumer effectCount() → 2, not 3.

Files and commands

Read the tests first, then replace each TODO in the starter until they pass. Java starter: OutboxStore.java · tests: OutboxStoreTest.java. Go starter: store.go · tests: store_test.go.

cd java
./gradlew test --tests 'labs.outbox.*'

cd go
go test -race -count=1 ./outbox

Reveal the reference solutions

Java: OutboxStore.java
Go: store.go

When to use

Use this lab when a database mutation must lead to an event despite process interruption. A producer transaction inserts one business ledger entry and one outgoing event with the same identity and positive integer amount. A duplicate identity with an identical amount is a no-op; a different amount conflicts. Producer and consumer normally use separate real SQLite files. Their transactions do not span the delivery boundary.

The consumer records an inbox identity and one local projection-effect row in a single transaction. Receiving a duplicate with the same payload produces no second effect; changed payloads conflict. The consumer's local effect is represented by a ledger row, not an external API call. There is no exactly-once network-delivery claim.

Steps

Write the outbox with business state. Start an immediate transaction, check duplicate identity and payload, insert the command ledger row, and insert the outgoing event. Commit both or neither. A test trigger rejecting the outgoing insert must roll back the earlier business insert.
Read a bounded delivery batch. Pending returns unsent events ordered by identity with a positive limit. This is a small relay core, not a continuously running broker client. A relay owner reads a batch, delivers each event, and marks it sent only after confirmed delivery. Ordering by identity is deterministic but is not domain causality or commit ordering.
Treat acknowledgment loss as replay. If delivery succeeds and the relay dies before marking sent, the next scan finds the same event. That is expected at-least-once behavior, not a bug to hide with early acknowledgment. A mark-sent call before delivery risks permanent event loss.
Deduplicate with the effect. In the consumer database, insert the inbox record and effect row inside one transaction. Checking inbox outside that transaction permits concurrent duplicates to apply multiple effects. Serialize SQLite writers before the check, and enforce primary keys even though the application also checks them.
Test the boundary without a fake guarantee. Produce an event, consume it, intentionally omit producer acknowledgment, reopen the consumer, and replay the pending event. Require one effect, then acknowledge and require no pending event. This tests the lost-ack sequence explicitly, but is not a broker outage, OS-kill, or power-loss test.
State the operational gaps. Multiple relays may read the same batch and deliver duplicates; consumers must remain idempotent. Add leases or claims for efficiency, not as a substitute for deduplication. Retention, cleanup, poison-message handling, delivery retries, backoff, event schema versions, and per-aggregate ordering are owner policies not implemented by this core.

Checklist

Relay and recall

Rebuild consumer receipt-plus-effect before reading Go or Java. The postgres adapters also execute one bounded relay batch: deliver, then acknowledge, leaving failures pending. Variation: the receiver commits and the delivery call then reports a lost acknowledgment. Acceptance: the next batch redelivers but creates no second effect.

For separately committed actions, model compensation as a new scoped request, not deletion of history. For example, retain a release command referring to an earlier reservation and version-check the current state; retrying that release must not release a replacement reservation. A charge that cannot be reversed enters reconciliation with its external reference. Compensation can itself fail and requires the same outcome retention and retry ownership.

Business state and outgoing event commit together
Delivery acknowledgment follows confirmed delivery
Consumer receipt and local effect commit together
Changed payloads conflict under the same identity
Duplicate replay after lost acknowledgment leaves one effect

Signals

Working: an injected outbox failure leaves no business row, while replay after lost acknowledgment leaves exactly one consumer effect.

Going wrong: a producer commits business state before creating the event, a relay acknowledges before delivery, or inbox cleanup permits replay of an old event still present in the transport. These tables have no automatic retention or size quota; define a replay horizon and a coordinated cleanup strategy before running indefinitely.

Implementing a Durable Idempotent Operation, for scoped retained responses within a database boundary.