Lab L18 · Java + Go

Implementing a streaming record processor

Build a function that reads a stream of newline-separated integers — one per line — and returns how many records there were and their sum. It reads through a buffer with memory bounded by the longest allowed line, never loading the whole input. It is strict: any malformed line, oversized line or numeric overflow fails the whole run with an IOException, and it never closes the caller's stream.

The task

A payment partner sends a nightly settlement file that can be gigabytes long; each line is an amount in cents. Loading it into memory is not an option, and silently skipping a bad row is unacceptable because these are money totals. A hostile or corrupt file must not be able to make one “line” consume unbounded memory, hence the per-record byte limit. The stream belongs to the caller, who opened it and must close it.

Bytes flow through a line buffer that can never exceed maxBytes; each completed line is validated and folded into the running count and sum. Any error aborts the whole run.

What you implement

public final class RecordProcessor {
    public record Result(long count, long sum) {}
    public static Result process(InputStream input,
                                 int maxBytes,               // max bytes per record, > 0
                                 BooleanSupplier cancelled)  // checked while reading
            throws IOException;
}

Rules

Records are separated by \n; one trailing \r is stripped, so CRLF files work. A final record without a newline counts; a trailing newline does not create an empty record. An empty stream → (0, 0).
Each record must be valid UTF-8 and match [+-]?[0-9]+ exactly — no spaces, no empty lines → otherwise IOException.
A record longer than maxBytes bytes → IOException as soon as the limit is exceeded. maxBytes <= 0 → IllegalArgumentException.
A value outside long or a sum that overflows (Math.addExact) → IOException.
cancelled becoming true → IOException. On any failure no partial result is returned.
Never call input.close().

Worked example

"12\n-5\r\n+3" → (3, 10).
"" → (0, 0). "7\n" → (1, 7).
"1\n\n2" → IOException (empty record). " 4" → IOException (space).
"9223372036854775807\n1" → IOException (sum overflow).

Files and commands

Read the tests first, then replace each TODO in the starter until they pass. Java starter: RecordProcessor.java · tests: RecordProcessorTest.java. Go starter: processor.go · tests: fuzz_test.go, processor_test.go.

cd java
./gradlew test --tests 'labs.streamio.*'

cd go
go test -race -count=1 ./streamio

Reveal the reference solutions

Java: RecordProcessor.java
Go: processor.go

When to use

Use this lab to revise streaming input without loading the complete file. The format is deliberately small: one signed 64-bit decimal integer per line, LF or CRLF delimiters, and an optional unterminated final record. An empty stream produces zero count and sum. Empty records, whitespace, invalid UTF-8, extra syntax, values outside the signed range, and intermediate sum overflow fail the operation. A trailing newline terminates the previous record rather than creating an extra empty record.

The caller owns the input stream. The processor does not close it and does not return a partial aggregate on failure. Reading can advance beyond the logical record through a buffered wrapper; callers should not expect to resume directly from the underlying stream after an error. These examples are single-pass processors, not restartable ingestion jobs.

Steps

Limit bytes before decoding. Configure a positive maximum record size. It excludes LF and includes an optional CR. Reject the next non-LF byte after the limit, before appending it. A maximum of three therefore admits 123 followed by LF but rejects four-digit input. The fixed reader buffer may hold more than one record; total memory is O(record limit) plus a fixed buffer, not O(file size).
Decode strictly. Java explicitly requests reporting of malformed UTF-8 rather than replacement characters. Go validates the byte slice. Strip only the optional final CR, not arbitrary whitespace, so malformed formats do not become silently valid.
Specify arithmetic. Accept an optional sign and ASCII decimal digits, parse within signed 64-bit range, and reject count or running-sum overflow. Intermediate overflow is an error even if later values could bring the mathematical total back into range. Java uses checked addition; Go checks the boundary before adding.
Own cancellation. Poll before reads. Cancellation does not interrupt an arbitrary blocked input read in either example. The owner must arrange a deadline, close, or cancellable transport when blocked-read interruption is required. Do not claim that checking a token creates a hard deadline.
Test the edges. Use empty input, CRLF, a final unterminated record, exact size and one byte over, invalid UTF-8, and positive and negative overflow. Add readers that return short chunks or injected errors when adapting to a transport. Chunk size must not change parsing semantics.
Discuss extensions honestly. A production ingestion path may require durable offsets, idempotent effects, framing recovery, record-level quarantine, or a bounded error policy. Those are separate state machines; do not add a retry around a partially consumed stream and call it recovery.

Checklist

Recall and variation

Rebuild byte-limit enforcement before reading Go or Java. Variation: feed the same bytes one byte at a time and through a buffered reader. Acceptance: framing and aggregate results agree. The paired httpio clients add bounded response bodies and real local-server deadline tests; closing the response is their responsibility, while the parser leaves caller-owned input open.

Byte limits are enforced before unbounded allocation
Decoding and numeric overflow fail explicitly
Partial results and stream ownership are documented
Cancellation during blocked IO has an owner-provided mechanism
Newline and final-record semantics are shared across languages

Signals

Working: the same valid bytes produce the same aggregate, without materializing the file; malformed input fails predictably.

Going wrong: a convenience line reader grows without a limit, invalid bytes are replaced silently, or a cancelled request leaves an unowned stream read running forever.