The Portfolio Ledger · A Real-Time Market Data Pipeline + Limit-Order-Book Matching Engine

Case I · ingestor / anomaly.py

The rolling z-score,
in O(1) per tick.

"Naive recompute would be O(window). Running sum and running sum-of-squares over a fixed-size deque is two scalar updates per tick."

Each symbol gets its own detector. Every trade tick produces a log-return; the return goes into a deque of length sixty, the running totals get the new value added and the soon-to-be-evicted value subtracted, and the z-score falls out of mean and standard deviation in two divides and a square root. An alert fires when |z| exceeds 2.5 and the window is warm.

No NumPy. No SciPy. The point of writing it this way is precisely to demonstrate the algorithmic technique, not to outsource it.

Operation	Cost
update(price)	O(1)
variance recompute	O(1)
window eviction	O(1)
warm-up tick count	60

ingestor / anomaly.py · RollingZScore.update

def update(self, price: float) -> AnomalyResult:    if price <= 0:        raise ValueError("price must be positive")    if self._last_price is None:        self._last_price = price        return AnomalyResult(z_score=None, is_anomaly=False)    r = math.log(price / self._last_price)    self._last_price = price    # subtract the about-to-evict value BEFORE append.    # keeps running totals in sync with the deque, O(1).    if len(self._returns) == self.window_size:        evicted = self._returns[0]        self._sum     -= evicted        self._sum_sq  -= evicted * evicted    self._returns.append(r)    self._sum    += r    self._sum_sq += r * r    if len(self._returns) < self.window_size:        return AnomalyResult(z_score=None, is_anomaly=False)    n        = self.window_size    mean     = self._sum / n    variance = self._sum_sq / n - mean * mean    if variance <= 0:        return AnomalyResult(z_score=None, is_anomaly=False)    z = (r - mean) / math.sqrt(variance)    return AnomalyResult(z_score=z, is_anomaly=abs(z) > self.threshold)

Case II · matching-engine / OrderBook.java

A price-time priority order book.

"Two TreeMaps for the price ladders, intrusive linked-list FIFOs at each level, and a HashMap order-id index so cancel is O(1) lookup, O(1) unlink, O(log n) only if the level empties."

The book is two NavigableMaps: bids reverse-ordered so the best bid is firstEntry, asks natural-ordered so the best ask is firstEntry. Each map value is a PriceLevel: a linked list of orders plus a running total quantity. Submit sweeps the opposite side, walks the FIFO at each price level, and posts what does not fill. Cancel goes through the HashMap, unlinks from its level, and removes the level only if empty.

Every public method on OrderBook is synchronized. Matching must be linearizable; one coarse lock is the right tool because the engine is single-threaded per book on the producer side.

Operation	Cost
submitLimit	O(k log n)
submitMarket	O(k log n)
cancel	O(1) + O(log n) if level empties
topOfBook	O(1)

matching-engine / OrderBook.java · match

private long match(        long takerId,        Side takerSide,        BigDecimal limitPrice,        long quantity,        long ts,        List<Trade> out) {    NavigableMap<BigDecimal, PriceLevel> opposite =            (takerSide == Side.BUY) ? asks : bids;    long remaining = quantity;    while (remaining > 0 && !opposite.isEmpty()) {        Map.Entry<BigDecimal, PriceLevel> best = opposite.firstEntry();        BigDecimal restingPrice = best.getKey();        if (limitPrice != null) {            int cmp = restingPrice.compareTo(limitPrice);            boolean acceptable =                    (takerSide == Side.BUY) ? cmp <= 0 : cmp >= 0;            if (!acceptable) break;        }        PriceLevel level = best.getValue();        while (remaining > 0 && level.head != null) {            Order maker = level.head;            long fillQty = Math.min(remaining, maker.remainingQuantity());            out.add(new Trade(maker.id(), takerId, symbol, restingPrice, fillQty, ts));            remaining -= fillQty;            maker.decreaseQuantity(fillQty);            level.recordFill(fillQty);            lastTradePrice = restingPrice;            if (maker.remainingQuantity() == 0) {                level.unlink(maker);                ordersById.remove(maker.id());            }        }        if (level.isEmpty()) opposite.remove(restingPrice);    }    return remaining;}

Case III · service boundaries

Four services, two protocols.

"The dashboard never touches the database. The ingestor never touches the API. Two protocols carry the whole stack: HTTP and Postgres."

Four services, but the rules between them are short. The ingestor writes ticks and alerts with raw psycopg in an autocommit connection, wrapping each tick and its derived alert inside one conn.transaction() so a crash mid-trade cannot leave a chart spike without its overlay marker.

The API reads with an AsyncConnectionPool in its lifespan, exposing three endpoints, and the dashboard polls those endpoints. The matching engine writes its own me_* tables on a HikariCP pool with prepareThreshold=1 so the very first execute uses a server-side named plan; subsequent inserts skip parse and plan.

No message queue. No ORM. No FastAPI WebSocket. Postgres is the bus.

Boundary	Protocol
ingestor → postgres	SQL · INSERT
matching-engine → postgres	SQL · INSERT (HikariCP)
api → postgres	SQL · SELECT
dashboard → api	HTTP · poll @ 10 s
dashboard → matching-engine	HTTP · poll @ 2 Hz

ingestor / main.py · _handle_trade

# A tick and its derived alert ship in one transaction so a crash# mid-trade cannot leave a chart spike without its alert marker.async with conn.transaction():    await conn.execute(        "INSERT INTO ticks (symbol, ts, price, volume) "        "VALUES (%s, %s, %s, %s) "        "ON CONFLICT (symbol, ts) DO NOTHING",        (symbol, ts, price, volume),    )    if result.is_anomaly:        await conn.execute(            "INSERT INTO alerts (symbol, ts, price, z_score, message) "            "VALUES (%s, %s, %s, %s, %s)",            (symbol, ts, price, result.z_score, message),        )

Real-Time
Market Data Pipeline.
Plus an Order Book.

By the numbers.

Twelve capabilities, one repo.

Four services. One Postgres.

Three pieces, in detail.

The rolling z-score,
in O(1) per tick.

A price-time priority order book.

Four services, two protocols.

Three commands. One stack.

Live Endpoints

Real-Time Market Data Pipeline. Plus an Order Book.

By the numbers.

Twelve capabilities, one repo.

Four services. One Postgres.

Three pieces, in detail.

The rolling z-score,in O(1) per tick.

A price-time priority order book.

Four services, two protocols.

Three commands. One stack.

Live Endpoints

Real-Time
Market Data Pipeline.
Plus an Order Book.

The rolling z-score,
in O(1) per tick.