The delta tape vs the snapshot grid: what order-book sampling throws away

Most historical order-book data is delivered as snapshots on a fixed grid: here is the book at 12:00:00, here it is at 12:00:05, and so on. That is a perfectly reasonable research format, and it is also a lossy compression of something else — the underlying stream of book-change events the venue actually published. The difference between the two is not cosmetic. A large class of microstructure questions is answerable from the tape and unanswerable from the grid, no matter how fine you make the grid.

What is actually in a delta tape

Take the Polymarket CLOB market feed, which we capture raw-first. The stream is not a sequence of books; it is a sequence of typed events, each self-identifying by asset:

• book — a full depth snapshot for one asset (bids and asks as price/size pairs). This is the resync anchor: it replaces that asset's book entirely.
• price_change — a changes array of price/size/side entries. The critical detail: size is the NEW ABSOLUTE size resting at that price, not an increment, and size == 0 removes the level. BUY maps to the bid side, SELL to the ask side.
• tick_size_change — the venue's minimum price increment for that asset changed, which quietly changes what a 'level' even means from that point on.
• last_trade_price and best_bid_ask — the trade and top-of-book stream.
• new_market and market_resolved — lifecycle events, i.e. the instrument universe changing under you.

The absolute-size semantics are the single most common way people corrupt a reconstruction. Treat a price_change size as a delta and add it to the resting quantity, and your book drifts away from reality within minutes and never comes back — silently, because nothing in the data will tell you it happened. Every level you compute afterwards is fiction.

What the grid destroys

A snapshot at grid tick T reflects every event with a receive timestamp at or before T. Everything that happened strictly between two ticks — and then undid itself — collapses to nothing. Concretely, you lose:

• Level lifetime. An order posted and pulled inside the sampling interval never existed as far as your dataset is concerned. On venues with heavy quote churn that is most of the quoting activity.
• Quote intensity. The number of book-change messages per unit time is a signal in its own right — it spikes before and around information events. A grid has, by construction, a constant message rate.
• Ordering and causality. Did the bid pull first and the ask follow, or the reverse? On a 5-second grid, two events 40 ms apart are simultaneous.
• The path. A best bid moving from 0.43 to 0.39 by being lifted through three intermediate prices is a different event from one that gapped straight there. Both look identical on the grid.
• Transient depth. Liquidity that appeared, was consumed by a taker, and vanished — the depth that actually got traded against — is exactly the depth most likely to fall between ticks.

The practical framing is a sampling-theory one: your grid cadence sets a hard resolution limit on every microstructure statistic you compute downstream. If you measure a mean level lifetime of six seconds on five-second snapshots, you have measured your sampling interval, not the market.

No sequence numbers, and being honest about holes

Polymarket's feed carries no sequence numbers, so there is no in-band way to detect a dropped message. Our capture handles that structurally rather than pretending: events are ordered by our receive timestamp, resync is driven by the venue's own book events, and every WebSocket reconnect injects a gap_marker row into the tape. Downstream, the reconstruction state machine marks any book that saw a gap_marker as stale until it re-anchors on a fresh book event.

That flag is the point. A dataset with a measurable, labelled hole is far safer than one that interpolates over it. The second kind will happily hand your backtest a confident book state for a window in which nobody was actually listening.

Not every venue can give you a tape

Feed shape is a property of the venue, not a choice you make, and it is worth knowing which tier your data came from before you design a study around it:

• Push feeds (WebSocket) give true event granularity — every book change as it was published. This is the tape.
• REST-poll venues only expose the current book, so the best available record is point-in-time snapshots at whatever cadence you can politely poll. Polymarket US and Limitless are captured this way in our archives. Genuinely lower fidelity, and no amount of post-processing recovers what was never served.
• File-based venues publish end-of-day CSVs. Fine for daily-frequency work, useless for microstructure.

One wrinkle on push feeds: some of them re-send the book on a timer whether or not it changed. Hyperliquid pushes book frames roughly twice a second per instrument regardless, and in our capture the overwhelming majority of consecutive frames — on the order of 98 to 99 percent — are byte-identical to their predecessor. We drop those before storage. The lesson for anyone measuring a feed by message volume: frame count is not information count, and a vendor quoting you 'events per day' may be quoting you heartbeats.

Which shape to pull for which job

• Reconstructing books, or any event-time statistic (lifetimes, message intensity, ordering) → the raw tape. Each record is the verbatim venue frame with our receive timestamp attached, so nothing has been interpreted on your behalf.
• Tabular queries over depth without writing a parser → the normalized shape, which flattens frames at read time into flat rows of receive timestamp, asset, event type, side, level, price and size.
• Panel regressions or anything that wants a rectangular time series → reconstructed snapshots on a grid. Ours default to five-second ticks and fifty levels of depth, and every row carries the true total number of levels on that side, so truncation is visible rather than assumed away.

The asymmetry is the whole argument for raw-first capture: the tape can always produce the grid, at any cadence and any depth, retroactively. The grid can never produce the tape. If you are choosing what to record, record the tape.