Building a Live Volume View You Would Defend

A live volume view is an ingestion path, a decoder, a bucketed aggregate and an API. The two decisions that determine whether the result is trustworthy are made early and are hard to change later: bucket by the block time of the event rather than by when you received it, and make every write idempotent on a stable key. Everything else is tuning.

This note assumes you already have a transport from the sources section and a decoder that produces keyed events. What follows is the arithmetic layer, where correct inputs are routinely turned into a wrong number by a scheduling decision nobody documented.

Four jobs pretending to be one

Receiving, decoding, aggregating and serving are separate concerns with separate failure modes, and the temptation in a first version is to collapse them into one process that reads a stream and updates a counter. That design works until the first time you need to reprocess, at which point there is nothing to reprocess from, because the raw material was consumed on the way past.

Keeping the stages separate buys you two things. The first is replay: if the decoder was wrong for a week, a durable log of raw transactions lets you fix it and rebuild. The second is attribution: when a number is questioned, you can point at the stage that produced it rather than at a monolith.

The cost is one extra store and a little more plumbing. It is the best trade available in this domain, because every pipeline of this kind gets its decoder wrong at least once, and the ones that kept the raw material recover in an afternoon while the others reconstruct from a competitor's chart.

The shape of the pipeline

The reconcile row is optional in the sense that the pipeline runs without it and dishonest in the sense that no figure should be published without it. It also produces the divergence metric that tells you how healthy the fast path is, which nothing else in the system can measure.

Event time versus arrival time

Every event has at least three clocks attached: the slot it landed in, the block time the cluster recorded for that slot, and the moment your process received it. Only the first two describe the market. The third describes your infrastructure and changes every time you deploy, backfill or recover from an incident.

Bucket by block time. It is stable across reprocessing, comparable with anybody else's chart, and unaffected by whether the event arrived through your live path or through a backfill six hours later. A system bucketed by arrival time produces a chart that visibly reshapes itself after every recovery, which is both wrong and alarming to look at.

Keep the slot as well. Block time is an estimate the cluster produces and is not guaranteed to be strictly monotonic between adjacent slots, whereas the slot number is an exact ordering. Use block time for the bucket and the slot for ordering, comparison and gap detection, and store both on every row.

Choosing windows and bucket widths

Fixed windows aligned to the clock are the right default. They are trivially comparable across systems, they roll up cleanly, and a user asked to reason about the last hour understands a set of minute bars. Rolling windows feel smoother and are considerably harder to store, because there is no natural row to write.

The practical pattern is fixed base buckets with rolling views computed at read time. Store one minute buckets, and answer questions about the last five minutes by summing five rows. This gives you rolling behaviour in the interface without rolling storage, and it means a correction to a single minute automatically fixes every window containing it.

Base bucket width is a cost decision as much as a resolution one. Halving the width doubles the row count for the same coverage, which is fine at small scale and expensive when the token list grows. Choose the width that supports the shortest window you intend to display, and no narrower.

Watermarks and when a bucket may close

Late data is not an anomaly, it is the normal condition. A backfilled range, a retried webhook delivery, or a reconnect that repaired a gap all produce events whose block time is well behind the current head. If a bucket closes the moment its clock boundary passes, all of those events are either lost or applied to a total that is already being displayed as final.

A watermark solves this cleanly. Track the highest block time you have durably processed across all ingestion paths, subtract a grace period sized to the lateness you actually observe, and treat that as the line below which buckets may close. Buckets above the line remain open and are marked incomplete.

1. Advance the watermark only on durable writes. A value held in memory is a value that lies after a crash.
2. Take the minimum across paths. If the live stream is at one point and the backfill worker is behind it, the watermark is the lower of the two, otherwise you close buckets the slower path has not finished.
3. Size the grace period from observation. Record the distribution of how late events arrive and set the period above the tail you care about rather than at a round number somebody guessed.
4. Apply post-close events as visible corrections. Update the stored total, mark the row as revised, and count the revision so that a rising revision rate becomes a signal.
5. Expose the watermark. Publish it on a status endpoint, because it is the single most useful number for anyone debugging why a chart looks incomplete.

The last step is worth the small effort. A user who can see that the pipeline has processed up to a particular moment can distinguish a quiet market from a stalled ingestion without asking anybody, which removes an entire category of support conversation.

Idempotent aggregation

Aggregation must be safe to run twice on the same input. Backfills overlap, retries repeat, and a reprocessing run deliberately reads a range that has already been processed. If any of those adds to a counter, the totals drift upward and there is no way to tell by how much.

The pattern that holds is an events table with a unique key of transaction signature plus instruction index, and aggregates computed from that table rather than incremented as events arrive. Writing an event twice is a no-op at the database level, and the aggregate is a function of a set rather than a running sum, so it cannot inherit an error.

Denomination and the pricing problem

Decide first whether you are reporting in the quote asset or in a fiat equivalent. Quote-denominated volume, meaning the SOL or stable side of each trade, is exact, reproducible and needs no external data. Fiat-denominated volume needs a price for every trade, which imports a second data source with its own latency and its own disagreements.

If you must convert, three rules keep the result defensible. Use a price contemporaneous with the trade rather than a current price, because valuing yesterday's activity at today's price produces a total that changes retroactively. Record which source supplied the price. And never mix a quote-denominated total with a fiat-denominated one in the same chart.

Prices are least reliable exactly when they matter most. In the first minutes after a launch a thin market can quote wildly, and different sources will disagree by margins that make a fiat total meaningless. A quote-denominated figure has no such failure mode, which is one reason serious tooling tends to report in SOL first and convert only for presentation.

Storage shape, with the arithmetic

The numbers below are illustrative, chosen to show how cardinality grows rather than to describe any real deployment. Assume you cover 500 tokens across 3 venues at one minute resolution.

Two decisions fall out of that table. The first is that most of those rows are zero, because most tokens do not trade in most minutes, so storing only non-empty buckets reduces the real count enormously and costs you nothing except remembering that a missing row means zero. The second is retention: keeping minute resolution for a month is a choice, and rolling old data up to hourly is the standard way to keep the cost bounded.

A composite primary key on token, venue and bucket start gives you idempotent upserts and an index that matches the query you will actually run. That is usually enough for a long time. Reach for a specialised time series store when this arithmetic, done with your own numbers, says you must.

Serving the view honestly

The API in front of the aggregate has one obligation beyond returning data: it must say how fresh the data is and which parts are incomplete. Three fields do almost all of that work.

• The watermark. The highest block time fully processed, so a client can compute its own staleness.
• A per-bucket completeness flag. The most recent bucket is open by definition; say so rather than letting a user read a partial minute as a fall in activity.
• The commitment level. Whether these totals came from confirmed or finalized data, because the two are not comparable, as set out in confirmed versus finalized.

For delivery to a browser, server-sent events are usually sufficient and considerably simpler than a WebSocket, since the traffic is one-directional. Whatever you choose, remember that a client polling every fifteen seconds imposes a fifteen second floor on freshness no matter how good the pipeline is, a point covered in why dashboards lag.

Venue coverage and the footnote you owe

A home-built view is accurate exactly where it was built to be. It covers the programs you decoded, the token programs you handled, and the edge cases you have already met once. Everything outside that region reads as zero, and zero is indistinguishable from nothing happening.

So publish the list. Say which venues are covered and which are not, and treat an uncovered venue as unknown rather than absent. This is the same discipline any serious tool applies to itself, and it is why a coverage statement matters more than a feature list when comparing platforms, whether that is your own panel or a hosted SOL volume bot reporting on the venues it touches.

Coverage also has a maintenance cost that scales with the venue's design. Constant product pools are the simplest case. Concentrated liquidity venues require you to handle position accounts, and a binned design such as the one behind a Meteora volume bot spreads a single trade across discrete price bins, which changes what a fill even looks like in the metadata. Budget for the venue you are adding, not for the average one.

Build checklist

• Raw transactions are stored durably before decoding, so the decoder can be fixed and rerun.
• Every event carries slot, block time, venue, mint, owner and amounts in base units.
• The unique key is transaction signature plus instruction index, enforced by the database.
• Buckets are keyed by block time, and rolling windows are computed at read time from fixed base buckets.
• A watermark gates bucket closure and is the minimum across all ingestion paths.
• Aggregates are recomputed from events rather than incremented, so reprocessing is safe.
• Volume is quote-denominated by default, and any fiat conversion records its price source.
• The API exposes the watermark, per-bucket completeness and the commitment level.
• The covered venue list is published next to the numbers, not buried in documentation.
• A reconciliation job compares the live totals against a stricter pass and alerts on growing divergence.

With totals that behave, the next problem is telling somebody about them without becoming background noise, which is the subject of alerting without noise.

Questions this desk keeps getting