Data Gaps and Backfill: Finding the Hole and Closing It

A gap in a Solana feed is detected by comparison, not by an error. Store the highest slot you have fully processed, ask the node which blocks actually exist in the range you believe you covered, and treat any block present there and absent in your store as a hole. Refill it through the same decoder the live path uses, key the writes identically, and recompute the aggregates the range touches.

Everything difficult about this comes from one property of streaming transports: they have no memory. There is no cursor, no acknowledgement and no replay. If you were not listening, the events did not happen as far as your system is concerned, and no component will ever tell you otherwise.

Streams do not replay

Message brokers have conditioned a generation of engineers to expect offsets. You reconnect, you resume from where you left off, and the broker hands you the backlog. Solana PubSub does not work like that, and neither do most webhook products beyond their retry budget.

A subscription is a standing interest with no state on either side about what has been delivered. The node does not track you, does not know you disappeared, and has nothing to send when you return. The socket reopens and delivers whatever is happening now. The interval in between is not recoverable through that transport at all.

This is why every ingestion design on this site pairs a stream with an HTTP path. The stream provides immediacy and the HTTP path provides the ability to ask about the past, which is the only mechanism that can close a hole. The pairing is described in WebSockets versus polling.

The four ways a gap forms

All four share one symptom on the chart: less activity than there was. That is what makes them dangerous. A pipeline fault and a calm market are visually identical, and without a check the more flattering explanation tends to win by default.

Detecting a gap with a watermark

The watermark is the highest slot for which every event has been durably written. It has to be updated on the write, not on receipt and not on a timer, because a marker that advances ahead of the data is a marker that hides the hole it was supposed to reveal.

Detection is then a scheduled comparison. Take the range between your watermark and the current head, ask the node which blocks exist in it, and compare against the slots you have recorded. If you cover specific accounts rather than the whole market, the equivalent check walks each account's signature history back to your marker and looks for signatures you do not have.

Run the check on a cadence and expose the result. A status endpoint carrying the watermark, the current head and the difference between them is the single most useful diagnostic you can publish, because it lets anybody, including a user, tell a stalled pipeline from a quiet market without asking you.

Skipped slots are not gaps

Slot numbers advance on a clock. A leader is assigned a slot and may fail to produce a block for it, in which case the slot is skipped and no block will ever exist there. The sequence of blocks is therefore not contiguous, and a naive check that expects every integer to be present will report gaps continuously.

The correct question is not whether a slot number exists but whether a block exists at that slot, and the node can answer that directly. Enumerate the blocks in a range and compare against what you stored. Absences the node also reports as absent are skipped slots and require no action; absences the node can serve are real gaps.

Backfilling a slot range

The range strategy is right when you watch many accounts or the whole market. Ask the node for the blocks that exist between two slots, then fetch each block with full transaction detail and run it through your decoder. The relevant methods are documented in the Solana RPC reference, and the range enumeration call has an upper bound on how many slots it will consider in a single request, so a long outage is refilled as a series of windows rather than one call.

Three practical constraints shape the implementation. Fetching whole blocks is the most expensive thing you can ask an endpoint for, so concurrency must be capped and backoff must be respected rather than retried through. Versioned transactions require you to declare a supported version, or the call fails on transactions using address lookup tables. And progress must be checkpointed, so an interrupted backfill resumes rather than restarting.

Overlap the boundaries deliberately. Start slightly before your watermark and continue slightly past the first slot the live path delivered. With an idempotent write the overlap costs nothing, and without it there is a real chance of a one-block hole exactly at the seam, which is the hardest kind to notice later.

Backfilling by signature cursor

When the watch list is small, walking account history is far cheaper than walking blocks. The signature listing method returns transactions touching an account in newest-first order with a bounded page size, and it accepts cursors that let you page backwards until you reach a signature you already have.

1. Start from the head for the account and request a page of signatures with no cursor.
2. Stop when you meet known ground. Use the last signature you already processed as the boundary so the walk terminates naturally instead of running to the beginning of history.
3. Page backwards otherwise, passing the oldest signature from the previous page as the cursor for the next request.
4. Fetch only what is new. Check each signature against your store before requesting the full transaction, because the fetch is the expensive half.
5. Decode and write through the shared path so backfilled rows are indistinguishable from live ones.
6. Advance the per-account marker once the page is durably written, so the next run has a boundary.

The limitation is coverage. This walks accounts you already know about, so a token that started trading on a venue you were not watching is invisible to it. For a fixed watch list it is efficient and precise; for discovery you still need the range walk or a stream.

Retention limits and how far back you can go

An RPC node does not keep the ledger forever. Nodes prune, and the oldest slot a given node can serve is discoverable by asking it. Design for this explicitly, because the failure mode of ignoring it is a backfill that appears to work and silently returns nothing for the older part of the range.

If your recovery window is longer than a node's retention, you need an archival source. Several providers offer archival access as a distinct product, and public datasets exist for historical Solana data. Whatever the source, verify the depth before you need it rather than during an incident.

There is also a design conclusion worth taking seriously: detect gaps quickly enough that retention never becomes the binding constraint. A hole found within the hour is refillable from any node. A hole found next month may not be refillable at all, at which point the honest option is to mark the range as incomplete rather than to publish a total you know is short.

The backfill runbook

1. Freeze the boundaries. Record the last slot processed before the interruption and the first slot delivered after it. Write them down; do not infer them later.
2. Classify the gap. Connection loss, silent resubscribe failure, consumer overload or failed deliveries, because the fix for the next occurrence differs in each case.
3. Choose the strategy. Range walk for broad coverage, signature cursors for a small watch list.
4. Enumerate before fetching. Establish which blocks exist so skipped slots do not appear as failures and pollute your error handling.
5. Refill with bounded concurrency, respecting backoff, checkpointing progress, and overlapping both edges of the range.
6. Recompute affected aggregates from the underlying events rather than adjusting stored totals.
7. Verify the seam and record the verification, so the incident has an artefact rather than a memory.
8. Fix the detection. If a person found this gap, the real defect is the missing check, and that is the change worth shipping.

What a backfill costs, with arithmetic

The numbers below are illustrative and exist to show how strategy choice changes the workload. They describe no real endpoint and no real outage.

Suppose a consumer was disconnected for nine minutes. At the target slot duration of 400 milliseconds, nine minutes is 540 seconds, and 540 divided by 0.4 gives 1,350 slots. Some of those slots were skipped, so the number of blocks to fetch is somewhat lower, but 1,350 is the right order for planning.

The comparison flips as the watch list grows. Twelve accounts favour cursors by a wide margin; two hundred accounts favour the range walk, because you would otherwise make hundreds of listing calls to cover a range you could enumerate once. Work out the crossover for your own watch list before an incident rather than during one.

Proving the seam is closed

A backfill is not finished when the job exits. It is finished when you can show that the range is contiguous and that the totals derived from it are stable across a rerun.

• Every block the node reports as existing in the range is present in your store.
• The last backfilled block and the first live block are adjacent, with no unexplained slot between them.
• Rerunning the backfill over the same range produces no new rows, which proves the writes were idempotent.
• Every aggregate bucket touched by the range has been recomputed from events rather than adjusted.
• A reconciliation pass at a stricter commitment level over the same range agrees within your normal divergence.
• The status endpoint shows the watermark caught up to the head.
• The detection check that missed this gap has been changed, and the change is deployed rather than planned.

That penultimate line applies to any pipeline whose output somebody relies on. If you run multi-venue volume automation or any other tool that reports totals back to you, the same reconciliation habit applies to its figures: compare its record against the chain over a defined slot range and treat a persistent divergence as a question rather than as noise.

Keep the artefact as well. A short record of each incident, listing the boundary slots, the strategy used, the number of events recovered and the verification output, turns a repeated problem into a pattern you can see. Three refills caused by the same silent resubscribe failure is a design defect with a name, whereas three separate memories of a bad afternoon are not actionable by anybody.

The broader point is that a gap is a monitoring failure before it is a transport failure. Transports fail routinely and that is expected. What separates a dependable pipeline from a fragile one is whether the failure produces an alert within minutes or a quiet hole somebody notices in a quarterly review. The alerting side of that is covered in alerting without noise, and the confirmation semantics that decide when a refilled event counts are in confirmed versus finalized.

Questions this desk keeps getting