anza-xyz

Solana Shred Recovery: When Valid Shreds Form an Invalid Erasure Batch Originally published on xroot.dev. A Solana block reaches validators as shreds, in batches of 32 data shreds and 32 coding shreds tied together by one Merkle root that the leader signs. Every shred in a batch can carry a valid signature and a valid Merkle proof while the batch itself is wrong, because nothing in the signature says the coding shreds are the parity of the data. SIMD-0504 says the rules for a batch apply to rebuilt shreds as much as to received ones, and between 24 October and 5 November Agave rebuilt its recovery path to enforce that: every validator now rebuilds every batch, so a batch like this fails everywhere. We modelled the old and new paths with the Reed–Solomon library Agave uses and ran four kinds of malformed batch past simulated validators losing packets. Under the old code, what a validator ended up doing depended on which shreds it happened to receive. Under the new code, every validator that collects 32 shreds rejects the batch: none of 800,000 rebuilds of a malformed batch passed. - * * How an erasure batch is built The leader cuts its block into data shreds and, for every 32 of them, computes 32 coding shreds with a Reed–Solomon code. Any 32 of the 64 are enough to rebuild the other 32, so a validator can lose half of a batch to the network and still have the block. Then the leader hashes all 64 shreds into a Merkle tree, signs the root and attaches to each shred its path to that root. A validator checks each shred on arrival: the signature over the root, and the path from the shred to the root. Shreds per batch — 32 data + 32 coding Needed to rebuild the batch — any 32 Payload: data · coding shred — 1,203 · 1,228 bytes Erasure-coded part of each — 987 bytes Merkle tree — 64 leaves, 6 levels - * * What the signature does not cover The Merkle root commits to the bytes of each of the 64 shreds. It says nothing about how they relate. A leader can compute its coding shreds from different data, or change one byte of a shred after encoding, then build the tree and sign. Each shred still verifies on its own. The batch is not a Reed–Solomon codeword: rebuilding it from different subsets of shreds gives different answers. Until 29 October, Agave ran Reed–Solomon only when data shreds were missing and at least 32 shreds of the batch were present, tried again after every new insert of shreds, and logged a failed attempt at info level before moving on. The Merkle check made sure a rebuilt shred was byte-for-byte what the leader had signed, so a bad batch could never smuggle in different block data. But whether a validator noticed anything depended on which shreds it held. In our simulation of 2,000 validators per row, adding one shred per insert, each row counts the validators that completed the batch from data shreds alone · recovered without noticing the bad shred · were left waiting for repair: First coding shred corrupted, 1% loss — 1,492 · 0 · 508 First coding shred corrupted, 5% loss — 379 · 0 · 1,621 Last coding shred corrupted, 1% loss — 62 · 1,934 · 4 Last coding shred corrupted, 5% loss — 29 · 1,895 · 76 Those in the first group never looked at a coding shred. With the first coding shred corrupted, every recovery that used it failed, about 31 failed attempts per validator even with no loss, and at 5% loss four validators in five could not complete the batch from Turbine. Repair fixes that, because it hands out the shreds the leader signed, but it takes longer. With the last coding shred corrupted, most recoveries never touched it and quietly succeeded. A corrupted data shred, or coding shreds computed from other data, behaved like the first case. Nothing in any row was rejected. - * * SIMD-0504: the batch is the unit of validity SIMD-0504, written by Philip Taffet of Firedancer, replaces the shred rules each client inferred from Agave’s code with a written list. Sixteen checks apply to each shred: index ranges, the fixed 32:32 layout, a Merkle tree of exactly six levels, flag and size rules. A shred that fails them is dropped. Eight checks apply to the batch as a whole: one slot, one shred version, one chained Merkle root, one parent across the data shreds, padding only at the end. A block containing a batch that fails them is invalid. Under TowerBFT a validator must not vote for it, and under Alpenglow it must vote skip. None of the sixteen or eight checks says the coding shreds must be the parity of the data. That follows from the proposal’s note on recovery: the batch checks apply to received and rebuilt shreds alike, so a validator that received all 32 data shreds must still check the coding shreds. The proposal says why: to eliminate any divergence based on whether a validator received or recovered a shred. A rule that only the validators who happened to need recovery could check would split the cluster the first time a leader broke it. Agave enforces the note by rebuilding every batch, and a batch whose coding shreds aren’t the parity of its data cannot be rebuilt to the root the leader signed. - * * “Always recover”: how Agave checks every batch Agave landed the change in a series of pull requests: #16194 wired the rebuild into blockstore on 29 October, #16378 added the feature gate and enforcement a day later, and #16639 stopped retrying a batch that cannot be rebuilt once SIMD-0504 is active. In the code at the time of writing, a batch is rebuilt exactly once, in the insert that first brings it to 32 shreds, whether or not any data shred is missing: received exactly 32: reconstruct the other 32 shards received more than 32: reconstruct missing data from 32 of them, re-encode all 32 coding shards, compare with the coding shards received then: rebuild the Merkle tree from all 64 leaves, compare with the signed root fail either check: with SIMD-0504 active, mark the slot dead (unless it was already full before this insert) The gate is stricter_shred_validation, key s5o4YoA9zUrw9j4v335ncdG5PpznDBsrgM6CvUGrasA. On 6 November at 03:18 UTC its feature account did not exist on mainnet, testnet or devnet. Until it is activated, Agave runs every check and reports failures to its metrics, but rejects nothing it did not reject before. - * * Same verdict, whatever arrived Our model follows that code with the same library, version and features (reed-solomon-erasure 6.0.0 with simd-accel), 987-byte shards and Agave’s Merkle construction. We built four malformed batches, the first or last coding shred flipped by one bit after encoding, one data shred flipped, and all coding shreds computed from different data, and rebuilt each from 200,000 random subsets of every size from 32 to 64 shreds: Honest batch, 200,000 subsets — all pass Malformed batches, 800,000 subsets — 0 pass …failed as Merkle mismatch — 629,095 …failed as wrong parity — 170,905 The reason is simple once written down. The rebuild produces one Reed–Solomon codeword from the shreds it used. For the batch to pass, that codeword must equal the signed shreds at every position the validator did not receive (or the Merkle root differs) and at every coding position it did receive (or the parity check fails). The positions it used to rebuild match by construction. So passing requires the codeword to match all 64 signed shreds, which only happens if the batch was a valid codeword to begin with. In the packet-loss simulation every validator that reached 32 shreds flagged every malformed batch, at every loss rate from 0 to 45%, and passed the honest one. That held with one shred per insert, and again with inserts of a random 1 to 16 shreds, where about half the failures of the last-coding-shred batch came from the parity comparison rather than the Merkle root. The only validators without a verdict were those that never collected 32 shreds through Turbine, 16 to 19% of them at 45% loss and none at 20% or less; they run the same check when repair brings the batch to 32. - * * What rebuilding every batch costs Less than it sounds. The old code already rebuilt a batch whenever a data shred was missing when it reached 32 shreds, and that is nearly always. The leader sends a batch’s data and coding shreds together, and Turbine routes each shred down its own tree. If they arrive in random order, the first 32 are exactly the 32 data shreds with probability 1 in C(64, 32), about 5.5 × 10⁻¹⁹, which fits the pull request’s estimate that the old code skipped recovery less than once in a trillion batches. What is new is re-encoding the received coding shreds when an insert pushes a batch past 32 at once, and rebuilding batches made only of data shreds. Timed on our model, per batch: Old recovery, 16 data + 16 coding — 299 µs New rebuild, same 32 shreds — 299 µs New rebuild, 24 data + 16 coding — 325 µs New rebuild, 32 data and no coding — 299 µs New rebuild, all 64 received — 305 µs Most of that is hashing the 64 leaves of the Merkle tree, about 214 µs here, which both versions do; encoding 32 coding shreds takes about 83 µs. For the same 32 shreds the new path costs exactly what the old one did, and re-encoding the received coding shreds adds about 9% when more than 32 are present. These are timings of our model on an Apple M1 Pro; absolute numbers depend on the hardware, SHA-256 speed above all. - * * Two Turbine changes around it Rebuilt coding shreds are now retransmitted. A validator forwards rebuilt shreds to its children in Turbine, so one dropped packet upstream doesn’t starve everyone below. When an insert brings only data shreds, the rebuild produces coding shreds the validator never received, and the first version of always-recover, keeping Agave’s old output, threw them away. #16622 forwards them as well, which its description notes Firedancer already did. Leaders no longer rebroadcast. When a leader’s own slot had not been seen by enough stake, Agave’s replay stage asked the broadcast stage to send every data and coding shred of that slot again, after 5 seconds and then at doubling intervals up to 320 seconds. On 5 November #16595 removed Tower consensus from Agave’s replay stage as part of its Alpenglow cleanup, and the request went with it. Hours later #16713 deleted the broadcast worker that served those requests. The issue behind it had called the mechanism redundant if repair works. A validator missing shreds now has two ways to get them: rebuild them from the batch, or ask a peer through repair. - * * What this does not show We ran a model of Agave’s rebuild, not Agave: shred headers, signatures, the per-shred checks and the batch header checks are not in it, and a malformed batch here differs from an honest one only in its erasure coding. Arrival order and loss are random and independent, which real Turbine is not, and inserts are one shred or a random 1 to 16. We had no cluster, so nothing here was measured on a network; the timings are from an Apple M1 Pro, not validator hardware. Firedancer’s implementation of the same rules was out of scope. - * * Shred Recovery: Questions People Actually Ask What is a Solana erasure batch? A group of 64 shreds: 32 data shreds carrying the block and 32 coding shreds that are their Reed–Solomon parity, so any 32 of the 64 rebuild the rest. All 64 are leaves of one Merkle tree, and the leader signs its root. Agave and SIMD-0504 also call it an FEC set. What does SIMD-0504 change about shred validation? It replaces the checks validators inherited from Agave’s code with a written list: sixteen checks on each shred and eight on each erasure batch as a whole. A batch that fails them makes the block invalid: under TowerBFT a validator must not vote for it, under Alpenglow it must vote skip. The checks apply whether a shred was received or rebuilt, so a validator has to check the coding shreds even when all 32 data shreds arrived. What is “always recover” in Agave? Agave rebuilds every erasure batch once, in the batch of incoming shreds that first brings it to 32 shreds, even when no data shred is missing. It re-encodes the coding shreds from the data, compares them with the coding shreds it received, and rebuilds the Merkle tree from all 64 leaves to compare with the signed root. With the SIMD-0504 feature active, a failure marks the slot dead unless it was already full. Is SIMD-0504 active on Solana mainnet? No. Agave’s gate, stricter_shred_validation, was merged on 30 October 2026 and had no feature account on mainnet, testnet or devnet on 6 November. Until it is activated, Agave runs the checks and reports failures to metrics without rejecting anything new. Does a Solana leader still rebroadcast slots that did not propagate? Not in Agave’s main branch. The replay code that asked a leader to resend an unpropagated slot was removed on 5 November 2026 along with Tower consensus in replay, and the broadcast worker that served those requests was deleted the same day. Missing shreds are recovered from erasure coding or fetched through repair. - * * The one-line version A leader can sign an erasure batch whose shreds each verify but whose coding shreds aren’t the parity of its data; SIMD-0504 makes that batch invalid, and Agave now rebuilds every batch once it has 32 shreds, so every validator rejects it the same way, whichever shreds reached it. Sources: SIMD-0504 at commit 33c75cff; Agave #16194, #16378, #16639, #16622, #16595 and #16713, read at commit 9eacc06d; the old recovery path at dd6f564f. The model, simulation and timings are our own.

Solana WebSockets That Stay Connected but Miss Updates Originally published on xroot.dev. Your indexer subscribes to a program’s logs at finalized, because it only wants data that can never roll back. The socket stays open. No error arrives, no disconnect, no lag warning. And some transactions that are finalized on chain never reach it. We measured how often on 5 November 2026, against the public mainnet RPC endpoint, which reported Agave solana-core 4.4.0. Over half an hour, a finalized logsSubscribe stream never looked at about 3% of slots and missed 61 of 1,803 finalized transactions that getSignaturesForAddress returns for the same slots. A processed stream on the same connection missed none. The cause is a seven-year-old design decision meeting a consensus protocol that has run mainnet for two weeks. It was reported on 3 November, and a fix was proposed the same day. Here is what the stream reads, why it skips, which subscriptions are exposed, what the fix changes for clients, and the reconciliation loop that catches the gap today. - * * One finalized transaction in thirty never arrived The setup was three programs and three commitment levels. The Address Lookup Table program and the first Memo program land roughly one transaction every slot or two, so each of their transactions is an independent test of whether a stream looked at its slot. The second Memo program lands a transaction in almost every slot, so the slots a stream reported for it are the slots that stream scanned. Ground truth was getSignaturesForAddress at finalized, polled over the same window. All nine subscriptions shared one WebSocket connection, so one node served all three commitment levels. The run covered 05:36 to 06:07 UTC, slots 465,589,591 to 465,598,091. The endpoint dropped our connection about once a minute under the high-volume Memo stream (close code 1006, no close frame), so each level is judged only on slots strictly inside a stretch where its connection was live. finalized — slots never scanned — 53 of 1,664 · 3.2% finalized — finalized transactions missed — 12 of 438 · 2.7% confirmed — slots never scanned — 4 of 1,664 · 0.24% confirmed — finalized transactions missed — 4 of 440 (3 in one slot) processed — slots never scanned — 0 of 1,664 processed — finalized transactions missed — 0 of 440 A second run, with one connection per commitment level and so most likely three different nodes behind the load balancer, agreed. Finalized never scanned 93 of 3,152 slots (2.95%) and missed 49 of 1,365 transactions; confirmed skipped 8 slots (0.25%) and 2 of 2,415 transactions; processed skipped none. Across both runs, finalized missed 61 of 1,803. programSubscribe behaves the same way. A third connection subscribed to the Lookup Table program’s accounts at all three levels for 30 minutes. In the stretch all three covered, the confirmed stream delivered 37 table updates (3.0% of 1,253) that the finalized stream never sent, and the finalized stream delivered none that confirmed did not. > The gaps are not random. Almost every jump skipped exactly one slot. Of the 146 slots the finalized streams never scanned across both runs, 135 were the third slot of a leader’s four-slot window (slot number mod 4 = 2, which is how Agave numbers positions in a window), and so were all 12 the confirmed streams skipped. Neither the issue nor the pull request mentions it, and we have not traced the cause. Something deterministic in when that slot becomes rootable is a better guess than thread scheduling. - * * Each update reads exactly one slot Agave’s PubSub service is driven by commitment updates. When the processed or finalized slot moves, the notifier thread visits every subscription at that level. For each one, it loads _one_ bank, the one at the new commitment slot, and runs a query against it. What the query returns depends on the subscription type, and for two of them the answer is “only what happened in this slot”. // rpc/src/rpc_subscriptions.rs — one bank: the new commitment slot's let bank = bank_forks.read().unwrap().get(slot); // programSubscribe's query against that bank bank.get_program_accounts_modified_since_parent(&params.pubkey) // runtime/src/bank.rs — which scans only that slot's own storage self.rc.accounts.load_by_program_slot(self.slot(), self.bank_id(), Some(program_id)) // logsSubscribe reads the same bank's log collector: its own transactions only bank.get_transaction_logs(pubkey) programSubscribe sees the accounts written _in_ the slot it was handed, and logsSubscribe sees the transactions executed _in_ it. Nothing records which slots a subscription has already been shown: the “last notified slot” passes through these two filters untouched. So if the finalized slot moves from 5 to 7, slot 7 is scanned and slot 6 is not, by this update or by any later one. The one-slot read for program subscriptions dates to March 2019. It is not a regression. The change is in how the commitment slot moves. - * * Three ways the commitment slot jumps The issue names three paths, and each one is in the source. - Coalescing. The commitment service drains its queue with try_iter().last(): if two updates are waiting when it wakes, it keeps the newer one and the older slot is never announced. Both the TowerBFT and the Alpenglow inputs are read this way. - A skip vote the cluster overrules (processed). Under Alpenglow, processed follows the node’s own notarize votes. If the node votes to skip a slot the cluster then notarizes, it never notarizes that slot, and a parent check stops it notarizing the rest of that leader window. Processed then jumps to the next window. - One root for many slots (finalized). Votor roots the highest finalized block it can root, and a single Rooted message moves the node’s finalized and root slots to that slot at once. It is published only after the bank tree has been rooted and pruned, so the banks in between are gone by the time the notifier runs. The third path is the one our data shows. Alpenglow has decided finality on mainnet since 21 October 2026, and under it a block is rootable only once the node has voted on it and replayed it. A finalized block that is not rootable yet when a later one is gets folded into the later root. The processed paths exist in the code, but they did not fire on this node during our half hour. - * * Which subscriptions are exposed, and why confirmed is not a cure The one-slot read is specific to two subscription types. The others either read through ancestors or walk back over the slots they missed: programSubscribe · logsSubscribe — one slot per update accountSubscribe — reads through ancestors — latest state arrives signatureSubscribe — status cache, through ancestors — arrives blockSubscribe — walks back to the last slot sent — arrives rootSubscribe — every newly rooted slot, in order — arrives, twice Confirmed takes a different route, which is why it did twelve times better. Under Alpenglow there is no optimistic confirmation in the TowerBFT sense, so votor sends the RPC layer an “optimistically confirmed” notification for each new root, with a comment explaining it is critical for compatibility with the RPC API. The tracker that receives it walks every ancestor above the last confirmed slot and issues one update per slot. On the same node, confirmed and finalized notifications for a slot arrived a median of 9 milliseconds apart: on an Alpenglow Agave node, confirmed is already root-timed. It still skipped 0.25% of slots, always the same window position as the finalized gaps. Our reading is a race with pruning. The confirmed walk follows parent links that squashing a root clears, and each slot it queues is then loaded from a bank tree that rooting prunes. Both happen at the moment a slot becomes final. We have not proven which of the two loses in our samples. - * * The fix on the table, and what it changes for clients Pull request 16689 changes the read from “one slot” to “every slot since the last one this subscription was shown”: - Snapshots before rooting. Each bank’s account writes and transaction logs are copied into a retained cache before rooting runs, so pruning and accounts cleaning cannot erase an update that has not been delivered yet. Rooted banks are captured from the same function that rooted them. - Delivery in order, once. On each commitment update, the notifier walks from the last bank this subscription was shown to the new slot and delivers each in order, deduplicating frozen banks by bank ID. The confirmed path gets the same retained chain. - A visible change in context.slot. Today a notification’s context.slot is the commitment slot that triggered it. With the fix it is the slot where the change happened, so a jump from 5 to 7 arrives as an update for 6 and then one for 7. Code that treats context.slot as “the current finalized slot” will see it lag behind. - Bounded, and loud when exceeded. The cache is capped at the PubSub queue budget (--rpc-pubsub-queue-capacity-bytes, 256 MiB by default) and at 512 banks, under two minutes at the 4.7 slots a second we measured. A subscription that needs a bank already evicted is disconnected through the same path a lagging client takes today, instead of being skipped silently. As of 5 November the pull request is open with no reviews, and no release contains it. The public endpoint we measured runs 4.4.0, which has the one-slot read, the coalescing and the root-timed confirmed path exactly as described above. - * * Catching the gap in your own pipeline No subscription setting makes a PubSub stream complete today, so the dependable move is to diff it against the ledger. For a logs stream, getSignaturesForAddress at finalized is the list of everything that should have arrived. This is the loop we ran: // ws: an open WebSocket to your RPC node · rpc(): a plain JSON-RPC POST helper const PROGRAM = 'AddressLookupTab1e1111111111111111111111111'; const GRACE_SLOTS = 150; // ~30 s for the stream to deliver before we call it missed const seen = new Set(); // signatures the stream delivered let fromSlot = 0; // first slot the stream was live for let cursor; // newest signature already reconciled ws.send(JSON.stringify({ jsonrpc: '2.0', id: 1, method: 'logsSubscribe', params: [{ mentions: [PROGRAM] }, { commitment: 'finalized' }], })); ws.onmessage = ({ data }) => { const msg = JSON.parse(data); if (msg.method !== 'logsNotification') return; const { context, value } = msg.params.result; fromSlot ||= context.slot; seen.add(value.signature); }; async function reconcile() { if (!fromSlot) return; const sigs = await rpc('getSignaturesForAddress', [ PROGRAM, { commitment: 'finalized', limit: 1000, ...(cursor && { until: cursor }) }, ]); if (!sigs.length) return; // Only judge signatures the stream has had time to deliver. const settled = sigs.filter((s) => s.slot <= sigs[0].slot - GRACE_SLOTS); if (!settled.length) return; cursor = settled[0].signature; for (const s of settled) { if (s.slot > fromSlot && !seen.has(s.signature)) { // fetch with getTransaction and apply it console.log('missed', s.signature, 'in slot', s.slot); } } } setInterval(reconcile, 30_000); Against mainnet from 07:57 UTC for three and a half minutes, it checked 239 finalized signatures and flagged 5 that the stream had skipped, every one of them in the third slot of a leader window. The grace period matters: the HTTP call and the WebSocket can land on different nodes, and without it a signature the stream is about to deliver looks missed. - Subscribe at confirmed to shrink the backlog. On an Alpenglow Agave node it arrived with finalized and skipped a twelfth as often. Gate on rootSubscribe if you need finality to be explicit rather than a property of today’s implementation: the source says the confirmed notification may be deprecated. Make that gate idempotent. On the same endpoint, rootSubscribe sent each root twice, once on its own and again just before the next one (127 notifications for 66 roots in 15 seconds). - Treat every disconnect as a gap. Reconcile from the last slot you finished. After the fix, falling behind the retention window is exactly how a gap will announce itself. - For account state, re-read what the missed transactions touched. getMultipleAccounts with minContextSlot, or accountSubscribe for a known set, which reads through ancestors. - Know which path you are on. Geyser plugins are fed from replay, not from this notifier, so a Geyser-based stream does not go through the one-slot read. A provider’s own WebSocket product may or may not be Agave’s PubSub underneath; the reconciliation loop works either way. - * * What this does not show One endpoint, one morning, half an hour per run, and three programs. The rate will move with load and with the node. The logs numbers are checked against the ledger; the programSubscribe number is a comparison with the confirmed stream on the same connection, because a lookup table can change without an account notification under that owner (a closed table, for one). We did not build or run the patched node, so section 05 describes the fix from its code and tests, not from measurement. Processed showed no gaps here, although the paths that would skip it are real. TowerBFT is not measured at all: mainnet no longer runs it. The one-slot read is as old under TowerBFT as under Alpenglow, so the exposure is not new, but we cannot say how often it fired before 21 October. The leader-window clustering is unexplained. - * * Missed WebSocket Updates: Questions People Actually Ask Does programSubscribe miss updates on Solana? At processed and finalized commitment on an Agave node, yes, whenever the commitment slot advances by more than one slot. Each update reads only the accounts written in the one slot it was triggered for, so writes in the slots it jumped over are never sent. On mainnet on 5 November 2026 a finalized stream jumped over about 3% of slots. Is confirmed commitment safe to use instead of finalized? It is much better but not complete. On an Agave 4.4 node under Alpenglow, confirmed notifications are sent when a block is rooted and walk every slot since the last one, so they arrive within milliseconds of finalized and skipped about 0.25% of slots in our measurement against 3% for finalized. Reconcile against the ledger either way. Does accountSubscribe have the same problem? No. accountSubscribe reads the account through all ancestors of the commitment slot and sends whenever its last-modified slot changes, so the latest state always arrives. signatureSubscribe, blockSubscribe and rootSubscribe also cover the slots in between. How do I detect missed logsSubscribe notifications? Diff the stream against getSignaturesForAddress for the same program at finalized commitment, after giving the stream a grace period to deliver, and fetch anything missing with getTransaction. Treat every disconnect as a gap to reconcile as well. Has the fix shipped? Not as of 5 November 2026. Pull request 16689 in the Agave repository, opened on 3 November, retains the skipped slots and delivers them in order. It is open and unreviewed, and no release contains it. - * * The one-line version On Agave today, a finalized programSubscribe or logsSubscribe stream is a sample of the finalized slots, not the set. On mainnet it is about 97% of them. Subscribe at confirmed, reconcile against the ledger, and treat a disconnect as a gap. When the fix ships, a gap will close your socket instead of passing unnoticed. The commitment levels themselves are explained in how many confirmations a deposit needs, and the certificates behind them in Alpenglow finality in numbers. A pipeline built on these streams is the subject of ingesting the Solana firehose with Rust, and the errors a reconnect can return are in the RPC error codes. Sources: Agave master at commit 32f5b38fd6, rpc_subscriptions.rs, bank.rs, commitment_service.rs, votor’s event_handler.rs and optimistically_confirmed_bank_tracker.rs; the same paths checked in the v4.4.0 release; issue 16654 and pull request 16689 as of commit cfa411b669. Measurements: our own WebSocket and JSON-RPC calls to the public mainnet endpoint on 5 November 2026, 05:34 to 06:09 and 07:57 to 08:01 UTC.

When a Solana Feature Activates but Its Program Migration Fails Originally published on xroot.dev. On Solana, a feature gate activating and the change it carries actually happening are two separate events. For most gates the difference never shows. For a gate that replaces a core program’s bytes, it can: the replacement runs once, at the epoch boundary, and if it fails, the gate stays active anyway. SIMD-0690, merged on 26 October, adds a deliberate way for that replacement to fail: the runtime hashes the new program bytes and refuses to deploy them unless they match a hash shipped inside the validator. Agave implemented it on 29 October. On 2 November it queued the first migration that relies on it, p-ATA, a rewrite of the Associated Token Account program, and Firedancer followed on 3 November. We pulled the real p-ATA buffer from mainnet, hashed it the way both clients do, and replayed every way the activation can go through the loader Agave uses. The check works. It also depends on something no code enforces: the two clients agree only if the gates activate in the right order. - * * Activated is not the same as replaced In the first bank of each epoch, Agave works out which feature gates have just activated, writes the activation slot into each gate’s account, and then runs the migrations tied to those gates. The p-ATA one is a single call, and its failure path is a log line: if new_feature_activations.contains(&feature_set::replace_ata_with_p_ata::id()) && let Err(e) = self.upgrade_loader_v2_program_with_loader_v3_program( &feature_set::replace_ata_with_p_ata::SPL_ASSOCIATED_TOKEN_ACCOUNT_PROGRAM_ID, &feature_set::replace_ata_with_p_ata::PROGRAM_BUFFER, feature_set::replace_ata_with_p_ata::BUILD_HASH, true, "replace_ata_with_p_ata", ) { warn!("Failed to replace SPL Associated Token Account with p-ATA buffer '{}': {e}", ...); } Every check runs before anything is written, so a failure changes nothing except the gate. The gate is now active, so it is never a new activation again and the call never repeats. The proposal says so in one line: on a mismatch, “the feature is still marked as activated, but the program data remains unchanged.” A retry means a new gate in a new release. That leaves four states a migration can be in: Gate account does not exist — not submitted Gate exists, no activation slot — pending, next epoch Active, program under Loader v3 — migrated Active, program under its old loader — activated, migration failed - * * What the hash check adds A migration copies a Loader v3 buffer account, uploaded ahead of time, into the target program’s address. Before SIMD-0690, the runtime checked that the buffer existed, was owned by the upgradeable loader and deserialized as a buffer, then deployed whatever it held. Agave’s implementation adds one step between those checks and the deploy: let buffer = Self::new_checked(bank, buffer_address)?; let data = buffer.buffer_account.data(); let elf = &data[UpgradeableLoaderState::size_of_buffer_metadata()..]; let end_offset = elf.iter().rposition(|&x| x != 0).map_or(0, |i| i + 1); let hash = solana_sha256_hasher::hash(&elf[..end_offset]); if hash != expected_hash { return Err(CoreBpfMigrationError::BuildHashMismatch(hash, expected_hash)); } Skip the 37-byte buffer header, drop trailing zero bytes, take SHA-256, compare. Only then does the deploy step load and verify the ELF in the runtime’s program environment. That step deliberately still accepts SBPF v0 programs “for core program migrations”, which matters here because the p-ATA ELF is v0. What came before is the reason for the change. When p-token replaced SPL Token in place on 13 May 2026, the function that did it took a program address, a buffer address, a flag and a log name. There was no hash. A Loader v3 buffer always has a write authority (the loader refuses to remove it), so until the activation slot, any change that authority made to the buffer, as long as it still loaded and verified, would have become SPL Token. Our p-token explainer covers that migration. Agave’s SIMD-0690 change also turns the expected hash from an optional parameter into a required one, so every future migration must name its bytes, while the check itself waits for its own gate, core_bpf_hash_validation. - * * The real buffer, hashed SIMD-0567 merged on 1 November at 11:02 UTC. Twenty-seven minutes later, the p-ATA buffer appeared on mainnet: 37 transactions in slots 464,075,848 to 464,075,854, between 11:30:19 and 11:30:21 UTC. No transaction has touched it since. We read its bytes and hashed them both ways: Account size — 35,869 bytes Loader v3 buffer header — 37 bytes, authority set ELF — 35,832 bytes, SBPF v0 Trailing zero bytes — 15 SHA-256, zeros removed — 4e008052⁠…⁠7f7d9e Agave, Firedancer and SIMD-0567 expect — 4e008052⁠…⁠7f7d9e SHA-256 of the whole ELF — 8dc8d134⁠…⁠3605a4 The buffer matches. The 15 zero bytes are worth a second look, because they are not padding. The buffer holds exactly one ELF file, and the file itself ends in zeros: its last section header has an alignment of 1 and an entry size of 0. Hash the ELF as a file and you get a different value. The expected hash is defined over the stripped bytes, the way solana-verify computes it, so solana-verify get-buffer-hash reproduces it straight from the chain. The header also shows the buffer’s authority is still set, to 8pEc9HbS…7pkzf. Until the gate activates, whoever holds that key can rewrite or close the buffer. Before SIMD-0690 a rewrite would have been deployed. Now it can only make the activation fail. - * * Seven ways the activation can go To see the success and failure paths without a cluster, we built a harness around the real buffer bytes. It runs Agave’s check copied verbatim, Firedancer’s check ported from C, and the deploy step through solana-sbpf 0.26.0, the loader version Agave pinned at the commit we read, configured the way Agave’s migration path configures it. With the SIMD-0690 gate active, both clients give the same answer for every case: Buffer as it is on mainnet — migrated Plus 4,096 zero bytes — migrated, same hash Minus its last 15 zero bytes — hash passes, loader rejects One .rodata letter changed — hash mismatch Instruction 17 immediate, 83 → 84 — hash mismatch First opcode set to 0xff — hash mismatch Buffer closed before activation — account not found Every failure leaves the gate active and ATA exactly as it was. Two of the rows are worth a closer look. The hash pins the content but not the length. A buffer missing its last 15 bytes hashes identically, because those bytes are zeros. It is still a broken ELF: its section header table now runs past the end of the file, and the loader stops with “Offset or value is out of bounds.” The check and the loader cover each other. Neither alone would catch both a substituted program and a truncated one. The authority problem has changed shape rather than disappeared. A changed letter in a string constant and a changed constant in an instruction both pass the loader and the verifier. Only the hash notices them. The buffer’s authority can no longer decide what ATA becomes, but closing or rewriting the buffer before activation still decides that ATA does not change, and a retry then needs a new gate. - * * When the order slips, the clients split The two clients compute the same hash, but they decide differently whether to compute it. Agave hashes only if core_bpf_hash_validation is already active and otherwise skips the check. Firedancer always passes the p-ATA hash and checks whenever one is passed. Its feature map says why: it “assumes this feature gate is activated for all migrations”, because the SIMD-0690 gate will activate first. The same harness, with the SIMD-0690 gate inactive when p-ATA activates: Buffer as it is on mainnet — both migrate One .rodata letter changed — Agave deploys, Firedancer refuses Instruction 17 immediate, 83 → 84 — Agave deploys, Firedancer refuses First opcode set to 0xff — both refuse, for different reasons In the highlighted rows the clients finish the epoch’s first slot with different bytes at the ATA address, so they compute different bank hashes. Validators executing with Firedancer’s runtime would disagree with validators executing with Agave’s about the same block. Frankendancer, the hybrid, executes with Agave’s runtime, so the line falls between runtimes rather than client brands. Which software each validator runs is visible in gossip and in block footers. None of this is a bug today. The buffer on mainnet matches, so the order makes no difference to the bytes that would be deployed. SIMD-0567 names SIMD-0690 as a dependency, and both clients document the order. Activating both in the same epoch is also safe: Agave installs the new feature set before it runs any migration, so the check is already on. But the order is a scheduling commitment, not a check in code. Anza’s activation guidelines fix a priority order and say exceptions will occur. The thing to watch is simple: gate FCsQnd4V…zp1u must be active no later than gate pata6uFa…ksx3. Neither gate account had been created on mainnet when this was written. - * * Telling the states apart A failed migration leaves no transaction and no event on chain. The only direct record is a warning in each validator’s log. It can still be detected from three accounts, read in one call: const FEATURE_OWNER = 'Feature111111111111111111111111111111111111'; const LOADER_V3 = 'BPFLoaderUpgradeab1e11111111111111111111111'; async function migrationState(rpc, gate, program, buffer) { const res = await fetch(rpc, { method: 'POST', headers: { 'content-type': 'application/json' }, body: JSON.stringify({ jsonrpc: '2.0', id: 1, method: 'getMultipleAccounts', params: [[gate, program, buffer], { encoding: 'base64' }] }), }); const [g, p, b] = (await res.json()).result.value; // Feature account: owned by the Feature program, byte 0 = Some(activated_at) const data = g?.owner === FEATURE_OWNER ? Buffer.from(g.data[0], 'base64') : null; if (!data) return 'gate not submitted'; if (data[0] !== 1) return 'gate submitted, pending'; const slot = data.readBigUInt64LE(1); if (p.owner === LOADER_V3) return `activated at ${slot}: migrated`; return `activated at ${slot}: MIGRATION FAILED (buffer ${b ? 'still there' : 'gone'})`; } // p-token: activated at 419472000: migrated // p-ATA: gate not submitted The two comments are the script’s output against mainnet, for p-token and p-ATA. Checking the owner of the gate account matters: a gate’s address can hold lamports in an ordinary system account before anyone submits it, and reading those as feature data would report nonsense. Our upgrades tracker follows which protocol upgrades are live. When p-ATA does migrate, the program account at ATokenGP…8knL changes owner from BPFLoader2111… to BPFLoaderUpgradeab1e…, gains a program-data account and has no upgrade authority. SIMD-0567 adds that ATA cannot be invoked during the migration slot, because Loader v3 programs take effect one slot after deployment. Tools that decide whether a program is upgradeable by its loader will see the change. - * * Two smaller things in the code The log reads backwards. The mismatch error is built as BuildHashMismatch(hash, expected_hash), with the buffer’s hash first, but its message labels the first value “Expected”. For our altered-letter buffer, the warning prints: Invalid build hash. Expected: 7c10bb1d…a3b147e730, Got: 4e008052…674e7f7d9e The value after “Expected” is what the buffer holds, and the value after “Got” is what the validator expected. The labels have been this way since the error was introduced in April 2025. Anyone diagnosing a failed migration from logs should swap them. A panic fixed in passing. The 2025 version of this check searched for the last non-zero byte across the whole account, header included. When the program bytes were all zero, as in a buffer created but never written, that search could land inside the 37-byte header and produce a slice that ends before it starts. Our harness reproduces the panic for a header with no authority, and for an authority whose last byte is zero. On a real cluster it would have stopped every validator at the activation slot. The SIMD-0690 commit fixed it under the line “strip trailing zeros from ELF bytes only”. It was never reachable on mainnet: the only migration that ever passed a hash before then was the slashing program’s, whose gate was never created on mainnet, and Agave removed it in June 2026. Firedancer’s version never had the problem. - * * What this does not show The harness replays the checks and the loader on real bytes. It does not run a cluster through an epoch boundary. The deploy column uses Agave’s loader for both clients; Firedancer has its own loader, which we did not run. The altered buffers are our edits, not anything seen on chain. We checked the on-chain buffer against the published hash but did not rebuild p-ATA from source with solana-verify. We read mainnet only, not testnet or devnet. - * * Program Migrations: Questions People Actually Ask Can a Solana feature gate be active while the program it was meant to replace is unchanged? Yes. A core program migration runs once, inside the first bank of the epoch in which its feature gate activates. If any check fails, the runtime logs a warning and moves on: the feature account records its activation slot and the target program keeps its old bytes. Nothing retries, because the gate is never newly activated again. A second attempt needs a new feature gate in a new validator release. What exactly does SIMD-0690 hash? The program bytes in the source buffer account: everything after the 37-byte Loader v3 buffer header, with trailing zero bytes removed, hashed with SHA-256. The result must equal a hash compiled into the validator next to the feature gate. solana-verify computes executable hashes the same way, so solana-verify get-buffer-hash reproduces the value from the chain. Is the p-ATA buffer on mainnet the one Agave expects? Yes. The buffer at pataAAkF439TTjwRqmxRkTTEfZemRdoriXusez4PQ6S hashes to 4e008052…7f7d9e, the value in Agave’s replace_ata_with_p_ata gate, in Firedancer’s constant and in the SIMD-0567 text. It was created and written in 37 transactions on 1 November 2026 and has not been written since. Do Agave and Firedancer run the hash check the same way? They compute the same hash but differ on when they check it. Agave checks only once the SIMD-0690 gate, core_bpf_hash_validation, is active. Firedancer always checks the p-ATA hash and records that it assumes the SIMD-0690 gate activates first. If p-ATA activated before SIMD-0690 with an altered buffer, Agave would deploy the altered bytes, Firedancer would refuse them, and the two clients would compute different bank hashes. How do I check whether a migration actually happened? Read three accounts. The feature account is owned by the Feature program and holds an activation slot once active. The target program is owned by BPFLoaderUpgradeab1e… after a successful migration. The source buffer is consumed on success. An active gate with the program still under its old loader means the migration failed. - * * The one-line version SIMD-0690 makes a core program migration deploy the reviewed bytes or nothing. A failure leaves the gate active and the program unchanged, and Agave and Firedancer agree on which happened only if the hash-check gate activates first. The last in-place swap is in SPL Token was replaced in place, the prefunded-account rule p-ATA builds on is in SIMD-0312’s CreateAccountAllowPrefund, and how to tell which client built a block is in reading the block footer. Sources: SIMD-0690 at commit eac20f29; SIMD-0567 at 93cdf03a; Agave #16077 and #16524 (source read at 9260869d); Firedancer #12680 (37c65819); the buffer and gate accounts read from mainnet with getAccountInfo, getMultipleAccounts and getSignaturesForAddress. Scenario results are from our own harness.

Why Solana’s getFeeForMessage Is Not a Transaction-Validity Check Originally published on xroot.dev. getFeeForMessage answers one question: what would this message pay? For years Agave also answered null when the message’s blockhash was gone or its durable nonce did not match, and code learned to read that null as “get a new blockhash”. Agave 4.5 stops doing it. And the null was never a reliable expiry signal in the first place. Agave #16519, merged on 29 October, removes both checks, because the fee depends on neither. We ran the old behaviour on a local Agave 4.2.1 validator, which has the same check as the 4.4 release, and found a 150-block window in which it priced transactions that the same node was already rejecting with “Blockhash not found”. - * * What the fee is made of The fee has two parts: 5,000 lamports for each signature, counting the signatures inside Ed25519, secp256k1 and secp256r1 precompile instructions, and the priority fee, the compute-unit limit times the compute-unit price in micro-lamports, divided by a million and rounded up. Both come from the message itself: Transfer, one signature — 5,000 Two signatures — 10,000 200,000 CU at 1,000,000 µlamports — 205,000 200,000 CU at 1 µlamport — 5,001 Values in lamports, from our local validator. The last row is the rounding: 0.2 lamports of priority fee costs 1. How that fee ranks a transaction against others is a separate calculation. Nothing here depends on the recent blockhash or on a durable nonce. - * * What the old version checked first Up to 4.4, Agave’s bank refused to price a message unless its blockhash was in the bank’s blockhash queue or, failing that, the message was a durable-nonce transaction whose nonce account stored exactly that value. Either check failing meant null. So did invalid compute-budget instructions, such as two SetComputeUnitLimit instructions. Three different problems, one answer. The catch is in the queue. A bank keeps the last 300 blockhashes, but accepts a transaction only if its blockhash is at most 150 blocks old (blockhash_queue.rs, is_blockhash_valid). The fee check asked “is it in the queue?”, not “is it young enough?”. We held one blockhash on the local validator and recorded when each method gave up on it: Blockhash issued at block height 360 — +0 lastValidBlockHeight 510 — +150 blocks isBlockhashValid false, at height 511 — +151 blocks getFeeForMessage still — 5,000 sendTransaction — Blockhash not found getFeeForMessage null, at height 661 — +301 blocks For 150 blocks, about a minute at mainnet’s 400 ms slots, the old method priced a transaction that the network rejected. Code that waits for a null before refreshing its blockhash spends that minute sending transactions that are dropped. - * * What Agave 4.5 changes The new function reads the compute-budget instructions and computes the fee, nothing else. The changelog lists it under 4.5.0, which was unreleased at the merge, when the stable line was 4.3 and the beta 4.4. Side by side, with the old column measured on 4.2.1 and the new one read from the merged code: Fresh blockhash — 5,000 → 5,000 Blockhash 151–300 blocks old — 5,000 → 5,000 Blockhash dropped or never seen — null → 5,000 Durable nonce, stored value — 5,000 → 5,000 Durable nonce, any other value — null → 5,000 Duplicate SetComputeUnitLimit — null → null v0, lookup table missing — error → error The last row is a JSON-RPC error, -32602 “invalid transaction message: Attempted to lookup addresses from a table that does not exist”: the RPC method resolves lookup tables before it asks the bank. The issue behind the change gives one more reason for it: the nonce check was the only part of the fee calculation that needed resolved addresses, which kept RPC from using a lighter message type. In 4.5 a null means one thing only: the compute-budget instructions are invalid. Like the 4.4 RPC changes, it will reach clients gradually, as RPC providers upgrade, so code meets both behaviours for a while. - * * What a fee does not promise A fee is computed from the message alone, in every version. It never checked the things that decide whether a transaction lands. We signed and sent three transactions the local node had just priced at 5,000 lamports: Fee payer with no account: “no record of a prior credit” — rejected Transfer above the payer’s balance: custom program error 0x1 — rejected Blockhash past its last valid height: “Blockhash not found” — rejected All three were rejected in preflight simulation. A fee also says nothing about signatures, account locks, the compute the instructions will use or whether a program succeeds. From 4.5 it says nothing about the blockhash or the nonce either. - * * What to call instead Is the blockhash still usable? getLatestBlockhash -> lastValidBlockHeight, then getBlockHeight <= lastValidBlockHeight (or isBlockhashValid) Is the durable nonce current? getAccountInfo on the nonce account (jsonParsed): its stored blockhash must equal the message's Would it execute right now? simulateTransaction What will it pay? getFeeForMessage In our run isBlockhashValid first reported false at block height 511, one past the blockhash’s lastValidBlockHeight of 510, and sendTransaction rejected it there, so the height comparison matches what the node enforces. Simulation is the closest thing to a validity check, but it is one node’s view at one moment: an account can change between simulation and landing. - * * What this does not show We measured Agave 4.2.1’s test validator, whose fee check matches the 4.4 release line for line in behaviour; we did not run 4.4 or 4.5, so the 4.5 column comes from the merged code. A local validator makes blocks at its own pace, so the 150-block window is in blocks, not seconds. Every call used processed commitment on a single node. - * * getFeeForMessage: Questions People Actually Ask Does getFeeForMessage check whether a Solana blockhash is still valid? No. From Agave 4.5 it does not look at the blockhash at all. Before that it only checked that the blockhash was still in the bank’s queue of the last 300, while transactions are accepted only up to 150 blocks, so it returned fees for blockhashes the network had already stopped accepting. Use the lastValidBlockHeight returned by getLatestBlockhash, or isBlockhashValid. When does getFeeForMessage return null in Agave 4.5? Only when the message’s compute-budget instructions are invalid, for example two SetComputeUnitLimit instructions, bad instruction data or an invalid heap size. A message that cannot be read at all, such as a v0 message whose address lookup table does not exist, gets a JSON-RPC error instead of null. How long is a Solana blockhash valid for transactions? 150 blocks after the block that produced it. getLatestBlockhash returns lastValidBlockHeight, the last block height at which a transaction using it can land. In our test isBlockhashValid turned false at lastValidBlockHeight + 1, and sendTransaction rejected the blockhash from then on. Does getFeeForMessage include the priority fee? Yes. It adds the priority fee set by the message’s compute-budget instructions, compute-unit limit times compute-unit price in micro-lamports, divided by a million and rounded up, to 5,000 lamports per signature, counting the signatures in Ed25519, secp256k1 and secp256r1 precompile instructions too. Does a fee from getFeeForMessage mean the transaction will succeed? No. It is computed from the message alone. It does not check signatures, whether the fee payer exists or can pay, the blockhash or durable nonce (from Agave 4.5), account locks or whether the instructions succeed. Simulate the transaction to check those. - * * The one-line version getFeeForMessage prices a message and nothing more: from Agave 4.5 it returns a fee for expired blockhashes and wrong nonces, and before that it kept pricing blockhashes for 150 blocks after the network stopped accepting them, so check expiry with lastValidBlockHeight and validity with simulateTransaction. Sources: Agave #16519, merged as 8ffb33c9, and issue #15876; the 4.4.0 release source for the old check, the blockhash queue and the processing age. Measurements are from our own client against a local solana-test-validator 4.2.1.

🚨 Mainnet-beta validators: Agave v4.4.0 is recommended for general adoption. Upgrade now. github.com/anza-xyz/agave/releases/tag/v4.4.0

I used REA and a swarm of 40 parallel agents to reverse engineer the entire Solana blockchain. Leader schedule, Gulf Stream, Sealevel, Tower BFT. Rebuilt in 72 hours, zero humans in the loop. Releasing it all open source. No token, no paywall. anza-xyz/agave

🚨 Mainnet-beta validators: we're looking for volunteers to bring 10% of stake to Agave v4.4.0-rc.1 by EOD Friday, Oct 30 per routine upgrade procedure. github.com/anza-xyz/agave/releases/tag/v4.4.0-rc.1

Get your funds off any Solana or SVM chains and move them to Ethereum Mainnet. f5df29e4e18d99626f6676cfebfd026f86ff2e14643b5391d7c8df22f18e2494 anza-xyz/agave

🚨 Mainnet-beta validators: Agave v4.3.2 is the recommended version for all validators. Upgrade now. This release includes important performance and stability fixes for Alpenglow coming epoch 1064, Oct 21. github.com/anza-xyz/agave/releases/tag/v4.3.2

Anza Advances Solana Agave Client with Series of V4 Pre-Releases AI disclosure: drafted by Basis Desk's AI newsroom and machine-checked against the primary sources listed below — how we use AI. Originally published on Basis Desk. > Multiple pre-releases for the Agave validator client have been published on GitHub. Key points - Anza released Agave v4.3.1 as a pre-release on Oct. 13. - Agave v4.4.0-beta.1 was also released as a pre-release on Oct. 13. - Earlier pre-releases include v4.5.0-alpha.2, v4.5.0-alpha.1, and v4.4.0-beta.0. Anza has released a series of pre-release versions for the Solana Agave validator client [1]. Recent Agave Pre-Releases On Oct. 13, Anza released version v4.3.1, a pre-release for the Agave client [1]. Earlier the same day, version v4.4.0-beta.1 was also published as a pre-release [2]. These releases follow prior pre-releases, including v4.5.0-alpha.2 on Oct. 8 [3], v4.5.0-alpha.1 on Oct. 3 [4], and v4.4.0-beta.0 on Sept. 28 [5]. All of these versions are marked as pre-releases on GitHub [1][2][3][4][5]. The releases indicate ongoing development and testing for the Agave client, which is a fork of the original Solana Labs client [4]. Related: Anza Advances Solana Agave Client with Series of V4 Pre-Releases Sources 1. Release v4.3.1 — Agave (Solana) releases 2. Release v4.4.0-beta.1 — Agave (Solana) releases 3. Release v4.5.0-alpha.2 — Agave (Solana) releases 4. Release v4.5.0-alpha.1 — Agave (Solana) releases 5. Release v4.4.0-beta.0 — Agave (Solana) releases --- Basis Desk is a source-verified crypto newsroom. Market data, a free MCP server for AI agents and JSON APIs: basisdesk.news. Not investment advice.