A “race condition” after the system was reset prevented the sequencers from catching up, causing the second outage.
A sequencer bug was responsible for two outages of the Coinbase layer-2 network Base last week, according to a post-mortem.
The Base engineering team said in a Saturday post-mortem that they identified a bug in sequencer block-building logic that allowed “stale journal state” to persist after a transaction validation failure.
“An invalid transaction was received by the block builder and failed during execution, as expected, but erroneously did not clear the journal state that contained the accounts and storage slots that had been accessed,” said the team.
The Base layer-2 network runs a single sequencer, which means one bug can stop everything. It is a centralized blockchain component that decides the order of transactions and has been responsible for outages on other layer-2 chains, including Arbitrum, OP Mainnet and zkSync Era.
On Thursday and Friday, Base mainnet experienced two block production outages, the first incident lasted 116 minutes and the second lasted 20 minutes.
There was a complete halt of new layer-2 blocks, and the sequencer and validator nodes could not progress past the invalid block until sequencing was restored.
The team fixed the outages by applying a patch to the sequencers to ensure the journal state was properly updated during execution.
However, mitigation took longer than expected “due to infrastructure conditions unrelated to the original bug,” they said.
There was also a “race condition” after the system reset, which prevented the sequencers from catching up, causing the second outage.
Related: Coinbase's Base resumes block production after 2-hour outage
Going forward, the Base engineering team plans to improve protocol “fuzz testing,” which involves bombarding the system with large volumes of random, malformed, or unexpected inputs to find bugs, and building “graceful recovery” so that validator nodes don’t need manual restarts during future incidents.
Source
This article is syndicated for educational reading. For the latest updates, visit the original publisher.
Read on cointelegraph.com
