Solana Network Slowed by New Software Bug

Last night, Solana network infrastructure operators encountered a software issue that almost froze all on-chain activity on the network.

Solana devs fixing the network over the weekend
Solana devs fixing the network over the weekend

Last night, Solana network infrastructure operators encountered a major technical issue that caused the blockchain to start forking, creating conflicting versions of its transaction history. This led to a decrease in transaction throughput and almost froze all on-chain activity on the network.

By 2am, the Solana network was processing only 93 transactions per second, compared to 5000 TPS just 15 minutes earlier. Validator operators and network engineers suspect the forking issue was caused by a bug in the new version of Solana code that had recently come online.

To try and restore Solana's operations, some validators began downgrading to the previous version, which was soon followed by a supermajority of validators. This didn’t fix the performance issues. The effort then turned to a more drastic solution: restarting the chain to the point immediately prior to the forking.

Coordinating a restart attempt meant that the chain would be offline completely, which was a risky move. However, this solution seemed to work, and within a few hours the network was fully operational again.

This incident was reminiscent of the chain's rocky tech incidents in 2022, prompting reforms to how Solana manages inbound traffic. As infrastructure operators continue to work on improving the Solana network's stability and performance.

As of Saturday afternoon, it seems like the Solana network is mostly back up to speed, so this was a pretty quick resolution. Of course, Solana is famous for breaking, but no one is going to care that the network goes down if they can bring it back up quickly every time.