Polygon’s Network was down from 5:50 PM UTC on March 10th until about 4 AM UTC on March March 11th. All users’ funds are safe, and the team explained the particular issues for which the network was not operational during these 11 hours.
Yesterday, Polygon developers notified users of an upcoming potential downtime for the network because of a planned network upgrade – the Heimdall node.
The Heimdall implementation is used by one of the two layers of the network’s proof-of-stake chain. The team confirmed the downtime at 5:50 PM UTC on March 10th, providing further clarifications. It’s worth noting that no user funds were compromised during the outage.
While we are working on identifying the definitive cause, it seems to have originated from an earlier upgrade consisting of a minot parameterization fix to the Ethereum to Polygon PoS state sync/bridging module.
The team also revealed that there “may have been a bug” in the upgrade, which had an effect on the consensus and led to different Heimdall validators being on different versions of the blockchain. This prevented them from reaching the 2/3 consensus.
When using Tenderming consensus, this situation will cause the Heimdall chain to halt.
The developers also reaffirmed that Heimdall doesn’t handle user transactions but is used for validator-related bridging and transactions.
Furthermore, the team explained that the Bor chain, which is the user-facing PoS chain of Polygon, was also halted because it relies on Heimdall for “block proposer committee selection.”
At around 4 AM UTC on March 11th, the team successfully released a hotfix which restored operations on Polygon’s network, although the Bridge itself “will not be active until we fully resolve the issue,”
The most recent update reads:
We appreciate your support during this time and have an update: we have deployed a temporary hotfix to unblock the Bor chain and resume producing blocks on the Polygon chain. While this solution is temporary, the team is working on implementing a longer-term upgrade to fix the Heimdall issue.
The network’s bridge, however, will not be active until the final solution is released.