Starting on June 21st, 2022 and peaking on June 27th, Hop experienced a heavy increase in the number of transfers being made. This led to a multitude of issues for both Hop and its partners which resulted in some transfers being delayed. All transfers were eventually completed and no funds were lost or ever at risk of being lost.
Daily Transfer Count for Hop
All issues have been addressed and are detailed below.
The bonder’s throughput is limited in the speed it can process transactions. At the peak of activity last week, the number of transfers being made exceeded the number of transfers the bonder could process. This caused ETH transfers to go uncompleted on all chains while the bonder caught up with its backlog.
The bonder was also attempting to process transactions with very small or zero fees, which wasted valuable resources. Due to the high number of spam transactions during the spike, there was a serious slowdown in the bonder’s throughput. Our team put a fix in place to ignore transactions with too low of a fee before processing as well making several additional optimizations to greatly increase the Bonders’ throughput.
The Hop bonders rely on events emitted from on-chain transactions. When a bonder does not see a transaction’s event, it is as if the transaction never happened. In the past, the Hop team has worked with teams like Infura and ethers.js to identify and resolve issues they had around event reading. During this week, Alchemy has confirmed that there were intermittent issues with their Arbitrum nodes over the last few days, which resulted in 42 missed events.
The bonders were run with different RPC providers in order to see the observe the events from these transactions.
Since its inception, the gasPrice on Arbitrum has been consistently under 1 Gwei. During Odyssey week, the gasPrice on Arbitrum spiked to 250+ Gwei, which means the cost to transact on Arbitrum exceeded the cost to transact on Ethereum.
The Hop system had implemented extreme safeguards, allowing the protocol to work fluidly with any transaction up to 100 Gwei on Arbitrum, over 100x what is typical. When the Arbitrum gasPrice exceeded 100 Gwei, some L1 to Arbitrum transactions became stuck.
All of these transactions are now processed, and we will upgrade the protocol to work with any gasPrice that the Arbitrum chain is subject to.
The Hop protocol is trustless, and anyone can transact with it in any way they wish. However, the off-chain systems, such as bonders, have practical limitations, such as liquidity or a minimum fee they are willing to accept. While the Hop frontend respects these limitations, we found that one third party did not respect liquidity limits and allowed its users to transact on Hop even when there was no available liquidity. Another third party did not meet the minimum bonder fee for some transactions. The result of these issues is transfers that were not be bonded (at least immediately) by the bonder and the users who sent those transactions believing that the Hop system was not working as intended.
We have contacted all relevant and additional information will be provided in our documentation so that future integrators do not make this mistake.
The Alchemy RPC endpoints used by many Hop bonders have historically worked consistently. This week, Alchemy experienced downtime that resulted in various minor issues with the Hop bonders. Even though the Alchemy status page we monitor does not reflect any issues, their team has confirmed through multiple channels that they were experiencing intermittent issues. These issues resulted in slower than expected bond times.
The Hop bonders will implement additional redundancies to avoid intermittent RPC issues in the future.