This blog entry reveals a risk against the Ethereum network that existed from the Merge until the Dencun hard fork.
Context
Before the merge, various message size caps for RPC communication were established to safeguard clients from denial-of-service (DOS) assaults. These specifications, implemented on messages received through HTTP endpoints, were transferred to the engine API, which is essential for linking Execution and Consensus Layer clients during block formation. The involvement of the engine API in block generation made it feasible for blocks to be created that exceeded the RPC size limits of some clients while still complying with acceptable limits for others.
Should an assailant craft a message that surpasses the size cap of the client with the lowest configuration, whilst remaining within gas limit constraints, and then waits for a block to be generated, this could lead to a scenario where certain clients view the block as legitimate, whereas others dismiss it, producing an HTTP error code “413: Content Too Large.”
Consequences
A malefactor capable of formulating these messages could compel most nodes (=geth) to reject blocks that a minority would accept. These rejected blocks would be excluded, and the proposer would forfeit potential rewards.
Initially, we believed that generating these blocks was only feasible through builders or a modified client version. Geth imposes a built-in limit of 128KB for transactions, meaning a large transaction like the one discussed would not appear in any geth node’s transaction pool. However, it was still possible to activate the limit by having a client with a higher threshold propose the block, while the CL requested validation of this larger proposed block.
We suggested a temporary solution by reducing the RPC limit across all clients to the lowest setting (5MB). This would invalidate the block, and an assailant would find their capacity to instigate chaos in the network significantly diminished, as the majority of nodes would reject their blocks.
However, on February 7th, we found that it was indeed possible to assemble a block that reached the 5MB threshold using a collection of transactions, each below the 128KB limit and not surpassing 30 million gas.
This presents a greater problem since we realized an attacker could generate a multitude of high-paying transactions and propagate them through the network. Due to outbidding everyone else in the mempool, all nodes (including geth nodes) would incorporate the attack transactions in their block, thereby creating a block that would be rejected by the majority of the network, leading to numerous forks (all considered valid by the minority nodes), resulting in continuous reorganizations of the chain.
Later on February 7th, we concluded that an increase in RPC limits across all clients would be the more prudent alternative.
Chronology
- 2024-02-06 13:00: Toni (EF), Pari (EF) and Justin (Besu) attempt to submit a specifically crafted transaction to the network. This transaction contributes to blocks of up to 2.7 MB when optimally compressed.
- 2024-02-06 13:25: Pari encounters errors from his local Geth node despite the transaction’s valid status.
- 2024-02-06 15:14: Justin successfully integrates the transaction into a block and transmits it via the Besu client.
- 2024-02-06 20:46: Sam (EF) informs Pari (special thanks to mysticryuujin on X), Toni, and Alex about certain Sepolia nodes experiencing difficulties.
- 2024-02-06 21:05: The team verifies with Marius from Geth and affirms the existence of the bug.
- 2024-02-06 21:10: The group convenes to troubleshoot the issue.
- 2024-02-07 23:40: We resolved to limit all clients’ RPC request sizes to 5MB.
- 2024-02-07 6:40: We uncovered a potentially larger problem and recognized that the attack could be executed with transactions under 128KB in size.
- 2024-02-07 10:00: All clients decided to enhance the RPC request size limit.
- 2024-02-07 21:00: The fix was integrated into geth.
- 2024-02-09: Geth was launched.
While Geth was the sole client impacted by this flaw, other clients also updated their defaults to safeguard themselves against this attack, despite increasing gas limits.
The client teams have indicated that the following updates reflect the secure RPC limits:
Geth: v1.13.12
Nethermind: v1.25.4
Besu: 24.1.2
Erigon: v2.58.0
Reth: v0.1.0-alpha.18