<\/p>\n
Oh my, it\u2019s been quite some time… over 1.5 years since we introduced Geth v1.9.0<\/a>. We did manage 26 incremental releases during this span (approximately one every three weeks), but releasing a significant update holds a particular significance. The rush of excitement from unveiling new features, combined with the anxiety of potential mishaps. Still uncertain whether I love it or despise it. Regardless, Ethereum is progressing and we must extend our capabilities to keep pace with it.<\/p>\n <\/p>\n Without any more delay, please greet Geth v1.10.0<\/a> as part of the Ethereum community.<\/em><\/strong><\/p>\n <\/p>\n <\/p>\n Before we delve into the specifics of our latest release, it’s crucial to highlight that new features inherently bring new risks<\/strong>. To accommodate users and projects with varying risk thresholds, many of our significant features can be (for the time being) activated or deactivated individually. Whether you consume the entire content of this blog entry – or only glance over sections that captivate your interest – kindly take a moment to read the ‘Compatibility’ section at the conclusion of this document<\/strong>!<\/em><\/p>\n <\/p>\n Now that\u2019s addressed, let\u2019s plunge in and explore what Geth v1.10.0 has to offer!<\/p>\n <\/p>\n <\/p>\n Let\u2019s address the major issue first. Geth v1.10.0 does not include<\/strong> the Berlin hard-fork<\/em><\/strong><\/a> yet, as there were some last-minute concerns raised by the Solidity team regarding EIP-2315. Since v1.10.0 represents a major release, we prefer not to launch it too close to the fork. We will follow up shortly with v1.10.1 that will include the finalized list of EIPs and block numbers integrated.<\/p>\n <\/p>\n <\/p>\n We’ve been discussing snapshots<\/a> for such an extended period now; it feels surreal to finally witness their presence in a release. Without diving into excessive detail (see linked article), snapshots<\/em> are an accelerated data structure positioned atop the Ethereum state, enabling accessing<\/em> accounts and contract storage at a significantly quicker pace.<\/p>\n <\/p>\n To quantify this, the snapshot functionality reduces the expense of accessing an account from O(logN)<\/span> to O(1)<\/span>. This might not appear substantial at first glance, but in practical terms, on the mainnet with 140 million accounts, snapshots can save roughly 8 database lookups per account read. That’s nearly a tenfold reduction<\/strong> in disk lookups, guaranteed to remain constant regardless of state size.<\/p>\n <\/p>\n Wait, does this imply we can increase the gas limit by 10x?!– –><\/em> Regrettably, no. While snapshots indeed provide a tenfold improvement in read performance, EVM execution also entails writing<\/em> data, and these writes necessitate Merkle proof. The requirement for Merkle proof maintains the necessity for O(logN)<\/span> disk access during writes.<\/p>\n <\/p>\n Then, what\u2019s the advantage?!<\/em> While swift read access to accounts and contract storage isn’t adequate to raise the gas limit, it addresses several particularly challenging issues:<\/p>\n <\/p>\n <\/p>\n As with all features, it\u2019s a matter of trade-offs. Although snapshots offer immense advantages, which we believe are significant enough to make available to everyone, there are certain drawbacks:<\/p>\n <\/p>\n <\/p>\n If you have reservations about the snapshot capability, you can deactivate it<\/strong> in Geth 1.10.0 through –snapshot=false<\/span>, but please note that we will make it compulsory in the long run to ensure a fundamental network health.<\/em><\/p>\n <\/p>\n <\/p>\n If you believed snapshots took a considerable period to deliver, just wait until you learn about snap sync! We introduced the preliminary prototype of a novel synchronization algorithm back in October, 2017<\/a>… and then sat on that concept for over three years?! \ud83e\udd2f Before proceeding, a brief history recap.<\/p>\n <\/p>\n When Ethereum was launched, participants could select between two distinct methods for synchronizing the network: full sync<\/em> and fast sync<\/em> (excluding light clients from this conversation). Full sync<\/em> functioned by downloading the complete chain and processing all transactions; in contrast, fast sync<\/em> placed trust in a recently confirmed block and directly downloaded the state tied to it (after which it transitioned to block execution similar to full sync). Both operational modes led to the same final dataset, yet they favored different trade-offs:<\/p>\n <\/p>\n <\/p>\n Full sync<\/em> has stayed accessible for anyone who desired to expend the resources to authenticate Ethereum’s complete history, but for the majority of individuals, fast sync<\/em> proved to be more than satisfactory\u2122. There’s a paradox in computer science that states that once a system exceeds 50 times its intended usage, it will malfunction. The rationale is that regardless of how something functions, apply sufficient pressure and an unexpected bottleneck will emerge.<\/p>\n <\/p>\n In the context of fast sync<\/em>, the unexpected bottleneck was latency<\/em><\/strong>, arising from Ethereum’s data structure. Ethereum’s state trie is a Merkle tree, with leaves containing valuable data and each upper node being the hash of 16 children. Syncing from the tree’s root (the hash embedded within a block header) requires requesting each node one-by-one<\/strong>. With 675 million nodes to download, even by combining 384 requests at once, it necessitates 1.75 million round-trips. Assuming an overly generous 50ms RTT to 10 serving peers, fast sync<\/em> effectively amounts to waiting for over 150 minutes for data to arrive. However, network latency constitutes merely one-third of the issue.<\/p>\n <\/p>\n When a serving peer receives a query for trie nodes, it must retrieve them from disk<\/strong>. Ethereum’s Merkle trie does not alleviate this problem either. Since trie nodes are indexed by hash, there’s no significant method to store\/retrieve them in batches, with each necessitating its own database read. To exacerbate matters, LevelDB (utilized by Geth) arranges data across 7 levels, meaning a random read will generally involve multiple files. When everything is accounted for, a single network request for 384 nodes\u2014at 7 reads per query\u2014results in 2.7 thousand disk reads. With the fastest SATA SSDs achieving speeds of 100,000 IOPS, this results in an additional 37ms of latency. Assuming the same 10 serving peers as previously mentioned, fast sync<\/em> has just added an extra waiting time of 108 minutes<\/strong>. But serving latency represents only one-third of the issue.<\/p>\n <\/p>\n Requesting that many trie nodes individually necessitates uploading an equal number of hashes to remote peers for servicing. With 675 million nodes to download, that’s 675 million hashes to upload, or 675 * 32 bytes = 21GB. At a global average upload speed of 51Mbps (X Doubt), fast sync<\/em> has consequently added an extra 56 minutes of waiting time<\/strong>. Downloads are slightly more than double as large, so with global averages of 97Mbps, *fast sync* incurred a further 63 minutes<\/strong>. Bandwidth delays represent the final third of the challenge.<\/p>\n <\/p>\n Summing it all up, fast sync<\/em> ultimately spends an astonishing 6.3 hours not doing anything, merely waiting for data<\/strong>:<\/p>\n <\/p>\n “`html<\/p>\n <\/p>\n Snap sync<\/a><\/em> was developed to address all three of the outlined issues. The fundamental concept is relatively straightforward: rather than downloading the trie node-by-node, snap sync<\/em> retrieves contiguous segments of pertinent state data and reconstructs the Merkle trie locally:<\/p>\n <\/p>\n <\/p>\n While snap sync<\/em> is remarkably akin to Parity’s warp sync<\/em><\/a> – and indeed adopted numerous design concepts from it – there are notable enhancements over the latter:<\/p>\n <\/p>\n <\/p>\n To quantify snap sync<\/em> versus fast sync<\/em>, synchronizing the mainnet state (disregarding blocks and receipts, as those remain the same) against three serving peers, at block ~#11,177,000 yielded the following outcomes:<\/p>\n <\/p>\n \n <\/p>\n Please note that snap sync<\/em> is included, but not yet activated, in Geth v1.10.0. The rationale is that serving snap sync<\/em> demands nodes to possess the snapshot<\/em> acceleration infrastructure already constructed, which currently does not exist, as it is also included in v1.10.0. You may manually activate snap sync via –syncmode snap<\/span>, but be informed that we anticipate it not to locate<\/strong> to suitable peers until a few weeks following Berlin. We will enable it by default when we believe there are sufficient peers to depend on it.<\/p>\n <\/p>\n <\/p>\n We’re extremely proud of our accomplishments with Geth over recent years. Nevertheless, there’s always that one subject<\/em><\/strong>, which causes you to flinch when queried about. For Geth, that subject is state pruning<\/em>. But what exactly is pruning and why is it necessary?<\/p>\n <\/p>\n When processing a new block, a node takes the current state of the network as input and modifies it according to the transactions in the block, creating a new output state. The output state is primarily the same as the input, with only a few thousand items altered. Because we cannot simply overwrite the previous state (otherwise we would not be able to accommodate block reorganizations), both the old and new states ultimately reside on disk. (Sure, we’re a bit smarter and only write new diffs to disk if they persist and aren’t deleted in the following blocks, but let’s set aside that detail for the moment).<\/em><\/p>\n <\/p>\n Writing these new pieces of state data, block by block, to the database presents a challenge. They continue to accumulate. In theory, we could “just delete” state data that is old enough not to pose a risk of a reorg, but as it turns out, that’s quite a complex challenge. Given that state in Ethereum is stored in a tree structure – and that most blocks only modify a minor fraction of the state – these trees share substantial portions of the data with each other. We can easily determine if the root of an old trie is outdated and can be removed, but it is exceedingly expensive to ascertain whether a node deep within an old state is still referenced by any newer state.<\/p>\n <\/p>\n Over the years, we’ve implemented various pruning algorithms to eliminate leftovers (lost count, around ten), yet we’ve not found a solution that does not fail if sufficient data is presented. Consequently, people have become accustomed to Geth’s database starting lean following a fast sync<\/em>, and gradually expanding until you become frustrated and resync. This is aggravating to say the least, as re-downloading everything merely wastes bandwidth and contributes unnecessary downtime for the node.<\/p>\n <\/p>\n Geth v1.10.0 does not fully remedy the issue, but it represents a significant advancement toward a better user experience. If you have snapshots<\/em> activated and fully generated, Geth can utilize these as an acceleration framework to relatively swiftly identify which trie nodes should remain and which should be purged. Pruning trie nodes based on snapshots does carry the disadvantage that the chain may not progress during the pruning process. This necessitates stopping Geth, pruning its database, and subsequently restarting it.<\/p>\n <\/p>\n In terms of execution time, pruning requires a few hours (greatly depending on your disk speed and accumulated data), with one third dedicated to indexing recent trie nodes from snapshots, one third for deleting outdated trie nodes, and the final third for compacting the database to reclaim <\/p>\n Please be aware that pruning is a recent and potentially risky feature<\/strong>, a malfunction of which may lead to corrupted blocks. We are assured of its dependability, but if an issue occurs, it\u2019s possible there won’t be a way to recover the database. Our advice – at least until the feature undergoes rigorous testing – is to back up your database before pruning and to experiment with testnet nodes initially prior to fully engaging with mainnet.<\/em><\/p>\n <\/p>\n <\/p>\n Ethereum has been operational for a significant duration, and in its nearly 6 years of existence, Ethereum users have conducted over 1 billion<\/em><\/strong><\/a> transactions. That\u2019s quite a considerable figure.<\/p>\n <\/p>\n Node operators have always assumed they could retrieve any arbitrary transaction from the past using just its hash<\/em>. To be honest, it seems like an obvious task. However, when crunching the numbers, we find ourselves in an unexpected situation. To make transactions searchable, we must – at the very least – map the complete range of transaction hashes to their respective blocks. Despite all adjustments made to minimize storage, we still have to save 1 block number (4 bytes) linked to 1 hash (32 bytes).<\/p>\n <\/p>\n 36 bytes per transaction may not appear substantial, but when multiplied with 1 billion<\/em> transactions, it aggregates to a remarkable 36GB of storage<\/em><\/strong> needed to confirm that transaction 0xdeadbeef<\/span> is located in block N<\/span>. That’s a significant amount of data and a vast number of database entries to manage. Retaining 36GB seems a reasonable expense if one intends to search transactions from 6 years ago, but practically, most users do not desire this. For them, the additional disk usage and I\/O overhead are merely squandered resources. Furthermore, it\u2019s crucial to remember that transaction indices are not included in consensus and do not form part of the network protocol. They are purely a locally generated efficiency mechanism.<\/p>\n <\/p>\n Is it feasible to eliminate some – for us – superfluous information from our nodes? Absolutely! Geth v1.10.0 enables transaction unindexing by default and configures it to 2,350,000<\/span> blocks (approximately 1 year). The transaction unindexer will operate in the background, and with each new block that arrives, it guarantees that only the transactions from the most recent N<\/span> blocks are indexed, removing older ones. Should a user choose to access older transactions, they can restart Geth with a higher –txlookuplimit<\/span> value, and any blocks missing from the updated range will be reindexed (note, the activating event is still block import, so a wait for 1 new block is necessary).<\/p>\n <\/p>\n Given that around a third of Ethereum’s transaction volume<\/a> occurred in 2020, maintaining an entire year’s worth of transaction index will still exert a considerable burden on the database. The aim of transaction unindexing is not to discard an existing feature in the name of conserving space. The objective is to transition towards an operational mode where storage does not expand indefinitely with the history of the chain.<\/p>\n <\/p>\n If you wish to turn off<\/strong> transaction unindexing entirely, you can execute Geth with –txlookuplimit=0<\/span>, which reverts to the previous behavior of keeping the lookup map for every transaction since inception.<\/em><\/p>\n <\/p>\n <\/p>\n Ethereum organizes all its data within a Merkle Patricia trie. The values at the leaves represent the raw data being stored (e.g., content of storage slots, account details), while the path to the leaf acts as the key by which the data is stored. However, the keys are not<\/em><\/strong> the account addresses or storage addresses; rather they are the Keccak256<\/span> hashes of those. This approach helps balance the branch depths of the state tries. Utilizing hashes for keys is adequate as users of Ethereum reference only the original addresses, which can be hashed as needed.<\/p>\n <\/p>\n There is one scenario, however, where someone possesses a hash stored in the state trie and wishes to retrieve its preimage: debugging. While traversing EVM bytecode, a developer might desire to examine all the variables within the smart contract. The data is available, but without the preimages, it becomes challenging to identify which data corresponds to which Solidity variable.<\/p>\n <\/p>\n Initially, Geth implemented a partially effective solution. We stored in the database all preimages that resulted from user calls (e.g. sending a transaction), but not those emanating from EVM calls (e.g. accessing a storage slot). This was insufficient for Remix, so we enhanced our tracing API to include saving preimages for all SHA3 (Keccak256) operations. Although this resolved the debugging issue for Remix, it prompted inquiries regarding all the data unused by non-debugging nodes.<\/p>\n <\/p>\n The preimages are not particularly heavy. If you perform a full sync from inception – re-executing all transactions – you’ll only incur an additional 5GB load. Nonetheless, there is no justification for retaining that data for users who aren’t utilizing it, as it merely increases the load on LevelDB compactions. Therefore, Geth v1.10.0 disables<\/strong> preimage collection by default, yet there is no method to actively eliminate preimages that have already been stored.<\/p>\n <\/p>\n If you are utilizing your Geth instance to debug transactions, you can revert<\/strong> to the original behavior using –cache.preimages<\/span>. Please note, it is not possible to regenerate preimages after the fact<\/strong>. Should you run Geth with preimage collection turned off and have a change of heart, you will need to reimport the blocks.<\/em><\/p>\n <\/p>\n <\/p>\n The eth\/66<\/span><\/a> protocol is a relatively minor adjustment, yet carries numerous advantageous implications. In essence, the protocol introduces request and response IDs for all bidirectional packets. The intention behind these IDs is to facilitate a simpler alignment of responses to requests, particularly to effectively transmit a response back to the subsystem that initiated the original request.<\/p>\n <\/p>\n These IDsare not crucial, and indeed we\u2019ve been successfully navigating around their absence for the last six years. Regrettably, any code that needs to fetch data from the network becomes excessively complex if multiple subsystems can simultaneously request the same type of data. For instance, block headers can be fetched by the downloader<\/em> synchronizing the chain; it can be fetched by the fetcher<\/em> handling block announcements; and it can also be requested during fork challenges. Moreover, timeouts can result in delayed or unanticipated deliveries, prompting re-requests. In these scenarios, when a header packet<\/em> arrives, every subsystem checks the data to determine if it was intended for itself or for another subsystem. Consuming a response that was not designated for a specific subsystem can lead to failures elsewhere that require careful management. It becomes disorganized. Feasible, but disorganized.<\/p>\n <\/p>\n The significance of eth\/66<\/span> in the context of this blog post lies not in its resolution of a specific issue, but rather in the fact that it will be implemented before the Berlin hard-fork<\/strong>. As all nodes are expected to upgrade by the time of the fork, this paves the way for Geth to begin phasing out the outdated protocols following the fork. Only after eliminating all older protocols can we restructure Geth\u2019s internals to utilize request IDs. In line with our protocol deprecation schedule<\/a>, we intend to discontinue eth\/64<\/span> soon and eth65<\/span> by summer’s end.<\/p>\n <\/p>\n Some individuals might view Geth leveraging its influence to enforce protocol updates on other clients. We wish to stress that the typed transactions<\/a> feature from the Berlin hard-fork originally called for a new protocol version. Since only Geth fully implemented the suite of eth\/xy<\/span> protocols, other clients sought to “hack” it into older protocol versions to avoid focusing on networking at this time. The consensus was that Geth would backport typed transaction support into all its previous protocol code to give other developers time, but in return, would phase out the old versions within 6 months to prevent stagnation.<\/em><\/p>\n <\/p>\n <\/p>\nBeware of dragons<\/h2>\n
Berlin hard-fork<\/h3>\n
Snapshots<\/h3>\n
\n
\n
Snap sync<\/h3>\n
\n
\n
\n
\n
Offline pruning<\/h3>\n
\n“`freed space. By the conclusion of the procedure, your disk utilization should roughly equal what it would be after a complete synchronization. To trim your database, please execute geth snapshot prune-state<\/span>.<\/p>\nTransaction unindexing<\/h3>\n
Preimage discarding<\/h3>\n
ETH\/66 protocol<\/h3>\n
ChainID enforcement<\/h3>\n