After far too long in development, we’re excited to finally unveil version 1.8.0 of the Go Ethereum client: Iceberg! This release addresses numerous challenges faced by the community and delivers several remarkable new functionalities, totaling up to ~170 adjustments!
Please be aware, this release incorporates some breaking changes that could impact certain advanced users! If you’re operating a production setup, ensure you review the “Breaking changes” section at the conclusion of this blog post!
Client synchronization
A tremendous amount of effort was invested in this release that isn’t readily apparent; instead, these are behind-the-scenes modifications aimed at making everyone’s experience just a little bit better. We aimed to tackle many of the concerns our users reported regarding synchronization and block processing. We’re not entirely where we want to be yet, but the experience with v1.8.0 should far surpass all prior releases.
Reliable light client
Geth v1.7.3 – launched shortly after Devcon3 – marked the initial release of version 2 of the light client protocol. It was intended to be a significant enhancement over version 1, finally allowing log filtering from Ethereum contracts. Unfortunately, it ended up breaking the light client.
The disruption was considerable, with several experimental protocols (discovery v5, light client v2) clashing with one another. Geth v1.7.3 attempted to advertise both les/1 and les/2, which created conflicts in discovery, disrupting both; les/2 servers would crash while handling some light client requests; and discovery v5, operating behind an unmentioned port, wasn’t beneficial either.
Geth v1.8.0 aims to pick up the pieces and restore les/2 to what it was intended to be in v1.7.3. We’ve eliminated support for les/1 in the discovery phase, thus resolving issues around finding peers while we smooth out any remaining problems. Light servers have been enhanced to be more resilient with existing connections and are also refined to distinctly separate eth and les peers, preventing starvation on the server side. Versions 4 and 5 of the discovery protocols are now running on the same port, and will subsequently better avoid complications with firewalls or NAT traversals.
With all these adjustments, the light client in v1.8.0 should locate servers within a few seconds of startup, and synchronizing the mainnet should wrap up within a minute. Given that light clients depend on generous nodes for service, we encourage anyone running non-sensitive full nodes with available resources to contemplate enabling the light server to assist users with less capable hardware.
Reliable fast sync
For an extended period, we’ve been receiving reports from users facing fast sync stalls accompanied by a “stalling peer” error message, or experiencing crashes with an “out of memory” error while trying to synchronize on average machines. These issues have increasingly surfaced as the Ethereum mainnet expanded, yet they have remained elusive to us due to their infrequent nature.
The significant internal alterations enabled us to consistently reproduce and resolve these issues. The stall was a highly uncommon race condition that happened when state sync restarted; the remedy for which is quite humorous considering it took us a year to uncover. The memory issue was also tackled by strictly limiting the amount of memory sync could utilize.
The end result of these optimizations is the revival of stable fast sync. From one angle, there are no longer any stalls, so you won’t need to constantly oversee the sync progress. From another aspect, memory usage remains consistent, eliminating the necessity for machines with extravagant RAM.
The chart above illustrates memory usage during mainnet fast sync on two m4.2xlarge Amazon instance types (purple = Geth 1.8, blue = Geth 1.7). At the time of writing, fast sync concludes in approximately 3 hours on these instance types. However, the exponential growth of Ethereum has led to a state trie consisting of around 85 million nodes, the import of which can take even half a day on user laptops (with an SSD). We hope 1.9 will address this concern.
Initial state pruning
Ethereum structures its state into a colossal trie data architecture. At the lowest level – in the leaves – are the accounts – and atop these accounts is a 16th order Merkle trie that cryptographically ensures forgery resistance. Each block has its own of these massive tries, with the latest block comprising about 85 million nodes. While many of these nodes are shared across consecutive blocks, every new block contributes a few thousand new nodes to the trie.
If we wanted to ascertain our balance from years past, we would need to maintain every iteration of this Merkle trie since the genesis block, potentially accumulating almost 1TB of data at present. In reality, almost no one cares about historical data – as long as it can be recalibrated – but rather solely about the recent state of the network. Fast sync facilitates “quickly” reaching the recent state, but indiscriminately stacking blocks will continuously consume more disk space.
The crucial characteristic of the Merkle tries to keep in mind is that while every new block introduces thousands of new nodes, thousands of old ones simultaneously become obsolete. If we could effortlessly eliminate these outdated nodes, disk growth would be significantly curtailed. However, once data is stored on disk, discarding it becomes highly expensive.
Geth v1.8.0 makes an initial attempt at resolving this issue by implementing an in-memory cache for storing the recent trie nodes. As long as the nodes are in memory, they can be easily reference counted and garbage collected. Instead of continuously writing each trie node to disk, we hold onto them for as long as feasible, hoping that an upcoming block will render them obsolete, thus saving us a database write.
By default, Geth v1.8.0 utilizes 25% of the user’s cache capacity (–cache) for trie caching, and it will record to disk either when the memory limit is surpassed or when the block processing duration since the last record exceeds 5 minutes. This does not entirely resolve database expansion at this point, but observing the disk statistics between v1.8 (purple) and v1.7 (blue) over the span of a single week, pruning significantly impacts the outcome.
Transaction tracing
For a considerable time, Geth has provided tracing transactions by outputting the executed opcodes. These outputs can be extremely useful for detecting consensus discrepancies among clients, however, they aren’t particularly user-friendly. Although it’s feasible to process these traces afterward, accumulating so much data only to discard most of it is inefficient.
Custom tracing scripts
The Geth v1.5 series introduced a new method for tracing transactions, permitting users to compose custom JavaScript scripts that operate within the node during tracing. Instead of generating predefined traces, users could collect any data they found relevant, without the necessity to export all extraneous information. Despite internal use, this feature never truly reached a sufficient and robust level for widespread adoption.
However, Geth v1.8.0 completely overhauls the custom tracing functionality. Initially, we’ve substituted the ottovm previously utilized for running tracers with duktape, yielding a fivefold speed enhancement. We no longer need the state a transaction depends upon to trace it; the tracer can reconstruct any missing elements from past states (incurring the cost of re-executing the blocks in memory). Additionally, when tracing several transactions simultaneously (i.e., an entire block), they are executed in parallel, which reduces tracing time based on the available CPU cores.
In conclusion, crafting a custom tracer is intricate, requiring a significant time investment even for experienced Ethereum developers. Therefore, we have opted to provide a selection of tracers straight out of the box for users, which they can utilize and possibly enhance. We eagerly anticipate any community contributions to these, or the introduction of entirely new ones!
- The callTracer serves as a comprehensive transaction tracer that extracts and reports all internal calls executed by a transaction, including any pertinent information.
- The prestateTracer provides sufficient information to establish a local execution of the transaction from a custom assembled genesis block.
- The 4byteTracer looks for 4-byte identifiers and accumulates them for post-processing. It gathers method identifiers along with the size of the provided data, allowing for a signature to be mapped against the data size.
For instance, executing the callTracer against the same transaction referenced above yields a much more user-friendly output: debug.traceTransaction(“0xhash”, {tracer: “callTracer”}).
Streaming chain tracers
Tracing a complete block of transactions is considerably more efficient than tracing each transaction individually, as it negates the need to generate the pre-state for each one separately. This is even more pronounced when producing the initial state requires the re-execution of multiple past blocks (pruned state). Nonetheless, the same challenge arises when tracing several blocks as well: if the pre-state was pruned, regenerating state just to repeat this for the next block is wasteful.
To facilitate tracing several consecutive blocks with minimal additional workload, Geth v1.8.0 unveils a new API endpoint capable of tracing chain segments. This endpoint can utilize the computed states between blocks without the need to re-run transactions repeatedly. Furthermore, individual blocks are traced concurrently, thereby shortening total tracing time as more CPU cores are engaged.
Tracing a transaction or block requires a relatively brief duration. In contrast, tracing a chain segment can extend indefinitely, depending on the length of the chain and the transactions it comprises. It would be highly impractical to wait for all transactions to complete tracing before commencing the return of those already processed. This excludes chain tracing as a straightforward RPC method. Instead, Geth v1.8.0 adopts chain tracing through a subscription (IPC/WebSocket), enabling the user to initiate a background tracing operation while Geth streams the results until all transactions are processed:
$ nc -U /work/temp/rinkeby/geth.ipc ```html {"id": 1, "method": "debug_subscribe", "params": ["traceChain", "0x0", "0xfff", {"tracer": "callTracer"}]}
{"jsonrpc":"2.0","id":1,"result":"0xe1deecc4b399e5fd2b2a8abbbc4624e2"} {"jsonrpc":"2.0","method":"debug_subscription","params":{"subscription":"0xe1deecc4b399e5fd2b2a8abbbc4624e2","result":{"block":"0x37","hash":"0xdb16f0d4465f2fd79f10ba539b169404a3e026db1be082e7fd6071b4c5f37db7","traces":[{"from":"0x31b98d14007bdee637298086988a0bbd31184523","gas":"0x0","gasUsed":"0x0","input":"0x","output":"0x","time":"1.077µs","to":"0x2ed530faddb7349c1efdbf4410db2de835a004e4","type":"CALL","value":"0xde0b6b3a7640000"}]}}} {"jsonrpc" ``````html :"2.0","method":"debug_subscription","params":{"subscription":"0xe1deecc4b399e5fd2b2a8abbbc4624e2","result":{"block":"0xf43","hash":"0xacb74aa08838896ad60319bce6e07c92edb2f5253080eb3883549ed8f57ea679","traces":[{"from":"0x31b98d14007bdee637298086988a0bbd31184523","gas":"0x0","gasUsed":"0x0","input":"0x","output":"0x","time":"1.568µs","to":"0xbedcf417ff2752d996d2ade98b97a6f0bef4beb9","type":"CALL","value":"0xde0b6b3a7640000"}]}}} {"jsonrpc":"2.0","method":"debug_subscription","params":{"subscription":"0xe1deecc4b399e5fd2b2a8abbbc4624e2","result":{"block":"0xf47","hash":"0xea841221179e37ca9cc23424b64201d8805df327c3296a513e9f1fe6faa5ffb3","traces":[{"from":"0xbedcf417ff2752d996d2ade98b97a6f0bef4beb9","gas":"0x4687a0","gasUsed":"0x12e0d","input":"0x6060604052341561000c57fe5b5b6101828061001c6000396000f30060606040526000357c0100000000000000000000000000000000000000000000000000000000900463ffffffff168063230925601461003b575bfe5b341561004357fe5b61008360048080356000191690602001909190803560ff1690602001909190803560001916906020019091908035600019169060200190919050506100c5565b604051808273ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff16815260200191505060405180910390f35b6000600185858585604051806000526020016040526000604051602001526040518085600019166000191681526020018460ff1660ff1681526020018360001916600019168152602001826000191660001916815260200194505050505060206040516020810390808403906000866161da5a03f1151561014257fe5b50506020604051035190505b9493505050505600a165627a7a7230582054abc8e7b2d8ea0972823aa9f0df23ecb80ca0b58be9f31b7348d411aaf585be0029","output":"0x60606040526000357c0100000000000000000000000000000000000000000000000000000000900463ffffffff168063230925601461003b575bfe5b341561004357fe5b61008360048080356000191690602001909190803560ff1690602001909190803560001916906020019091908035600019169060200190919050506100c5565b604051808273ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff16815260200191505060405180910390f35b6000600185858585604051806000526020016040526000604051602001526040518085600019166000191681526020018460ff1660ff1681526020018360001916600019168152602001826000191660001916815260200194505050505060206040516020810390808403906000866161da5a03f1151561014257fe5b50506020604051035190505b9493505050505600a165627a7a7230582054abc8e7b2d8ea0972823aa9f0df23ecb80ca0b58be9f31b7348d411aaf585be0029","time":"658.529µs","to":"0x5481c0fe170641bd2e0ff7f04161871829c1902d","type":"CREATE","value":"0x0"}]}}} {"jsonrpc":"2.0","method":"debug_subscription","params":{"subscription":"0xe1deecc4b399e5fd2b2a8abbbc4624e2","result":{"block":"0xfff","hash":"0x254ccbc40eeeb183d8da11cf4908529f45d813ef8eefd0fbf8a024317561ac6b"}}}
Native events
For approximately one and a half years we have been offering the capability to generate Go wrappers for Ethereum contracts. These are incredibly beneficial as they facilitate the execution and interaction with contracts directly using Go. The primary advantage is that our abigen utility produces static types for nearly everything, guaranteeing that code engaging with contracts is safe at compile-time. This is particularly useful in development, as any alteration in the contract ABI promptly leads to compilation errors, thereby reducing the likelihood of runtime failures.
However, abigen has always lacked the feature for filtering Ethereum contract logs: it was not possible to filter past events, nor to subscribe to upcoming events. With Geth v1.8.0, it finally introduces event filtering for native dapps! From this point forward, Go wrappers created by abigen will provide two additional methods for every event, namely FilterMyEvent and WatchMyEvent. Maintaining abigen‘s stringent type safety, both event filters and the resultant logs are strongly and statically typed. Developers can exclusively work with Go types, while all other processes are handled seamlessly behind the scenes.
A great illustration is filtering for Akasha posts on the Rinkeby testing network. The publishing event is specified as event Publish(address indexed author, bytes32 indexed entryId). Filtering for posts created by addresses 0xAlice or 0xBob would appear as follows:
contract.FilterPublish(nil, []common.Address{"0xAlice", ```"0xBob"}, nil)
Devcon3 puppeth
As many of you might be aware, the Rinkeby testing network is largely overseen via puppeth. For those who are unfamiliar, puppeth serves as “a utility to assist you in establishing a new Ethereum network, from the genesis block, bootnodes, signers, ethstats server, cryptocurrency faucet, wallet browsers, block explorer, dashboard, and more; eliminating the complexity typically associated with manually configuring each service independently”.
Puppeth proved to be an essential resource for us in maintaining the Rinkeby network since its inception 10 months ago. It was adequate for its intended use – as an internal tool – even though it had several imperfections. Our aim was to enhance this tool to benefit not only Rinkeby but also all other developer networks, so for Devcon3, we’ve significantly refined it. It has become more user-friendly, added support for configuring Parity, C++ Ethereum, pyethapp, and Harmony (operating on ethash consensus), and can also deploy online wallets and basic block explorers.
It feels like an eternity since Devcon3 and the integration of Puppeth into the master branch, but v1.8.0 finally delivers the subsequent version of puppeth for those who have been waiting. Go ahead and launch your own Ethereum network!
Breaking modifications
- Discovery v4 and v5 have been unified to utilize the same UDP port (default is 30303). If you are manually managing peers and using the light client, ensure your v1.8.0 clients are directed to port 30303 instead of 30304 as before.
- Trie pruning is now active on all –syncmode variations (including –syncmode=full). If you are operating an archive node and wish to preserve all historical data, you should turn off pruning by using –gcmode=archive.
- Only the most recent 128 tries are stored in memory; the majority of tries are garbage collected. Should you be running a block explorer or another service that depends on transaction tracing without an archive node (–gcmode=archive), you must trace within this limitation! Alternatively, specify the reexec: 12345 tracer option to regenerate historical state; ideally, switch to chain tracing, which distributes overhead across all traced blocks.
- Native events rely on alterations to internal go-ethereum types within the generated code. If you are utilizing wrappers generated before v1.8.0, you’ll need to regenerate them to ensure compatibility with the new codebase.
- The HTTP/WS RPC endpoint has been updated with DNS rebind protection. If you are operating an RPC endpoint identified by name rather than IP, use –rpcvhosts=your.domain to continue receiving remote requests.
While we regard Geth 1.8.0 as our most exceptional release to date, we encourage users to proceed with caution during the upgrade and keep a close watch thereafter as it includes significant alterations. We also want to highlight that Geth 1.8.0 brings about state pruning, which is incompatible with earlier versions of Geth (previous versions will refuse the pruned database).
Similar to past major releases, our advice for production users is to sync from scratch, while keeping the old database securely backed up until you verify that the new release functions correctly for all your scenarios.
For a comprehensive overview of the modifications, please refer to the Geth 1.8.0 release milestone.
Binaries and mobile libraries can be accessed on our download page.
Acknowledgment
As a final note regarding this release, we wish to express our gratitude to Ming Chan for her incredible dedication as the former EF Executive Director! Among her numerous responsibilities, she always took the time to proofread our release announcements, correcting any lost-in-translation mistakes; while also ensuring that our less technical readers clearly understood. Thank you for all that you have done for the Foundation and the community!
¹ “Because the previous version was un-sync-able” ~Nick Johnson