Geth 1.9.0: Unveiling the Latest Features and Enhancements

After several months of quiet, we are thrilled to announce the v1.9.0 launch of Go Ethereum! While this version has been in development for considerably longer than we expected, we believe there will be some exciting features for all to appreciate!

Caution: We have made every effort to eliminate all issues, but as with all significant releases, we suggest everyone exercise additional caution when upgrading. The v1.9.0 release includes alterations to the database schema, making it impossible to revert once updated. We also suggest performing a fresh fast sync, as it can significantly decrease the database size.

Several of the features outlined here have been quietly integrated throughout the 1.8.x release series, yet we have considered them essential enough to explicitly showcase.

Performance

It’s fascinating to note that the “Performance” segment was located toward the conclusion of previous announcements, but over time it has evolved into one of the most desired enhancements.

Over the last six months, we have endeavored to analyze the various components crucial to block processing, striving to identify and enhance some of the bottlenecks. Among numerous enhancements, the most impactful ones included:

• The identification and enhancement of a quadratic CPU and disk IO complexity, stemming from the Go implementation of LevelDB. This issue caused Geth to become starved and halted, worsening exponentially as the database expanded. A huge thank you to Gary Rong for his tireless efforts, particularly as his contributions benefit the entire Go community.
• The examination and improvement of the account and storage trie access methods across blocks. This led to the stabilization of Geth’s memory consumption even during the import of the Shanghai DoS blocks and accelerated overall block processing through concurrent heuristic state prefetching. This effort was primarily executed by Péter Szilágyi.
• The assessment and enhancement of various EVM opcodes, with the goal of uncovering anomalies in both Geth’s EVM implementation and Ethereum’s protocol design overall. This work resulted in fixes within Geth as well as insights contributed to the Eth 1.x scaling conversations. A shoutout goes to Martin Holst Swende for initiating this initiative.
• The evaluation and optimization of our database schemas, aiming to eliminate redundant data and redesign indexes for lowered disk usage (sometimes at the cost of a slight CPU impact). Kudos for these efforts, which spanned 6-9 months, go to Alexey Akhunov, Gary Rong, Péter Szilágyi, and Matthew Halpern.
• The discovery of LevelDB compaction overhead during the state sync phase of fast sync. By temporarily allocating pruning caches to fast sync blooms, we have managed to streamline most data accesses in memory. This work was chiefly carried out by Péter Szilágyi.

[TL;DR] Fast sync

We executed a fast sync benchmark on two i3.2xlarge AWS EC2 instances (8 core, 61 GiB RAM, 1.9 TiB NVMe SSD) with –cache=4096 –maxpeers=50 (default settings in v1.9.0) on April 25th.

[TL;DR] Full sync

We conducted a full sync benchmark on two i3.2xlarge AWS EC2 instances (8 core, 61 GiB RAM, 1.9 TiB NVMe SSD) with –cache=4096 –maxpeers=50 –syncmode=full.

*Although the performance is comparable, we achieved that while reducing memory usage by approximately one-third and completely eliminating unnecessary memory spikes (Shanghai DoS). The reason for the elevated disk IO is due to utilizing less memory for caching, necessitating more aggressive pushes to disk.

[TL;DR] Archive sync

We performed an archive sync benchmark on two m5.2xlarge AWS EC2 instances (8 core, 32 GiB RAM, 3TiB EBS SSD) with –cache=4096 –syncmode=full –gcmode=archive.

* EBS volumes are considerably slower than physical SSDs connected to the VM. Improved performance can be attained on VMs with genuine SSDs or true physical hardware.

Freezer

Wouldn’t it be remarkable if we could conserve so much valuable storage on our costly and delicate SSDs to operate an Ethereum node, and instead transfer at least some of the information onto an economical and resilient HDD?

With the release of version v1.9.0, Geth divided its database into two segments (carried out by Péter Szilágyi, Martin Holst Swende and Gary Rong):

• Recent blocks, complete state, and acceleration structures are maintained in a swift key-value store (LevelDB) like before. This is intended to function atop an SSD, as disk IO performance is critical.
• Blocks and receipts older than a specific cutoff point (3 epochs) are transferred from LevelDB to a custom freezer database, supported by a set of append-only flat files. Since the node seldom requires access to these data, and only appends to them, an HDD should be more than adequate for this purpose.

A new fast sync at block 7.77M allocated 79GB of data to the freezer and 60GB to LevelDB.

Freezer fundamentals

By default, Geth will place your freezer within your chaindata directory, inside the ancient subfolder. The choice of using a sub-folder was to prevent disrupting any automated tools that might be relocating the database around or across instances. You can explicitly set the freezer in a different location using the –datadir.ancient CLI parameter.

Upon upgrading to v1.9.0 from a prior version, Geth will automatically begin transferring blocks and receipts from the LevelDB database into the freezer. If you didn’t specify –datadir.ancient at that moment, but wish to relocate it afterward, you will need to manually copy the existing ancient folder and then initiate Geth with –datadir.ancient pointed to the appropriate path.

Freezer hacks

Since the freezer (cold data) is saved separately from the state (hot data), an intriguing question arises regarding what occurs if one of the two databases is lost?

• If the freezer is removed (or an incorrect path is given), you essentially pull the rug from underneath Geth. The node would become inoperative, hence it categorically forbids this action at startup.
• Conversely, if the state database is the one that is deleted, Geth will reconstruct all its indices based on the frozen data, and then perform a swift sync on top to back-fill the absent state.

In essence, the freezer can function as a guerrilla state pruner to routinely eliminate accumulated debris. By deleting the state database, while retaining the freezer, the node will conduct a fast sync to retrieve the latest state, but will reuse all the pre-existing block and receipt data already downloaded beforehand.

You can initiate this via geth removedb (along with the –datadir and –datadir.ancient options if you utilized custom ones); instructing it to solely remove the state database while leaving the ancient database intact.

Please be aware that reindexing all the transactions from the ancient database can surpass one hour, and fast sync will commence only thereafter. This is likely to be transitioned into a background process in the near future.

GraphQL

Who doesn’t adore JSON-RPC? Me!

As its designation implies, JSON-RPC is a *Remote Procedure Call* framework. Its design intention is to facilitate invoking functions that perform some arbitrary processing on the remote side, subsequently returning the result of said processing. Naturally – as the protocol is generic – it allows for running data inquiries on top, but there are no standardized query semantics, hence individuals tend to develop their own.

In the absence of support for versatile queries, we end up squandering both computational and data transfer resources:

• RPC calls that yield substantial data (e.g. eth_getBlock) consume bandwidth if the user is only interested in a few fields (e.g. just the header, or even less, solely the miner’s address).
• RPC calls returning only a small amount of data (e.g. eth_getTransactionReceipt) deplete CPU resources if the user must repeat the call multiple times (e.g. retrieving all receipts one-by-one necessitates loading all of them from disk for each call).

In the context of Ethereum’s JSON-RPC API, the issues mentioned above are exacerbated by the mini-reorg characteristics of the blockchain, as executing multiple inquiries (e.g.eth_getBalance) must genuinely guarantee that they operate against the identical state and even the same node (e.g. load-balanced backends might experience minor synchronization delays, potentially serving differing content).

Indeed, we could devise a groundbreaking, highly efficient query mechanism that would allow us to obtain solely the information we require while reducing computational and data transfer strain… or we could opt to not-reinvent the wheel (once more) and instead utilize one that is already validated: GraphQL.

Querying with GraphQL

To begin, a massive thank you goes to Raúl Kripalani, Kris Shinn, Nick Johnson, Infura, and Pegasys, for spearheading both the GraphQL specification and its implementation, as well as to Guillaume Ballet for completing the final integrations!

Geth v1.9.0 brings inbuilt GraphQL query functionalities through the –graphql CLI option. Since GraphQL operates as a protocol on top of HTTP, the entire suite of sub-flags (restrictions, CORS, and virtual host guidelines) is applicable just like for HTTP RPC. But enough chatter, let’s dive into it!

For a quick demonstration, let’s seek out all the ENS domain registrations on the Görli testnet! Launch Geth v1.9.0 on Görli with GraphQL enabled (geth –goerli –graphql), allow it to sync (which should take roughly 1-2 minutes) and navigate your browser to the… gasp… integrated GraphQL explorer at http://localhost:8547!

To keep it straightforward, here’s a brief sample query that locates the ENS HashRegistered events and retrieves the address of the user who performed the registration together with the block number and timestamp it was included:

While the example may be somewhat forced and basic, it emphasizes that GraphQL empowers us to handle complex “join queries” that previously necessitated multiple RPC calls, returning much more data than was actually required.

As with all remarkable developer tools, Geth v1.9.0’s GraphQL explorer comes equipped with integrated code completion, field documentation, and real-time query execution! Go and query something extraordinary!

Hardware wallets

Geth has already supported certain hardware wallets previously, but with the v1.9.0 update, we’ve significantly enhanced that roster!

Ledger wallets

We have been backing the Ledger Nano S for several years, but Geth v1.9.0 now also incorporates native support for the Ledger Nano X (through USB)!

Moreover, v1.9.0 updates the default HD derivation path from the legacy one, originally promoted by Ledger, to the canonical one utilized by all Ethereum wallets (and currently by Ledger as well). No need to worry, Geth will identify all your previous accounts as well, just using the canonical path for new accounts! This development was executed by Péter Szilágyi.

If you haven’t utilized a Ledger through Geth until now, the process is as follows:

• Connect your Ledger Nano S or Ledger Nano X and unlock it using your PIN.
• Launch the Ethereum application on your Ledger (Geth will record Ethereum app offline).
• You can view all your accounts via personal.listWallets from the Geth console.
  
  • This will auto-derive any accounts that you’ve previously used + 1 additional empty one.
  • Alternatively, you can accomplish the same through RPC using personal_listWallets.

• Perform transactions using your preferred method and Geth will relay the signing request to the Ledger.

Linux users should note, you need to explicitly allow your user to access your Ledger wallet via udev rules!

Trezor wallets

For nearly two years, we have offered support for the Trezor One. Unfortunately, a firmware update (v1.7.0+) altered the USB protocol in an incompatible manner. While we recommend everyone to utilize the most recent software for security purposes, we also recognize the hesitance to frequently update firmware on a cold storage device.

Consequently, Geth v1.9.0 adopts the new WebUSB protocol compatible with updated Trezor One models, while still retaining support for the older USB HID protocol for devices that haven’t been updated. This compatibility was implemented by Guillaume Ballet and Péter Szilágyi (we have even released a new usb library for Go to facilitate support).

The Trezor One procedure is slightly more intricate due to its unique PIN entry:

“`html

• Connect your Trezor One; Geth will recognize it but will ask you to access it.
• Execute personal.openWallet(‘trezor://…’) using the device’s URL.
  
  • If you are unaware of the URL, you can verify it using personal_listWallets.
  • The console will continuously prompt for PIN input and password as required.
  • When calling via RPC, openWallet provides a thorough error if another call is necessary.

• You can enumerate all your accounts through personal.listWallets in the Geth console.
  
  • This will automatically derive any accounts you have previously utilized plus 1 new empty account.
  • Alternatively, you can accomplish this through RPC using personal_listWallets.

• Conduct transactions through your method of choice, and Geth will relay the signing request to the Trezor.

Along with enhanced support for the Trezor One, Geth v1.9.0 also brings direct support for the Trezor Model T. The workflow with the Model T is somewhat more straightforward, as the PIN entry occurs on the device:

• Plug in your Trezor Model T and unlock it using your PIN; Geth should identify it.
• You can list all your accounts through personal.listWallets from the Geth console.
  
  • This will automatically derive any accounts you have used before plus 1 new empty one.
  • You can alternatively achieve the same through RPC using personal_listWallets.

• Perform transactions through your preferred method, and Geth will send the signing request to the Trezor.

Linux users should note that you need to explicitly allow your user to access your Trezor wallet via udev rules!

Status keycards

Originally prototyped over a year ago, Geth v1.9.0 finally provides support for the Status keycard, a full HD hardware wallet built on Java SmartCards. Currently, the Status keycard can only be utilized through Geth via the PC/SC daemon (installation is required) and through USB (the +iD serves as a good USB smartcard reader). This effortwas heavily handled by Nick Johnson, initially integrated by Péter Szilágyi and completed by Guillaume Ballet (along with Andrea Franz and the rest of the Status team).

If you lack a pre-initialized Status keycard or are utilizing a developer card or have somehow rendered your existing card unusable (we’re developers, we must understand the outcomes in such cases), you can follow our technical guide for instructions on how to erase your keycard and reset it. Note that wiping it will result in the loss of your private key.

Clef

Wallets, wallets everywhere!

Upon Ethereum’s launch in 2015, there were no external tools available, which compelled client implementations to become comprehensive all-in-one solutions. This included everything from peer-to-peer networking to account management as well as contract and user interactions, all handled by the client. Although necessary, this was quite inefficient: accounts and networking do not securely integrate, and having all functionality within a single binary obstructs the creation of a modular ecosystem.

We have aimed to address this for a minimum of two years, and Geth v1.9.0 ultimately delivers the contribution of Martin Holst Swende (supported by many others): a standalone signer for the whole Ethereum ecosystem named Clef. While “standalone signer” may sound simple, Clef is the culmination of extensive architectural work to ensure it is secure, adaptable, and composable.

A brief release blog post alone cannot do justice to this project, but we will attempt to highlight the key features of Clef, the rationale behind the design, and the potential for new use cases.

Ecosystem composability

The primary motivation for the development of Clef was to detach account management from Geth (there’s no need to worry, the previous method will continue to function for the foreseeable future). This enables Geth to act as an “insecure” network gateway into Ethereum, potentially resolving many
“`many problems concerning unintentionally revealing accounts through RPC (and unlocking them, the lethal combination).

However, monopolizing all this effort for Geth would not be fair to us. Instead, we crafted Clef to be utilized by various programs, enabling you to have a single signer securely managing your keys, to which any number of applications (e.g. Geth, Parity, Trinity, Metamask, MyCrypto, Augur) can transmit signing requests!

To realize this, Clef offers a minimal external API (changelog) available via IPC (default) or HTTP. Any application that can access these endpoints (e.g. Geth via IPC, Metamask via HTTP) is capable of sending signing requests to Clef, which will then ask the user for manual approval. The API is intentionally minimal and utilizes JSON-RPC, making it easy to support in any project.

Our aspiration with Clef is not to solely be “The Geth Signer”, but rather to evolve into a standalone entity usable by any other project, whether distinct client implementations (Trinity), browser integrations (Metamask), service components (Raiden) or decentralized applications (Augur). If you wish to integrate Clef, contact us and we’ll HELP!

Pluggable interface

What constitutes the ideal user interface?

If you posed this question to me, I’d assert command line: straightforward, operates over SSH, and I can script it :D. However, I represent a minority here and even I frequently prefer a proper UI. So, Electron? Some consider it the greatest innovation since sliced bread, and numerous developers can create it; but it’s bulky and slow and JavaScript :P. What about Qt? It’s platform-agnostic, compact, and efficient, but only a few developers are versed in it and it has an unusual license. Android, GTK, iThingy?… Win32 😂?

The answer is all of them! The ideal UI depends on your intended use, and we don’t want to dictate that decision for you, but rather enable you to operate Clef in the manner that best integrates into your life:

• If you are constantly on the move, you might prefer an Android or iOS interface.
• If you own a secured remote server, you may favor CLI over SSH.
• If you possess a powerful laptop, the allure of Electron might be just what you need.
• If you have an offline signing machine, a Qt UI might be simple yet sufficient.
• If you are a financial institution, you may require a custom integration into your infrastructure.

We can’t realize all these options. But you can! We’ve designed Clef with sufficient adaptability to permit anyone to create a custom UI on top, without needing to alter Clef itself, or possess any knowledge of Go. The aim is to provide a foundation to the community so that designers and UI developers can excel in their craft, without needing to concern themselves with cryptography and related complexities.

In order to accomplish this, Clef provides an expanded internal API (changelog), solely through standard input/output. Any user interface is designed to initiate itself and internally start an instance of Clef, binding to its IO streams. The IO streams communicate using JSON-RPC, allowing the UI to send arbitrary trusted requests to Clef, and Clef will push notifications and confirmation prompts to the UI.

Clef itself comes equipped with a built-in CLI interface (otherwise it wouldn’t be particularly useful) and we’ve created a comprehensive Quickstart Guide to help you familiarize yourself with the overall features and concepts. Additionally, there are several proof-of-concept UIs that we’ve used to validate architectural choices, but to achieve a robust UI, we need the community’s help, as we lack the expertise ourselves!

Integrated 4bytes

By now, you likely grasp the general aim. We intend for Clef to be a reusable component of the puzzle. The challenge is to ensure it’s the right size! Too much functionality integrated (e.g. fixed UI) restricts potential uses. Too little (e.g. absence of hardware wallet) forces UI developers to reinvent the wheel. It’s a delicate balance between maximizing utility and security without sacrificing flexibility.

Thus, we concur that “fixed UI is undesirable, pluggable UI is favorable,” “absence of hardware wallet is bad, Ledger + Trezor + Keycard is advantageous.” What else do wallet implementations frequently reinvent? 4bytes!

In Ethereum, whenever a user interacts with a contract, they transmit a large blob of binary data, encoded in a very specific ABI format. This is necessary for the EVM to interpret it properly, and obviously, this is generated by some program (e.g. Augur). The problem arises when the user is then asked to confirm a transaction that resembles this:

The solution proposed by the Ethereum community was to compile a 4byte database, allowing users to look at the first 4 bytes of that data to infer what the remainder of the data represents, thus providing a meaningful overview of what they are about to approve (images above and below credited to Etherscan).

At present, all Ethereum wallet UIs are reinventing the wheel regarding 4bytes integration! The database is public, yet the integrations are bespoke. Clef includes the entire 4byte database embedded within itself,and each time a transaction occurs, it decodes the calldata behind the scenes. Clef not only transmits the decoded call to the interface but also incorporates warning alerts if the data fails to align with the method signature! Clef will manage Ethereum; you can concentrate on the interface!

Programmatic guidelines

Clef appears fantastic, what else could we desire? Well… drawing from the preceding segments, we’re capable of creating the ideal signer to approve any and every one of our transactions… manually. What occurs, though, if we wish to automate part of that (e.g. Clique signer, Raiden relay, Swarm exchange, etc.)? We could simply disregard it and permit the interface to resolve it… but then we’re back to square one, as all wrapping interfaces need to recreate the same systems, and many will likely do so insecurely.

Clef addresses this through an encrypted key-value repository and a clever rule engine! Instead of asking the user to confirm each request via a passphrase input, we can authorize Clef to sign on our behalf by saving our passphrase in its encrypted vault. This would enable passwordless signing but still necessitates manual approval!

As an additional step, we can also supply Clef with a JavaScript rule file, which will execute whenever a request is received and can decide to auto-approve, auto-deny, or send the request for manual confirmation. The JavaScript rules have access to the full request and can also save arbitrary information in a key-value store for durability. For instance, an academic demonstration rule file:

function ApproveSignData(req) {
  if (req.address.toLowerCase() == '0xd9c9cd5f6779558b6e0ed4e6acf6b1947e7fa1f3') {
    if (req.messages[0].value.indexOf('bazonk') >= 0) {
      return 'Approve';
    }
    return 'Reject';
  }
  // Otherwise goes to manual processing
}

The aim of these regulations is to enable you to establish arbitrary approval logic for whatever your requirements may be, whether that’s automated server-side transactions (Clique, Raiden, Swarm, Faucet) or minor client-side automation (approve X Wei / 24h to Augur). The programmable regulations guarantee that Clef stays true to its promise of composability, allowing anyone to construct their ideal integration on top.

For a thorough demonstration on how to configure automatic rules, please refer to the Clef Quickstart Guide.

Light clients

Light clients are challenging and they complicate everything more than it needs to be. The primary reason is more philosophical than technical: the best aspects in life are free, and the secondbest are inexpensive. In Ethereum client terminology, the “best” clients are those functioning with zero overhead (consider Metamask, Infura), while the next tier consists of the light clients.

The issue lies in the fact that relying on trusted servers contradicts the foundational principles of the project, but light clients tend to be excessively demanding for devices with limited resources (ethash drains your phone’s battery). Geth v1.9.0 introduces a novel mode for light clients, referred to as an ultra light client. This mode aims to find a balance on the security spectrum between a trusted server and a light server, substituting PoW verification with digital signatures from a majority of trusted servers.

With a sufficient number of signatures from independent entities, one could attain adequate security for non-critical DApps. That being said, ultra light client mode is not intended for the typical node, but rather for projects looking to integrate Geth into their own processes. This effort was led by Boris Petrov and Status.

Checkpoint oracle

Light clients are sneaky little shortcuts! Instead of downloading and authenticating each header from the genesis to the chain head, they utilize a hardcoded checkpoint (included within Geth) as an initial reference point. Naturally, this checkpoint holds all necessary information to cryptographically validate even older headers, ensuring that security is not compromised.

Nevertheless, as advantageous as the embedded checkpoints can be, they possess certain limitations:

• Since the checkpoints are hardcoded within our release binaries, older versions will always commence synchronization from an earlier block. This is acceptable for a few months, but eventually, it becomes tedious. You can certainly update Geth to retrieve a new checkpoint, but that will also introduce all our behavioral changes, which you may prefer to avoid for various reasons.
• Given that these checkpoints are incorporated into the code, you are out of options if you wish to support them in your own private network. You would need to either distribute a modified Geth, or configure the checkpoints through a configuration file, issuing a new one each time you update the checkpoint. It’s feasible, but not particularly practical for the long term.

This is where Gary Rong’s and Zsolt Felföldi’s contribution comes into play. Geth v1.9.0 incorporates support for an on-chain checkpoint oracle. Rather than depending on hard-coded checkpoints, light clients can connect with untrusted remote light servers (peer-to-peer, without any centralized nonsense) and request them to return an updated checkpoint stored within an on-chain smart contract. The most impressive aspect is that light clients can cryptographically validate that the returned data was signed by the required number of authorized signers!

But wait, how does a light client determine who is authorized to sign an on-chain checkpoint? For networks natively supported, Geth includes hard-coded checkpoint oracle addresses and lists of authorized signers (thus trusting the same developers who provide Geth itself). For private networks, the oracle parameters can be defined via a configuration file.

Although the old and new checkpoint systems appear similar (both necessitate hardcoded data in Geth or a configuration file), the new checkpoint oracle only needs to be configured once and can then be used indefinitely to release new checkpoints.

checkpoint-admin

Ethereum contracts are formidable, but engaging with them is not for the timid. Our checkpoint oracle contract is especially challenging, because a) it goes to great lengths to maintain security even amidst chain reorganizations; and b) it must support the sharing and verification of checkpoints for clients that are not yet synced.

As we do not anticipate anyone (ourselves included) to manually interact with the checkpoint oracle, Geth v1.9.0 also includes an administrative tool specifically for this contract, checkpoint-admin. Keep in mind, you only need to concern yourself with this if you intend to operate your own checkpoint oracle for your private network.

The checkpoint-admin can be utilized to inquire about the status of an already deployed contract (–rpc must target either a light node or a full node with –lightserv enabled, both exposing the les RCP API namespace):

$ checkpoint-admin --rpc ~/.ethereum/rinkeby/geth.ipc status
Oracle => 0xebe8eFA441B9302A0d7eaECc277c09d20D684540

Admin 1 => 0xD9C9Cd5f6779558b6e0eD4e6Acf6b1947E7fA1F3
Admin 2 => 0x78d1aD571A1A09D60D9BBf25894b44e4C8859595
Admin 3 => 0x286834935f4A8Cfb4FF4C77D5770C2775aE2b0E7
Admin 4=> 0xb86e2B0Ab5A4B1373e40c51A7C712c70Ba2f9f8E

Checkpoint (released at #4638418) 140 => 0x488c2eba92d31baeccfb6968fad5c21a3df93181b43b4cf253b4d572b64172ef

The admin command may also be utilized to install a new oracle, verify an upgraded checkpoint and distribute it within the network. Moreover, checkpoint-admin also functions in offline mode (without a live chain to provide information) and can additionally be supported by clef for signing instead of utilizing key files, but elaborating on all these is meant for another occasion.

Monitoring

This is possibly something that not many were aware of, but for quite some time, Geth has incorporated support for monitoring various subsystems and events. Naturally, the initial version was rather rudimentary 🤣 (text UI, RPC reporting), but it laid the foundation. We can achieve more than this!

Metrics collection

First things first, metrics must be collected before they can be exported and visualized. Geth can be directed to gather all its established metrics using the –metrics CLI flag. To make these readings available to the external environment, Geth v1.9.0 offers 3 independent systems: ExpVars, InfluxDB and Prometheus.

ExpVars represent a somewhat customized approach in the Go ecosystem for exposing public variables on an HTTP interface. Geth employs its debug pprof endpoint for this purpose. Operating Geth with –metrics –pprof will unveil the metrics in expvar format at http://127.0.0.1:6060/debug/metrics. Keep in mind, you should never expose the pprof HTTP endpoint to the public internet as it may trigger resource-heavy operations!

ExpVars are relatively well-supported within the Go ecosystem, yet they do not represent the industry standard. A comparable mechanism, but with a more standardized format, is the Prometheus endpoint. Executing Geth with –metrics –pprof will also reveal this format at http://127.0.0.1:6060/debug/metrics/prometheus. Again, please refrain from exposing the pprof HTTP endpoint to the public internet! Acknowledgements to Maxim Krasilnikov for adding this functionality.

While ExpVars and Prometheus are pull based monitoring systems (remote servers pull the data from Geth), we also accommodate push based monitoring through InfluxDB (Geth sends the data to remote servers). This capability requires several CLI flags to be configured to establish the database connection (server, database, username, password, and Geth instance tag). Please refer to the METRICS AND STATS OPTIONS section of geth help for specifics (–metrics.influxdb and subflags). This initiative was carried out by Anton Evangelatov.

Metrics visualization

Visualizing metrics can prove to be somewhat challenging as you need to choose a charting tool/service and invest considerable efforts into configuring all the hosts, charts, and dashboards.

We ourselves utilize Datadog internally and have been consistently refining our monitoring dashboards ever since we set them up 1.5 years ago. If you are currently employing Datadog or contemplating doing so, here’s a sneak peek of what you can create based on the metrics offered by Geth (this is the dashboard through which we evaluate PRs against the master branch):

Unfortunately, Datadog does not permit sharing dashboards with external parties (since those depend on how the monitored machines have been configured). Hence, we cannot readily share the aforementioned work with you, but we did export a JSON dump of it in case anyone wishes to follow in our path!

Of course, we also recognize that a paid service like Datadog might not always be ideal, especially if you are just beginning and do not have funds to expend on monitoring. An excellent no-cost monitoring tool is Grafana!

Maxim Krasilnikov created a Grafana dashboard some time ago against an earlier developmentversion of Geth. We integrated his remarkable work with the metrics we became fond of in Datadog, resulting in a considerable amount of additional effort on Geth. The ultimate outcome, however, is impressive (anticipate more updates in upcoming releases):

You can easily recreate the aforementioned graphs using my version of Maxim Krasilnikov’s project by executing docker-compose up at the repository’s root and accessing http://localhost:3000 with the admin/admin credentials. Alternatively, you can explore my test snapshot on Raintank or import this dashboard into your own Grafana instance

Puppeth explorer

Once upon a time in a distant place, Puppeth saw the first illumination (specifically, in Mexico, nearly two years ago). If you haven’t heard about it, “Puppeth is a utility to assist you in constructing a new Ethereum network all the way to the genesis block, bootnodes, signers, ethstats, faucet, wallet, explorer, and dashboard”. Initially developed to support the deployment and maintenance of the Rinkeby testnet, it has since been utilized by various groups for other private networks as well.

Puppeth is not intended for managing a high-value production network, but it has demonstrated enough resilience to keep Rinkeby relevant for over two years! If you’re interested in a more in-depth exploration of Puppeth, here’s my presentation from a while back. In this article, however, let’s concentrate on what’s new!

Puppeth is incredible! It has enabled you since day one to deploy a fully functioning Ethereum network across multiple machines, enriched with a stats page for maintenance, along with a faucet and a dashboard to facilitate user onboarding. Nonetheless, Puppeth was deficient in a reliable block explorer, as the only options back then were Etherscan and Etherchain, both of which were closed-source. We did implement a basic solution, but it was somewhat unremarkable…

With the introduction of Blockscout late last year, everything transformed! The Ethereum community finally acquired a genuine, open-source block explorer, thanks to the POA Network team. While Blockscout still has some ground to cover compared to the established players, it is undeniably an incredibly useful tool. Therefore, Geth v1.9.0 includes a preliminary integration of Blockscout into Puppeth, addressing a significant gap in our private network deployment tool!

This initiative was led by Gary Rong, but a significant recognition goes to Ayrat Badykov for his assistance in resolving issues, inquiries, and more.

Please be aware that we foresee the initial integration to be rough (e.g., due to a “bug” in Blockscout, the Puppeth explorer must fully sync a Geth archive node prior to launching the explorer web interface). Feel free to test it, run it, and report any concerns, but don’t be taken aback if it goes offline at 3 AM!

Discovery protocol

Now, here lies another remnant of legacy infrastructure! Besides a minuscule modification, Ethereum’s discovery protocol has been specified, implemented, and set in stone for practically eternity. For those curious about the discovery protocol, it’s the apparatus through which a new node can identify other Ethereum nodes on the internet and join them into a global peer-to-peer network.

So… what’s the issue then? Hasn’t it functioned adequately until now? If it ain’t broke, don’t fix it and all that?

Well, Ethereum’s original discovery protocol was designed for a different era, a time when there was solely one chain, when private networks were nonexistent, and when every node in the network was an archive node. We have outgrown these simplistic beliefs, which, while a success story, also presents new challenges:

• The Ethereum ecosystem today comprises numerous public, private, and test networks. Although the Ethereum mainnet consists of a multitude of machines, other networks tend to be much smaller (e.g., Görli testnet). The discovery protocol does not make any distinctions among these networks, leading to a continual trial and error process of locating unknown peers, connecting to them, and then realizing they are on a different network.
• The same original Ethereum network can end up dividing itself into several disconnected fragments, where participants might want to join one part or another. Ethereum Classic serves as one of the key examples here, but similar challenges arise each time a network upgrade (hard fork) occurs, and some nodes update late. Without details about the network’s rules, we revert to trial and error connectivity, which is computationally very costly.
• Even if all nodes belong to the same network and adhere to the same fork rules, there exists the potential for challenging peering: if there is connectivity asymmetry, where certain nodes depend on services provided by a limited subset of machines (i.e., light clients vs. light servers).

For the long term, we’re striving towards a completely fresh version of the discovery protocol. Geth’s light clients have been using a PoC version of this for a long time, but implementing such a significant alteration for the entire Ethereum network demands substantial time and careful attention. This initiative is primarily led by Felix Lange and Frank Szendzielarz in partnership with Andrei Maiboroda from Aleth/C++, Antoine Toulme with Java, Age Manning from Lighthouse/Rust, and Tomasz Stańczak from Nethermind/C#.

Ethereum Node Records

The preceding content was an extensive discussion about something we haven’t released! However, what we have released is the Ethereum Node Record (ENR) extension of the new discovery protocol, which can indeed operate on top of the previous protocol as well! An ENR is a small, 300-byte arbitrary key-value data structure that nodes can promote and inquire about through discovery. While the new discovery protocol will offer advanced methods for sharing these within the network, the old protocol can also directly query them.

The immediate advantage is that nodes can promote extensive metadata about themselves without an expensive TCP + crypto handshake, enabling potential peers to filter out undesirable connections without ever initiating them! All credit goes to Felix Lange for his relentless contributions in this area!

Alright, alright, we understand it’s advanced. But what is it actually good for, in simple terms?

Geth v1.9.0 introduces two extensions to the discovery protocol through ENRs:

• The current discovery protocol can only manage one type of IP address (IPv4 or IPv6). Since the majority of the internet still relies on IPv4, that’s what peers advertise and exchange with one another. Even though IPv6 is feasible, in reality you cannot find such peers. Felix Lange’s contribution on advertising both IPv4 and IPv6 addresses via ENRs enables peers to discover and maintain Kademlia routing tables for both IP types. Although further integration work remains, we aspire to elevate IPv6 to a first-class citizen within Ethereum.
• Locating a Rinkeby node today works similarly to connecting to random websites to determine whether they are Google or not. The discovery protocol retains a mix of internet addresses that use the Ethereum protocol, but otherwise has no information about which chain or forks they are associated with. The only method to find out is to connect and observe, which is a costly game of chance. Péter Szilágyi proposed an extension to ENR that allows nodes to convey their chain configuration via the discovery protocol, creating a 0-RTT mechanism to reject clearly unwanted peers.

However, the most impressive aspect of ENR – and its already implemented features – is that anyone can develop a UDP crawler to index Ethereum nodes without needing to establish a connection (most nodes won’t have available slots; and crawlers that do connect via TCP consume valuable resources). Accessing all nodes, their IPs/ports, capabilities, and chain configurations facilitates the development of a brand new discovery protocol based on DNS, enabling nodes with blocked UDP ports (e.g., via Tor) to join the network as well!

Bootnodes

We have experienced a varying quantity of bootnodes of differing quality, managed by various individuals since the Frontier launch. Although it functioned reasonably, from a devops perspective it left much to be desired, especially regarding monitoring and maintenance. In conjunction with our Geth v1.9.0 release, we have chosen to establish a new set of bootnodes that are managed through Terraform and Ansible, and monitored with Datadog and Papertrail. We have also enabled them to support light clients, hopefully enhancing the reliability of the light protocol along the way. Huge thanks to Rafael Matias for his efforts on this!

Our new collection of bootnodes is:

• enode://d860a01f9722d78051619d1e2351aba3f43f943f6f00718d1b9baa4101932a1f5011f16bb2b1bb35db20d6fe28fa0bf09636d26a87d31de9ec6203eeedb1f666@18.138.108.67:30303 (Singapore, AWS)
• enode://22a8232c3abc76a16ae9d6c3b164f98775fe226f0917b0ca871128a74a8e9630b458460865bab457221f1d448dd9791d24c4e5d88786180ac185df813a68d4de@3.209.45.79:30303 (Virginia, AWS)
• enode://ca6de62fce278f96aea6ec5a2daadb877e51651247cb96ee310a318def462913b653963c155a0ef6c7d50048bba6e6cea881130857413d9f50a621546b590758@34.255.23.113:30303 (Ireland, AWS)
• enode://279944d8dcd428dffaa7436f25ca0ca43ae19e7bcf94a8fb7d1641651f92d121e972ac2e8f381414b80cc8e5555811c2ec6e1a99bb009b3f53c4c69923e11bd8@35.158.244.151:30303 (Frankfurt, AWS)
• enode://8499da03c47d637b20eee24eec3c356c9a2e6148d6fe25ca195c7949ab8ec2c03e3556126b0d7ed644675e78c4318b08691b7b57de10e5f0d40d05b09238fa0a@52.187.207.27:30303 (Australia, Azure)
• enode://103858bdb88756c71f15e9b5e09b56dc1be52f0a5021d46301dbbfb7e130029cc9d0d6f73f693bc29b665770fff7da4d34f3c6379fe12721b5d7a0bcb5ca1fc1@191.234.162.198:30303 (Brazil, Azure)
• enode://715171f50508aba88aecd1250af392a45a330af91d7b90701c436b618c86aaa1589c9184561907bebbb56439b8f8787bc01f49a7c77276c58c1b09822d75e8e8@52.231.165.108:30303 (South Korea, Azure)
• enode://5d6d7cd20d6da4bb83a1d28cadb5d409b64edf314c0335df658c1a54e32c7c4a7ab7823d57c39b6a757556e68ff1df17c748b698544a55cb488b52479a92b60f@104.42.217.25:30303 (West US, Azure)

Our previous bootnodes will continue to operate for the time being, but will gradually be phased out in the coming months.

Other changes

In addition to all the remarkable features listed above, there are a few more significant changes that, while not large enough to constitute their own section, are nevertheless crucial enough to specifically mention.

RPC APIs:

• The origin verification on WebSocket connections (–wsorigins) is enforced solely when the Origin header is included. This facilitates establishing a connection to Geth from non-browser environments like Node.js, while restricting access to the RPC endpoint from random sites.
• You can establish the maximum gas for eth_call through the –rpc.gascap command line parameter. This is advantageous if you’re exposing the JSON-RPC endpoint to the public Internet.
• All RPC method calls are now recorded at the debug level. Methods that fail are logged as warning, ensuring you can always identify when something isn’t functioning correctly.
• Geth v1.9.0 introduces support for the eth_chainId RPC method as outlined in EIP 695.

Networking:

• The default number of peers has been increased to 50, replacing the previous 25. This modification enhances synchronization performance.
• A new CLI utility (cmd/devp2p) has been introduced to the source repository for debugging P2P network problems. Although this tool isn’t included in the alltools archive yet, it is quite helpful in diagnosing peer discovery issues.
• The P2P server now declines connections from IPs that attempt to connect too frequently.

Miscellaneous:

• Substantial efforts have been made to enhance the abigen tool. Go bindings now accommodate Solidity struct and function pointer arguments. Additionally, the Java generator has seen improvements. The mobile framework is now capable of creating deploy transactions.
• Considerable portions of the go-ethereum repository can now be built without CGO. Huge thanks to Jeremy Schlatter for this contribution.

Compatibility

While Go Ethereum v1.9.0 introduces a remarkable array of enhancements, there are also a few backward-incompatible modifications. This segment outlines everything that has been altered or deprecated in the release:

• Unlocking accounts via open HTTP, WebSocket, or GraphQL ports has been disallowed for security reasons. Advanced users can revert to the previous behavior using the –allow-insecure-unlock CLI flag at their own discretion.
• The older Ubuntu Docker images and the previous (monolithic) Alpine Docker images have been removed as they were deemed deprecated over a year ago. If you didn’t configure your cluster back in 2016, you likely used the slim Alpine images and remain secure.
• The original geth monitor CLI command was removed along with its associated debug_metrics RPC API endpoint. Individuals who depend on monitoring should utilize the ExpVar, InfluxDB, or Prometheus metrics reporting along with Datadog or Grafana.
• The geth bug CLI command has been removed, considered an unnecessary feature. If you encounter a bug, you can easily file an issue on our GitHub tracker and complete the template manually.
• The les/1 and eth/62 protocols have been eliminated. les/1 was only supported by Geth, and everyone on Constantinople already uses les/2. Furthermore, eth/62 was deprecated even before Frontier, but remained for cpp-ethereum.
• Google+ authentication has been removed from the Puppeth faucet due to Google discontinuing its social network in early April 2019.
• The Ledger HD wallet derivation path has been updated from the original legacy path to the canonical ecosystem path. Accounts from the previous paths can still be detected.
• The default cache limit is determined dynamically according to the network and synchronization modes. Mainnet full nodes default to 4GB, test networks and private networks to 1GB, while light clients default to 128MB. Explicit –cache settings are obviously respected.
• The PoW computation in Whisper v6 was incompatible with Parity as it did not fully comply with the specification. This has been fixed, but it also indicates that Whisped v6, included with Geth v1.9.0, is not compatible with earlier versions.
• The –lightserv and –lightpeers flags have been renamed to –light.serve and –light.maxpeers, respectively. The original versions are now deprecated but will function for the next year or so.
• The default data directory on Windows is now sourced from the LocalAppData environment variable. However, the previous location in $HOME/AppData/Roaming remains recognized. This location adjustment works better with Cygwin and configurations using remote user accounts.
• The JSON-RPC server has undergone a rewrite and now enables bi-directional communication. You can define method handlers on the client side using the rpc.Client.RegisterName method. Though we performed extensive testing, there might be compatibility challenges with the new server. Please report any RPC issues that arise.

Epilogue

We take great pride in this release! It took significantly longer than anticipated, but we aimed to bundle all the major changes to reduce unexpected challenges (upgrade issues); and to finalize the APIs of new features to avoid future disruptions. We hope you, too, will discover something valuable among our delivered ~370 changes.

As with all our earlier releases, you can locate the:

And as a final note before wrapping up (better safe than sorry):

Warning: We’ve done our utmost to eliminate all bugs, but as with any major release, we advise everyone to proceed with caution while upgrading. The v1.9.0 release includes changes to the database schema, making it impossible to revert once updated. We also recommend a fresh fast sync, as it can significantly decrease the database size.