Announcement of Solidity Optimizer and ABIEncoderV2 Bug
Through the Ethereum bug bounty initiative, we acquired a report regarding a defect found within the new experimental ABI encoder (known as ABIEncoderV2). Upon further examination, it became evident that the feature experiences several variations of a similar nature. The opening section of this announcement delves into the specifics of this issue. Though the new ABI encoder remains tagged as experimental, we still believe it warrants a significant announcement as it is currently deployed on the mainnet.
Furthermore, two minor issues in the optimizer have been recognized over the last fortnight, one of which was resolved with Solidity v0.5.6. Both were introduced with version 0.5.5. Refer to the latter section of this announcement for the specifics.
The 0.5.7 release encompasses the fixes for all issues detailed in this communication.
All the defects cited here should be readily observable in tests pertaining to the corresponding code paths, particularly when executed with all combinations of zero and nonzero values.
Acknowledgments to the Melonport team (Travis Jacobs & Jenna Zenk) and the Melon Council (Nick Munoz-McDonald, Martin Lundfall, Matt di Ferrante & Adam Kolar), who reported this through the Ethereum bug bounty initiative!
Who Needs to Be Aware
If you have deployed contracts utilizing the experimental ABI encoder V2, they may potentially be impacted. This implies that only contracts incorporating the following directive within the source code could be susceptible:
pragma experimental ABIEncoderV2;
Moreover, there are several conditions necessary for the bug to activate. Refer to technical details below for additional information.
To the best of our knowledge, there are approximately 2500 contracts operational on mainnet that utilize the experimental ABIEncoderV2. It remains uncertain how many of these may contain the flaw.
How to Determine if a Contract is Vulnerable
The defect manifests only when all of the following stipulations are satisfied:
- Storage data involving arrays or structs is transmitted directly to an external function call, to abi.encode or to event data without previous assignment to a local (memory) variable AND
- there exists an array containing elements smaller than 32 bytes or a struct that possesses elements sharing a storage slot or members of type bytesNN shorter than 32 bytes.
Additionally, under the following scenarios, your code is NOT impacted:
- if all your structs or arrays solely utilize uint256 or int256 types
- if you exclusively use integer types (which may be shorter) and only encode one array at a time
- if you only return such data and do not employ it in abi.encode, external calls, or event data.
Should you have a contract that fulfills these criteria, and wish to confirm its vulnerability, please contact us via security@ethereum.org.
Preventing Similar Flaws in the Future
To adopt a cautious approach regarding modifications, the experimental ABI encoder has been accessible only when explicitly activated, enabling users to engage with it and assess its functionality without excessive reliance prior to it being deemed stable.
We strive for excellence and have recently commenced ‘semantic’ fuzzing on certain sections of OSS-Fuzz (previously, we applied crash-fuzzing on the compiler, though this did not validate compiler accuracy).
For developers—detecting bugs within the Solidity compiler is challenging with tools such as vulnerability detectors, as instruments that function on source code or AST representations do not catch defects arising solely in the compiled bytecode.
The most effective method to safeguard against such flaws is to implement a rigorous set of end-to-end tests for your contracts (ensuring all code pathways are verified), as bugs in a compiler are likely to be apparent and result in erroneous data.
Potential Consequences
Naturally, any defect can yield significantly different consequences based on the program’s control flow; however, we anticipate that this is more inclined to cause malfunction rather than be exploitable.
When triggered, the bug under certain conditions will send corrupted parameters during method invocations to other contracts.
Timeline
2019-03-16:
- Report via bug bounty concerning corruption incurred when reading directly from arrays of booleans into the ABI encoder.
2019-03-16 to 2019-03-21:
- Investigation into the root cause and analysis of affected contracts. An unexpectedly high number of contracts must be addressed.compiled with the experimental encoder were discovered deployed on mainnet, several without authenticated source-code.
- Examination of the bug revealed additional methods to trigger the flaw, e.g. utilizing structs. Additionally, an array overflow bug was identified in the same process.
- A selection of contracts located on Github were examined, and none were observed to be impacted.
- A fix for the ABI encoder was implemented.
2019-03-20:
- Resolution to disclose information publicly.
- Rationale: It would not be practical to identify all compromised contracts and contact all creators promptly, and it would be beneficial to halt further spread of vulnerable contracts on mainnet.
2019-03-26:
- New compiler version released, version 0.5.7.
- This announcement was published.
Technical details
Background
The Contract ABI is a guideline on how data can be exchanged with contracts from the external environment (a Dapp) or during interactions among contracts. It accommodates various types of data, encompassing simple values like numbers, bytes, and strings, as well as more intricate data types, including arrays and structs.
Upon receiving input data, a contract must decode this information (this is accomplished by the “ABI decoder”) and before returning data or transmitting data to another contract, it must encode it (this is conducted by the “ABI encoder”). The Solidity compiler produces these two segments of code for every defined function within a contract (and for abi.encode and abi.decode). In the Solidity compiler, the subsystem responsible for generating the encoder and decoder is referred to as the “ABI encoder”.
In mid-2017, the Solidity team commenced work on a new implementation called “ABI encoder V2” aimed at producing more versatile, secure, efficient, and auditable code generation. This experimental code generator has been available to users since late 2017 with the launch of version 0.4.19, when explicitly enabled.
The flaw
The experimental ABI encoder does not appropriately manage non-integer values that are shorter than 32 bytes. This concern pertains to bytesNN types, bool, enum, and other types when they are included in an array or a struct and encoded directly from storage. This indicates that these storage references must be utilized directly within abi.encode(…), as parameters in external function calls or in event data without first assigning to a local variable. Employing return does not trigger the flaw. The types bytesNN and bool will lead to corrupted output, while enum could result in an invalid revert.
Moreover, arrays with elements shorter than 32 bytes may not be managed correctly even if the base type is an integer. Encoding such arrays in the manner outlined above can result in other data within the encoding being overwritten if the count of elements encoded is not a multiple of the number of elements that can fit in a single slot. If nothing follows the array in the encoding (noting that dynamically-sized arrays are always encoded after statically-sized arrays with statically-sized content), or if merely a single array is encoded, no additional data is overwritten.
Unrelated to the ABI encoder issue discussed previously, two defects have been discovered in the optimizer. Both were introduced with the release of version 0.5.5 (on 5th March). They are unlikely to appear in code produced by the compiler, unless inline assembly is utilized.
These two defects were recognized through the recent inclusion of Solidity in OSS-Fuzz – a security toolkit aimed at identifying discrepancies or problems across various projects. For Solidity, we have incorporated several different fuzzers that assess distinct aspects of the compiler.
- The optimizer transforms opcode sequences like ((x , where a and b are constants determined at compile-time, into (x while inadequately addressing overflow during the addition.
- The optimizer incorrectly manages the byte opcode if the constant 31 is utilized as the second argument. This may occur during index access on bytesNN types with a compile-time constant value (not index) of 31 or when employing the byte opcode in inline assembly.
This announcement was collaboratively authored by @axic, @chriseth, @holiman