Fuzz Testing
Forge supports property based testing.
Property-based testing is a way of testing general behaviors as opposed to isolated scenarios.
Let’s examine what that means by writing a unit test, finding the general property we are testing for, and converting it to a property-based test instead:
pragma solidity 0.8.10;
import {Test} from "forge-std/Test.sol";
contract Safe {
receive() external payable {}
function withdraw() external {
payable(msg.sender).transfer(address(this).balance);
}
}
contract SafeTest is Test {
Safe safe;
// Needed so the test contract itself can receive ether
// when withdrawing
receive() external payable {}
function setUp() public {
safe = new Safe();
}
function test_Withdraw() public {
payable(address(safe)).transfer(1 ether);
uint256 preBalance = address(this).balance;
safe.withdraw();
uint256 postBalance = address(this).balance;
assertEq(preBalance + 1 ether, postBalance);
}
}
Running the test, we see it passes:
$ forge test
Compiling 24 files with Solc 0.8.10
Solc 0.8.10 finished in 1.11s
Compiler run successful!
Ran 1 test for test/Safe.t.sol:SafeTest
[PASS] test_Withdraw() (gas: 19463)
Suite result: ok. 1 passed; 0 failed; 0 skipped; finished in 382.51µs (42.34µs CPU time)
Ran 1 test suite in 5.88ms (382.51µs CPU time): 1 tests passed, 0 failed, 0 skipped (1 total tests)
This unit test does test that we can withdraw ether from our safe. However, who is to say that it works for all amounts, not just 1 ether?
The general property here is: given a safe balance, when we withdraw, we should get whatever is in the safe.
Forge will run any test that takes at least one parameter as a property-based test, so let’s rewrite:
contract SafeTest is Test {
// ...
function testFuzz_Withdraw(uint256 amount) public {
payable(address(safe)).transfer(amount);
uint256 preBalance = address(this).balance;
safe.withdraw();
uint256 postBalance = address(this).balance;
assertEq(preBalance + amount, postBalance);
}
}
If we run the test now, we can see that Forge runs the property-based test, but it fails for high values of amount
:
$ forge test
Compiling 1 files with Solc 0.8.10
Solc 0.8.10 finished in 1.07s
Compiler run successful!
Ran 1 test for test/Safe.t.sol:SafeTest
[FAIL: EvmError: Revert; counterexample: calldata=0x29facca70000000000000000002c6b590ed97e6bf47d0c87c2dc70a0f5c0d7c71d86e005 args=[4254526348652830286417092903881653923087016841932365829 [4.254e54]]] testFuzz_Withdraw(uint256) (runs: 3, μ: 19531, ~: 19531)
Suite result: FAILED. 0 passed; 1 failed; 0 skipped; finished in 851.15µs (498.96µs CPU time)
Ran 1 test suite in 5.75ms (851.15µs CPU time): 0 tests passed, 1 failed, 0 skipped (1 total tests)
The default amount of ether that the test contract is given is 2**96 wei
(as in DappTools), so we have to restrict the type of amount to uint96
to make sure we don’t try to send more than we have:
function testFuzz_Withdraw(uint96 amount) public {
And now it passes:
$ forge test
Compiling 1 files with Solc 0.8.10
Solc 0.8.10 finished in 1.06s
Compiler run successful!
Ran 1 test for test/Safe.t.sol:SafeTest
[PASS] testFuzz_Withdraw(uint96) (runs: 257, μ: 19266, ~: 19631)
Suite result: ok. 1 passed; 0 failed; 0 skipped; finished in 4.75ms (4.41ms CPU time)
Ran 1 test suite in 6.12ms (4.75ms CPU time): 1 tests passed, 0 failed, 0 skipped (1 total tests)
You may want to exclude certain cases using the assume
cheatcode. In those cases, fuzzer will discard the inputs and start a new fuzz run:
function testFuzz_Withdraw(uint96 amount) public {
vm.assume(amount > 0.1 ether);
// snip
}
There are different ways to run property-based tests, notably parametric testing and fuzzing. Forge only supports fuzzing.
Interpreting results
You might have noticed that fuzz tests are summarized a bit differently compared to unit tests:
- “runs” refers to the amount of scenarios the fuzzer tested. By default, the fuzzer will generate 256 scenarios, but this and other test execution parameters can be setup by the user. Fuzzer configuration details are provided
here
. - “μ” (Greek letter mu) is the mean gas used across all fuzz runs
- “~” (tilde) is the median gas used across all fuzz runs
Configuring fuzz test execution
Fuzz tests execution is governed by parameters that can be controlled by users via Forge configuration primitives. Configs can be applied globally or on a per-test basis. For details on this topic please refer to
📚 Global config
and 📚 In-line config
.
Fuzz test fixtures
Fuzz test fixtures can be defined when you want to make sure a certain set of values is used as inputs for fuzzed parameters. These fixtures can be declared in tests as:
- storage arrays prefixed with
fixture
and followed by param name to be fuzzed. For example, fixtures to be used when fuzzing parameteramount
of typeuint32
can be defined as
uint32[] public fixtureAmount = [1, 5, 555];
- functions named with
fixture
prefix, followed by param name to be fuzzed. Function should return an (fixed size or dynamic) array of values to be used for fuzzing. For example, fixtures to be used when fuzzing parameter namedowner
of typeaddress
can be defined in a function with signature
function fixtureOwner() public returns (address[] memory)
If the type of value provided as a fixture is not the same type as the named parameter to be fuzzed then it is rejected and an error is raised.
An example where fixture could be used is to reproduce the DSChief
vulnerability. Consider the 2 functions
function etch(address yay) public returns (bytes32 slate) {
bytes32 hash = keccak256(abi.encodePacked(yay));
slates[hash] = yay;
return hash;
}
function voteSlate(bytes32 slate) public {
uint weight = deposits[msg.sender];
subWeight(weight, votes[msg.sender]);
votes[msg.sender] = slate;
addWeight(weight, votes[msg.sender]);
}
where the vulnerability can be reproduced by calling voteSlate
before etch
, with slate
value being a hash of yay
address.
To make sure fuzzer includes in the same run a slate
value derived from a yay
address, following fixtures can be defined:
address[] public fixtureYay = [
makeAddr("yay1"),
makeAddr("yay2"),
makeAddr("yay3")
];
bytes32[] public fixtureSlate = [
keccak256(abi.encodePacked(makeAddr("yay1"))),
keccak256(abi.encodePacked(makeAddr("yay2"))),
keccak256(abi.encodePacked(makeAddr("yay3")))
];
Following image shows how fuzzer generates values with and without fixtures being declared: