Overcoming Fuzzing Roadblocks

1. What is a roadblock?

Some firmware can pose a challenge to the fuzzer preventing it from reaching a booted or interesting state. The most common roadblocks are:
  • Unmapped memory regions.
    • When you first create your project, a “dry run” is performed to ensure the validity of the memory configuration. A dry run works by executing the first few functions in the firmware. If the execution succeeds without crashing, your project is created.
    • However, it is common for the dry run to miss memory regions that are not accessed until much later in the firmware. These memory regions show up as “unmapped memory” crashes at runtime.
    • This type of roadblock is trivially fixable by adjusting the project configuration.
  • Checks of external flash.
    • Some firmware assume the presence of an external flash with a certain configuration on it. Since the contents of the external flash are not contained in the firmware image we are fuzzing, the check (i.e. CRC) will most likely fail, resulting in hangs.
  • Long, hard-coded sleeps.
    • Some firmware have hard-coded sleeps during initialization, which can cause hangs (a deliberate termination of execution by the fuzzer). Shorter sleeps the fuzzer can get away with, but sleeps long enough to cause a hang are problematic.
  • RF calibration sequences.
    • This can be very long and are generally irrelevant to the fuzzing campaign.

Roadblock Example: unmapped memory

A missing memory region takes the the form of an eventless crash:
Image without caption
To fix this, simply navigate to Project Settings and add a memory region that encompasses the missing memory. We recommend you align the memory region on a 0x1000 boundary and assign it a generous size as there may be other accesses to this region at surrounding addresses.
In this case, we identify the missing memory region as a flash at 0x10800000 and choose to add it as a “MMIO” memory. This has the effect of the fuzzer supplying any bytes read by the firmware from this memory region.
Image without caption
Image without caption
Image without caption

Roadblock Example: long-sleep in non-essential function

We notice progress into our firmware flatlines fairly early. The block frequency map shows an outlier PalSysInit hogging most of our compute:
Image without caption
We also notice a hang pointing to the same PalSysInit function:
Image without caption
We can also confirm that PalSysInit is at fault by graphing the top-most test case.
Test cases are sorted in decreasing order of block coverage meaning that the “deepest” test case will always be at the top of the Test Cases section.
Image without caption
We see the following flame graph:
Image without caption
Panning and zooming to the very end, we confirm. that PalSysInit is the last function that executes:
Image without caption
A fourth way we can confirm that this where is getting stuck is by dumping the coverage and importing it into our decompiler (See Visualizing CoverageVisualizing Coverage).
Inspecting the disassembly for this function, we immediately notice the hard-coded sleep causing execution to Hang:
Image without caption
We could patch out the loop or the entire function. In this case, because the function is neither interesting nor required for our fuzzing campaign, we choose to patch it out in the next step.

2. How to overcome a roadblock?

A common approach to overcoming roadblocks is binary patching. We make this very easy to do from the Images page of your project:
Image without caption
Ensure you have the right image selected, then select Add Patch:
Image without caption
The patches work at the instruction level, meaning we can patch any instruction in our program by replacing it with a custom instruction.
In this situation, we choose to patch the first instruction of PalSysInit by making it return 0.
Image without caption
Scroll down and click Save.
Image without caption
Start a new run.
Image without caption
The fuzzer is now no longer hanging on PalSysInit and is reaching a lot of new code:
Image without caption
The coverage plot also shows >1000 blocks being reached within the first few seconds, whereas we were only seeing ~300 prior to patching the roadblock:
Image without caption