Table of Contents
Overview
Fnrec is a tool that collects a trace of functions called before a crash or hang. It can be used to determine what led to the issue and which function was called last.
How tracing works
Fnrec uses the gcc -finstrument-functions compiler flag to trace each function call and record it into a memory buffer. The memory buffer has a magic number at the start to identify it as valid and not just random bytes in memory. As gPXE runs, it records each function call into the memory buffer. When the crash occurs and the system is reset, we assume that memory is not cleared or overwritten (this is true on many machines but you may have to experiment with pressing Ctrl+Alt+Del versus the reset switch versus other alternatives).
When the system loads gPXE again, it checks for the memory buffer and if the magic number is still there, the contents of the memory buffer are printed out. This output is all the recorded function calls, which allow developers to see what code paths were taken and what the last function called was.
Using fnrec
Build with FNREC=1
Build from source and add FNREC=1 to the make line:
make FNREC=1 [...]
Reproduce the crash or hang
Boot gPXE and reach the point where it hangs. Now reset your system and boot into gPXE again.
You can try Ctrl+Alt+Del, the reset button, or any other method to reboot the system. The aim is to keep the recorded trace in memory, most machines will not clear memory on reboot. Even a quick power off/power on cycle has been known to work.
Collect function trace
This time gPXE should print out lots of numbers before the banner and prompt. Copy these numbers into a text file (serial console is useful for capturing this output). If you can't use serial or easily copy the numbers, please take at least the last two lines of numbers.
Post-process the trace
The raw trace data contains memory addresses. There is a tool to post-process them and convert addresses to symbol names:
$ util/fnrec.sh bin/sky2.hd.tmp fnrec.dat
In this example the sky2.usb image was used and the fnrec.dat file contained the raw trace data. An ELF file containing symbols is build as .hd.tmp for .usb images. Other formats have similar .tmp files.
