==== Overview ====
Fnrec is a tool that collects a
trace of functions called before a crash or hang.  It can be used to
determine what led to the issue and which function was called last.

==== How tracing works ====
Fnrec uses the gcc
-finstrument-functions compiler flag to trace each function call
and record it into a memory buffer.  The memory buffer has a magic
number at the start to identify it as valid and not just random bytes
in memory.  As gPXE runs, it records each function call into the
memory buffer.  When the crash occurs and the system is reset, we
assume that memory is not cleared or overwritten (this is true on many
machines but you may have to experiment with pressing Ctrl+Alt+Del
versus the reset switch versus other alternatives).

When the system loads gPXE again, it checks for the memory buffer and
if the magic number is still there, the contents of the memory buffer
are printed out.  This output is all the recorded function calls,
which allow developers to see what code paths were taken and what the
last function called was.

==== Using fnrec ====

== Build with FNREC=1 ==

Build from source and add ''FNREC=1'' to the make line:

  make FNREC=1 [...]

== Reproduce the crash or hang ==

Boot gPXE and reach the point where it hangs.  Now reset your
system and boot into gPXE again.

You can try Ctrl+Alt+Del, the reset button, or any other method to reboot the system.  The aim is to keep the recorded trace in memory, most machines will not clear memory on reboot.  Even a quick power off/power on cycle has been known to work.

== Collect function trace ==

This time gPXE should print out lots of numbers before the banner
and prompt.  Copy these numbers into a text file (serial console is
useful for capturing this output).  If you can't use serial or easily
copy the numbers, please take at least the last two lines of numbers.

== Post-process the trace ==

The raw trace data contains memory addresses.  There is a tool to post-process them and convert addresses to symbol names:

  $ util/fnrec.sh bin/sky2.hd.tmp fnrec.dat

In this example the ''sky2.usb'' image was used and the ''fnrec.dat'' file contained the raw trace data.  An ELF file containing symbols is build as ''.hd.tmp'' for ''.usb'' images.  Other formats have similar ''.tmp'' files.