GDB remote debugging

You can use the GNU Project Debugger (GDB) to debug gPXE. You either need two computers or virtualization software (e.g. QEMU). One host runs gPXE while the other runs GDB.

gPXE supports debugging via serial port or over the network. For serial, you need a null modem serial cable. For network, you need the machine to be connected to a network with UDP port 43770 traffic allowed.

Create the following file in src/config-local.h:

/* @BEGIN general.h
 *
 * Local config.h changes that do not get committed to git.
 *
 */

#define  GDBSERIAL               /* Remote GDB debugging over serial */
#define  GDBUDP                  /* Remote GDB debugging over UDP
                                  * (both may be set) */

/* @END general.h */

This file overrides src/config.h but is not under version control. Therefore you will never accidentally commit a patch that enables GDB debugging. Sometimes you also need to remove/define GDBSERIAL in config/general.h to get serial debugging to work.

#define GDBSERIAL               /* Remote GDB debugging over serial */

Now build gPXE:

$ cd src
$ make

Notice that the bin directory contains not only the gPXE image you built, but also a file with the same name but ending with .tmp. This file is an ELF with debugging symbols that we can load in GDB.

You can break into the debugger from the gPXE shell with the gdbstub command:

Usage:
  gdbstub <transport> [<options>...]

Start remote debugging using one of the following transports:
  serial           use serial port (if compiled in)
  udp <interface>  use UDP over network interface (if compiled in)

For example:

gdbstub serial     # debug via serial port
gdbstub udp net0   # debug over network interface 'net0'

gPXE will be waiting for a connection from a remote GDB.

First we load debugging symbols:

$ cd gpxe/src
$ gdb
(gdb) file bin/gpxe.hd.tmp   # or equivalent for your gPXE image filename

The .tmp file must correspond to your gPXE image. For example, gpxe.dsk → gpxe.dsk.tmp, gpxe.pxe → gpxe.pxe.tmp, and gpxe.usb → gpxe.hd.tmp.

Next you should set up the serial port in GDB if you are debugging via serial. The set remotebaud N command is used to set the serial port baud rate to N. See Remote Configuration in the GDB Manual.

Now connect to gPXE, which is already waiting since we entered the gdbstub command in the gPXE shell:

target remote /dev/ttyS0            # for serial debugging
target remote udp:192.168.1.2:43770 # for network debugging on gPXE
                                    # host 192.168.1.2
target remote 192.168.1.2:43770     # if you are forwarding over TCP
                                    # using QEMU -serial

You should soon see the normal GDB prompt and be able to start debugging. If you have trouble, try the set debug remote 1 GDB command to see the traffic between GDB and gPXE.

To run gPXE in QEMU with the serial port redirected to a TCP port:

$ qemu -serial tcp::43770,server bin/gpxe.usb

You can break into the debugger without using the gPXE shell. The following breaks into the debugger for serial debugging:

#include <gpxe/gdbserial.h>
#include <gpxe/gdbstub.h>
[...]
gdbstub_start ( gdbserial_configure() );

Here is the same thing for network debugging:

#include <gpxe/gdbudp.h>
#include <gpxe/gdbstub.h>
[...]
struct sockaddr_in sa = {
    /* FILL IN LISTEN ADDR/PORT OR ZERO FOR DEFAULTS */
};
gdbstub_start ( gdbudp_configure ( "net0", &sa ) );

Interrupt (CTRL + C) in GDB does not work. Implementing this is not possible since gPXE does not use interrupts. This means that you cannot hit CTRL + C if gPXE is in an infinite loop.

Debugging works for 32-bit protected mode. There is no support for 16-bit real-mode.

Debugging can safely be used after initialise() and startup() have been called in main(). If you need to use it during initialise() or startup(), take special care and modify the code if necessary (e.g. by hardcoding calls to initialize the GDB stub before the thing you are debugging).

Backtraces are sometimes wrong. I believe this is because the optimization settings gPXE is built with confuse GDB.

When debugging over the network, gPXE will drop network packets while you are in the debugger. This means you should use serial debugging if you need to minimize side-effects on gPXE. You may also try using a dedicated NIC for debugging so that traffic to the primary NIC is not dropped while you are in the debugger. Note that traffic may still be dropped if you stay in the debugger so long that the receive buffers fill up (i.e. gPXE is not running while you are in the debugger).

• Breakpoint (b) marks code where execution should stop and break into the debugger. To break on an instruction's virtual memory address, use b *ADDR.

(gdb) b http_rx_response
Breakpoint 1 at 0x25de: file net/tcp/http.c, line 157.

• Continue (c) resumes execution until the next breakpoint.

(gdb) c
Continuing.

Breakpoint 1, http_rx_response (http=0x595e4, response=0x594b4 "HTTP/1.0 200 OK") at net/tcp/http.c:157
157     static int http_rx_response ( struct http_request *http, char *response ) {
(gdb)

• Step (s), next (n), finish (fin), and step instruction (si) resume execution over a limited region of code. Step executes until the next line of source code, it steps inside function calls. Next executes until the next line of source code in the function, thereby stepping over function calls. Finish executes the remainder of the current function. Step instruction executes the current machine instruction.

• Backtrace (bt) shows the call stack. Some stack frames may not be detected correctly by GDB due to the compiler optimisations used when building gPXE.

(gdb) bt
#0  http_rx_response (http=0x595e4, response=0x594b4 "HTTP/1.0 200 OK") at net/tcp/http.c:157
#1  0x0000282a in http_socket_deliver_iob (socket=<value optimized out>, iobuf=0x5a3ac, meta=<value optimized out>)
    at net/tcp/http.c:363
#2  0x00034118 in xfer_deliver_iob_meta (xfer=<value optimized out>, iobuf=0x5a3ac, meta=0xa4aa0) at core/xfer.c:143
#3  0x00035247 in tcp_rx (iobuf=0x5a3ac, st_src=0xbe470, st_dest=<value optimized out>, pshdr_csum=<value optimized out>)
    at net/tcp.c:768
#4  0x00002510 in tcpip_rx (iobuf=0x5a3ac, tcpip_proto=<value optimized out>, st_src=0xbe470, st_dest=0xbe450,
    pshdr_csum=<value optimized out>) at net/tcpip.c:54
#5  0x000be470 in ?? ()

• Local variables (info locals) can be displayed though their values may be unavailable due to compiler optimizations.

(gdb) info locals
spc = <value optimized out>
rc = <value optimized out>

• CPU registers (i r) shows many but not all of the registers.

• Print (p) shows the value of program variables and arbitrary expressions in C-like syntax (see GDB Manual).

(gdb) p response                    
$1 = 0x594b4 "HTTP/1.0 200 OK"      
(gdb) p *http                       
$2 = {refcnt = {refcnt = 2, free = 0x28e8 <http_free>}, xfer = {intf = {dest = 0x595c8, refcnt = 0x595e4}, op = 0x42e20},
  uri = 0x59554, socket = {intf = {dest = 0x59434, refcnt = 0x595e4}, op = 0x42e38}, process = {list = {next = 0x59608,
      prev = 0x59608}, step = 0x286b <http_step>, refcnt = 0x595e4}, response = 0, content_length = 0, rx_len = 0,
  rx_state = HTTP_RX_RESPONSE, linebuf = {data = 0x594b4 "HTTP/1.0 200 OK", len = 15, ready = 1}}

• Debugging with GDB (official manual)
• A cheatsheet, note that there are several others available.