Stefan Hajnoczi: GDB Remote Debugging

Milestone: Get GDB stub into mainline.

Git commit: 0004992b5bb35b4d5df324d8afffae04e3ee8285

I hit an interesting bug thing while writing the GDB stub test suite. The interrupt handler shares the stack with the main thread of control. Update: hpa informed me that the i386 ABI requires ESP to be top of stack, and therefore this issue cannot happen in conforming code.

If the main thread uses stack memory below the stack pointer when an interrupt comes in, then that memory may be corrupted. The interrupt handler uses the stack to store the CPU state and assumes that memory below the stack pointer is unused.

One option is a seperate stack for the interrupt handler and GDB stub. This stack doesn't need to be very big (definitely less than 4K, probably 512 bytes is fine).

Another option is to subtract an arbitrary amount to “bump” the stack pointer in the interrupt handler. This is a hack and would not guarantee anything.

Finally, I am going to leave this issue for now because I think there is very little code (if any) which behaves as described. GCC tends to reserve space for locals variables at the beginning of a function.

Committed a test suite for the GDB stub. It works in two parts: a GDB script that executes tests and some assembly to set things up on gPXE's end. The two parts pass control through each other using the GDB continue command and the x86 int $3 breakpoint.

The tests include register and memory read/write. I plan to add more as needed by the feature set.

To run the test suite, build gPXE with GDBSTUB and make sure tests/gdbstub_test.S gets linked in. Then run:

$ make bin/gpxe.hd.tmp
$ make
$ qemu -serial tcp::4444,server bin/gpxe.usb

[From another terminal]
$ gdb -x tests/gdbstub_test.gdb

Git commits: 813fd7500972968b19f17326c5cac308c43b59b6, 6b0bc333ef9123a2f255db49f1346a3cb2d28285

librm.S #ifdef removal. The GDBSTUB config.h option was implemented using #ifdef. Today I got some advice from mcb30 and removed the #ifdefs. The GDB stub interrupt handlers are now linked in using weak symbols. That means certain symbols are overridden when GDBSTUB is enabled, causing GDB IDT setup code to be called.

Split serial console from serial driver. This patch makes the serial driver interface available in include/gpxe/serial.h. It moves the serial console code from core/serial.c to core/serial_console.c, leaving just the serial driver itself.

There is currently no check against GDBSTUB and CONSOLE_SERIAL being enabled at the same time. Just don't do it .

Git commit: b739cae42af8a7b37def48c9284381e98a91043e, 20a525f3df08af982989663d4c3718af3e985d9b, 2507e1c548bb98f9448e4a343f45c1d55dcef227, 8e93a5edf8e3a91a42c88081c64a255a930f8713

Removed unused arch/i386/core/gdbsym.c. This was an attempt at GDB debugging via QEMU and mcb30 suggested it could be removed. Its GDBSYM config.h option has potential to confuse users wanting to build the GDB stub, so I removed it.

I feel the code is ready for mainline review. The GDB remote debugging functionality is enabled with #define GDBSTUB in config.h. Debugging works over the serial port. Here's how I test in QEMU:

$ make # with GDBSTUB enabled in config.h
$ make bin/gpxe.hd.tmp # for debugging symbols
$ qemu -serial tcp::4444,server bin/gpxe.usb

[From another terminal]
$ gdb
(gdb) file bin/gpxe.hd.tmp
(gdb) target remote localhost:4444

To run the test suite, add REQUIRE_OBJECT ( gdbstub_test ) in core/config.c and rebuild. The gdb invocation works like this:

$ gdb tests/gdbstub_test.gdb

Once all tests are done, you will be left at the GDB prompt and can type quit.

Git commits: 77b559fe736608856b3d5812e6f6fe4c40858732, f1a3608cbaba4c6777d88aaf3c703c14fc8e8812

GDB remote debugging is in mainline gPXE. Thanks to mcb30 for reviewing and merging the code.

Documentation is available here. I also spent a couple of hours yesterday and today making a screencast (higher quality version 14 MB):

I hope others find it helpful to add GDB to the (small) set of gPXE debugging tools.

Please note that changes and git commits described from here onward may not be in mainline gPXE. When something gets merged, I will make a note of it.

Detach and kill are now handled. In the new gdbstub2 branch, I've added code to leave the GDB stub when GDB detaches or asks to “kill” it. It might be a good idea to manually clear all breakpoints before detaching. The GDB stub does not keep track of breakpoints and therefore has no way to automatically clear them when GDB disconnects.

Atomic read/write for device memory. Memory reads/writes of 2 or 4 bytes are now done atomically. This is important when operating on device memory where a memory operation can have side-effects. Users should do single reads/writes in GDB to deal correctly with device memory, e.g. x/w $eax. Do not attempt to read more than one 2- or 4-byte device register at a time.

Had the weekly meeting today and discussed next week's goal: hardware watchpoints and UDP transport.

Hardware watchpoints shouldn't require major changes so I want to defer that towards late next week.

UDP transport is challenging because we want to support remote debugging over UDP while affecting the network stack as little as possible. The GDB stub is designed to be isolated from the rest of gPXE so that using the debugger does not affect the state of the program.

To send a UDP packet, we'll craft a Ethernet, IP, UDP packet by hand. This side-steps the network stack and reduces dependencies on gPXE functions. Using netdev_tx() a raw packet can be queued for transmission.

Receiving UDP packets is more difficult. When the GDB stub has control, the program is paused inside an interrupt handler. If we receive a packet not destined for the GDB stub we are in trouble since there is only limited memory available to buffer received packets.

Ideally we could queue up all non-GDB packets so that they are processed when gPXE regains control. But due to finite memory, I am going to implement a strategy that drops all non-GDB packets first. Depending on how that works in practice, I might add something fancier to deal with the memory issue or an eviction policy.

Week 3