Stefan Hajnoczi: GDB Remote Debugging

Milestone: Set up IDT and write an interrupt handler.

Some notes after chatting with mdc and mcb30:

• Place IDT code in arch/i386/transitions/librm.S - in similar places to lgdt and sgdt.
• GDB stub should be written with portability in mind, separate out arch-specific parts.
• GDB stub should be a build option.
• Source-level debugging and symbols should work (mostly) out-of-the-box due to ELF build.

Git commit: ac29ad53aff6e89f12bd5a163861d1afb1846049

Implemented an interrupt handler in arch/i386/transitions/librm.S. It currently sets eax to 0xcafebabe and spins in an infinite loop. Have tested that it is working using QEMU.

Git commits: 12144ffbfadef9c6597f9ac754685223bb736368, 1ff72edaa0c68966e1bc102ae5167d714eeb03e6, c15542a614961acc1051296fc2367d1539db57ff

When GDB reads or writes registers on x86, it wants a snapshot like this: EAX, ECX, EDX, EBX, ESP, EBP, ESI, EDI, EIP, EFLAGS, CS, SS, DS, ES, FS, GS. This snapshot is a blob that gets sent between the GDB stub and GDB.

The interrupt handler now takes this register snapshot and passes it to the GDB stub. It also applies the register snapshot to the actual CPU state when the GDB stub returns. So if the GDB stub changes EAX in the register snapshot, then the EAX register will be changed when the interrupt handler returns.

Each interrupt is mapped onto a POSIX signal number (e.g. SIGSEGV). The GDB protocol communicates these numbers when reporting that execution was interrupted.

Git commit: f6c6b14468fffff0cf55df77ee7bb796113bcb4a, 6f8c3b03af1fa4733958a0ad66496a0acc8ce882

Asked mcb30 for feedback on the code so far. The latest git commit includes his suggested clean ups and simplifications.

The interrupt handler calls gdbstub_handler(regs), where regs is a pointer to the register snapshot. The GDB stub may change the values in the register snapshot. When the interrupt handler exits, it applies the snapshot to the CPU state. Changing ESP is currently not supported, since it is more difficult to implement and we do not anticipate it ever being changed.

Git commit: d1e823a19d9c847fb7a965f8fbb9345f68875c3a

The GDB stub has initial support for:

• Register read/write
• Memory read/write
• Continue and step
• Breakpoints
• Source-level debugging

Here is a screenshot:

The stub currently uses the serial driver directly. I need to design a clean GDB transport interface.

To try it out:

$ git clone git://git.etherboot.org/scm/people/stefanha/gpxe.git gdbstub
$ cd gdbstub/src
$ make
$ qemu -serial tcp::4444,server bin/gpxe.usb

[From a different terminal]
$ cd gdbstub/src
$ gdb
(gdb) file bin/gpxe.hd.tmp
(gdb) target remote localhost:4444

Git commit: 6f5d000a673209278400b9a04e12ee36cab07d28

Slow day today because I need to do university work for Thurs and Fri, my last ever assessment. Will get back to gPXE tomorrow afternoon.

Talked to mcb30 about improvements to the GDB stub. Checked that the register ordering is indeed correct and that GDB uses the reverse order to the pushal instruction.

I tried running a process that polls for serial activity and breaks into the GDB stub on activity. This eliminates the need for a hardcoded breakpoint during gPXE startup. gPXE will boot normally when GDB is not being used. If GDB is connected, then it will break into the GDB stub.

I am not sure if this is the best solution, but I'll use it for a while and see how effective it is.

Git commits: d828b65e7182372fdf3f9174f76a6d31341d16a9, 00f6bbbb61348865d57ebc979617f5f962bd037c

I cleaned up the code today and did manual testing to see if things behave the way they are expected to.

Moved register snapshot and single step support into its own header file arch/i386/include/gdbmach.h. This keeps the GDB stub portable.

Implemented retransmit support if GDB sends NACK after receiving a corrupted reply from the GDB stub.

Had the weekly mentor meeting today with mdc. The aim for next week is to push the GDB stub into mainline. Mainline support will make using the GDB stub easy for developers and advanced users. I hope that others will find debugging useful way to develop faster. Mainline exposure will also help the GDB stub to improve.

The mainline effort requires making the GDB stub a config.h option and writing usage documentation. I think the GDB stub should not be enabled in default builds since only a small number of users will ever need it. When these tasks are complete I will ask mcb30 to consider merging it into mainline.

I have been reading the GDB Manual and playing with GDB command files (aka scripts). I am writing a test suite for the GDB stub using GDB scripts.

To run the test suite you start gPXE and launch GDB with the test suite scripts. The scripts step through a series of test procedures, like writing to memory and reading that memory back to check the write was correctly performed.

Having a test suite is important to me because I want to have the freedom of experimenting with the GDB stub code without worrying about breaking things. The test suite will make sure I don't introduce regressions.

Git commit: 42838a3f3147236be9a586dabc3524dc128235ec, 95b94a699fb0e62613885f0936890e902f624ae0

Added GDBSTUB config.h option. GDB remote debugging support is not built in when config.h contains #undef GDBSTUB. I have enabled GDBSTUB by default for now, but will make it default to off when proposing a patch to mainline.

This option is implemented by placing a few #ifdefs in arch/i386/transitions/librm.S. Hopefully there is a way to eliminate the need for #ifdefs. Please let me know if you have ideas .

Introduced config-local.h to prevent accidental commits. Since we often need to tweak config.h options during development, for example enabling GDBSTUB , it sometimes happens that we forget about these temporary changes before committing. Then we have to go back and amend the commit to avoid introducing changes to config.h.

A neat solution proposed by mdc was to have config-local.h. This file never gets checked in to git and can override the defaults in config.h. I have implemented this idea by adding support for the @TRYSOURCE directive to util/mkconfig.pl. The main config.h now tries to source config-local.h if it exists.

I played with the ipv4_arp_check hang bug. This was a great test for the GDB stub. The GDB stub worked well except for the usual issues with optimized code.

The hang was caused by NULL pointer memory corruption. I wanted to use a watchpoint to find out where the NULL pointer was being written from. Unfortunately, the current GDB stub does not support hardware breakpoints or watchpoints.

I have not looked at implementating this on i386 using the debug registers yet, but think it would be a very useful feature. I had to work around this missing functionality by dumping memory at several points in time to find out when the corruption occurred.

Thinking about this class of bugs also suggests having a default watchpoint on 0x00000000 whenever gPXE is built with GDBSTUB. A memory read or write to 0x00000000 will result in breaking into the debugger. This is like the NULL_TRAP feature on steroids .

On to Week 2!