This is an old revision of the document!
====== EtherBoot developers' manual ====== This manual is probably outdated and needs some rework. Submitted unchanged into this wiki to allow transition to the upcoming wiki-only website by Anselm Martin Hoffmeister. ==== Revision information ==== Written by Ken Yap, Georg Baum Copyright (C) 2001,2002,2003 Ken Yap, (C) 2003 Georg Baum Revision history: * Revision 5.2.2 (current), 2003-09-30, revised by KY * Revision 5.2.0, 2003-08-11, revised by KY, GB This document explains the internals of the EtherBoot package. The information here applies to version 5.2 of EtherBoot. ===== About this Developers Manual ===== === Obtaining the most recent version of this document === This document and related documents are also kept online at the [[http://www.etherboot.org/|Etherboot Home Page]]. This will in general have the latest source distributions and documentation. === Feedback === Comments on and corrections for this Developers Manual may be directed to the primary author. === Copyrights and Trademarks === This manual may be reproduced in whole or in part, without fee, subject to the following restrictions: * The copyright notice above and this permission notice must be preserved complete on all complete or partial copies. * Any translation or derived work must be approved by the author in writing before distribution. * If you distribute this work in part, instructions for obtaining the complete version of this manual must be included, and a means for obtaining a complete version provided. * Small portions may be reproduced as illustrations for reviews or quotes in other works without this permission notice if proper citation is given. Exceptions to these rules may be granted for academic purposes: Write to the author and ask. These restrictions are here to protect us as authors, not to restrict you as learners and educators. * All trademarks mentioned in this document belong to their respective owners. === Acknowledgements and Thanks === Thanks to all the people who have contributed information and corrections to this document. ===== Characteristics of Etherboot ===== For an introduction to Etherboot from the user perspective you may wish to read the User Manual first. In order to understand Etherboot development, it is necessary to first understand how Etherboot differs from normal application programs. - Etherboot runs as a standalone program, not under an operating system. The only services it relies on are those provided by the BIOS of the motherboard, or LinuxBIOS. - It has complete access to the "bare metal" of the machine. - It needs to have as small a memory footprint as practicable, in order to maximise the amount of memory available to the code that it loads. ===== The execution environment of Etherboot ===== ==== The network booting process ==== Since this is the part that the user sees first, let us first demystify how network booting works. From time immemorial, well actually since the ''IBM XT'' appeared on the market, the PC architecture has a mechanism for invoking "extension BIOSes". The original reason for this mechanism was to allow adaptor cards that the main BIOS didn't know how to deal with to carry ROMs with initialisation code or drivers. An early example was the XT hard disk controller. The main BIOS of XTs only knew how to boot from floppies. When an XT hard disk controller is added, the code in the ROM on the controller appears in the memory space of the PC and is called as part of the machine initialisation. Another example is the BIOSes on VGA video adaptor cards, although strictly speaking that is a special case in terms of ROM address. When network adaptors were made for the PC, it was a natural step to put ROMs on them that could contact a server for network booting. How does the main BIOS know that the code in the ROM is to be executed and why does it not execute some random code by accident? The ROM code has several conditions placed on it. * The ROM must start on a 2kB boundary in the memory space, between ''0xC8000'' and ''0xEE000'', although some main BIOSes scan outside these limits. * The first two bytes of the ROM must be ''55 AA'' hex. * The third byte of the ROM should contain the number of bytes in the ROM code divided by 512. So if the ROM code is 16kB long, then this byte would hold ''20'' hex (''32'' decimal). * All the bytes in the ROM (specified by the length byte just mentioned) must checksum to 8 bits of binary zero. The sum is formed by 8 bit addition of all the bytes, throwing away the carry. Note that there is not a particular location designated as the "checksum byte". Normally the ROM building process alters an unused byte somewhere to fulfil the checksum condition. If such a ROM is detected and validated by a scan, then the main BIOS does a far call to ''ROMSEG:3'', where ROMSEG is the segment of the ROM and 3 is the offset to transfer control to the discovered extension BIOS. Typically a network boot ROM does not take full control at this point. Instead the normal procedure to do some initialisation or probing of the hardware and then plant a vector that will be called when the BIOS is ready to boot the OS. The vector used for this purpose is normally interrupt ''0x19'' although interrupt ''0x18'' is sometimes used. For PCI plug and play ROMs things are more complicated. For the full story, you need to get the specifications from Phoenix and Intel. Here is a quick summary. * The boot ROM must satisfy the requirements for ROMs listed above (called //legacy ROMs//). * There are two additional structures in the ROM, the PCIR structure and the PnP structure. These structures are pointed to by offsets in two 16-bit words at ''0x18'' and ''0x1A'' bytes respectively from the beginning of the ROM. As a double check, the structures each begin with 4 magic bytes, ''PCIR'' and ''$PnP'' respectively. * The PCIR structure contains the vendor and device IDs of the network adaptor, and these must match the IDs that is stored in the adaptor's PCI configuration memory, or the ROM will be ignored. * The PnP structure contains various vectors. The one of interest to us is the Boot Execution Vector (BEV). This points to the starting point of the boot ROM code. The first time the ROM is detected, it is called at the ''ROMSEG+3'' entry point as for legacy ROMs. This entry point must indicate, by returning ''0x20'' in register ''AX'', that it is a network boot device. When the BIOS is ready to boot, it calls the BEV. Note that the BIOS only calls the BEV if the BIOS configuration specifies the device in the boot sequence. * There is a checksum for the PnP structure in addition to the overall checksum in legacy ROMs. The network boot process then works like this: - The main BIOS detects the Etherboot ROM as an extension BIOS and passes control to it with a far call. - For legacy ROMs, the Etherboot code hooks itself to interrupt ''0x19'' and returns control to the main BIOS. For PnP ROMs the Etherboot code indicates that it is a bootable device. - The main BIOS finishes initialising other devices and boots the operating system by calling interrupt ''0x19''. - The Etherboot code gains control. - It initialises the network hardware so that it is ready to send and receive packets. - It sends a Boot Protocol (BOOTP, [[http://www.ietf.org/rfc/rfc951.txt|RFC 951]]) or Dynamic Host Configuration Protocol (DHCP, [[http://www.ietf.org/rfc/rfc2131.txt|RFC2131]]) broadcast query packet. An alternative is Reverse Address Resolution Protocol (RARP, [[http://www.ietf.org/rfc/rfc903.txt|RFC 903]]) - Assuming a reply is received, the Etherboot code decodes the fields of the reply, sets its IP address and other parameters, and sends a Trivial File Transfer Protocol (TFTP, [[http://www.ietf.org/rfc/rfc1350.txt|RFC1350]]) request to download the file. Be aware that the 16-bit counter field of TFTP may limit transfers to 16 MB (signed interpretation), 32 MB (unsigned interpretation) or 90 MB (unsigned, large block size option active). This is a rollover bug in many TFTP servers, but quite prevalent. An alternative loading protocol is Network File System (NFS, [[http://www.ietf.org/rfc/rfc1094.txt|RFC 1094]]) protocol. In this instance a mount of the remote filesystem is done (with the bare minimum of features) and the boot file is read off the filesystem. - The file to be loaded is in a special format, it contains a "directory" in the first block that specifies where in memory the various pieces of the file are to be loaded. Formats that are supported are tagged *FIXME*Appendix A or [[http://www.acm.uiuc.edu/sigops/rsrc/pfmt11.pdf|Execution and Loader Format (ELF)]]. One small extension to ELF has been made, the top bit in the e_flags longword of the ELF header has been used as ELF_PROGRAM_RETURNS_BIT, meaning that the program transferred to intends to return to Etherboot, e.g. a menu program, and that Etherboot should not disable the network interface yet. See the file ''core/osloader.c''.\\ Eric Biederman adds: The ELF format is documented in the [[http://www.sco.com/developer/devspecs/gabi41.pdf|SysV Generic ABI doc]]. [[http://www.sco.com/developer/devspecs/abi386-4.pdf]] Also check out the [[http://www.linuxbase.org/spec/|linux standard base]], it has good links to all of these documents, in its related documents section. - EtherBoot transfer control to the loaded image. Notice no assumption was made that the image is a Linux kernel. Even though loading Linux kernels is the most common use of Etherboot, there is nothing in the procedure above that is Linux specific. By creating the loaded file appropriately, different operating systems, e.g. FreeBSD, DOS, can be loaded. In the case of a Linux kernel, there is some additional work to be done before the kernel can be called, so the segment of the file that Etherboot transfers to is not the startup segment of the kernel, but an initial stub, whose code is in ''first32.c'' in the mknbi package. This stub has several tasks, which either cannot be done by Etherboot, or should not be done by Etherboot because they are Linux specific. These are: to append kernel arguments from ''option 129'' of the BOOTP or DHCP reply; to copy and expand special kernel parameters, in particular the ''vga='' and the ''ip='' parameters and then to point the kernel to the location of the parameter area; to move the RAMdisk, if there is one, to the top of memory (this last cannot be done at image creation time since the size of the RAM of the machine is not known then). The kernel parameters are passed to the kernel as a pointer to the string written in a certain location in the original bootblock (''boot.S'' from the Linux kernel sources). This is a 16-bit pointer and is the offset of the parameter area from the base of the bootblock. This is one reason why the parameter area must be in the same 64kB as the bootblock. If the components of Etherboot are to be relocated elsewhere, e.g. ''0x80000'' upwards, then they should be relocated together. In version ''0x0202'' and above of the Linux setup segment, this can be passed instead as a absolute 32-bit pointer in a certain location in the setup segment. This eases the relocation requirements. The address of the RAM disk, if it exists, is passed to the kernel as a 32-bit address in a particular location in the setup segment (''setup.S'' from the Linux kernel sources). This is filled in with the final location of the RAM disk after it has been moved to the top of memory. ==== To compress or not to compress, that is the question ==== We simplified things a little when we talked about how the main BIOS detects the Etherboot ROM and passes control to it. At this point the code is executing from ROM. There are two problems with executing from ROM where x86 PCs are concerned. - The x86 architecture does not easily support Position Independent Code (PIC). The main drawback when executing C code is referencing global entities (global and static variables). Since the ROM address is not known when building the image, addresses cannot be assigned to global entities. More advanced environments have a dynamic loader for adjusting references just before use. Etherboot has no such help. If the code were written in assembler, we could use a convention like always referencing global entities as offsets from a particular register. But we don't want to write in assembler and we don't have control over what the C compiler generates. - C code assumes that data locations are writable. ROM locations are not writable, by definition. One could remedy this by locating the writable entities in a RAM segment, but this causes more complication. For the reasons above, Etherboot copies itself to the top of memory or 4 GB, whichever is the lesser, and executes there. However that is the physical address. The logical address for execution is set in the Makefile and currently is ''0x20000'' for the x86 version. The translation is done using the address translation facility of the MMU. This allows Etherboot global variables and functions to be assigned known addresses at the linking step. (Actually top of memory isn't completely true. On the x86 platform it executes in an even megabyte just below the top of memory. This is normally the second last megabyte. The reason has to do with the setting of the A20 gate. If Etherboot executes in an odd megabyte it will get gated out of existence when the A20 gate is disabled. By running in an even megabyte this is avoided. Note that we still have to disable the A20 gate because some operating systems react strangely if they find it enabled when they are expecting it not to be, but now we can do it from C instead of from real mode asm running in the lower 640kB.) This gives us about 1 MB for code, data, and stack. There is no heap as is normally understood; Etherboot does not have dynamic storage allocation. This keeps things simple, makes it a bit less prone to programming errors, and also acts as a check against unthinking use of memory by programmers. There is however a small heap which can be used to provide small allocations of memory below 640 kB. The ROM loader copies the payload to a temporary location starting at ''0x80000'' and continues execution from there. The reason for this first relocation is so that execution goes faster. Typically ROM has longer access times than RAM and is often accessed 8 bits at a time. The glue chipsets used in motherboards add wait states so that the CPU can interface with relatively slow ROM. This is to reduce ROM costs, as execution speed is not important at boot time. By copying itself to RAM first, the decompression goes faster. After relocation and resumption of execution, for the x86 platform, the loader then jumps to ''start16'', which switches into protected mode, then this jumps to ''start32''. On other platforms, there is no need to switch modes so it jumps to the beginning of the code directly. Other loaders are used when the code is not run from ROM but they also have the ultimate action of jumping to ''start16''. We usually want to minimise the size of the ROM used to hold Etherboot. Even if the network adapter accommodates large ROMs, there are many claims on the area between ''0xC8000'' to ''0xEE000'', by other extension BIOSes, by peripherals that use shared memory, and so forth. Compressed images are suffixed with ''.z*''. The payload is a ''.zimg'' file. The first part of a ''.zimg'' file is the decompressor which decompresses its own payload to the intended execution location. The decompressor is assembled from ''arch/i386/prefix/unnrv2b.S''. The second part is the ''.img'' result of a compilation. In non-compressed images, the ''.img'' is the one and only part in the payload. Compressed ''.zrom'' images should work just the same as ''.rom'' images, but are smaller (but this saving may or may not bring it below a particular ROM chip capacity which are powers of two). The compressor is a C program called ''util/nrv2b.c''. This is run on the host platform when building images. Here is a diagram of a full prefix stack for the x86 architecture. Other architectures will not have the real mode to protected mode transition and will have different runtime routines and decompressor: <file> +-----------------------+ | | | Payload | *.img +-----------------------+ | | | Decompressor | unnrv2b.S +-----------------------+ | | | PM runtime routines | start32.S +-----------------------+ | | | RM to PM transition | start16.S +-----------------------+ | | | Media specific loader | loader.S, *prefix.S +-----------------------+ </file> ==== Real and protected mode ==== One of the complications of the x86 architecture is the existence of the real and protected modes of execution. To simplify a long story, if we want to execute 32-bit code as generated by the C compiler, we need to be in protected mode. However the processor boots up in real mode, which is 16-bit. So the loaders must execute in real mode. In addition, the main BIOS calls the extension BIOS in real mode. So at least the prologue of the main code must be in real mode. BIOS calls must also be in real mode. In Etherboot this is handled by a pair of functions (written in assembler of course) called ''prot_to_real'' and ''real_to_prot'' that do the switching. The C code doesn't call this directly. They are implicitly called from routines that use BIOS services, such as printing characters to the screen or reading characters from the keyboard. In Etherboot 5.2 and above, the 16-bit code is executed inside special segments of memory created on the fly for running real-mode code fragments, as opposed to previous versions, where they ran inline with the 32-bit code. This allows the 32-bit stack to be separate from the 16-bit one. The file ''first32'' establishes a 32-bit stack which is separate from the 16-bit one and uses that thereafter. ===== The architecture of Etherboot ===== Leaving aside the peripheral code for the loaders, the rest of Etherboot can be divided into the core, the drivers and miscellaneous (catch-all category). <file> +------------------+------------------+ | Etherboot core | Miscellaneous | +------------------+------------------+ | Driver support | +-------------------------------------+ | Hardware drivers | +-------------------------------------+ | Hardware | +-------------------------------------+ </file> These parts are mapped on the directory structure in the src directory as follows: * ''arch'': Architecture specific files. Each architecture has an own subdirectory, currently we have ''i386'' and ''ia64''. * ''bin'': The directory where all binary files are built. * ''core'': Etherboot core. * ''drivers/disk'': Drivers for floppies and hard disks. * ''drivers/net'': Drivers for NICs. * ''firmware'': BIOS-specific files. * ''include'': Include files. * ''util'': Utility programs that are needed for building, but not included in the resulting roms. These directories are duplicated under arch where appropriate. ==== Etherboot core ==== The core of Etherboot handles the protocol for obtaining a network identity and for loading data over the network. This comprises the files ''main.c'' (main program), ''config.c'' (probe routines), ''osloader.c'' (support for various load image formats), ''nfs.c'', ''proto*.c'' (support for network I/O) and ''vsprintf.c'' (printing routines). (More here.) ==== Drivers ==== Generally each file represents a family of network adapters. For example ''tulip.c'' handles all adapters that use a Tulip compatible network controller, even if they are from different manufacturers. ''3c90x.c'' handles a whole family of related adapters from 3Com. The interface between the core and the drivers is well-defined and explained in the Section called *FIXME* Writing an Etherboot Driver. The files ''timer.h'' and ''timer.c'' provide routines for access to the second hardware timer of the PC. This is used for implementing microsecond timeouts. The files ''pci.h'' and ''pci.c'' provide a PCI initialisation subsystem that is executed for PCI adapters. ''skel.c'' is a skeleton driver that an aspiring driver writer can use as a starting point. ==== Miscellaneous ==== In this category are all the files that don't fit into the first two categories. ==== External auxiliary programs ==== In Etherboot 5.0 and later there is the facility to load and run external auxiliary programs. As Etherboot runs in high memory. All the memory below that and above ''0x10000'' is fair game for loading. What if we did not load the target operating system but instead loaded an external program, one that returned to Etherboot after doing something. This something could be a fancy menu system. The advantage of this approach is that the menu system does not have to be compiled into Etherboot, which means it can be changed without changing ROMs, and can be much larger. How does such a menu program indicate to Etherboot which image is to be loaded next? Several new features of Etherboot make this possible. Firstly, a bit in the header of the loaded image is used to indicate that the loaded program intends to return to Etherboot, as opposed to never returning, in the case of an operating system. Next, the external program is passed a pointer to the BOOTP/DHCP structure obtained by Etherboot, which includes the filename field. Finally, the external program can return one longword of status to Etherboot to indicate various conditions. So here's how the external menu program would work. Etherboot is told to load an image which is a menu program. The program would present the user with a list of images to choose from. The presentation can be as simple as a numbered menu or as fancy as a point and click interface (if you can arrange to interface to the pointer). When a particular image has been chosen, the menu program alters the filename field in the BOOTP/DHCP structure and returns a longword of status indicating that Etherboot should retry the loading. Since the filename has been altered, Etherboot will end up loading the desired image instead of the menu this time around. Think of it as a program chaining facility. ===== The development environment of Etherboot ===== Etherboot is written in C, with certain routines written in x86 assembler for access to machine resources or to do things not possible in C. The tools used are the GNU C compiler and GNU x86 assembler. ''ld'' is used to link the object files. An auxiliary program called ''makerom.pl'' converts this binary into a form suitable for writing into a ROM. A Makefile coordinates all the building procedures. However there are many network adapters that differ only in the PCI vendor and device IDs. This information must be programmed into the ROM header using ''makerom''. An auxiliary script called ''genrules.pl'' scans all driver source files for the device names and IDs and outputs make rules to a file called Roms that is included by the main Makefile to generate all the ROM images. Another file included by Makefile is ''Config'', which contains user configurable build parameters. Let us take an example of a ROM image that we could build, ''bin/mx98715.zrom''. According to this line in the file tulip.c: <file>PCI_ROM(0x10d9, 0x0531, "mx98715", "Macronix MX987x5"),</file> the driver code is in ''tulip.c'' and the vendor ID is ''0x10d9'' and the device ID is ''0x0531''. If one were to say ''make bin/mx98715.zrom'', the following actions happen: * Any needed utilities such as ''bin/nrv2b'' are built. * ''drivers/net/tulip.c'' is compiled to ''bin/tulip.o''. * The startup routine, ''arch/i386/core/start32.S'' is assembled to ''bin/start32.o''. * All the core Etherboot files, e.g. ''core/main.c'', ''osloader.c'', etc, are compiled to corresponding object files in ''bin/*.o'' and combined into an ''ar'' archive, ''bin/bootlib.a'', for convenience. * The startup routine (which must be first in the list of objects), the driver object, ''bootlib.a'', the platform specific objects are linked to produce an ELF binary, linked to run at the relocation address of Etherboot, currently ''0x20000''. The ELF binary is converted to an Etherboot binary using GNU utilities. This binary is can be on non-x86 platforms by conversions. (To fill in later.) For the x86 platform, we need to prepend loaders for the chosen load method. * The prepended ROM loaders, ''rloader'', and ''prloader'', are produced by assembling ''arch/i386/prefix/loader.S'' with different defines. The meaning of the prefixes are: * ''rloader'' = basic ROM loader, and * ''prloader'' = PCI header basic ROM loader. * The loaders contain the needed headers for the BIOS to recognise the code as an extension ROM. See the Section called *FIXME* The execution environment of Etherboot on the extension BIOS mechanism. * For example, in the case of ''bin/mx987x5.zrom'', this is for a PCI adaptor and we want a compressed ROM, so the loader is ''prloader''. This is prepended to the Etherboot image to generate a ROM image. * ''makerom.pl'' is run over this ROM image to pad it to the size of a standard EPROM. (8 kB, 16 kB, 32 kB, etc.) The PCI IDs are written into the image into the PnP structure at the right spots. A string identifying the device and Etherboot version is written into the unprogrammed bytes at the end of the ROM image, if space is available, and a pointer to that written into the PnP structure. The vendor string and the device string are normally printed by the PCI PnP BIOS at boot time. Makerom also calculates the checksum (both for the PnP structure, if present and for the whole image) and alters designated spare bytes in the image so that the checksums come out right. The image is then ready to be written into an EPROM. * By prepending alternate loaders to the Etherboot image, we can load the image from floppy, LILO/SYSLINUX, PXE or DOS. In the case of floppy loading, the preloader is called ''floppyload.bin''. It is one sector (512 bytes) long and loaded from the floppy, starting at the beginning of the floppy. It then loads the blocks following itself, which it assumes to be an Etherboot image, into memory, then jumps to the start of it. * In the case of LILO/SYSLINUX booting, the preloader is ''bin/liloprefix.bin''. It contains what looks like a floppy sector and a startup segment to LILO/SYSLINUX and makes LILO/SYSLINUX "think" that this is a Linux kernel. * The loader for chaining from PXE is in ''bin/pxeprefix.bin'', and when prepended to an image, produces a file that can be downloaded and executed by a PXE conforming boot rom as a first stage loader. Etherboot then takes over the remainder of the load process. * Yet another preloader is not for a device but for an OS environment. This is ''bin/comprefix.bin'', and when prepended to an image, makes it look like a DOS .com executable, including the peculiarity of starting at ''COM_SEGMENT:0x100''. All it does however is jump to the Etherboot image. * All the preceeding special preloaders have associated ''Makefile'' rules and are created by asking for images with the appropriate suffix. For example, if one wanted an image for writing onto a floppy one would say: <file>make bin/mx98715.zdsk</file> ===== Frequently asked questions ===== ==== How portable is Etherboot ? ==== Work leading up to 5.2 made Etherboot much more portable. The platforms on which Etherboot runs, other than the x86, are the Itanium and the Hammer. The portability issues can be looked at in two categories: Support routines needed for platforms need to be written. Platform independent and dependent code have been separated and it should be straightforward to identify the support that is needed for running Etherboot on a new platform. Byte order, operand size and alignment issues in the core routines and PCI drivers have been mostly dealt with, although there are some known places where the code has not been made portable, e.g. longword access of registers in the eepro100 driver. We hope to find these bugs as time goes by. In some cases, e.g. ISA bus NICs, there is no need to fix the problem as the ISA bus only exists on the PC platform (unless somebody attaches a NE2000 to a big-endian microcontroller and plans to run Etherboot on it). Here are the low-level services that need to provided: writing and reading from the console, determining the size of memory, obtaining the an elapsed time, inserting a boot vector to be called, and a microsecond timer for short delays. Depending on the target environment, these services may be provided in different ways. ==== How can I get Etherboot to boot a Cardbus (PCMICA) or a USB NIC? ==== ===== Writing an EtherBoot Driver ===== ==== Preliminaries ==== ==== Background information ==== ==== Structure of the code ==== ==== Rom naming rules ==== ==== Booting the code from a floppy ==== ==== Booting the test code with another EtherBoot ROM ==== ==== Writing the code ==== ==== Things to watch out for ==== ==== Tidying up ==== ==== Keeping your driver in sync with the Linux version ==== ===== A potted history of Etherboot ===== ===== Appendix A. Draft Net Boot Image Proposal 0.3, June 15, 1997 ===== ==== Preamble - the why ==== ==== The target ==== ==== Net Boot Process Description ==== ==== Image Format with Initial Magic Number ==== ==== Boot prom entry points ==== ==== Example of a boot image ==== ==== Terms ==== ==== References ==== ===== Appendix B. Compression algorithm, 1 July 1997 ===== ===== Appendix C. Copying conditions of the compression code, 1 July 1997 =====
