Network Booting Linux (PXE)

Written by Kevin Korb as a presentation for GOLUG Presented on 2008-05-07

• What is Network Booting

Network Booting is when a computer's boot device is a network interface instead of a local storage device. It downloads and runs an operating system from a boot server. Many computers and network interfaces have this capability built into their firmware however some require some form of local storage to store the initial bootstrap code.

• Why would you network boot

• Reduced management: It is possible to have multiple computers using the same OS image. That way when you make changes to one the others are all changed as well. This is also usually done in a read only mode which enhances security (if someone gets root on a client they still can't change anything).
• Reduced power consumption: Every running hard drive consumes about 25W of power and generates some amount of heat. If you believe in redundancy then a computer is likely to have two hard drives in it. Switching to a netboot/diskless configuration will eliminate these running disks and save power. When I switched 4 of my computers to a network boot configuration I got rid of 4 hard drives that had been spinning 24x7.
• Concentrated storage: The OS on a computer (especially Linux) usually isn't that big. In fact you can't buy a hard drive small enough to be a tight fit anymore. This means that all of your computers have at least 1 hard drive with only a small portion of it used by the OS. Then you are faced with the choice of either leaving all that space unused or storing data on a system that wasn't really intended to be a server. In a network boot configuration all of your OS images are in the same place. Probably consuming a bigger portion of a single disk (or RAID). The clients then have no need of a hard drive so they will not have the extra space and you will not be tempted to setup server services on them so the space can be used.
• Even more power consumption: Once you have freed up a bunch of hard drives you can use them for data storage. Data storage disks can be spun down when not in use which causes them to use very little power.
• Less noise: When a computer has no hard drives in it it will generate much less noise. You will probably find that you don't need case fans either which means even less noise. This is great if you want to have a computer in your bedroom or living room.

• What is the down side?

• Swap: If a computer is short on memory and swaps often you probably don't want this. Swapping over NFS is slow and not very reliable. However, it is possible to swap to a USB thumb drive which is reasonably fast. It is also possible to leave a single hard drive in just for swap but that then negates some of the advantages.
• Performance: If a computer is often used for heavy disk IO applications you probably don't want this. NFS is of course slower than a local disk. You will find though that for most things it doesn't make much of a performance difference.
• Single point of failure: This should be obvious but it should be mentioned too. With a network boot setup the boot server becomes a single point of failure for any client that boots from it.

• Technologies involved

• PXE: PXE (Preboot eXecution Environment) is firmware level software that usually resides on the NIC or the motherboard. If your computer is PXE capable you will see your network controller listed in your BIOS as a potential boot device. Note that most integrated NICs have an option to disable the NIC BIOS to make the computer boot faster so you will have to enable that before this will work.
• gPXE: gPXE is an open source software implementation of PXE. It is mainly for computers that don't have PXE capabilities but it is also has more features (like http download of the kernel). In some cases it is possible to attach gPXE to the computers BIOS and then flash it adding gPXE to the motherboard. If that isn't possible or you just don't want to do that it can also be stored on a local disk, floppy, CD, or USB thumb drive. It can even be a choice within grub on a computer with local hard drives. You can get binary gPXE images from http://rom.etherboot.org/rom/gpxe/gpxe-0.9.3/r-o-m/
• DHCP: DHCP (Dynamic Host Configuration Protocol) is a method of having a server tell a client what IP address to use. DHCP is required for PXE booting as it will use DHCP to determine an IP address to use while downloading the files needed to start booting.
• TFTP: TFTP (Trivial File Transfer Protocol) is a file transfer protocol that PXE (and other network boot systems) uses to download configuration files and kernels from the boot server. TFTP is used instead of a more elaborate protocol because it doesn't require any kind of authentication and it is usually locked into a single directory where it can only read the files it needs to serve out.
• PXELINUX: PXE boot loader for the Linux kernel. Comes with the syslinux package.
• NFS: NFS (Network File System) is a system for mounting a directory on an NFS server into a directory on an NFS client. When network booting the root filesystem on the client will be NFS mounted from the boot server.
• rarpd: Rarpd is a service that provides IP addresses to clients based on their MAC addresses. It predates and has been mostly replaced by DHCP however some non-Linux systems (OpenBSD) still use it to determine their IP during boot up when network booting.
• bootparamd: Bootparamd is another service that has been mostly eliminated. It tells network boot clients what NFS path to use to find their root filesystem. In the Linux world this is passed through the kernel parameters but some operating systems (OpenBSD) still use it.
• BOOTP: Bootp is the oldest service for a client to determine its IP address. Hopefully you will never see it because it somewhat conflicts with DHCP.

• How it all works in Linux

1. A client is powered on and goes through POST.
2. PXE is loaded from firmware or gPXE is loaded from wherever.
3. PXE then sends out DHCP requests to determine an IP address and boot information
4. The DHCP server tells the client what IP address to use and tells it to load "pxelinux.0
5. The client uses TFTP to download and run pxelinux.0 from the server.
6. pxelinux.0 (on the client) then looks for a config file on the server using TFTP. It looks for pxelinux.cfg/[mac address] first and then pxelinux.cfg/default if it can't find a config file specific to its IP address.
7. The client then uses TFTP to download a kernel file (and an initrd/miniroot if needed) from the server. The file names will be specified in the pxelinux config file.
8. The client then executes the kernel file using the parameters (such as nfsroot=) listed in the pxelinux config file.
9. The Linux being booted across the network should not be configured with an IP address or with DHCP. The DHCP has already happened and Linux will just continue to use that information.
10. The /etc/fstab file should list the appropriate NFS mount as the root filesystem. It should be mounted normally (no fsck of course).
11. At this point the client will work just as it would if it had booted from a hard drive.

• Setting it all up on the server

1. DHCP: First you must have a DHCP server setup. The initial setup of a DHCP server is beyond the scope of this presentation so I will only concentrate on what has to be added to make PXE work.

Add these lines to the global section of your dhcpd.conf file:

option space PXE;
option PXE.mtftp-ip               code 1 = ip-address;
option PXE.mtftp-cport            code 2 = unsigned integer 16;
option PXE.mtftp-sport            code 3 = unsigned integer 16;
option PXE.mtftp-tmout            code 4 = unsigned integer 8;
option PXE.mtftp-delay            code 5 = unsigned integer 8;
option PXE.discovery-control      code 6 = unsigned integer 8;
option PXE.discovery-mcast-addr   code 7 = ip-address;

Then add these lines to the subnet section:

  # Provide PXE clients with appropriate information
  class "pxeclient" {
    match if substring(option vendor-class-identifier, 0, 9) = "PXEClient";
    vendor-option-space PXE;

    # At least one of the vendor-specific PXE options must be set in
    # order for the client boot ROMs to realize that we are a PXE-compliant
    # server.  We set the MCAST IP address to 0.0.0.0 to tell the boot ROM
    # that we can't provide multicast TFTP.

    option PXE.mtftp-ip 0.0.0.0;

    # This is the name of the file the boot ROMs should download.
    filename "pxelinux.0";
  }

  # Provide Etherboot clients with appropriate information
  class "etherboot" {
    match if substring(option vendor-class-identifier, 0, 9) = "Etherboot";
    filename "vmlinuz_arch;
  }

You will also want to have a dynamic range setup for generic images such as Knoppix and you will want to hard code the IP to MAC address mappings for all systems with static IP addresses.

My dhcpd.conf file is available here for reference: http://www.sanitarium.net/unix_stuff/config_files/dhcpd.conf.txt

2. TFTP: The next service you will need is a tftpd. I use tfpd-hpa from http://www.kernel.org/pub/software/network/tftp/. Since the tftp service is completely unauthenticated it is normally restricted to a directory where only the files it needs are stored. This means the kernels, the miniroot/initrd files, and the pxelinux files. From now on I will assume that the tftp root is /netboot/tftpboot since that is what I use. The tftp daemon can be run stand alone service or from /service with a command line like 'exec /usr/sbin/in.tftpd -L -v -s /netboot/tftpboot'. You can use the command line tftp client to test this service with commands similar to the command line ftp client.
3. PXELINUX: The next thing you need is the pxelinux.0 file from the syslinux package. You must place a copy of that file into the tftpboot directory so that the tftpd can serve it out. You also make a pxelinux.cfg directory in the same place to hold the config files for different systems. I have some of my config files posted here: http://www.sanitarium.net/unix_stuff/config_files/pxelinux.cfg
4. NFS: The directories on the server that contain the OS images much be exported via NFS to the clients that need to use them. NFS server setup is also beyond the scope of this presentation. I use the naming convention of /netboot/[hostname] for host specific images and /netboot/[distro name][-version] for generic images such as Knoppix.

Once all these services are up and running on the server you should be ready to serve out client images.

• Setting it up different types of clients

Now it is time to actually install some operating systems. Sine most OS installers do not support installing to an NFS mount (I actually don't know of any) we will install the OS to a local temporary disk and then move it later.

• Gentoo:
1. Install Gentoo like normal.
2. When you compile the kernel make sure to enable the following options:
   • Networking / Networking Options / TCP/IP Networking / IP: kernel level autoconfiguration / IP: DHCP support
   • File Systems / Network File Systems / NFS file system support / Provide NFSv3 client support
   • File Systems / Network File Systems / NFS file system support / Root file system on NFS
   • Whatever NIC driver you use should be compiled in as well unless you want to use an initrd to get the module.
3. For /etc/conf.d/net use config_eth0=( "noop" )
4. In the /etc/fstab file set the root to be: nfsserver:/netboot/hostname / nfs sync,hard,intr,rw,nolock,rsize=8192,wsize=8192 0 0
5. Then copy the entire OS to the remote path listed above and make note of the computer's MAC address.
6. Set the BIOS to boot from the network interface or setup gPXE.
7. On the server, hard link the kernel into the tftpboot directory with something like:
   ln /netboot/hostname/vmlinuz-2.6.24 /netboot/tftpboot/vmlinuz.hostname
8. On the server, cp -a /dev/console /dev/null /netboot/hostname/dev/
9. On the server, add the MAC address to the dhcpd.conf file with a static IP address. Something like:

  host comatose {
    hardware ethernet 00:50:2C:05:5E:7B;
    fixed-address 192.168.100.20;
  }

10. On the server, add a config file for the system in the /netboot/tftpboot/pxelinux.cfg directory. The file name must be generated by the MAC address of the computer. The format is 01-[MAC] where the MAC has the letters converted to lower case and the colons converted to dashes. So, the file name for the MAC address listed above would be: 01-00-50-2c-05-5e-7b. The syntax of the config file is similar to what you would put into a grub.conf file for a kernel entry except that it also has the NFS information. An example would be:

DEFAULT /vmlinuz.comatose
APPEND ip=dhcp root=/dev/nfs nfsroot=192.168.100.202:/netboot/comatose

11. The system should now boot from the network into the Gentoo image that you created.
12. Use the system like normal and forget that it has no hard drive!
• Ubuntu:
1. Install Ubuntu like normal.
2. Copy the entire OS to your NFS server in /netboot/hostname
3. In the NFS root change the root entry in the /etc/fstab file to be: nfsserver:/netboot/hostname / nfs sync,hard,intr,rw,nolock,rsize=8192,wsize=8192 0 0
4. Set the BIOS to boot from the network interface or setup gPXE.
5. On the server, hard link the kernel into the tftpboot directory with something like:
   ln /netboot/hostname/boot/vmlinuz-2.6.24­16-generic /netboot/tftpboot/vmlinuz.hostname
6. On the server, hard link the initrd into the tftpboot directory with something like:
   ln /netboot/hostname/boot/initrd.img-2.6.24-16-generic /netboot/tftpboot/initrd.hostname
7. On the server, add the MAC address to the dhcpd.conf file with a static IP address. Something like:

  host deranged {
    hardware ethernet 00:D0:B7:BE:E7:1C;
    fixed-address 192.168.100.2;
  }

8. On the server, add a config file for the system in the /netboot/tftpboot/pxelinux.cfg directory. The file name must be generated by the MAC address of the computer. The format is 01-[MAC] where the MAC has the letters converted to lower case and the colons converted to dashes. So, the file name for the MAC address listed above would be: 01-00-d0-b7-be-e7-1c. The syntax of the config file is similar to what you would put into a grub.conf file for a kernel entry except that it also has the NFS information. An example would be:

DEFAULT /vmlinuz.deranged
APPEND ip=dhcp root=/dev/nfs nfsroot=192.168.100.202:/netboot/deranged panic=60 initrd=/initrd.deranged quiet splash

9. The system should now boot from the network into the Ubuntu image that you created.
10. Use the system like normal and forget that it has no hard drive!
• Knoppix:

Knoppix is a bit more complex. Because of the need to support any kind of hardware the software within Knoppix must be used to generate the kernel and initrd images. The ones on the CD/DVD will NOT work with an NFS root. The Knoppix people packaged together network booting support and terminal server support so that has to be at least partially setup.

1. Boot a Knoppix CD or DVD into the full GUI.
2. Unplug the network cable (The next step will start a dhcpd which will conflict with yours).
3. Run K / KNOPPIX / Services / Start KNOPPIX Terminal Server. This will start a wizard that will allow you to setup the Knoppix system as a boot and terminal server. You can take the defaults for the network settings. The important part is that you have to select all of the NIC drivers you need your boot server to support Knoppix on.
4. Kill off the dhcpd that Knoppix started
5. Copy the vmlinuz, miniroot.gz, and default files that were just created in /tftpboot to your server in /netboot/Knoppix-version
6. Shutdown Knoppix
7. Copy the Knoppix CD/DVD to the same place on your server without overwriting any of those files.
8. Hard link the kernel and miniroot.gz files into your /netboot/tftpboot directory.
9. The default file is the pxelinux.cfg file for the Knoppix image. Copy it to your /netboot/tftpboot/pxelinux.cfg directory and update the NFS paths.
10. Now any system that attempts to network boot from your server but finds no matching configuration file will use the default configuration file and boot up Knoppix.
• OpenBSD:

NOTE: OpenBSD does not officially support this. Use it at your own risk. Also, swapping to NFS on OpenBSD is not very reliable so make sure you have enough RAM that you will not need to swap.

1. Install OpenBSD like normal and boot it from the temporary hard drive.
2. Install the kernel sources from the CD
3. In /usr/src/sys/arch/i386/conf create a new kernel config from the GENERIC one. Edit it and swap the comment on these two lines:

config         bsd     swap generic
#config         bsd     root on nfs swap on nfs

4. Compile that kernel and install it as /bsd.nfsroot
5. Copy the OS up to the NFS server.
6. The new root entry for the /etc/fstab file should be: nfsserver:/netboot/hostname / nfs rw,intr,nolock 0 0
7. On the server, hard link over the kernel. Note that the OpenBSD bootloader will only look for /netboot/tftpboot/bsd so you can't use the bsd.hostname notation like in Linux. The command would be: ln /netboot/hostname/bsd.nfsroot /netboot/tftpboot/bsd
8. On the server, hard link over the boot loader with: ln /netboot/hostname/usr/mdec/pxeboot /netboot/tftpboot/pxeboot.0
9. The pxelinux.cfg config file would simply contain: DEFAULT /pxeboot.0
10. On the server, add an entry to /etc/ethers for rarpd such as: 00:02:B3:A2:9C:AE hellmouth
11. On the server, add an entry to /etc/bootparams for bootparamd such as:
    hellmouth root=192.168.100.202:/netboot/hellmouth swap=192.168.100.202:/netboot/hellmouth/swapfile
12. On the server, create a small swapfile just so there will be one:
    dd if=/dev/zero of=/netboot/hostname/swapfile bs=1024k count=16 ; chmod 0600 /netboot/hostname/swapfile
13. On the server, in /netboot/tftpboot symlink the pxeboot.0 file to a file named to the hex translation of the IP of the OpenBSD client. For instance:
    ln -s pxeboot.0 C0A86401 (for 192.168.100.1)