Building and Debugging the Linux Kernel

Dennis J. McWherter, Jr. bio photo By Dennis J. McWherter, Jr. Comment

Recently, for the first time in a few months, I had built the latest version of the Linux kernel. In fact, since I don’t drink coffee, I decided to write this post while I waited for it to build. We’ll take a quick look through where to get the official source code, how to compile it, and what tools there are to debugging it (short of rebooting your machine for every patch, of course). That said, I will shift much of the focus of this article on to actually running the built kernel and debugging it.

Where’s teh codez?

The official linux source tree can be found at Kernel.org. From here you can download the source in tarball form or take the latest release (at time of writing, this is kernel v3.19 which is the tag I checked out). However, I recommend ingesting your source via git. Using git allows you to keep the tree up-to-date with the latest revisions and checkout any previous version of kernel source that you care about. In particular, I recently grabbed the source directly from the torvalds/linux.git repo. You will want to run:

git clone https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/

Once you have this copy of the source code, you can move around to any source revision you like or take a tagged build. To see a list of available tags, cd into the source directory and try:

git tag -l

Great, this is the first step to building your kernel. If you want more help with git, see this git primer or try to search around for others. There are many git resources available out there.

Building the Kernel

Since we’re ready with our source code, we can build the kernel. Building the kernel is reasonably straightforward if you’re familiar with typical Makefile projects, but there are other resources that also contain more detail on this.

Ramdisk (optional). The first thing I like to do is create a ramdisk. I have found that about 3GB is sufficient for this size. You can try to run something like the following to create a ram disk at the location of /media/ramdisk:

sudo mkdir -p /media/ramdisk
sudo mount -t tmpfs -o size=3072M tmpfs /media/ramdisk
# The next command copies the repo over
cp linux /media/ramdisk

This improves the overall performance of the build since it can write the compiled code and read the source files out of RAM instead of off disk (i.e. disk is orders of magnitudes slower than memory). However, the initial copying of source will take some time since it is reading from disk and copying into memory.

NOTE: This is an optimization and in no way necessary for compiling the Linux kernel. Likewise, upon reboot, the contents of the RAM disk will be cleared (since RAM is volatile storage).
Caveat: Though I recommend performing all of your builds on your RAM disk for speed, this does cause an obvious synchronization problems between the files on your disk and the files in your ram disk. I will leave the coping mechanism to you if you’re performing active development, but one feasible solution is to simply continue using git to track your changes (perhaps an alternate remote would be useful here). Just be sure to push your changes before you delete the ram disk or reboot.

Configuring the Kernel. The next step in the process is to create a configuration. You can use your current kernel’s configuration or use the default one.

make defconfig

The above command will create a default kernel configuration for you. Alternatively, you can use make menuconfig instead to manually configure your kernel. Open up the generated configuration (i.e. the .config file) and make sure you have set the following parameters (for debugging):

CONFIG_FRAME_POINTER=y
CONFIG_KGDB=y
CONFIG_KGDB_SERIAL_CONSOLE=y
CONFIG_DEBUG_INFO=y

Build the kernel. After that, you’ll want to actually build the kernel. The machine I’m using to compile the kernel has 4 cores, so I run the following command:

make -j4

At this point you’ll want to walk away and get a cup of coffee if you drink it (otherwise, write a blog post).

The number passed with -j option to make signifies the number of threads you wish make to use for the compilation process (i.e. you should adjust this to be reasonable for the number of cores on your machine). With a 3GB RAM disk and 4 threads, this process takes about 15 minutes and 26 seconds on my virtual machine to compile (not bad for a VM on a dinky little laptop).

Install the kernel. Congratulations, you’ve successfully built the Linux kernel on your own. Finally, we need to install the kernel so we can actually use it.

sudo make modules_install install

This command will put the kernel files alongside your current kernel. When this is done, you should be able to see the kernel files in /boot.

Qemu

Our kernel is built and now we want to test it. But naturally, we don’t want to reboot for every change (and we didn’t necessarily install it in the “right” place to do that, anyway). To do this, we’ll use qemu. Qemu is a pretty nifty little, “machine emulator and virtualizer.” This tool will allow us to boot up our compiled kernel right in software and observe its behavior. We can similarly attach gdb for debugging this process and it’s much faster than performing full reboots on our machine to test the kernel we just built.

NOTE: If you’re building a custom kernel for production, it is best to use qemu for prototyping and development, but obviously you should run thorough tests on the bare-metal machines. Since qemu is software that virtualizes the hardware, it is possible that there is a bug in emulation and behavior may vary.

To load our kernel in qemu, we’ll run something similar to the following line:

qemu-system-x86_64 -kernel /boot/vmlinuz-3.19.0 -initrd /boot/initrd.img-3.19.0 -hda ~/rootfs.img -nographic -serial stdio -append “root=/dev/sda console=ttyS0”

This little gem will start qemu and boot your recently built kernel. However, we are missing a piece at this point. Particularly, we have specified a rootfs (root filesystem) image, but we never created it. While the kernel should drop a shell even without the rootfs, it isn’t really all that useful without it. I was running this on an ubuntu VM, so we will use “qemu-debootstrap” to build this for us.

dd if=/dev/zero of=~/rootfs.img bs=200M count=1
mkfs.ext2 -F ~/rootfs.img
sudo mkdir /mnt/rootfs
sudo mount -o loop ~/rootfs.img /mnt/rootfs
sudo qemu-debootstrap –arch=i386 precise /mnt/rootfs
sudo cp /etc/passwd /etc/shadow /mnt/rootfs/etc
sync
sudo umount /mnt/rootfs

This set of commands will create a viable rootfs using ubuntu’s “precise” release as the base distribution. A quick summary follows:

  1. Create a file of zeros ~200MB in size called rootfs.img
  2. Format rootfs.img to ext2 file system
  3. Make a mount point for image
  4. Mount image to created mount point
  5. Create the rootfs for “precise” in the mounted location
  6. Copy over current system user data for login
  7. Flush file buffers (i.e. make sure we write changes)
  8. Umount disk image

As you can see, all of the real work is handled for us by qemu-deboostrap. However, after we have run this, we should now be able to start qemu. After issuing the command above, you should eventually she a login prompt appear in the qemu window. Login with the credentials to your current machine (i.e. we copied /etc/passwd and /etc/shadow) and you should be able to use the system like normal. Similarly, a uname -a should show you that you’re running whatever kernel version which you just compiled.

Debugging

We have successfully built our kernel and loaded it up in our emulator. Next up is debugging. Note that printk (kernel’s version of printf) is still very much your friend while working in the kernel, but some things are simply better looked at under the debugger when possible. For this purpose, KGDB was born and that is why we enabled it earlier. Though the official KDB/KGDB docs are quite thorough on this topic, I will summarize here.

Start the kernel. Shutdown your running kernel and modify the boot options slightly. We are going to tell kgdb to send the output to our ttyS0 at 115200 baud:

qemu-system-x86_64 -kernel /boot/vmlinuz-3.19.0 -initrd /boot/initrd.img-3.19.0 -hda ~/rootfs.img -nographic -serial tcp::4444,server -append “root=/dev/sda kgdboc=ttyS0,115200”

Similarly, we change how we will be interacting with our serial port. In particular, we have told qemu to listen on 0.0.0.0 port 4444 for a client until it starts. Since we only have a single serial device, this is now treated as ttyS0 instead of our console. We’ll show why this is important later. In any case, you can start your debugger now by simply running:

gdb </path/to/kernel/build>/vmlinux
(gdb) target remote 127.0.0.1:4444

This connects gdb at the network location we told qemu to listen on. Though this is a network connection from the host’s perspective, your kernel sees this as a direct serial connection by specifying the -serial option.

Drop the kernel into debug mode. The first thing we want to do is actually put the kernel in debug mode. Login as root (or use sudo) and run the following command:

echo g > /proc/sysrq-trigger

If you compiled with the KDB frontend and don’t have gdb attached, you should see kdb open up in your console. If, however, you followed the steps above and attached with gdb, control of the kernel should have been passed to the debugger. From here, you can operate gdb as you would with any other program.

NOTE: If you have trouble with gdb (i.e. it times out before you drop to debug mode) simply reconnect using the “target remote” command above. If it appears to be hanging when you first get into debug mode, simply stop debugging the process in gdb (i.e. CTRL + C) and reconnect. You should see a (gdb) prompt when the debugger has successfully connected to the kernel. If you want to continue kernel execution, simply type “continue.”

There you have it; a more or less step-by-step guide to building and running your own version of the Linux kernel. This is more useful than just being the coolest kid amongst your friends. With this knowledge you can keep your kernel up-to-date with the latest fixes (before full point releases) and bake in any custom code into your kernel that you find useful. Happy kernel hacking!

comments powered by Disqus