Compiling the Linux Kernel

THIS PAGE IS A WORK IN PROGRESS

Official Debian Page
Real-Time Linux Wiki
http://subversion.ffado.org/wiki/IrqPriorities

Direct building of the vanilla kernel can be highly complex and confusing unless you are familiar with the intricacies of the Linux kernel. A much easier way is to use the method employed by developers of whatever distribution you run. For Debian this approach use the "test_patches" script included with the Debian kernel source. This method is the easiest way of applying a few patches to the latest Debian kernel. This is usually what a user may need to do if they have hardware that has only recently been supported by Linux. Here is my email from the debian-user list in reply to the author of the above document. There is also a related discussion here.

Soon after writing this message and after studying your document I
realised I'd forgotten something. As I found there are complexities
such as the revision string that need to be known about, as are
covered in your document, and your document will certainly come in
useful for any future problems !

The thing I forgot is that I originally used test-patches (in
debian/bin). I had the problem with the conflict of revision strings;
test-patches adds "a~test". So I thought, "why not remove this string"
? I ended up editing test-patches ...

# Append 'a~test' to Debian version; this should be less than any official
# successor and easily recognisable
# version="$(dpkg-parsechangelog | sed 's/^Version: //; t; d')"
# if [ "${version%a~test}" = "$version" ]; then
# version="$version"a~test
# dch -v "$version" --distribution UNRELEASED "Testing patches $*"
#fi

I commented out lines 50 to 56. Now test-patches builds the kernel
without the revision string mismatch. I proceeded to install Nvidia
DKMS without any problems (therefore fixing my original problem).

sudo apt-get source linux
cd linux-3.2.23
sudo chown -R djbarney.djbarney *
sudo chown -R djbarney.djbarney .*
Config options added to debian/config/featureset-rt/config
bash debian/bin/test-patches -f 686-pae -s rt -j 4
../alsa-snd-usb-audio-Replace-mixer-for-Electrix-Ebox-44.patch
../alsa-snd-usb-audio-Skip-un-parseable-mixer-units.patch

This builds ...

linux-headers-3.2.0-3-rt-686-pae_3.2.23-1_i386.deb
linux-image-3.2.0-3-rt-686-pae_3.2.23-1_i386.deb
linux-image-3.2.0-3-rt-686-pae-dbg_3.2.23-1_i386.deb

After installing the RT kernel and booting on it I run ...

sudo apt-get install --reinstall nvidia-kernel-dkms

I know this actually creates (I think) a revision string conflict with
official packages. But I can live with that. I just need to reinstall
after any future updates.

Testing currently shows qjackctl (Jack Audio) reporting a working
real-time system, with my external USB audio hardware (the patches for
ebox) with ZERO X-Runs and a 10.7ms latency.

So I seem to have stumbled upon a method that seems to be reserved by
Debian kernel developers for testing patches that actually makes this
process pretty painless. Of course some of this may be dependent on
the current state of Wheezy testing and other Debian kernel policies
that seem to be in flux. Hopefully what I've discovered here can help
other people who need to patch the kernel (to get hardware working,
etc) and encourage the development and documentation of procedures to
make all this a relatively painless process.

Barney Holmes

Old document below this point

Note (30/09/12). I have been referred to this document via the debian-user email list. I noticed this quote ...

"I have found that the documentation for kernel building in Debian is spread out over several different sources. Some of these documents are out of date. Furthermore, there are a number of small but important steps that are often left out."

There is a more official way something like ...

make oldconfig
make-kpkg clean
make-kpkg --rootcmd fakeroot --initrd kernel-image kernel-headers
make-kpkg clean
dpkg -i ../linux-image-${KERNEL_UNAME}_${KERNEL_UNAME}-10.00.Custom_*.deb
dpkg -i ../linux-headers-${KERNEL_UNAME}_${KERNEL_UNAME}-10.00.Custom_*.deb

But I have found it very difficult to find out how to A., alter a few config options of the stock Debian build and B., add patches in a way that is SIMPLE and does not involve checking numerous little settings, caveats and on and on. The Linux kernel has got more complex over the years but there's no way it needs to be this complicated !

By the way the test-patches editing above works but may depend on future developments of Debian Wheezy testing as well as the way Debian devs handle kernel building. But it is SIMPLE and does WORK on my current system allowing me to simply use a piece of hardware on a real-time system set up for music production - I am a professional musician. This document has become a little messy because of the politics (if you ask me) that seems to surround Linux kernel building. This is a shame as this is the MAJOR strength of having a Linux system. Any user can alter it to their needs EASILY - as least that is the way it is supposed to be. Maybe Linux devs and enthusiasts out there will work to make all this easier in the future.

~~~

My purpose for building the kernel has been to apply patches to allow me to use a new piece of music hardware (an Electrix ebox44). Either I missed the relevant documentation or this is not clearly stated in FAQ's. But it is actually very easy these days to apply patches to your current stock distribution kernel (which is stable and tested with all your hardware). This is what many users looking for hardware support will want to do. In Debian this is as simple as ...

Steps ... http://kernel-handbook.alioth.debian.org/ch-common-tasks.html

1. sudo apt-get source linux
2. apt-get install build-essential fakeroot
3. apt-get build-dep linux
4. apt-get source linux ( INSTALLS IN CURRENT DIRECTORY, INSTALLED TO /home/djbarney/Build/linux-3.2.21 )
SIMPLE PATCHING FIRST NO CHANGE IN KERNEL CONFIG
5. apt-get install devscripts
6 bash debian/bin/test-patches ../alsa-snd-usb-audio-Replace-mixer-for-Electrix-Ebox-44.patch ../alsa-snd-usb-audio-Skip-un-parseable-mixer-units.patch
7. YES ! CONGRATULATIONS. YOU HAVE BUILT YOUR FIRST DEBIAN KERNEL WITH EBOX PATCHES.

It's as simple as that ! No need to go through complicated kernel settings because that has all be handled by the Debian devs. If I have to compile vanilla kernel one day then I'll complete the rest of this guide.

Introduction

Linux is a trademarked name protected by the Linux Trademark Institute.

Despite its large code base (over 7 million lines of code), the Linux kernel is the most flexible operating system that has ever been created. It can be tuned for a wide range of different systems, running on everything from a radio-controlled model helicoptor, to a cellphone, to the majority of the largest supercomputers in the world. By customizing the kernel for your specific environment, it is possible to create something that is both smaller and faster than the kernel provided by most Linux distributions.

Kroah-Hartman, Greg. Linux Kernel in a Nutshell: Linux 2.6 Kernel in Detail (In a Nutshell. 1st ed. O’Reilly Media, 2006. Quoted from Chapter 2. Introduction.

For this important book see the attachements to download the single page HTML version. Or view online.

At least for someone who last built a kernel around a decade ago this seems a somewhat confusing process. There is this guide ... Compiling the Linux Kernel but I'm not sure how useful/accurate it is. A better example of a guide is here but it lacks much for the newcomer to Linux. A much better example is ...

When the topic of this book was first presented to me, I dismissed it as something that was already covered by the plentiful documentation about the Linux kernel. Surely someone had already written down all of the basics needed in order to build, install, and customize the Linux kernel, as it seemed to be a very simple task to me. [1]

After digging through the different HOWTOs and the Linux kernel Documentation directory, I came to the conclusion that there was not any one place where all of this information could be found. It could be gleaned by referencing a few files here, and a few outdated websites there, but this was not acceptable for anyone who did not know exactly what they were looking in the first place.

Kroah-Hartman, Greg. Linux Kernel in a Nutshell: Linux 2.6 Kernel in Detail (In a Nutshell. 1st ed. O’Reilly Media, 2006.

The main Linux Documentation Project Kernel guide page which has been used to explain the config process in the past, gives this message ...

The Kernel-HOWTO has been removed because it don't fitted [sic] the LDP standard.

If you qualify for writing a replacement, please contact us, we need you :-)

This page aims to solve this problem ;) ... when ready it will be put through the LDP submission process.

Getting Linux

Get latest stable kernel

Get latest stable kernel .... http://kernel.org/. Hint: It's clearly labelled on that page.

Get the .sign file

Get the .sign file as well to verify integrity and security. Hint: It's the same name as what you just downloaded but remove the ".bz2" and add ".sign" after the ".tar".

Extract BZIP archive

Extract TAR archive from bzip compressed file ... "bunzip2 linux-3.3.7.tar.bz2".

Verify

Verify with "gpg --verify linux-3.3.7.tar.sign". Note: I could not finalise this step as I have a problem with GPG keys on my system.

Extract TAR archive

DO NOT extract to /usr/src. Extract to somewhere like your home directory. Extract TAR archive ... "tar -xvf linux-3.3.7.tar".

Read the README

Make sure you read the README file. This is considered good practice and will save you a ot of trouble.

Configure the kernel

Configure the kernel for your machine ... "make menuconfig" (text console GUI)), or you can use "gconfig" (Gtk GUI) or "xconfig" (Qt GUI) or even "config", or nconfig (text). Hint: You have to be in the top of the directory you just extracted.

The options are the same as you'd see in xconfig, with one crucial difference -- there's no way to "grey out" an option in menuconfig. If you don't select "Prompt for development or incomplete code/drivers", you won't even see the options for Firewire or Bluetooth. For this reason many of us encourage kernel newbies to use xconfig, and try menuconfig later when they're more familiar with kernel configuration.

Creating custom kernels with Debian's kernel-package system: 4. Configuring the kernel

Config interfaces


menuconfig


gconfig


xconfig


nconfig


config

Searching for config symbols (options)

See "linux-3.3.7/Documentation/kbuild". Other interfaces have a search function but the menuconfig one seems to be the most advanced.

======================================================================
menuconfig
--------------------------------------------------

SEARCHING for CONFIG symbols

Searching in menuconfig:

The Search function searches for kernel configuration symbol
names, so you have to know something close to what you are
looking for.

Example:
/hotplug
This lists all config symbols that contain "hotplug",
e.g., HOTPLUG, HOTPLUG_CPU, MEMORY_HOTPLUG.

For search help, enter / followed TAB-TAB-TAB (to highlight
) and Enter. This will tell you that you can also use
regular expressions (regexes) in the search string, so if you
are not interested in MEMORY_HOTPLUG, you could try

/^hotplug

Simplifying the config process

There are over 3000 possible kernel symbols (options) that can be set, but don't run away ! Fortunately there are a number of ways to masssively simplify the entire process. This involves configuring all the config options to known working settings for your architecture. From that point you can almost guaruntee that you will at least be able to build a working kernel and you won't have to worry anymore about setting all the config options or missing important ones. The next stage is customising and fine tuning it by looking at the setup of a stock (package manager) kernel image on your system. See Finding Which Module Is Needed.

If you are all ready running a stock kernel image either on a Linux installation or another similiar Linux machine then use its ".config" file. You can also get this file by booting a live Linux distribution CD. See Using a Distribution Kernel in Linux Kernel in a Nutshell.

On Debian the config file is in /boot ... in my case "/boot/config-3.2.0-2-686-pae". Copy it to ".config" at the top of the directory you extracted.

Run "make oldconfig" if you are using a .config that does not match the kernel version that you downloaded. The actual version of a stock kernel is at the top of it's .config file in /boot.

If you can't use a preexisting config file then create a default kernel configuration (see the 4.1.2. Default Configuration Options chapter of Linux Kernel in a Nutshell).

djbarney@djbarney-osiris:~/Build/linux-3.3.7$ make defconfig
*** Default configuration is based on 'i386_defconfig'
#
# configuration written to .config
#
djbarney@djbarney-osiris:~/Build/linux-3.3.7$

This provides a sane starting point for customisation if you can't use the distribution config method.

Now for customisation and tuning. Finding out what hardware you are using can be done with a bit of sleuthing and with the help of a running Kernel (the stock linux-image again). This is achieved by running a script that does all the work for you (see Let the kernel tell us what we need).

Genkernel

There is genkernel which is a Gentoo project, but it may be possible to use this on other distributions. I'm looking into this (so beware this section may not be accurate and may be changed or removed).

http://www.gentoo.org/main/en/mirrors2.xml

http://www.gentoo.org/doc/en/genkernel.xml

Setting config symbols (options)

The kernel configuration is kept in a file called .config in the top directory of the kernel source tree. If you have just expanded the kernel source code, there will be no .config file, so it needs to be created.

Kroah-Hartman, Greg. Linux Kernel in a Nutshell: Linux 2.6 Kernel in Detail (In a Nutshell. 1st ed. O’Reilly Media, 2006.

Any features you add to the kernel increase its size (and the time to build it), even if you choose to add them as modules. To make their kernels useable by most people on most hardware, Linux distributors generally include support for most hardware and functions. Debian is no different; pre-compiled kernels include support for hardware you'll never have and languages you'll never read. My general rule for kernel configuration is, "If in doubt leave it out." If you test your new kernel and find that some of your hardware doesn't work, it's easy to tweak the configuration and build another kernel.

Creating custom kernels with Debian's kernel-package system: 4. Configuring the kernel

Apart from larger and larger sizes, another feature of recent kernel images is their use of initrd. A special dictionary I keep next to my computer defines initrd as "one more thing to worry about". You can avoid the need to worry about initrd by ensuring that you compile directly into the kernel (not as modules) support for your boot hardware and root filesystem. If the hard drive you boot from is IDE, compile IDE support into the kernel. If your root filesystem is Reiserfs be sure Reiserfs support is built not as modules but directly into the kernel.

Creating custom kernels with Debian's kernel-package system: 4. Configuring the kernel

For the specification of Config Language see "linux-3.3.7/Documentation/kbuild/kconfig-language.txt".

Now you can run "make xconfig" and run through some of the options. For my purposes I just want to apply some patches to the kernel in order to be able to use a piece of audio hardware (see the next section). So here I could just build the kernel as it is with all it's modules and it should run just like the stock kernel that the config file is based upon. Of course it's never as easy as that ! This is a new kernel release so there may be some problems to solve.

So, for the moment I'll work through some of the kernel options until I get bored or tired and just go ahead with the kernel compile. I can always come back later and do some more tuning. For example on my system I am thinking of music production so real time performance is important. For example I just found out that my kernel can use the high resolution HPET timer which I have support for on my system board. I never knew Linux could handle this so I never turned it on. Now I know because I actually went ahead with building my own kernel. I think HPET is related to system performance and scheduling so i'm interested in it for music production. This is another good example of why building a custom kernel is so important for special applications and production systems. I can tune the kernel towards a particular goal - in this case high performance for music production and real time video mixing. This simply cannot be done on proprietary systems such as Microsoft Windows or even on the MAC OS X (to be fair OS X may be built to favour music makers). So this is an important lesson for Linux penguins ... don't let evil forces turn Linux into a monolithic slab of binary data that you just "download" through a package manager. This is not what Linux is and in a way defeats the entire point of using it in the first place ! Anyway, enough advocacy for the moment.

Here's another example ... I set "Preemptible Kernel (Low-Latency Desktop)" to Y in the CPU configuration because that provides better performance for music production.

Dealing with Patches

See [http://djbarney.org/lkian/#linuxkian-CHP-6-SECT-1 Download the New Source].

"Quilt" is a tool that can be used to handle patches

http://en.gentoo-wiki.com/wiki/ALSA
http://www.gentoo.org/doc/en/handbook/handbook-x86.xml?part=1&chap=7
http://wiki.linuxaudio.org/wiki/lowlatency_deprecated

Scholarly Lite is a free theme, contributed to the Drupal Community by More than Themes.