Cross-compilers Part 2

In part 1 I described cross-compiling, what it means and why you might want to, or even need to use it.

I also described how to create a cross-compiler tool-chain using crosstool-ng.

In this show I will demonstrate using one of the cross-compilers which I created as described in the last show to compile a Raspberry Pi Linux kernel.

make targets

Before we talk about how to configure things for a kernel build it might be worth taking a couple of minutes to review a number of things about the make system.

Gnu make files contain, among other things, targets. These could equally be described as recipes which detail how tasks like building programs from source and then installing those programs is performed.

Targets can be found in a Makefile by looking for labels seperated from the details of the target by a colon.

The label before the colon is exactly that, a label. The values after the colon indicate what is to be done when the target is run.

make will pick and run the first target if none is selected on the command-line. If a target is specified, then that is the one which is run.

This is why the familiar sequence:

$ ./configure
$ make
$ sudo make install

Is like that. The first call to make does not specifty a target and the first in the Makefile is selected and run. It is conventional for the first target to be the one which actually builds the software.

The second call to make selects the install target, the recipe for which is to install the just-built program.

The Makefile in the top-level directory of the kernel source tree is hugely complex and contains many targets. Many of which run programs that help us with selecting kernel configuration options before we do an actual build.

In the text that follows we will see many of these targets and kernel configuration tools discussed.

Getting the Source Code

The first thing we need to do is get the Raspberry Pi kernel source.

You could just get the source from kernel.org in the normal way, but the Raspberry Pi kernel source will include stuff like the drivers for the LEDs on the board, as well as for various GPIO functions.

As with my cross-compilers, I like to put the kernel source in /opt.

I am going to assume you have done what I do on all my development machines, and that is, create a developers group and make myself a member of it:

I am using a Debian machine for this show, as for part 1.

$ sudo groupadd developers
$ sudo usermod -a -R developers mike

And I assume /opt downwards already belongs to root:developers.

Then:

$ mkdir -p /opt/linux/rpi
$ sudo chown -R root:developers /opt/linux/rpi
$ cd /opt/linux/rpi
$ git clone https://github.com/raspberrypi/linux src
$ sudo chown -R root:developers src/

I put the source in a subdirectory of linux called rpi because I also have Banana Pi source in /opt/linux/bpi...

I use the last argument to git clone to put the source into src because I like to do what is called out-of-tree builds, which I will explain shortly.

I also need to create two other sub-directories:

$ mkdir /opt/linux/rpi/build
$ mkdir /opt/linux/rpi/modules
$ sudo chown root:developers build
$ sudo chown root:developers modules

And make sure the permissions of everything under /opt/linux/rpi is 0775.

The build directory is where our out-of-tree compile will put everything it makes.

The modules directory is where we will specify we want the module-installation stage to put our modules. From there we can gzip them up and transfer them to the Pi.

The kernel Configuration File .config

Compiling a Linux kernel involves a lot of configuration. Selection of options etc.

These settings are stored in a file called .config, see Klaatu's excellent kernel compilation show to hear more about it.

There are a number of ways in which you can get a .config file to use:

  1. Create a new one from scratch (not recommended).
  2. Start with a default configuration.
  3. Pull the configuration from a running Pi.

Number 1 above is not recommended because it will ask you thousands of questions, will take you ages and is, because it is so complex, prone to error.

Option 2 is the easiest, but option 3 will give you a kernel configuration which matches the configuration of a currently running Pi.

Current Configuration

While you are logged into a running Pi, follow these steps:

$ cd
$ zcat /proc/config.gz > .config
$ scp .config youruser@yourdesktopmachine:

Or transfer the file in some other way to the machine where you intend to do the cross compile.

The reason I say a running Pi is that the /proc/config.gz file only exists in the file-system of a Pi while it is running. If you mounted the second partition of the SD card in another Linux machine you would not find it.

Kernel Makefile targets

In the kernel Makefile, as with all Makefiles used to compile C or C++ programs, there are a number of targets.

These are effectively tasks which will be run when you type make in the same directory.

In the long and hugely complex kernel Makefile there are a number of targets which can be used for operations that concern manipulation of the .config file.

Some of these are:

Next we will look at using these targets.

Default Configuration

In the source-tree which you got from the Raspberry Pi Foundation github.com repository, there is a default configuration that you can start with.

If you placed the source as detailed above it is in:

${KERNEL_ROOT}/src/arch/arm/configs/bcmrpi_defconfig

There is a special way in which we can grab this default configuration and use it for our kernel compile, using the defconfig target in the kernel Makefile.

Before I describe this I need to detail two environment variables which we will need to set throughout our building process:

The KERNEL_ROOT variable we will use to hold a path to the root of our kernel source tree.

The CCPREFIX variable will hold a vital path prefix which will let our make commands find the right versions of gcc, ld and other tools.

Let's look at settings for those two variables which will suit our current configuration, where our source is, where our cross-compiler is etc:

$ export KERNEL_ROOT=/opt/linux/rpi
$ export CCPREFIX=/opt/toolchains/armv7/arm-unknown-linux-gnueabihf/bin/arm-unknown-linux-gnueabihf-

You can either put these export commands in a script you might want to use for the builds, put them in your .bashrc file or just set them manually before you perform a compile.

You could also just put them in a text file which you then source like this:

$ source kernel_compile_exports

The above value for CCPREFIX is correct for the arm-cortex-a7 cross-compiler tool-chain we created for compiling a kernel for the Raspberry Pi version 2 in the last cross-compilers show.

Notice that the CCPREFIX value does not just contain a path to the version of gcc in our tool-chain, but also that portion of the name of the executable which is unique in that path to our compiler, in this case arm-unknown-linux-gnueabihf.

From this point on I will assume that these two variables contain the right values, however they have been set.

To grab the default configuration:

$ cd ${KERNEL_ROOT}/src
$ make ARCH=arm PLATFORM=bcmrpi O=${KERNEL_ROOT}/build defconfig

The second command-line above demands some detailed explanation, let's break down the options to the make command:

The following two parts:

And our current working directory, contribute to the path from where the above command will take the default configuration we want. Remember that the file is at:

${KERNEL_ROOT}/src/arch/arm/configs/bcmrpi_defconfig

You will see how the values assigned to ARCH and PLATFORM contribute to the finding of the default configuration file bcmrpi_defconfig.

There are also other configuration files in here with different prefixes we could select with different values of PLATFORM=, but I have no idea what any of them are, nor even whether they refer to the Pi in particular.

The:

O=${KERNEL_ROOT}/build

Part of the above command is what will produce an out-of-tree build, that is, a build where everything that is compiled, linked or otherwise is temporary, is put, outside of our original source-tree.

The defconfig is one of the Makefile targets which tells make what we want to do.

It places a version of the bcmrpi_defconfig file in our build path with the name .config.

Doing the build out-of-tree in this way makes cleaning-up after a build very easy:

$ cd ${KERNEL_ROOT}
$ rm -rf build/
$ mkdir build

After the defconfig command completes, we will now have the following file, among other stuff:

${KERNEL_ROOT}/build/.config

Preparing for the Kernel Compile

So, whether we pulled a .config file from a running Pi, or whether we used the above command to get a default .config, we can now go ahead with the build.

But there is one more step if we took the .config from a running Pi.

oldconfig

We need to run another make target, oldconfig:

$ cd ${KERNEL_ROOT}/src
$ make O=${KERNEL_ROOT}/build CROSS_COMPILE=${CCPREFIX} oldconfig

What this does is check that there have been no changes to the Kconfig files since the configuration exercise which created the .config file we are using.

The Kconfig files are to be found throughout the kernel source tree and these are files which define the structure of kernel compile options available when you configure a build.

It may re-start the configuration program, which will ask you for the setting of any options which have been added.

The compile I ran, described below, only asked me for one value which was something about LED triggers. It prompted me to say the default was m, so I chose that option.

The kernel cross-compile I did for this show was with a fresh kernel source tree from the Raspberry Pi repository, and a .config file from a Raspberry Pi version 2 running the image dated 2015-01-31 from the Raspberry Pi Foundation download page.

After I booted the Pi with this image and did:

$ sudo apt-get update
$ sudo apt-get upgrade

I ended up with kernel version 3.18.7-v7+. The date I did this was the 25th of February 2015.

I got the kernel version on the Pi with:

$ uname -r

If you look at the contents of:

${KERNEL_ROOT}/src/Makefile

On your build machine, you will see:

VERSION = 3
PATCHLEVEL = 18
SUBLEVEL = 7

Or something similar depending on when you got the source, right at the top.

So where has the -v7+ bit of the kernel version gone?

In the .config file there is a variable called CONFIG_LOCALVERSION. In our file it is set to -v7. The plus is added by some kind of function of the state of the current commit in the github repository. I don't really understand that at the moment.

The CONFIG_LOCALVERSION variable can be set manually by you by using the menuconfig target.

menuconfig

You can also do this to get an ncurses screen which will allow you to make changes to your configuration:

$ cd ${KERNEL_ROOT}/src
$ make ARCH=arm O=${KERNEL_ROOT}/build CROSS_COMPILE=${CCPREFIX} menuconfig

If you use menuconfig you must run oldconfig again after it.

vermagic Numbers

It might be worth describing here why it will probably be necessary to compile the kernel when all you want to do is add or remove a module.

Each kernel module has written into it the version of the kernel. This is the version magic or vermagic number.

As far as I know, this number must match the version of the kernel. If there is a way to change the vermagic number in a kernel module, I don't know what it is. If you know, please explain it to me.

The Kernel Compile

Now we are prepared, we can do our full compile of the kernel. It is worth running down a check-list of what we have done so far before we kick off the compile:

If all the above is set, we can kick off our compile:

$ cd ${KERNEL_ROOT}/src
$ make V=1 ARCH=arm O=${KERNEL_ROOT}/build CROSS_COMPILE=${CCPREFIX} -j$(cat /proc/cpuinfo|grep processor|wc -l)

If this long command wraps on your screen bear in mind it is one line of text.

You might notice that in the second line above there is no target specified. As discussed earlier, make will select the first target in the Makefile when we do not provide a target. As with other correctly written Makefiles the target to build the kernel is first.

Here is an explanation of each part of this command-line:

This just controls the verbosity of the messages we see during the build. The default is zero and I believe it goes up to six (not sure). I like one.

As before, the ARCH=arm variable will control part of the build process, in this case it contributes to where the kernel will be placed when it is built. And probably other stuff as well.

Controls the path to where our out-of-tree build will put stuff, including the kernel.

This controls which version of gcc, ld and other executables used for the build are chosen. This is our cross-compiler tool-chain.

Will set the number of jobs which make will use during the compile.

The number of jobs is set by using grep to find a line which appears in /proc/cpuinfo once for each CPU or core, and piping it to wc in line-count mode.

On my quad-core machine this is the same as typing -j4.

Depending on how fast your desktop machine is, this will take a while. But hopefully not the 15 hours it took me to compile a kernel on a Raspberry Pi Model B!

On my Debian machine, which has 8GB of RAM but not the fastest clock-speed in the world it takes about fifteen minutes.

Some of the slowness in the compile on a Raspberry Pi can no doubt be explained by saying there are many disk accesses during a compile, and SD-card writes are slow relative to spinning disks. Especially for cheaper SD-cards.

Compiling the Modules

Assuming the above has finished successfully, the next stage is to compile the kernel modules:

$ cd ${KERNEL_ROOT}/src
$ make V=1 ARCH=arm O=${KERNEL_ROOT}/build CROSS_COMPILE=${CCPREFIX} modules

Similar to the command-line for the kernel compile, but with the target modules to specify that target in the Makefile

You could repeat the -j option, but I never do for this stage because it's much faster than a kernel compile.

Module Installation

If we were compiling a kernel on our desktop machine, to run on the same machine, this would place the modules in the right place. But we are going to specify a directory from which we can zip up the modules and firmware and move them to the Pi.

This is the command-line for this operation:

$ cd ${KERNEL_ROOT}/src
$ make V=1 ARCH=arm O=${KERNEL_ROOT}/build CROSS_COMPILE=${CCPREFIX} INSTALL_MOD_PATH=${KERNEL_ROOT}/modules modules_install

Again, the -j switch is left out as this is just a file-moving exercise.

If the second command-line above wraps on your screen remember it's a single line of text.

Where's the Kernel?

So I've got a kernel, where is it?

When we specified the out-of-tree path as:

O=${KERNEL_ROOT}/build

And the architecture using:

ARCH=arm

And we had KERNEL_ROOT set to /opt/linux/rpi, the compile put our kernel in:

/opt/linux/rpi/build/arch/arm/boot/

The file we are interested in here is:

Image

Note the upper-case I.

This file we will need to move to the boot partition of our Raspberry Pi as kernel.img. Probably it is a good idea to save the old one first.

Where are the Modules?

If the modules_install stage was done as above, then in:

${KERNEL_ROOT}/modules/

We will find:

${KERNEL_ROOT}/modules/lib

In which are these two directories:

${KERNEL_ROOT}/modules/lib/firmware
${KERNEL_ROOT}/modules/lib/modules

In the second of these sub-directories you will see another sub-directory bearing the same version number we were expecting, something like:

3.18.7-v7+

Your version number may be different, depending on:

These two directories firmware and modules should be zipped-up and then extracted in the /lib directory on your Pi.

But this directory also contains a couple of symlinks back to the kernel source, which we don't want on our SD card so remove them:

cd ${KERNEL_ROOT}/modules/lib/modules/<kernel version>
rm -rf source/
rm -rf build/

After the kernel, the firmware and the modules have been transferred to the Pi and it has been rebooted, it will probably be necessary to run this on the Pi:

$ sudo depmod

I believe this does something to the dependency map of the kernel so that it can find modules that may not have been there before.

This is another stage I am not totally familiar with.

Note that between kernel versions 3.2.x and 3.6.x there was a change to the firmware interface, so if you are bridging this divide you will have to take this into account.

However, it is unlikely you are doing this and I don't intend to cover this here.

More information can be found at this elinux page about Raspberry Pi kernel cross-compilation.

Conclusion

Congratulations!

If you created a cross-compiler tool-chain from the first part of this two-part series, and you have a running Pi after following the steps in this document, you have cross-compiled a Raspberry Pi kernel and saved about fifteen hours of your life.