Cross-compilers Part 1

In this show I'll introduce the concept of cross-compiling software, explain what it is and why you might want/need to do it.

I'll also talk about a great piece of kit for creating cross-compiler tool-chains on Linux; crosstool-ng.

What is Cross-compiling?

We have all probably sat in front of a desktop Linux machine and compiled something with the Gnu tools, typically the Gnu C and C++ compilers gcc and g++, and the linker ld.

The following sequence will be familiar:

$ ./configure
$ make
$ sudo make install

As will:

$ gcc -o helloworld helloworld.c

That's all very well if you're sitting at, for example, an x86_64 machine which has a screen (or in my case a screen-reader), a keyboard, and the Gnu tools.

But what happens if, for example, your boss at 'Kitchen Widgets' announces the new flag-ship product will be the 'Kitchen Widgets 3000', a microwave oven which has an embedded micro-controller to control all it's bells and whistles.

It might be able to do your chicken ding in two minutes flat, so you can get back to Minecraft, but what it doesn't have is a console, screen or a C compiler.

You will have to write, compile and possibly even test (with some kind of virtual machine) the code that will run on it.

Let's say the 'Kitchen Widgets 3000' will contain an ARM CPU, to drive the display, sense the user input controls, switch on the motor that spins the platten and the klystron that zaps your food. And also to do stuff like cut the power to the klystron if you open the door. You don't want 800 Watts of about 6GHz microwave radiation at just above waist height, do you?

Yes, I know, ARM don't make CPUs, they just design the architecture and licence other people to make them.

Enter the Cross-compiler

You need a collection of tools to allow you to write and build the code, on a platform that DOES have a screen, a keyboard, a console and a C compiler; your x86_64 Linux box for example.

This is where the term cross-compiler comes from, you will compile the code on x86_64 architecture, to run on the ARM CPU in the product.

Tool-chain

The term tool-chain is commonly used to describe the collection of stuff you need, which includes:

It's possible to generate all this stuff from scratch, but it is an exhausting process and fraught with the potential for errors which might not rear their ugly heads until your chicken isn't properly cooked by the 'Kitchen Widgets 3000'.

Luckily there is a great tool, called crosstool-ng that will automate the process to some extent, after you have taken some steps to define what you want, such as what the host machine is, and what the target architecture is.

Gotcha!

Before we get down to installing ct-ng and creating our first tool-chain, a few words about the process, and some potential 'gotchas' to avoid...

sysroot

When I first started mucking about with cross-compilers, there was one thing I really did not understand, and that relates to shared libraries, or even static ones, which need to be linked into our code.

Leaving aside the C libraries, or any libraries that need to be linked into the actuall cross-compiler, what about libraries we will need for our software?

For example, what about pthreads, a very commonly used lib.

If we are compiling on x86_64 for ARM, we can't link in the pthreads library we might find at:

/usr/lib/x86_64-linux-gnu/libpthread.so

Because that is x86_64 code, obviously.

So where do we put our ARM version of pthreads?

This is a chunk of knowledge that eluded me for a long time. It's one of those things you look for for so long that you begin to think it must be something that is so obvious that most authors of blog posts and tutorials think they don't need to say it. But if it's so obvious, why does it elude me? Am I stupid?

The answer is 'sysroot'.

If you do this on any Linux machine that has the Gnu C compiler installed:

$ gcc -print-sysroot

Chances are you will get back a single slash: "/", or just an empty string.

The sysroot variable is a path which is prepended to any search paths used by both the C compiler, and by the linker. And probably some of the other Gnu tools too, like ar etc, not sure.

There is also a switch to set this value:

$ gcc --sysroot=/path/to/my/sysroot

Which will set the sysroot value and then pass it on to the linker. Or it can be set specifically by LDFLAGS in a Makefile.

This switch is probably available in properly written autotools configurations, so that a generated configure script allows you to set it.

It is in the sysroot path that you will place any target architecture libraries that need to be linked into your code.

Now, here is something that bit me badly the first time I created a tool-chain with crosstool-ng, so watch it!

When you configure crosstool-ng to generate a tool-chain, one of the things you will set is the installation path. In other words where the tool-chain is going to live.

When you specify this installation path, the sysroot variable, internal to the tools which are generated, will be set to this path and then some.

So for example if my tool-chain configuration specifies that my tool-chain will be installed in:

/home/mike/toolchains/armv7

Then the sysroot in gcc and ld etc. will contain:

/home/mike/toolchains/armv7/arm-unknown-linux-gnueabihf/arm-unknown-linux-gnueabihf/sysroot

Assuming some of the other settings in the configuration of the tool-chain build process set those values. See later for the .config file.

The gnueabihf bit means; Gnu, embedded application binary interface, hard-float. The 'unknown' is the vendor string.

Let's actually break that down, it's called a 'tuple' I think:

It doesn't need to actually consist of all of those parts, see the comment above about where the pthread library may be for our desktop x86_64 machine. In that tuple, the vendor string part is missing.

Perhaps we should set our's to:

arm-kitchenwidgets-baremetal-eabihf

Assuming the micro-controller in our microwave oven contains no operating system as such, and certainly not Linux, and it has a hard-float processor.

Now I like to have my toolchains in:

/opt/toolchains/xxx

And generating a tool-chain in one place and then moving it does not work. At best you will end up having to provide the '--sysroot=xxx' switch all the time, and at worst, some soft-links will be broken by moving it.

So, I have to configure my tool-chain builds to install the tool-chain where it will eventually live permanently.

I like to always create a group called 'developers', make myself a member of the group and then make anything in:

/opt

Owned by root:developers, like this:

$ sudo chown -R root:developers /opt/

I like to put kernel source in /opt as well. Not sure if using /opt for these things busts any kind of Linux file-system hierarchy standard but hey, it's my machine.

Canadian Build

For completeness, I will describe this, but it's not something you will need to do every day.

You might need to create a cross-compiler tool-chain to run on one kind of architecture, to compile code for another architecture, and actually build the cross-compiler on yet another.

For example, you might be creating a tool-chain on Linux x86_64, to run on Windows, to compile code for ARM-cortex-a7.

This is called a 'Canadian build'. It's so called because when the term was coined, the political landscape in Canada was dominated by three major parties. I don't know whether that has changed. Personally I have never forgiven the Canadians for Bryan Adams.

I have never created a Canadian build tool-chain. It looks complicated.

Installing crosstool-ng

This is how I installed crosstool-ng on my Debian machine.

At the time of writing the stable version of ct-ng was 1.20.0.

I like to install ct-ng to /opt/crosstool-ng and then a subdirectory of that path named for the version number, so:

/opt/crosstool-ng/1.20.0

You can find the current version from the top of the crosstool-ng page.

In this way I can have different versions of ct-ng and test new features or bleeding-edge git code.

First, install dependencies:

$ sudo apt-get install gperf libtool automake libncurses5-dev bison flex gawk

You may have some or all of the above if you have already installed build-essential.

Next set an environment variable to the version:

$ export CTNGVERSION=1.20.0

Now:

$ wget http://crosstool-ng.org/download/crosstool-ng/crosstool-ng-${CTNGVERSION}.tar.bz2
$ tar xjf crosstool-ng-${CTNGVERSION}.tar.bz2
$ cd crosstool-ng-${CTNGVERSION}
$ ./configure --prefix=/opt/crosstool-ng/${CTNGVERSION}
$ make
$ sudo make install

And you should now have:

/opt/crosstool-ng/1.20.0

Bung it in your path, in whatever way you set that, perhaps in your .bashrc:

export PATH=/opt/crosstool-ng/1.20.0:$PATH

Using crosstool-ng

Example: a Raspberry Pi Tool-chain

Here is a good use of cross-compiling.

When I first started mucking about with the Pi, I needed to compile kernels and kernel modules.

I once did it on a Model B Pi, and if took fifteen hours. Ouch. Not something you want to do every day. Much better to make use of a faster desktop machine.

My x86_64 Debian Wheezy machine with 1.4GHz clock-speed and 8GB RAM takes...fifteen minutes. Not a mega-fast machine by any stretch but 15 minutes is better than 15 hours.

The Raspberry Pi Foundation Tool-chain

It is possible to get a tool-chain thusly:

$ git clone https://github.com/raspberrypi/tools

But it only works on 32-bit machines. I think it's something to do with 32-bit host libraries.

I recently got a couple of version 2 Model B+ Raspberry Pi boards, so I needed to create two tool-chains to run on x86_64, so i could compile stuff for both flavours of Pi in my stable:

  1. For the original Pi, which is arm-cortex-a6
  2. For the 2B+, which is arm-cortex-a7

I think arm-cortex-a7 is also the right architecture for the Banana Pi.

I took a long time looking at virtualbox and vagrant as a way of creating a 32-bit VM in which to run the Pi Foundation tools.

I was struggling to understand at first how to create an arm-cortex-a7 tool-chain, as the Foundation chain only works for arm-cortex-a6.

I found a few downloads which would run on 64-bit. I tried to get both:

VMs going in virtualbox and vagrant. But I could never get any of the 64-bit machines to work correctly. Something about connection timeouts. I will return to virtualbox and vagrant some time in the future.

I only picked two different versions of Ubuntu because these were the first two Ubuntu boxes I found online.

I needed a 64-bit VM because I could only find a 64-bit arm-cortex-a7 cross-compiler tool-chain, and my Debian Wheezy box has a version of the libglibc library older than the downloaded tool-chain required.

I didn't want to get into upgrading the C lib too far in advance of where Wheezy is now. That way madness lies.

Then I discovered that at the Arch Linux ARM site there are crosstool-ng configurations for download for all of these architectures:

Taking the arm-cortex-a7 tool-chain build as an example, create a build place for your tool-chain and then grab configuration files:

$ mkdir ~/armv7tc
$ cd ~/armv7tc
$ wget http://archlinuxarm.org/mirror/development/ct-ng/xtools-dotconfig-[v5|v6|v7] -O .config

Make sure to select either v5, v6 or v7 for the tool-chain you want to build, by editing the '(v5|v6|v7)' in the above. v7 for our tool-chain for the 2B+ Raspberry Pi.

This will download a .config file.

The name of this file, and it's format will be familiar if you have ever compiled a Linux kernel and used the kconfig tools to set-up for the kernel build.

I can recommend the excellent HPR episode by Klaatu on kernel compilation, I don't really want to cover it here.

Just suffice to say that the following command will bring up an ncurses screen which is the same as the one used for kernel build configuration:

$ ct-ng menuconfig

In the directory where you have your .config file.

There is a line in the config file which is prefixed with:

CT_PREFIX_DIR=

You need to set this value to where you want your tool-chain to be installed, for example:

CT_PREFIX_DIR="/opt/toolchains/armv7/${CT_TARGET}"

I think the configuration tool adds the target variable at the end.

A note about accessibility at this point...

I don't find ncurses screens very easy to work with. They are easier on a native console than they are on SSH. It can be done but I often resort to hacking the .config file directly with nano or emacs. Which I don't recommend.

You will not need to change anything else in the .config file if you got it from the Arch Linux ARM site as above.

Now build your tool-chain with:

$ ct-ng build

It will take quite a long time.

Note that it is a good idea to create the directory:

~/cross/src

As doing so will save the tarballs of binutils etc. as they are downloaded for tool-chain generation. This is another setting that can be changed in the .config file.

Next time

I will do another show about how to use the tool-chain, including how to use it to build a Raspberry Pi kernel.

There is a .tar.gz file which goes with this show, which contains: