Embedded Linux – What kernel repo/version to use?

Objectives

In this article, I’m going to tell you about a tricky subject. The topic which came up in one of my workshops was “Which Linux kernel version should be used for an industrial embedded device?” The customer decided to start with a 5.12.x kernel.

Prerequisites

At the time of writing (2021), some attendees of one of my workshops started a new headless product based on an i.mx6. This applications processor saw the light of the day in 2011/2012. However, despite the fact that this silicon is more than 10 years old it will be supported until 2027 as you can see here:

Clearly, a big advantage of using an old chip like the i.mx6 is mature kernel support. A disadvantage might be that its old technology.

Please read Greg’s blog post What Stable Kernel Should I Use? and Robert’s blog post What Kernel Should I Use (Embedded Edition) to get more context to this topic.

This should help you to better understand my comments and own conclusions with respect to those blog posts.

What issue do we try to tackle?

How to interpret what Greg and Robert say: As opposed to Greg and Robert I am not making money from upstreaming stuff to kernel.org. Let us keep in mind: the “i.mx6” family is pretty well supported in the upstream kernel. Especially when you don’t need multimedia, but just headless support.

Which kernel to pick?

That’s what kernel.org currently (Oct 2021) shows:

Here’s Greg’s shortlist of what kernel you should use, ranked from best to worst options. He’ll go into the details of all of these below, but if you just want the summary of all of this, here it is:

Hierarchy of what kernel to use, from best solution to worst according to Greg’s post

• Supported kernel from your favorite Linux distribution
• Latest stable release
• Latest LTS release
• An older LTS release that is still being maintained

This would mean the following:

• Supported kernel from your favorite Linux distribution
  • the preferred kernel version is the one that comes with poky upstream:
    • 5.14.x – latest stable,
    • 5.10.x – latest LTS.
    • This requires some extra work to run those on your custom board.
• Latest stable release
  • 5.14.19 as the latest stable
• Latest LTS release
  • 5.10.70, which is the latest LTS.
• An older LTS release that is still being maintained
  • Last, but not least is an older LTS, which means 5.4.150.

No matter whether you plan to use LTS or the latest stable plan for an update once per month. It seems like the third day of each month is a good date for a kernel update.

What kernel to never use

• Unmaintained kernel release

Apparently, there are many unmaintained kernels out there which you should not use like 5.12.x, 5.13.x, 5.15-rc3,…

Kernel statistics

You can extract from the kernel git repository the release date for each kernel version. So let’s say you do something like that:

git tag -l --sort=-creatordate --format='%(creatordate:short):  %(refname:short)' | grep v5.10

In the table above you see in bold the recommended kernel versions. You just need to decide whether you prefer LTS or the latest stable.

If you use the Yocto Project® I would recommend to start with a default kernel config that comes from kernel.org. You can add configuration fragments over it in order to configure the kernel as you need it. Like this, it is pretty straightforward to make your kernel updates, since you practically just need to adjust the new branch and commit id.

Just for fun, I upgraded from my good old trusty 5.10.27 LTS kernel to stable 5.14.9 without any major surprises as long as you run it with a rootfs from an SD card.

The big surprise was, that it was not possible anymore to boot with a rootfs over nfs out of the box. And to find out why was a lengthy procedure with lots of trial and error and guesswork.

Building the new kernel went flawlessly and in runtime, I just observed that more kernel threads were created with the 5.14.9 kernel compared to the 5.10.27 LTS.

Generally speaking, an LTS kernel might make your life a bit easier with respect to surprises caused by the upgrade.

In case you need to use “out-of-tree” patches like it’s the case with preempt-rt, xenomai, evl, or something similar, LTS (and sometimes not event the latest version) is the way to go. As a matter of fact you might not even be able to find matching patches for each and every latest stable kernel version. On top of that it might get more tricky to create your own patches against stable kernel versions compared to crafting your own against an LTS version. Preempt-rt patches seem only to be actively maintained for LTS kernels.

Conclusion

I would give the latest LTS kernel a try. In case you want to check out a blog post about the Yocto Project® kernel tooling have a look here. In order to switch from a workflow with “bitbake” to a developer workflow you can use an SDK to build the kernel. If you want to learn more about how Embedded Linux works have a look here. To learn more about the Yocto Project® have a look here.