Show “degrees of real-time” by observing maximum jitter of cyclictest over an upstream kernel
This post shows you “degrees of real-time” by observing maximum jitter of cyclictest over an upstream kernel which is configured and patched in various ways without and with load produced by stress-ng.
Those are the kernels/kernel configurations we’ll have a look at:
|Kernel Version||Patch applied||SCHED Configuration||Notes|
|5.4.109||preempt-rt patch||PREEMPT_NONE (preempt-rt patch not required)||At the time of writing, only a patch against upstream 5.4.106 was available. Unfortunately, the CAN driver crashes with it. This was fixed in upstream 5.4.109. The 5.4.106 patch applied on 5.4.109 without problems and was used here.|
(preempt-rt patch not required)
|5.4.109||preempt-rt patch||PREEMPT (preempt-rt patch not required)||see above|
|5.4.109||preempt-rt patch||PREEMPT_RT (preempt-rt patch required)||see above|
|5.4.109||Ipipe patch||PREEMPT_NONE (but does not matter since we’ll run Xenomai on it)||At the time of writing, only a patch against upstream 5.4.107 was available. Unfortunately, the CAN driver crashes with it. This was fixed in upstream 5.4.109. The 5.4.107 patch did not apply out of the box against 5.4.109, but it was adjusted to apply.|
As already mentioned above we’ll use cyclictest for our benchmarks. We’ll need two versions of cyclictest. We’ll compile one against standard Linux, and the other one against Xenomai. For the graphs below we tried to run one high-priority process periodically every 500 microseconds and measured the jitter.
We’ll use stress-ng to apply stress to the system. The real-time behavior is visible via the worst-case jitter under heavy load. For the “load” test-cases loadavg needs to reach “number of CPUs * 3” to stop measuring.
The worst-case jitter in the graphs to the left looks pretty promising. Unfortunately jitter in an “idle system” does not mean much. Only after we’ll apply a heavy load to the system more realistic worst-case jitter values can be seen. This gives us a much better understanding of how real-time a system actually is. You’ll see those results in the graphs to the right.
PREEMPT_NONE is the scheduler preemption model here. This is possible without any kernel patches. Please note the maximum jitter under load in the graph on the right.
PREEMPT_VOLUNTARY is the preemption model here. This is possible without any kernel patches. Please note the improvement in maximum jitter under load in the graph on the right.
PREEMPT is the preemption model here. This is possible without any kernel patches. Please note the improvement in maximum jitter under load in the graph on the right.
PREEMPT_RT is the scheduler preemption model here. For this, to work we’ll need to apply the preempt-rt kernel patch. Please note the improvement in maximum jitter under load in the graph on the right.
We use a special version of cyclictest which was built against Xenomai here. This means that the preemption model configured here is irrelevant for the test case. Cyclictest does not run over the Linux scheduler.
The improvement in maximum jitter is pretty good and typically a factor of two better than with preempt-rt. The disadvantage of Xenomai compared to preempt-rt is, that preempt-rt is the only real-time Linux solution which can be upstreamed to kernel.org.
As you can see above, depending on kernel configuration and patch level we can reduce the maximum jitter in our test case. The graphs here should give you some initial idea about the options you have. For your own, specific real-time problem you’ll need most likely to come up with custom test cases.
For a similar blog post with the 5.10 Linux kernel, but also test-cases with release and debug kernels check here. For a blog post about the Yocto Project® kernel tooling have a look here. If you want to learn more about how Embedded Linux works plus some Linux with additional real-time, have a look here. To learn more about the Yocto Project® have a look here.