Show “degrees of real-time” by observing maximum jitter of cyclictest over an upstream kernel
Objectives
This post shows you “degrees of real-time” by observing maximum jitter of cyclictest over an upstream kernel which is configured and patched in various ways without and with load produced by stress-ng.
Prerequisites
Kernel/Patches/Configurations
Those are the kernels/kernel configurations we’ll have a look at:
| Kernel Version | Patch applied | SCHED Configuration | Notes |
| 5.4.109 | preempt-rt patch | PREEMPT_NONE (preempt-rt patch not required) | At the time of writing, only a patch against upstream 5.4.106 was available. Unfortunately, the CAN driver crashes with it. This was fixed in upstream 5.4.109. The 5.4.106 patch applied on 5.4.109 without problems and was used here. |
| 5.4.109 | preempt-rt patch | PREEMPT_VOLUNTARY (preempt-rt patch not required) | see above |
| 5.4.109 | preempt-rt patch | PREEMPT (preempt-rt patch not required) | see above |
| 5.4.109 | preempt-rt patch | PREEMPT_RT (preempt-rt patch required) | see above |
| 5.4.109 | Ipipe patch | PREEMPT_NONE (but does not matter since we’ll run Xenomai on it) | At the time of writing, only a patch against upstream 5.4.107 was available. Unfortunately, the CAN driver crashes with it. This was fixed in upstream 5.4.109. The 5.4.107 patch did not apply out of the box against 5.4.109, but it was adjusted to apply. |
cyclictest
As already mentioned above we’ll use cyclictest for our benchmarks. We’ll need two versions of cyclictest. We’ll compile one against standard Linux, and the other one against Xenomai. For the graphs below we tried to run one high-priority process periodically every 500 microseconds and measured the jitter.
stress-ng
We’ll use stress-ng to apply stress to the system. The real-time behavior is visible via the worst-case jitter under heavy load. For the “load” test-cases loadavg needs to reach “number of CPUs * 3” to stop measuring.
Jitter Measurements
The worst-case jitter in the graphs to the left looks pretty promising. Unfortunately jitter in an “idle system” does not mean much. Only after we’ll apply a heavy load to the system more realistic worst-case jitter values can be seen. This gives us a much better understanding of how real-time a system actually is. You’ll see those results in the graphs to the right.
PREEMPT_NONE
PREEMPT_NONE is the scheduler preemption model here. This is possible without any kernel patches. Please note the maximum jitter under load in the graph on the right.
PREEMPT_VOLUNTARY
PREEMPT_VOLUNTARY is the preemption model here. This is possible without any kernel patches. Please note the improvement in maximum jitter under load in the graph on the right.
PREEMPT
PREEMPT is the preemption model here. This is possible without any kernel patches. Please note the improvement in maximum jitter under load in the graph on the right.
PREEMPT_RT
PREEMPT_RT is the scheduler preemption model here. For this, to work we’ll need to apply the preempt-rt kernel patch. Please note the improvement in maximum jitter under load in the graph on the right.
Xenomai
We use a special version of cyclictest which was built against Xenomai here. This means that the preemption model configured here is irrelevant for the test case. Cyclictest does not run over the Linux scheduler.
For this, to work we’ll first patch the Linux kernel with the Ipipe patch. Next, the userspace application needs to be custom-built against Xenomai.
The improvement in maximum jitter is pretty good and typically a factor of two better than with preempt-rt. The disadvantage of Xenomai compared to preempt-rt is, that preempt-rt is the only real-time Linux solution which can be upstreamed to kernel.org.
Conclusion
As you can see above, depending on kernel configuration and patch level we can reduce the maximum jitter in our test case. The graphs here should give you some initial idea about the options you have. For your own, specific real-time problem you’ll need most likely to come up with custom test cases.
For a similar blog post with the 5.10 Linux kernel, but also test-cases with release and debug kernels check here. For a blog post about the Yocto Project® kernel tooling have a look here. If you want to learn more about how Embedded Linux works plus some Linux with additional real-time, have a look here. To learn more about the Yocto Project® have a look here.
