Last Updated on January 23, 2025

Show “degrees of real-time” by observing the maximum jitter of cyclictest over an upstream kernel

Objectives

This post shows you “degrees of real-time” by observing the maximum jitter of cyclictest over an upstream kernel configured and patched in various ways without and with load (produced by stress-ng).

Prerequisites

Kernel/Patches/Configurations

Those are the kernels/kernel configurations we’ll have a look at:

cyclictest

As mentioned above, we’ll use cyclictest for our benchmarks. For the graphs below, we tried running one high-priority process periodically every 500 microseconds and measuring jitter/latency.

stress-ng

We use stress-ng to apply stress to the system. The real-time behavior is visible via the worst-case jitter under heavy load. For the “load” test cases, loadavg needs to reach “number of CPUs * 3” to stop measuring.

Jitter Measurements

The worst-case jitter in the graphs to the left looks pretty promising. Unfortunately, jitter in an “idle system” does not mean much. Only after we apply a heavy load to the system can more realistic worst-case jitter values be seen. This gives us a significantly better understanding of our system’s real-time behavior. You’ll see those results in the graphs to the right. Looking at the graphs, we can deduce that the kernel configuration is crucial.

PREEMPT_NONE (RELEASE)

PREEMPT_NONE is the scheduler preemption model here. This is possible without any kernel patches. Please note that the maximum jitter under load in the graph on the right got worse, as expected.

PREEMPT_NONE (DEBUG)

PREEMPT_NONE is the scheduler preemption model here. This is possible without any kernel patches. Please note the significant increase in maximum jitter compared to the release version on an idle system. Funny enough, the debug version shows less jitter under load than the release version, which is unexpected.

PREEMPT_VOLUNTARY (RELEASE)

PREEMPT_VOLUNTARY is the preemption model here. This is possible without any kernel patches. Please note that the jitter under load in the release version got worse than PREEMPT_NONE, which is unexpected.

PREEMPT_VOLUNTARY (DEBUG)

PREEMPT_VOLUNTARY is the preemption model here. This is possible without any kernel patches. Please note that the debug version under load shows less jitter than the release version, which is unexpected.

PREEMPT (RELEASE)

PREEMPT is the preemption model here. This is possible without any kernel patches. Please note the improved maximum jitter under load in the graph on the right compared to the previous preemption models.

Please note: loadavg is mostly below 1 on an idle system here, as one might expect.

PREEMPT (DEBUG)

PREEMPT is the preemption model here. This is possible without any kernel patches. Please note that the maximum jitter is worse with the debug version than with the release version, as one might expect.

PREEMPT_RT (RELEASE)

PREEMPT_RT is the preemption model here. The preempt-rt kernel patch was applied for this to work. Please note the overall improvement in maximum jitter.

Please note: the loadavg rarely goes below 1 and goes up to 4, which is not expected, and I also don’t have a good explanation for it. Ideas are welcome!

Conclusion

As you can see above, depending on kernel configuration and patch level, we can influence the maximum jitter in our test case. The graphs here should give you some initial idea about your options. For your specific real-time problem, you’ll most likely need to come up with custom test cases. It is important to note that you need to be extra careful with your kernel configuration not to significantly increase the worst-case jitter.

For a similar blog post with the 6.6 Linux kernel, including Xenomai test cases, check here. If you want to read about the Yocto Project® kernel tooling, check here. If you want to learn how Embedded Linux works plus some Linux with additional real-time, click here. To learn more about the Yocto Project® take a look here.