Last Updated on March 29, 2026

Objectives

This post shows you “degrees of real-time” by observing the maximum jitter of cyclictest over an upstream kernel configured and patched in various ways without and with load (produced by stress-ng), illustrating how to show “degrees of real-time” by observing the maximum jitter of cyclictest over an upstream kernel.

Prerequisites

Kernel/Patches/Configurations

Those are the kernels/kernel configurations we’ll take a look at:

cyclictest

As mentioned above, we’ll use cyclictest for our benchmarks. For the graphs below, we tried running one high-priority process periodically every 500 microseconds and measuring jitter/latency.

stress-ng

We use stress-ng to apply stress to the system. The real-time behavior is visible via the worst-case jitter under heavy load. For the “load” test cases, loadavg needs to reach “number of CPUs * 2” to stop measuring.

Jitter Measurements

The worst-case jitter in the graphs to the left looks pretty promising. Unfortunately, jitter in an “idle system” does not mean much. Average and especially the worst-case jitter after we apply a heavy load to the system show more realistic values. This gives us a significantly better understanding of our system’s real-time behavior. You’ll see those results in the graphs to the right. Looking at the graphs, we can deduce that the proper kernel configuration is crucial.

PREEMPT_NONE (DEBUG)

PREEMPT_NONE is the scheduler preemption model here. This is a configuration option of a Linux kernel without any patches applied. Please note that the maximum jitter under load in the graph on the right got worse, as expected. The small difference in the worst case jitter between the graph on the left and the right is quite unusual. One would expect a much smaller worst case jitter value on the graph to the left.

PREEMPT_NONE (RELEASE)

PREEMPT_NONE is the scheduler preemption model here. This is possible without any kernel patches. Please note the significant decrease in maximum jitter compared to the debug version on an idle system. Funny enough, the debug version shows less jitter under load than the release version, which is unexpected. A similar behavior was observed starting with the 6.6.x kernel. It might be caused by a scheduler change introduced in the 6.6.x kernel. The “Completely Fair Scheduler” (CFS) was replaced by the “Earliest Eligible Virtual Deadline First” (EEVDF) scheduler.

PREEMPT_NONE/EVL PATCHED KERNEL (RELEASE)

What we can see above is similar to the previous graphs. It was measured with an EVL patched kernel, but instead of EVL the PREEMPT_NONE scheduler was selected.

PREEMPT_VOLUNTARY (DEBUG)

PREEMPT_VOLUNTARY is the preemption model here. This is possible without any kernel patches. Please note that the debug version under load shows less jitter than the release version (below), which is unexpected.

PREEMPT_VOLUNTARY (RELEASE)

PREEMPT_VOLUNTARY is the preemption model here. This is possible without any kernel patches. Please note that the jitter under load in the release version got worse than PREEMPT_NONE/EVL, which is unexpected but better compared against PREEMPT_NONE.

PREEMPT (DEBUG)

PREEMPT is the preemption model here. This is possible without any kernel patches. Please note that the debug version under load shows less jitter than the release version (below), which is unexpected.

PREEMPT (RELEASE)

PREEMPT is the preemption model here. This is possible without any kernel patches. Please note that the maximum jitter is better with the debug version than with the release version, which is unexpected. Anyway, the maximum jitter under load is better than with PREEMPT_NONE and PREEMPT_VOLUNTARY here.

PREEMPT_RT (RELEASE)

PREEMPT_RT is the used in the kernel configuration here. The preempt-rt kernel patch was applied for this to work. Please note the overall improvement in maximum jitter.

EVL (RELEASE)

EVL is used in the kernel configuration as the preemption model here. The EVL kernel patch was applied for this to work. Please note the overall improvement in maximum jitter.

Look at the numbers

Average jitter

Analyzing the average jitter of release versions under load, everything looks as expected. The only weird thing is, that the debug versions of kernels without patches without load perform worse than the release versions.

Maximum jitter

Analyzing the maximum jitter of release versions under load, EVL/PREEMPT_RT/PREEMPT look as expected. For some reason, the EVL patched kernel with a PREEMPT_NONE scheduler behaves better than PREEMPT_VOLUNTARY, which seems a bit off. What can be seen is, that the real-time kernels behave significantly better than the others.

Conclusion

As you can see above, depending on kernel configuration and patch level, we can influence the maximum jitter in our test case. The graphs here should give you some initial idea about your options. For your specific real-time problem, you’ll most likely need to come up with custom test cases. It is important to note that you need to be extra careful with your kernel configuration not to significantly increase the worst-case jitter.

For a similar blog post with the 6.6 Linux kernel, including Xenomai test cases, check here and here. If you want to read about the Yocto Project® kernel tooling, check here. If you want to learn how Embedded Linux works plus some Linux with additional real-time, click here. To learn more about the Yocto Project® take a look here.