Show “degrees of real-time” by observing maximum jitter of cyclictest over an upstream kernel

Objectives

This post shows you “degrees of real-time” by observing maximum jitter of cyclictest over an upstream kernel which is configured and patched in various ways without and with load produced by stress-ng.

Prerequisites

Kernel/Patches/Configurations

Those are the kernels/kernel configurations we’ll have a look at:

cyclictest

As already mentioned above we’ll use cyclictest for our benchmarks. For the graphs below we tried to run one high-priority process periodically every 500 microseconds and measured the jitter.

stress-ng

We use stress-ng to apply stress to the system. The real-time behavior is visible via the worst-case jitter under heavy load. For the “load” test-cases loadavg needs to reach “number of CPUs * 3” to stop measuring.

Jitter Measurements

The worst-case jitter in the graphs to the left looks pretty promising. Unfortunately jitter in an “idle system” does not mean much. Only after we’ll apply a heavy load to the system more realistic worst-case jitter values can be seen. This gives us a much better understanding of how real-time a system actually is. You’ll see those results in the graphs to the right. If we compare release (default config) and debug (lots of debug stuff added) versions of the same kernel we’ll see that the kernel configuration is essential for worst-case jitter.

PREEMPT_NONE (RELEASE)

PREEMPT_NONE is the scheduler preemption model here. This is possible without any kernel patches. Please note the maximum jitter under load in the graph on the right.

PREEMPT_NONE (DEBUG)

PREEMPT_NONE is the scheduler preemption model here. This is possible without any kernel patches. Please note the significant increase in maximum jitter under load in the graph on the right. It is caused by changes to the kernel configuration.

PREEMPT_VOLUNTARY (RELEASE)

PREEMPT_VOLUNTARY is the preemption model here. This is possible without any kernel patches. Please note the improvement in maximum jitter under load in the graph on the right.

PREEMPT (RELEASE)

PREEMPT is the preemption model here. This is possible without any kernel patches. Please note the improvement in maximum jitter under load in the graph on the right.

PREEMPT_RT (RELEASE)

PREEMPT_RT is the preemption model here. For this, to work we’ll need to apply the preempt-rt kernel patch. Please note the improvement in maximum jitter under load in the graph on the right.

PREEMPT_RT (DEBUG)

Please note the significant increase in maximum jitter under load in the graph on the right. It is caused by changes to the kernel configuration.

Conclusion

As you can see above, depending on kernel configuration and patch level we can reduce the maximum jitter in our test case. The graphs here should give you some initial idea about the options you have. For your own, specific real-time problem you’ll need most likely to come up with custom test cases. It is important to note that you need to be extra careful with your kernel configuration not to significantly increase the worst-case jitter.

For a similar blog post with the 5.4 Linux kernel, but also Xenomai test-cases check here. If you want to read about the Yocto Project® kernel tooling have a look here. If you want to learn more about how Embedded Linux works plus some Linux with additional real-time, have a look here. To learn more about the Yocto Project® have a look here.