Embedded Linux Kernel Internals and Device Drivers

In this five-day course, the concepts of Linux kernel internals and device driver development are illustrated through hands-on exercises combined with instructions. The practical exercises are carried out with free and open-source software such as the Yocto Project® software development kit (SDK) on a target board (e.g., i.mx8). The hardware can be purchased and will be shipped to you on request. This enables you to work with the course material at your premises even after the workshop. The training is structured in such a way that you can quickly familiarize yourself with the subject. We describe the processes, concepts, and commands necessary to write Linux device drivers through a combination of theory and hands-on exercises. Learn from an expert trainer and take home the knowledge and ability to effectively use it in your own embedded development project.

• Basic familiarity with using a Linux system (e.g., Ubuntu) as an end-user in user space
• Basic familiarity with a command-line shell
• Basic knowledge of Linux user space and application development
• Intermediate C programming knowledge
• You should have attended ”Embedded Linux – From System Architecture to Real-Time (5 days)” or ”Introduction to Embedded Linux in Theory and Practice – a Crash Course (3 days)” (strongly recommended!)
• …except if you are already familiar with the material presented in those courses.
• To follow this course, you should have experience with embedded Linux (how to build the kernel, user space, root file system, and use the tools).
• The sessions mentioned above can be delivered on-site and online. You are welcome to ask for dates and pricing here.

People with an interest in or who are tasked with the development or evaluation of developing (embedded) Linux device drivers. The workshop is aimed at managers, software developers, system engineers, testers, administrators, engineers, and other parties interested in (embedded) Linux device drivers.

Day 1

Introduction

• A quick history of Linux | licensing | kernel versioning | release cycle | kernel trees | mainline | kernel-space compared to user-space | difference between mechanism and policy | kernel driver alternatives | Real-Time-Operating-Systems (RTOS) in contrast to Linux

Memory Management

• virtual memory versus physical memory | memory allocation in kernel | pages | zones | API | slab

Kernel Patch

• patch life cycle | git | kernel source code | create patch | check patch | fix patch | ship the patch | audited source code (git)

Day 2

Kernel Modules

• Get kernel sources | configure kernel | build kernel | install kernel | kernel module (statically linked | loaded at run-time) | __init/__exit | licensing | tainted module/kernel | EXPORT_SYMBOL | out of tree.ko makefile | module-init-tools | put the module in the kernel tree | Kconfig | parameter passing | sparse

Character Drivers

• architecture | user/kernel interface | I/O subsystem | virtual file system (VFS) | sysfs (devices | bus | drivers | classes) | kobject | ktype | kset | Linux Kernel Driver Model | device files
• Character Driver
  • Driver initialization | Registration of Character Driver | Driver opening | Driver release | cdev | cdev_add | cdev_del | … | major/minor numbers | udev | udevmonitor | udevadm

Day 3

Character Driver Functions (advanced)

• ioctl | unlocked_ioctl | compat_ioctl | user-space API | kernel-space API | process life cycle | sleeping/blocking | sleeping/waking up | wait queue | thundering herd | poll/select

• bug | debugging | debugging the kernel | binary search with git | debug support from the kernel | printk | syslogd | klogd | loglevels | rate limit | debug levels | debug selective subsystems | debugging by querying | debugfs | oops debugging | programmatically asserting an oops | MagicSysRqKey | kgdb/kdb | JTAG (optional)

Day 4

Kernel Profiling/Tracing

• gcov|lcov|oprofile (deprecated and replaced by perf) |ftrace
  • nop tracer | function tracer | sched_switch tracer (viewer) | function_graph tracer | dynamic tracer
• perf| LTTng | Kernel Probes
  • Kprobes | JProbes | KretProbes (partially deprecated)

Interrupts

• interrupts instead of polling | interrupt | program sections | reentrancy | events | interrupt handler | shared interrupt handler | interrupt flow | interrupt control

Deferring Work

• top/bottom halves | Softirqs | Tasklets | Workqueues | Threaded interrupts

Day 5

Concurrency

• critical region/section | atomic | race condition | synchronization | locking | locking solutions | deadlock | contention | What needs to be locked?
• atomic operations | spin locks | reader-writer spin locks | semaphore | binary semaphore | mutex | reader-writer semaphore | completion variables | sequential locks | disable Preemption | sequential order, and Hardware/Software barriers

Time

• HZ | jiffies | big/small delays | kernel timers | high-resolution timers

Hardware I/O

• I/O Ports | I/O Memory | How to deal with side effects of accessing registers?

User-Kernel Communication

• put(get)_user() | copy_to(from)_user() | Kernel I/O | memory mapping | procfs | sysfs | debugfs | relayfs | netlink | ioctl

Portability

• word size | opaque data types | signed/unsigned char | data alignment (padding) | integral promotion | code reuse | endianness | system tick | page size | In which order are software instructions executed? | Symmetric Multi Processing (SMP) | Preemption | High Memory

• ”Embedded Linux Kernel Internals and Device Drivers (5 days)”
  • https://rlbl.me/ldd-en-pdf

• ”Embedded Linux – From System Architecture to Real-Time (5 days)”
  • https://rlbl.me/elisa-en-pdf
• ”Refresher to Embedded Linux & Intro to the Yocto Project (5 days)”
  • https://rlbl.me/intely-r-en-pdf
• ”The Yocto Project – A thorough Overview (4 days)”
  • https://rlbl.me/yocto-r-en-pdf
• ”Introduction to Embedded Linux & Real-Time, bird’s eye view of the Yocto Project (4 days)”
  • https://rlbl.me/entirety-en-pdf
• ”Introduction to Embedded Linux in Theory and Practice – a Crash Course (3 days)”
  • https://rlbl.me/elin-en-pdf
• ”(Embedded) Linux debugging (3 days)”
  • https://rlbl.me/lindeb-r-en-pdf
• ”FreeRTOS in Theory and Practice (3 days)”
  • https://rlbl.me/freertos-r-en-pdf