This three-day class uses hands-on exercises combined with instruction to illustrate the concepts of Linux debugging and profiling. It’s a collection of debugging/profiling chapters of various of my workshops plus some add-ons. Hands-on sessions are performed with free and open-source software like the Yocto Project® SDK on a target board (e.g. i.mx6) which can be purchased/shipped to you on request. This makes it possible for you to work with the course material at home even after the workshop.

• Basic familiarity with using a Linux system (e.g. Ubuntu) as an end-user in user space
• Basic familiarity with a command-line shell
• Basic knowledge of user/kernel space programming with Linux
• Intermediate C programming knowledge
• Ideally, you have basic experience with Embedded Linux (kernel, userspace, root file-system, tools) to follow this course. You should have attended ”Embedded Linux – From System Architecture to Real-Time (5 days)” or ”Introduction to Embedded Linux in Theory and Practice – a Crash Course (3 days)”. (highly recommended)
• It helps to follow the kernel-related stuff if you attended ”Embedded Linux Kernel Internals and Device Drivers (5 days)”
• The sessions mentioned above can be delivered on-site and online. Feel free to ask for dates and pricing here.

People with interest in or who are tasked with the development, bug fixing or evaluation of Embedded Linux systems and want to dig a bit deeper like managers, project managers, software-, development- and systems engineers, field engineers, engineers, . . .

Day 1

Introduction

• Overview | History | Linux Kernel Versioning | Release Cycle | Mainline | Tools | Kernel specifics | Licensing | FLOSS | Standards | Working with free Software | Spelunking | Unix Philosophy

Eval Board

• Booting
  • How does Linux boot on a PC? | How is an embedded system different?
• Booting the target system
  • Boot Sequence | SD card partitions

What’s needed on… (partially optional content)

• host
  • Toolkit
    • The Yocto Project® | Install the Yocto Project® SDK
  • SD card preparation | terminal emulation
    • screen | byobu | minicom
  • servers
    • tftp | NFS
  • root file systems
• target system
  • git | u-boot
    •  scripting | fancy stuff | checkout | config | build | install | variables and commands
  • Flat device tree (fdt)
    • what is a fdt? | device tree compiler | examples
  • kernel
    • checkout | config-kconfig | build-kbuild
  • Flat device tree (fdt)
    • build | install
  • kernel modules
    • build | install

User Space Debugging

• What’s a bug? | Types of bugs | Debugging strategy | errno
• Debugging: Simple Tools
  • lsof | ltrace | strace | netstat | procfs | gcc/binutils | toolchain | ldd | static libs/shared libs | strings | nm | objdump | readelf | addr2line

• Preventing Faults | Modeling/Abstraction | Prototypes | Training | Requirements/Architecture/Design | Firmware Standard
  • Coding Standard | splint | cppcheck

Day 2

• Debugging: Advanced Tools
  • gdb | host gdb | target gdb | gdb remote debugging | gdb remote debugging with GUI
  • Memory: System wide | meminfo | free | vmstat | nmon
  • Memory overflow: What is Mem Munch?
  • Memory: Per Process | ps | smem | procrank | top | htop | atop | memstat
  • Memory allocation: Others | dmalloc | stack smash
  • Valgrind: array | uninitialized memory | memory leak | core | cachegrind | helgrind

Kernel Debugging

• bug | debugging | debugging the kernel | binary search with git | debug support from kernel | syslog
• Debugging by Printing
  • printk | klogd | loglevels | rate limit | debug levels | debug selective subsystems | dynamic debug | issues with printk | printk from user space
• Debugging by Querying
  • debugfs
• oops debugging
  • disassemble | gdb | oops | further reading
• kgdb/kdb | console + debug
• Magic SysRq Key

Day 3

JTAG (optional)

• BDI setup | U-Boot: debugging before memory relocation | Linux Kernel module debugging | Other possibilities?

User Space Profiling/Tracing

• Profiling/Tracing tools
  • syslog | ptrace | strace | ltrace | system timers
  • gprof | gcov | uftrace

• top and similar programs
  • top | htop | powertop | iotop | atop | crash

Kernel Profiling/Tracing

• gcov | lcov | ftrace
  • nop tracer | function tracer | sched_switch tracer (viewer) | function_graph tracer | dynamic tracer | trace-cmd/kernelshark | trace_printk() | Additional Reading
• perf: perf top | hackbench | flame graphs
• Kernel Probes: Kprobes | KretProbes
• BPF: intro | ply

Optimizations/Benchmarks

• bootgraph | bootchart
• Phoronix Test Suite

• ”Embedded Linux – From System Architecture to Real-Time (5 days)”
  • https://rlbl.me/elisa-en-pdf
• ”Refresher to Embedded Linux & Intro to the Yocto Project (5 days)”
  • https://rlbl.me/intely-r-en-pdf
• ”Embedded Linux Kernel Internals and Device Drivers (5 days)”
  • https://rlbl.me/ldd-en-pdf
• ”The Yocto Project – A thorough Overview (4 days)”
  • https://rlbl.me/yocto-r-en-pdf
• ”Introduction to Embedded Linux & Real-Time, bird’s eye view of the Yocto Project (4 days)”
  • https://rlbl.me/entirety-en-pdf
• ”Introduction to Embedded Linux in Theory and Practice – a Crash Course (3 days)”
  • https://rlbl.me/elin-en-pdf
• ”FreeRTOS in Theory and Practice (3 days)”
  • https://rlbl.me/freertos-r-en-pdf