(Embedded) Linux Debugging

(Embedded) Linux Debugging

3-day class

The course content could be tailored to better suit your needs and objectives. If time permits consulting during the training can be included. Please click the public tab to see the public sessions.

Private online
Onsite at your premises
We'll never share your email with anyone else.
Pick Quarter Quarter Status Access Delivery Location Language Timezone Book Until
3rd quarter 2024 private remote virtual German/English (to be defined) UTC+2 (to be defined) Oct 01, 2024
4th quarter 2024 private remote virtual German/English (to be defined) UTC+2 (to be defined) Jan 01, 2025

Content as described below. Starting from three attendees from the same company please check our private sessions.

Public online
Public workshop in classroom
We'll never share your email with anyone else.
Pick Date Date Status Access Delivery Location Language Timezone Book Until
Nov 26, 2024 public remote virtual German/English (to be defined) UTC+2 Oct 17, 2024

This course explains tools and methods for Linux user/kernel space debugging, profiling and tracing.

Course Objectives

We’ll show various approaches which can be used for debugging and profiling software in Linux user and kernel space. Theory and hands-on exercises provide you with the necessary experience to pick the right approach when you need to find and fix faults on your own.

Description

This three-day class uses hands-on exercises combined with instruction to illustrate the concepts of Linux debugging and profiling. It’s a collection of debugging/profiling chapters of various of my workshops plus some add-ons. Hands-on sessions are performed with free and open-source software like the Yocto Project® SDK on a target board (e.g. i.mx6) which can be purchased/shipped to you on request. This makes it possible for you to work with the course material at home even after the workshop.

Prerequisites

Course roadmap

This image has an empty alt attribute; its file name is shedule2.jpg


Who should attend?

People with interest in or who are tasked with the development, bug fixing or evaluation of Embedded Linux systems and want to dig a bit deeper like managers, project managers, software-, development- and systems engineers, field engineers, engineers, . . .

Course Outline

Day 1

Introduction

  • Overview | History | Linux Kernel Versioning | Release Cycle | Mainline | Tools | Kernel specifics | Licensing | FLOSS | Standards | Working with free Software | Spelunking | Unix Philosophy

Eval Board

  • Booting
    • How does Linux boot on a PC? | How is an embedded system different?
  • Booting the target system
    • Boot Sequence | SD card partitions

What’s needed on… (partially optional content)

  • host
    • Toolkit
      • The Yocto Project® | Install the Yocto Project® SDK
    • SD card preparation | terminal emulation
      • screen | byobu | minicom
    • servers
      • tftp | NFS
    • root file systems
  • target system
    • git | u-boot
      •  scripting | fancy stuff | checkout | config | build | install | variables and commands
    • Flat device tree (fdt)
      • what is a fdt? | device tree compiler | examples
    • kernel
      • checkout | config-kconfig | build-kbuild
    • Flat device tree (fdt)
      • build | install
    • kernel modules
      • build | install

User Space Debugging

  • What’s a bug? | Types of bugs | Debugging strategy | errno
  • Debugging: Simple Tools
    • lsof | ltrace | strace | netstat | procfs | gcc/binutils | toolchain | ldd | static libs/shared libs | strings | nm | objdump | readelf | addr2line
  • Preventing Faults | Modeling/Abstraction | Prototypes | Training | Requirements/Architecture/Design | Firmware Standard
    • Coding Standard | splint | cppcheck
Day 2
  • Debugging: Advanced Tools
    • gdb | host gdb | target gdb | gdb remote debugging | gdb remote debugging with GUI
    • Memory: System wide | meminfo | free | vmstat | nmon
    • Memory overflow: What is Mem Munch?
    • Memory: Per Process | ps | smem | procrank | top | htop | atop | memstat
    • Memory allocation: Others | dmalloc | stack smash
    • Valgrind: array | uninitialized memory | memory leak | core | cachegrind | helgrind

Kernel Debugging

  • bug | debugging | debugging the kernel | binary search with git | debug support from kernel | syslog
  • Debugging by Printing
    • printk | klogd | loglevels | rate limit | debug levels | debug selective subsystems | dynamic debug | issues with printk | printk from user space
  • Debugging by Querying
    • debugfs
  • oops debugging
    • disassemble | gdb | oops | further reading
  • kgdb/kdb | console + debug
  • Magic SysRq Key
Day 3

JTAG (optional)

  • BDI setup | U-Boot: debugging before memory relocation | Linux Kernel module debugging | Other possibilities?

User Space Profiling/Tracing

  • Profiling/Tracing tools
    • syslog | ptrace | strace | ltrace | system timers
    • gprof | gcov | uftrace
  • top and similar programs
    • top | htop | powertop | iotop | atop | crash

Kernel Profiling/Tracing

  • gcov | lcov | ftrace
    • nop tracer | function tracer | sched_switch tracer (viewer) | function_graph tracer | dynamic tracer | trace-cmd/kernelshark | trace_printk() | Additional Reading
  • perf: perf top | hackbench | flame graphs
  • Kernel Probes: Kprobes | KretProbes
  • BPF: intro | ply

Optimizations/Benchmarks

  • bootgraph | bootchart
  • Phoronix Test Suite

Course Brochure (pdf)

What customers are saying