Debug
Debugging latenciesPREEMPT_RT
ftrace - Kernel function tracer
Infrastructure that can be used for debugging or analyzing latencies and performance issues in the kernel.
- Very well documented in Documentation/trace/ftrace.txt
- Negligible overhead when tracing is not enabled at run-time.
- Can be used to trace any kernel function!
Using ftrace
- Tracing information available through the debugfs virtual fs (CONFIG_DEBUG_FS in the Kernel Hacking section)
- Mount this filesystem as follows: mount -t debugfs nodev /sys/kernel/debug
- When tracing is enabled (see the next slides), tracing information is available in /sys/kernel/debug/tracing.
- Check available tracers in /sys/kernel/debug/tracing/available_tracers
Scheduling latency tracer
CONFIG_SCHED_TRACER (Kernel Hacking section)
- Maximum recorded time between waking up a top priority task and its scheduling on a CPU, expressed in us.
- Check that wakeup is listed in /sys/kernel/debug/tracing/available_tracers
- To select, reset and enable this tracer:
echo wakeup > /sys/kernel/debug/tracing/current_tracer
echo 0 > /sys/kernel/debug/tracing/tracing_max_latency
echo 1 > /sys/kernel/debug/tracing/tracing_enabled
- Let your system run, in particular real-time tasks. Dummy example: chrt -f 5 sleep 1
- Disable tracing:
echo 0 > /sys/kernel/debug/tracing/tracing_enabled
Read the maximum recorded latency and the corresponding trace:
cat /sys/kernel/debug/tracing/tracing_max_latency
Exemple of worst-case latencies
Tests done with cyclictest
- x86 Geode 500 Mhz: 96 us
- x86 Pentium Dual-core T4500 2.3 Ghz: 36 us
- ARM Freescale i.MX35 533 Mhz: 120 us
- ARM Marvell 88FR131 1200 Mhz: 54 us See more results at https://www.osadl.org/QA-Farm-Realtime.qa-farm-about.0.html
About real-time support in the standard Linux kernel
- Inside the RT patch, Steven Rostedt, Red Hat, ELC 2013 http://elinux.org/images/b/ba/Elc2013_Rostedt.pdf Video: http://j.mp/1apUtu6
- The Real-Time Linux Wiki: http://rt.wiki.kernel.org “The Wiki Web for the CONFIG_PREEMPT_RT community, and real-time Linux in general.” Contains nice and useful documents!