perf-workshop/scenario5-debug-symbols/README.md

Scenario 5: Debug Symbols - The Missing Map

Learning Objectives

• Understand what debug symbols are and why they matter
• Compare profiling output with and without symbols
• Use perf annotate to see source-level hotspots
• Understand the trade-offs of shipping debug symbols

Background

When you compile C code, the compiler translates your source into machine code. By default, the connection between source lines and machine instructions is lost.

Debug symbols (enabled with -g) preserve this mapping:

• Function names
• Source file names and line numbers
• Variable names and types
• Inline function information

Files

• program.c - A program with nested function calls
• Makefile - Builds nodebug and withdebug versions

Exercise 1: Build Both Versions

make all

Compare file sizes:

make sizes

The withdebug binary is larger due to DWARF debug sections.

Exercise 2: Profile Without Debug Symbols

perf record ./nodebug 500 5000
perf report

You'll see something like:

  45.23%  nodebug  nodebug    [.] 0x0000000000001234
  32.11%  nodebug  nodebug    [.] 0x0000000000001456
  12.45%  nodebug  libm.so    [.] __sin_fma

The hex addresses (0x...) tell you nothing useful!

Exercise 3: Profile With Debug Symbols

perf record ./withdebug 500 5000
perf report

Now you see:

  45.23%  withdebug  withdebug  [.] compute_inner
  32.11%  withdebug  withdebug  [.] compute_middle
  12.45%  withdebug  libm.so    [.] __sin_fma

Much better! You can see which functions are hot.

Exercise 4: Source-Level Annotation

With debug symbols, you can see exactly which lines are hot:

perf record ./withdebug 500 5000
perf annotate --stdio -l compute_inner

The -l flag shows a summary of hot source lines at the top:

Sorted summary for file .../withdebug
----------------------------------------------
   72.75 program.c:29
   27.25 program.c:28

This tells you line 29 (result = result * 1.0001 + 0.0001) consumed ~73% of cycles!

Below the summary, you see interleaved source and assembly:

 Percent | Source code & Disassembly of withdebug
--------------------------------------------------
         : 9    double compute_inner(double x, int iterations) {
         : 10   double result = x;
   27.25 :   130f:   addsd  %xmm0,%xmm1 // program.c:28
   27.67 :   1313:   mulsd  ...         // program.c:29

Note on line numbers: The left margin (9, 10, etc.) shows objdump output line numbers, NOT source file lines. The actual source lines are shown in the summary and in // program.c:XX comments on assembly lines.

Exercise 5: Understanding Symbol Tables

# Show symbols in each binary
make symbols

# Or manually:
nm nodebug | head -20
nm withdebug | head -20

# Show more detail about sections
readelf -S withdebug | grep debug

Debug sections you might see:

• .debug_info - Type information
• .debug_line - Line number tables
• .debug_str - String table
• .debug_abbrev - Abbreviation tables

Exercise 6: Stripping Symbols

Production binaries are often "stripped" to reduce size:

make stripped
ls -l withdebug stripped

# Try to profile the stripped binary
perf record ./stripped 500 5000
perf report

The stripped binary loses symbol information too!

Exercise 7: Separate Debug Files

In production, you can ship stripped binaries but keep debug info separate:

# Extract debug info to separate file
objcopy --only-keep-debug withdebug withdebug.debug

# Strip the binary
strip withdebug -o withdebug.stripped

# Add a link to the debug file
objcopy --add-gnu-debuglink=withdebug.debug withdebug.stripped

# Now perf can find the debug info
perf record ./withdebug.stripped 500 5000
perf report

This is how Linux distros provide -dbg or -debuginfo packages.

Discussion Questions

1. Why don't we always compile with -g?

   • Binary size (can be 2-10x larger)
   • Exposes source structure (security/IP concerns)
   • Some optimizations may be affected (though -O2 -g works well)

2. Does -g affect performance?

   • Generally no: debug info is stored in separate sections
   • Not loaded unless a debugger attaches
   • Some edge cases with frame pointers

3. What about release vs debug builds?

   • Debug build: -O0 -g (no optimization, full debug)
   • Release build: -O2 -g (optimized, with symbols)
   • Stripped release: -O2 then strip

Key Takeaways

1. Always compile with -g during development
2. Debug symbols don't meaningfully affect runtime performance
3. Without symbols, profilers show useless hex addresses
4. Production: ship stripped binaries, keep debug files for crash analysis

Bonus: Flamegraph Generation

# Record with call graph
perf record -g ./withdebug 500 5000

# Generate flamegraph (requires FlameGraph scripts)
perf script | /path/to/FlameGraph/stackcollapse-perf.pl | /path/to/FlameGraph/flamegraph.pl > profile.svg