193 lines
4.9 KiB
Markdown
193 lines
4.9 KiB
Markdown
# Scenario 5: Debug Symbols - The Missing Map
|
|
|
|
## Learning Objectives
|
|
- Understand what debug symbols are and why they matter
|
|
- Compare profiling output with and without symbols
|
|
- Use `perf annotate` to see source-level hotspots
|
|
- Understand the trade-offs of shipping debug symbols
|
|
|
|
## Background
|
|
|
|
When you compile C code, the compiler translates your source into machine code.
|
|
By default, the connection between source lines and machine instructions is lost.
|
|
|
|
**Debug symbols** (enabled with `-g`) preserve this mapping:
|
|
- Function names
|
|
- Source file names and line numbers
|
|
- Variable names and types
|
|
- Inline function information
|
|
|
|
## Files
|
|
- `program.c` - A program with nested function calls
|
|
- `Makefile` - Builds `nodebug` and `withdebug` versions
|
|
|
|
## Exercise 1: Build Both Versions
|
|
|
|
```bash
|
|
make all
|
|
```
|
|
|
|
Compare file sizes:
|
|
```bash
|
|
make sizes
|
|
```
|
|
|
|
The `withdebug` binary is larger due to DWARF debug sections.
|
|
|
|
## Exercise 2: Profile Without Debug Symbols
|
|
|
|
```bash
|
|
perf record ./nodebug 500 5000
|
|
perf report
|
|
```
|
|
|
|
You'll see something like:
|
|
```
|
|
45.23% nodebug nodebug [.] 0x0000000000001234
|
|
32.11% nodebug nodebug [.] 0x0000000000001456
|
|
12.45% nodebug libm.so [.] __sin_fma
|
|
```
|
|
|
|
The hex addresses (`0x...`) tell you nothing useful!
|
|
|
|
## Exercise 3: Profile With Debug Symbols
|
|
|
|
```bash
|
|
perf record ./withdebug 500 5000
|
|
perf report
|
|
```
|
|
|
|
Now you see:
|
|
```
|
|
45.23% withdebug withdebug [.] compute_inner
|
|
32.11% withdebug withdebug [.] compute_middle
|
|
12.45% withdebug libm.so [.] __sin_fma
|
|
```
|
|
|
|
Much better! You can see which functions are hot.
|
|
|
|
## Exercise 4: Source-Level Annotation
|
|
|
|
With debug symbols, you can see exactly which lines are hot:
|
|
|
|
```bash
|
|
perf record ./withdebug 500 5000
|
|
perf annotate --stdio -l compute_inner
|
|
```
|
|
|
|
The `-l` flag shows a summary of hot source lines at the top:
|
|
|
|
```
|
|
Sorted summary for file .../withdebug
|
|
----------------------------------------------
|
|
72.75 program.c:29
|
|
27.25 program.c:28
|
|
```
|
|
|
|
This tells you line 29 (`result = result * 1.0001 + 0.0001`) consumed ~73% of cycles!
|
|
|
|
Below the summary, you see interleaved source and assembly:
|
|
|
|
```
|
|
Percent | Source code & Disassembly of withdebug
|
|
--------------------------------------------------
|
|
: 9 double compute_inner(double x, int iterations) {
|
|
: 10 double result = x;
|
|
27.25 : 130f: addsd %xmm0,%xmm1 // program.c:28
|
|
27.67 : 1313: mulsd ... // program.c:29
|
|
```
|
|
|
|
**Note on line numbers**: The left margin (9, 10, etc.) shows objdump output
|
|
line numbers, NOT source file lines. The actual source lines are shown in the
|
|
summary and in `// program.c:XX` comments on assembly lines.
|
|
|
|
## Exercise 5: Understanding Symbol Tables
|
|
|
|
```bash
|
|
# Show symbols in each binary
|
|
make symbols
|
|
|
|
# Or manually:
|
|
nm nodebug | head -20
|
|
nm withdebug | head -20
|
|
|
|
# Show more detail about sections
|
|
readelf -S withdebug | grep debug
|
|
```
|
|
|
|
Debug sections you might see:
|
|
- `.debug_info` - Type information
|
|
- `.debug_line` - Line number tables
|
|
- `.debug_str` - String table
|
|
- `.debug_abbrev` - Abbreviation tables
|
|
|
|
## Exercise 6: Stripping Symbols
|
|
|
|
Production binaries are often "stripped" to reduce size:
|
|
|
|
```bash
|
|
make stripped
|
|
ls -l withdebug stripped
|
|
|
|
# Try to profile the stripped binary
|
|
perf record ./stripped 500 5000
|
|
perf report
|
|
```
|
|
|
|
The stripped binary loses symbol information too!
|
|
|
|
## Exercise 7: Separate Debug Files
|
|
|
|
In production, you can ship stripped binaries but keep debug info separate:
|
|
|
|
```bash
|
|
# Extract debug info to separate file
|
|
objcopy --only-keep-debug withdebug withdebug.debug
|
|
|
|
# Strip the binary
|
|
strip withdebug -o withdebug.stripped
|
|
|
|
# Add a link to the debug file
|
|
objcopy --add-gnu-debuglink=withdebug.debug withdebug.stripped
|
|
|
|
# Now perf can find the debug info
|
|
perf record ./withdebug.stripped 500 5000
|
|
perf report
|
|
```
|
|
|
|
This is how Linux distros provide `-dbg` or `-debuginfo` packages.
|
|
|
|
## Discussion Questions
|
|
|
|
1. **Why don't we always compile with `-g`?**
|
|
- Binary size (can be 2-10x larger)
|
|
- Exposes source structure (security/IP concerns)
|
|
- Some optimizations may be affected (though `-O2 -g` works well)
|
|
|
|
2. **Does `-g` affect performance?**
|
|
- Generally no: debug info is stored in separate sections
|
|
- Not loaded unless a debugger attaches
|
|
- Some edge cases with frame pointers
|
|
|
|
3. **What about release vs debug builds?**
|
|
- Debug build: `-O0 -g` (no optimization, full debug)
|
|
- Release build: `-O2 -g` (optimized, with symbols)
|
|
- Stripped release: `-O2` then `strip`
|
|
|
|
## Key Takeaways
|
|
|
|
1. **Always compile with `-g` during development**
|
|
2. **Debug symbols don't meaningfully affect runtime performance**
|
|
3. **Without symbols, profilers show useless hex addresses**
|
|
4. **Production: ship stripped binaries, keep debug files for crash analysis**
|
|
|
|
## Bonus: Flamegraph Generation
|
|
|
|
```bash
|
|
# Record with call graph
|
|
perf record -g ./withdebug 500 5000
|
|
|
|
# Generate flamegraph (requires FlameGraph scripts)
|
|
perf script | /path/to/FlameGraph/stackcollapse-perf.pl | /path/to/FlameGraph/flamegraph.pl > profile.svg
|
|
```
|