180 lines
4.4 KiB
Markdown
180 lines
4.4 KiB
Markdown
# Scenario 5: Debug Symbols - The Missing Map
|
|
|
|
## Learning Objectives
|
|
- Understand what debug symbols are and why they matter
|
|
- Compare profiling output with and without symbols
|
|
- Use `perf annotate` to see source-level hotspots
|
|
- Understand the trade-offs of shipping debug symbols
|
|
|
|
## Background
|
|
|
|
When you compile C code, the compiler translates your source into machine code.
|
|
By default, the connection between source lines and machine instructions is lost.
|
|
|
|
**Debug symbols** (enabled with `-g`) preserve this mapping:
|
|
- Function names
|
|
- Source file names and line numbers
|
|
- Variable names and types
|
|
- Inline function information
|
|
|
|
## Files
|
|
- `program.c` - A program with nested function calls
|
|
- `Makefile` - Builds `nodebug` and `withdebug` versions
|
|
|
|
## Exercise 1: Build Both Versions
|
|
|
|
```bash
|
|
make all
|
|
```
|
|
|
|
Compare file sizes:
|
|
```bash
|
|
make sizes
|
|
```
|
|
|
|
The `withdebug` binary is larger due to DWARF debug sections.
|
|
|
|
## Exercise 2: Profile Without Debug Symbols
|
|
|
|
```bash
|
|
perf record ./nodebug 500 5000
|
|
perf report
|
|
```
|
|
|
|
You'll see something like:
|
|
```
|
|
45.23% nodebug nodebug [.] 0x0000000000001234
|
|
32.11% nodebug nodebug [.] 0x0000000000001456
|
|
12.45% nodebug libm.so [.] __sin_fma
|
|
```
|
|
|
|
The hex addresses (`0x...`) tell you nothing useful!
|
|
|
|
## Exercise 3: Profile With Debug Symbols
|
|
|
|
```bash
|
|
perf record ./withdebug 500 5000
|
|
perf report
|
|
```
|
|
|
|
Now you see:
|
|
```
|
|
45.23% withdebug withdebug [.] compute_inner
|
|
32.11% withdebug withdebug [.] compute_middle
|
|
12.45% withdebug libm.so [.] __sin_fma
|
|
```
|
|
|
|
Much better! You can see which functions are hot.
|
|
|
|
## Exercise 4: Source-Level Annotation
|
|
|
|
With debug symbols, you can see exactly which lines are hot:
|
|
|
|
```bash
|
|
perf record ./withdebug 500 5000
|
|
perf annotate compute_inner
|
|
```
|
|
|
|
This shows the source code with cycle counts per line!
|
|
|
|
```
|
|
│ double compute_inner(double x, int iterations) {
|
|
│ double result = x;
|
|
1.23 │ for (int i = 0; i < iterations; i++) {
|
|
45.67 │ result = sin(result) * cos(result) + sqrt(fabs(result));
|
|
23.45 │ result = result * 1.0001 + 0.0001;
|
|
│ }
|
|
│ return result;
|
|
│ }
|
|
```
|
|
|
|
## Exercise 5: Understanding Symbol Tables
|
|
|
|
```bash
|
|
# Show symbols in each binary
|
|
make symbols
|
|
|
|
# Or manually:
|
|
nm nodebug | head -20
|
|
nm withdebug | head -20
|
|
|
|
# Show more detail about sections
|
|
readelf -S withdebug | grep debug
|
|
```
|
|
|
|
Debug sections you might see:
|
|
- `.debug_info` - Type information
|
|
- `.debug_line` - Line number tables
|
|
- `.debug_str` - String table
|
|
- `.debug_abbrev` - Abbreviation tables
|
|
|
|
## Exercise 6: Stripping Symbols
|
|
|
|
Production binaries are often "stripped" to reduce size:
|
|
|
|
```bash
|
|
make stripped
|
|
ls -l withdebug stripped
|
|
|
|
# Try to profile the stripped binary
|
|
perf record ./stripped 500 5000
|
|
perf report
|
|
```
|
|
|
|
The stripped binary loses symbol information too!
|
|
|
|
## Exercise 7: Separate Debug Files
|
|
|
|
In production, you can ship stripped binaries but keep debug info separate:
|
|
|
|
```bash
|
|
# Extract debug info to separate file
|
|
objcopy --only-keep-debug withdebug withdebug.debug
|
|
|
|
# Strip the binary
|
|
strip withdebug -o withdebug.stripped
|
|
|
|
# Add a link to the debug file
|
|
objcopy --add-gnu-debuglink=withdebug.debug withdebug.stripped
|
|
|
|
# Now perf can find the debug info
|
|
perf record ./withdebug.stripped 500 5000
|
|
perf report
|
|
```
|
|
|
|
This is how Linux distros provide `-dbg` or `-debuginfo` packages.
|
|
|
|
## Discussion Questions
|
|
|
|
1. **Why don't we always compile with `-g`?**
|
|
- Binary size (can be 2-10x larger)
|
|
- Exposes source structure (security/IP concerns)
|
|
- Some optimizations may be affected (though `-O2 -g` works well)
|
|
|
|
2. **Does `-g` affect performance?**
|
|
- Generally no: debug info is stored in separate sections
|
|
- Not loaded unless a debugger attaches
|
|
- Some edge cases with frame pointers
|
|
|
|
3. **What about release vs debug builds?**
|
|
- Debug build: `-O0 -g` (no optimization, full debug)
|
|
- Release build: `-O2 -g` (optimized, with symbols)
|
|
- Stripped release: `-O2` then `strip`
|
|
|
|
## Key Takeaways
|
|
|
|
1. **Always compile with `-g` during development**
|
|
2. **Debug symbols don't meaningfully affect runtime performance**
|
|
3. **Without symbols, profilers show useless hex addresses**
|
|
4. **Production: ship stripped binaries, keep debug files for crash analysis**
|
|
|
|
## Bonus: Flamegraph Generation
|
|
|
|
```bash
|
|
# Record with call graph
|
|
perf record -g ./withdebug 500 5000
|
|
|
|
# Generate flamegraph (requires FlameGraph scripts)
|
|
perf script | /path/to/FlameGraph/stackcollapse-perf.pl | /path/to/FlameGraph/flamegraph.pl > profile.svg
|
|
```
|