# Scenario 6: USDT Probes - Custom Instrumentation

## Learning Objectives
- Understand the difference between static and dynamic probes
- Add USDT probes to C code
- Trace probes with bpftrace
- See how probes enable production debugging

## Background

**Dynamic probes** (like `perf probe`): Added at runtime, can break on any function
**Static probes** (USDT): Compiled into the binary, designed by the developer

USDT probes are:
- Nearly zero overhead when not traced
- Stable API for debuggers/tracers
- Self-documenting: probe names describe what's happening
- Used by Python, Ruby, MySQL, PostgreSQL, and many other projects

## Prerequisites

```bash
# Install USDT support and bpftrace
sudo apt install systemtap-sdt-dev bpftrace
```

## Files
- `server.c` - Simulated server with USDT probes at key points

## Exercise 1: Build and Run

```bash
make
./server 3 10
```

This processes 3 batches of 10 requests each.

## Exercise 2: List the Probes

```bash
# Using readelf
readelf -n ./server | grep -A2 stapsdt

# Or using perf
sudo perf probe -x ./server --list
```

You should see probes like:
- `myserver:server_start`
- `myserver:request_start`
- `myserver:request_end`
- `myserver:request_error`

## Exercise 3: Trace with bpftrace

### Count requests by type
```bash
# In terminal 1:
sudo bpftrace -e '
usdt:./server:myserver:request_start {
    @types[arg1] = count();
}
END {
    print(@types);
}
'

# In terminal 2:
./server 5 50
```

### Measure request latency
```bash
sudo bpftrace -e '
usdt:./server:myserver:request_start {
    @start[arg0] = nsecs;
}
usdt:./server:myserver:request_end {
    $latency = (nsecs - @start[arg0]) / 1000000;
    @latency_ms = hist($latency);
    delete(@start[arg0]);
}
END {
    print(@latency_ms);
}
'
```

### Track errors
```bash
sudo bpftrace -e '
usdt:./server:myserver:request_error {
    printf("ERROR: request %d failed with code %d\n", arg0, arg1);
    @errors = count();
}
'
```

## Exercise 4: Trace with perf

```bash
# Add probe
sudo perf probe -x ./server 'myserver:request_start'

# Record
sudo perf record -e 'probe_server:*' ./server 3 20

# Report
sudo perf report
```

## How USDT Probes Work

The `DTRACE_PROBE` macro inserts a NOP instruction:

```c
DTRACE_PROBE2(myserver, request_start, req->id, req->type);
```

Compiles to something like:
```asm
nop    ; placeholder for probe
```

When you activate tracing:
1. Tracer finds the probe location (stored in ELF notes)
2. Replaces NOP with a trap instruction (INT3 on x86)
3. Trap triggers, tracer runs, returns control
4. When tracing stops, NOP is restored

**Overhead when not tracing**: ~0 (just a NOP)
**Overhead when tracing**: trap + handler execution

## Real-World Uses

### Python has USDT probes:
```bash
# If Python was built with --enable-dtrace
sudo bpftrace -e 'usdt:/usr/bin/python3:python:function__entry { printf("%s\n", str(arg1)); }'
```

### MySQL probes for query tracking:
```bash
sudo bpftrace -e 'usdt:/usr/sbin/mysqld:mysql:query__start { printf("%s\n", str(arg0)); }'
```

## Discussion Questions

1. **When would you use USDT vs dynamic probes?**
   - USDT: Known important points, stable interface
   - Dynamic: Ad-hoc debugging, no source changes

2. **What's the trade-off of adding probes?**
   - Pro: Always available for debugging
   - Con: Must plan ahead, adds code complexity

3. **Why not just use printf debugging?**
   - Printf has overhead even when you don't need it
   - USDT has zero overhead until activated
   - USDT can be traced without rebuilding

## Advanced: Creating Custom Probes

The probe macros from `<sys/sdt.h>`:

```c
DTRACE_PROBE(provider, name)                    // No arguments
DTRACE_PROBE1(provider, name, arg1)             // 1 argument
DTRACE_PROBE2(provider, name, arg1, arg2)       // 2 arguments
// ... up to DTRACE_PROBE12
```

Arguments can be integers or pointers. Strings need special handling.

## Key Takeaways

1. **USDT probes are designed-in observability**
2. **Zero overhead when not actively tracing**
3. **bpftrace makes probe usage easy**
4. **Many production systems already have probes (Python, databases, etc.)**

This is an advanced topic - the main takeaway for beginners is that
such instrumentation exists and enables powerful production debugging.