XRef

Linux live analysis: architecture and acceptance contract

On this page
  1. Delivery order
  2. Architecture and platform boundaries
  3. L1: identify the process and translate addresses
  4. L1: search memory and show a compact context window
  5. Region selection and scanning
  6. Context window and pointer hints
  7. Result and failure contract
  8. L1 validation and integration
  9. L2: bounded uprobe hit counts, caller evidence and register samples
  10. L3: bounded hardware data watchpoints (DR0–DR3) and dependent watches (--follow)
  11. L4: opt-in PID-pinned live MCP tool (xref_live)
  12. Sampling profiles
  13. Bounded call trace windows (live trace)
  14. Window lifecycle, static probe expansion and bounds
  15. Text output
  16. JSON schema (operation: "live_trace", schema_version: 1)
  17. Primary references

Status: implemented across Stages L1 (PR #49), L2 (PR #53), L3 Increment 1 (PR #54), L3 Increment 2 (PR #55), and L4 for issue #46. Available starting with v0.3.0; the published v0.2.0 and earlier binaries do not contain xref live or the opt-in xref_live MCP tool.

For a recorded, step-by-step example with real commands and their output (memscan → resolve → refs/c → probe → watch → --follow after a buffer move), see the Live Analysis walkthrough, also published at xrefcode.com/live-walkthrough.

The goal is to connect a running Linux x86-64 process to XRef's static address/function evidence: find state, observe selected calls, and identify candidate instructions that modify selected memory. Each result must distinguish an observation from an inference and state its coverage and limits.

Delivery order

  1. L1: process identity, mappings, address resolution and memory search. Use safe standard-library reads, bounded output and controlled synthetic target processes. This is the first independently reviewable deliverable.
  2. L2: function hit counts, followed by caller/register evidence. Validate the Linux tracing adapter on a synthetic fixture before adding it.
  3. L3: hardware write watchpoints. Validate event/IP/value semantics and thread/resource coverage separately; preserve unknowns.
  4. L4: explicitly enabled live MCP. Add a PID-bound live context only after cancellation, process lifetime and backend cleanup have been validated.

The issue remains open while requested later stages are outstanding. Source changes use isolated worktrees, focused development tests, a completed-change final gate, independent review and explicit integration. Implementing L1 does not imply publishing a new release or tracing a production process.

Architecture and platform boundaries

L1: identify the process and translate addresses

Stage L1 (maps, resolve, memscan) command shapes:

xref live maps --pid PID --json
xref live resolve --pid PID LIVE_VA... --json
xref -b app.elf live resolve --pid PID LIVE_VA... --json
  1. Validate the PID. Open and retain proc/executable handles, record the PID's start identity and executable identity, and hash the actual executable bytes. A numeric PID alone is insufficient because a PID can be reused.
  2. If --bin is supplied, verify it identifies the process's executable build. An identical copy can be accepted by digest; a coincidental filename cannot. Resolve the executable automatically when no binary path is supplied. Static commands retain their requirement for a selected binary.
  3. Read bounded maps/smaps metadata. Preserve paths as reported, including spaces, escape spelling and deleted-file suffixes; paths are labels rather than authority to open arbitrary files.
  4. Obtain ELF load-segment geometry and flags. Do not infer executable status from the existing Segment { va, offset, size } fields alone.
  5. Distinguish static VA, file offset and live VA explicitly. Check the relevant load segment, its file-backed extent, mapping identity and load bias. Never apply one unexplained global subtraction to every mapping.
  6. Preserve multiple valid interpretations as candidates. Bytes in an anonymous mapping or a zero-filled segment tail are not automatically file-backed code.
  7. Recheck process/executable identity at operation boundaries. Report process exit, replacement, image mismatch and mapping changes explicitly; do not quietly attribute data from a changed process to the original image.

Return raw addresses and mapping provenance when static function attribution is unavailable. An .eh_frame FDE range is not proof of a source-level function.

L1: search memory and show a compact context window

Proposed command shape:

xref live memscan --pid PID U64_VALUE... --before-words 2 --after-words 6 --max-hits 32 --json

The input is a bounded sequence of little-endian u64 values. Matches start at 8-byte-aligned live addresses. Limit pattern length, requested/read bytes, result count, context words and total serialized output.

Region selection and scanning

Context window and pointer hints

Return a compact window of aligned words around each match, rather than an unbounded memory dump. The proposed defaults are two words before and six words after, with at most 32 hits. These are upper bounds: output/region/read limits can shorten a window or stop the result earlier, and that loss must be visible.

Each slot carries its address, exact observed word when readable, classification and evidence. Serialize 64-bit addresses/words without losing integer precision. Do not silently read across an excluded mapping or guard gap to fill a window.

Candidate classifications:

Result and failure contract

The human CLI and JSON output share the same typed live result. Keep a versioned envelope with tool/version, operation, process/image identity, observation interval, mapping identity, data/error, warnings, coverage, limits and stops. The existing static JSON/MCP schema does not silently acquire arbitrary PID access.

Examples of necessary distinctions:

The live reader does not write target memory or start/restart the target. Permissions remain governed by Linux; do not automatically elevate privileges or change host policy when an access check fails.

L1 validation and integration

Use deterministic public fixtures and a controlled synthetic Linux process. That fixture may grant the test reader access to its own memory; the product must not change another process's ptrace policy as part of attachment.

Acceptance covers:

Run only new/directly affected tests during implementation. Follow the repository's final formatting/lint/compile gate and reuse successful checks on unchanged code. Independent review must inspect the actual diff and evidence, not just an exit code.

L2: bounded uprobe hit counts, caller evidence and register samples

Stage L2 (probe) command shape:

xref live probe --pid PID ADDR... [--backend perf|tracefs] [--count] [--callers] [--regs] [--duration-ms 1000] [--max-events 4096] [--json]
xref live probe --backend tracefs --pid PID ADDR... --fetch 'amount=+0x30(%rcx):u64' --filter 'amount > 0' --every 2 --first 64 --until-hits 128 --json
xref live probe --backend tracefs --pid PID ADDR... --duration-ms 0 --heartbeat-ms 1000 --group-by ret --mark-file /tmp/app.log --mark-regex '^TX-[0-9]+'

L3: bounded hardware data watchpoints (DR0–DR3) and dependent watches (--follow)

Stage L3 hardware watchpoint command shape:

xref live watch --pid PID [ADDR...] [--follow SPEC]... [--len 1|2|4|8] [--access write|rw] [--regs] [--duration-ms 1000] [--max-events 4096] [--json]

L4: opt-in PID-pinned live MCP tool (xref_live)

Stage L4 opt-in MCP startup command:

xref -b app.elf mcp --allow-live-pid PID

Sampling profiles

On Linux x86-64, profile a verified executable and a running process with hardware performance events. The kernel and system policy must permit the requested events; unavailable events return an error rather than silently changing the event or backend.

xref -b ./app live profile --pid "$PID" --duration 10s --freq 4000 \
  --callers dwarf --out capture.xprof
xref -b ./app profile report capture.xprof --by function --top 30
xref -b ./app profile report capture.xprof --by block
xref -b ./app profile report capture.xprof --by line
xref -b ./app profile annotate capture.xprof 0x1234 --json
xref -b ./app profile annotate capture.xprof 0x1234 --c
xref -b ./app profile diff before.xprof after.xprof --other ./app-new --json
xref -b ./app profile import perf.data --out imported.xprof --base 0x400000 --pid "$PID"

--callers dwarf unwinds a bounded captured user stack with CFI from the target and verified shared objects; incomplete unwinds are reported and can omit callers when a module or its CFI is unavailable. fp uses kernel call chains, lbr needs supported hardware and provides a bounded branch history, and none records leaf samples. Shared-library addresses are attributed to their module when mappings are known; a file hash is a snapshot of the backing file, not proof of unchanged in-memory code. Line reports need DWARF line information; stripped images may leave samples unresolved. Pseudo-C statement attribution is approximate and separately reports unmapped samples.

--trigger-probe 0x1234 --window 200ms uses a bounded in-memory flight recorder around a uprobe hit. Thread attachment, buffer eviction and asynchronous events can leave gaps; lost and dropped records, incomplete coverage, and the trigger status are reported. The capture thread CPU measurement excludes costs incurred by the target and interrupt contexts, so it is not a measurement of the total workload slowdown. Sample shares are not exact execution time.

Importing text from perf script, or binary perf.data directly, currently requires --pid and either a caller-supplied runtime image --base or --maps snapshot. The base or snapshot and target identity are unverified, mappings are synthetic, and timestamps or event periods may be missing. Native import preserves recorded sample periods, timestamps and loss records when present, but cannot prove the target identity or recover all mappings. For recordings with multiple event types, it imports the first and marks the report incomplete. It accepts bounded zstd-compressed perf records, including bounded AUXTRACE records in compressed streams. A supported Intel PT stream can be decoded from an AUX-only capture or alongside one ordinary sampled event. Reconstructed instruction events are reported separately from CPU samples: they do not measure elapsed time and have no reconstructed timestamps. The importer requires a matching build ID and executable mapping and currently rejects multi-thread or CPU-wide PT streams, loss, unsupported packet sequences, and ambiguous image mappings. Other AUX formats remain undecoded and mark a sample report incomplete; an unsupported AUX-only capture cannot be profiled. Runtime code modifications cannot be ruled out from perf metadata. The PT integration has been tested with synthetic trace bytes, a compiled ELF and real perf sideband, not with a hardware Intel PT capture. Mixed profiles currently support function, instruction, block, caller-path and line reports; folded output, annotations and profile diffs reject them. For this initial decoder, record a per-thread user-mode trace and restrict the PT address filter to executable code in the selected target, for example:

perf record --per-thread -e intel_pt/mtc=0/u \
  --filter 'filter 0xSTART/0xSIZE@/path/to/target' -p "$PID" -- sleep 1

Replace START and SIZE with the file-relative executable range supported by the host's perf address filter. CPU-wide streams, transitions outside the verified target and unsupported packet sequences are rejected. Reaching the instruction or callchain budget retains the decoded prefix and marks it as incomplete with intel_pt_instruction_budget_exhausted. Sample-READ records with the basic grouped or single-value formats are accepted; their counter values are not included in the profile. Unsupported READ format flags are rejected. Compressed streams use a maximum 64 MiB zstd window and each compressed record may expand to at most 64 MiB. Input is limited to 16 GiB, feature metadata to 64 MiB and each perf event round to 64 MiB or 131,072 records; exceeding a limit returns an error. Native import first makes a private temporary copy of the input, so it needs temporary disk space up to the input size and may take longer on large files. Reports mark imports incomplete rather than asserting equivalence with a native capture. For a recording or snapshot made on a different system, use --target-path to identify the target's original path.

Native perf imports can infer the target slide from initial executable MMAP2 records when --base and --maps are omitted. The target path and ELF segment layout must match; if the binary was moved, a unique matching recorded build ID can identify its original path. Ambiguous slides, paths or changed mapping identities are rejected. The recorded mappings and build ID cannot prove that runtime code bytes remained unchanged. profile report --within FUNCTION_OR_VA keeps matching call chains; profile annotate --percent function --min-samples N shows function-relative shares and counts. --hz, --duration-ms and --callgraph are available on live profile; profile diff --by function --min-delta 0.5% filters small absolute changes in sample share.

For multiple post-trigger windows, use --trigger-probe VA --window-ms 500 --max-windows 20. Add --trigger-fetch 'op=+40(%sp):u64' --trigger-filter 'op == 1' to evaluate a named value captured by a tracefs uprobe at the hit. The latter requires writable tracefs and perf tracepoint access. Report a single window with profile report FILE --window 1, or their union without double counting overlaps with --merge-windows; either can be combined with --within. Trigger captures use .xprof schema 4: the schema 3 layout (window records) written by a capture with the system-wide trigger. This version reads schemas 1–4; older versions cannot read schema 3 or 4. A trigger profile with schema 3 or lower may come from per-thread triggers, so its report keeps the trigger_misses_threads_shorter_than_the_proc_poll_interval warning. Sampling is enabled only after the observer receives a matching trigger, so the beginning of a window can be missed; profiles explicitly mark this coverage limitation and report trigger losses. The trigger is a tracefs uprobe read system-wide (one perf sampler per online CPU, filtered to the target process), so it fires in every thread that executes the code, including threads created after attach and threads that exit right after the hit. CPU samplers still attach per thread once a thread is seen, so a thread that exits within that latency adds no samples to its window (trigger_window_samples_miss_threads_that_exit_before_attach). Without writable tracefs or system-wide perf access (root or CAP_PERFMON) a --trigger-probe capture fails with that reason; there is no per-thread fallback, so every saved trigger profile was taken with the system-wide trigger. Hits drained from several CPU rings are applied in timestamp order, so --max-windows keeps the earliest triggers. Existing centered single-trigger captures remain available with the original --window option.

The existing opt-in xref_live MCP tool also accepts the profile operation for the pinned process with the same 5-second duration and 4096-event caps. Its capture buffer is bounded to 16 MiB and the existing response-size cap applies.

Bounded call trace windows (live trace)

xref live trace (issue #111, discussion #110) records ordered, thread-scoped call trace windows on Linux x86-64 using the isolated tracefs uprobe backend (trace_clock=mono, set_event_pid + event-fork, record-tgid, and the out-of-process cleanup watchdog):

xref live trace --pid PID \
  --start 0x1000 \
  --start-filter "op == 1" \
  --stop-return 0x1000 \
  --callees-of fn_a --depth 2 \
  --fetch-at 0x1000 "op=%di:u64" \
  --fetch-ret 0x1020 "rv=%ax:u64" \
  --max-windows 4 --max-events 1024 --duration-ms 0 \
  --json

Window lifecycle, static probe expansion and bounds

Text output

pid 4242  match proc_exe_auto  slide +0x555555554000  backend tracefs  return_mode mixed_uretprobe_best_effort
warning: uretprobe_returns_are_best_effort_under_unwind
window 1  tid 5001  start 0x1000  status complete  stop 0x1000  op=0x1
   0  > 0x1000 @0x1050 (fn_a)  op=0x1
   1    > 0x1020 @0x1004 (fn_b)
   1    < 0x1020 (fn_b)  rv=0x2
   1    > 0x1030 @0x1009 (fn_c)
   2      > 0x1040 @0x1033 (fn_d) [tail]
   1    < 0x1030 (fn_c)
   0  < 0x1000 (fn_a)
summary: windows=1 events=7 lost=0 stop=max_windows elapsed_ms=12

JSON schema (operation: "live_trace", schema_version: 1)

--json emits a single-line JSON document with stable static VAs (PIE slide removed) suitable for offline comparison (trace diff):

{
  "schema_version": 1,
  "tool": "xref",
  "tool_version": "0.3.0",
  "operation": "live_trace",
  "backend": "tracefs",
  "process": {
    "pid": 4242,
    "starttime_ticks": 100,
    "pid_alive_after": true,
    "process_changed": false,
    "binary_match": "proc_exe_auto",
    "exe_sha256": "...",
    "pie_slide": 93824992231424,
    "warnings": ["uretprobe_returns_are_best_effort_under_unwind"]
  },
  "summary": {
    "windows": 1,
    "complete_windows": 1,
    "incomplete_windows": 0,
    "events": 8,
    "lost": 0,
    "lost_events": {
      "status": "known",
      "overrun": 0,
      "commit_overrun": 0,
      "dropped_events": 0,
      "perf_lost_records": 0,
      "total": 0
    },
    "stop_reason": "max_windows",
    "return_mode": "mixed_uretprobe_best_effort",
    "elapsed_ms": 12,
    "unparsed_events": 0,
    "excluded_other_tgid_events": 0,
    "ambiguous_tid_events": 0
  },
  "windows": [
    {
      "window_id": 1,
      "tid": 5001,
      "start_va": "0x1000",
      "stop_va": "0x1000",
      "start_timestamp_ns": 10000010000,
      "end_timestamp_ns": 10000050000,
      "duration_ns": 40000,
      "status": "complete",
      "possible_loss": false,
      "start_fetches": [{ "name": "op", "value": "0x1" }],
      "events": [
        {
          "seq": 0,
          "kind": "entry",
          "depth": 0,
          "target_va": "0x1000",
          "symbol": "fn_a",
          "call_site_va": "0x1050",
          "call_site_verified": true,
          "caller_va": "0x1050",
          "tail_call": false,
          "timestamp_ns": 10000010000,
          "fetches": [{ "name": "op", "value": "0x1" }]
        },
        {
          "seq": 1,
          "kind": "entry",
          "depth": 1,
          "target_va": "0x1020",
          "symbol": "fn_b",
          "call_site_va": "0x1004",
          "call_site_verified": true,
          "caller_va": "0x1000",
          "tail_call": false,
          "timestamp_ns": 10000015000
        }
      ]
    }
  ]
}

Field notes:

Primary references

Source: docs/live.md · generated for the site; edit the Markdown, not this page.