TTY BPF Instrument

Demonstrations and preliminary documentation can be found at https://github.com/edurange/demo-bpf-tty-logger and https://github.com/edurange/prototype-tty-bpf-instrument. Wiki documentation is pending. Preliminary draft files are available in the Discord #fileshare channel and will be linked here once Wiki configuration is adjusted to support attached non-image documents.

Summary

The TTY instrument is a BPF-based kernel observation tool for capturing terminal activity across a host system. Its purpose is to observe TTY byte activity at the kernel boundary and emit records that downstream components can reconstruct, validate, and analyze. The current design treats terminal activity as factual kernel observations first, and reserves higher-level ideas like sessions, users, commands, and learning activity for later interpretation rather than things the probe is responsible for determining.

Although the immediate target is TTY activity, the design is also meant to serve as a template for future host-level instruments. It separates kernel observation, bounded transport, user-space draining, reconstruction, and downstream interpretation in a way that can be reused for other event sources. The TTY instrument is intentionally demanding: it involves raw byte payloads, ambiguous identity, ordering challenges, fragmentation, loss accounting, and privacy-sensitive data. If this pattern works for TTY activity, it provides a strong example for building other instruments with similar fidelity and reliability requirements - for the kernel-level services like the filesystem or network interfaces, common user-level processes like `bash`, etc.

Why TTY Capture Is Difficult

This project is difficult because TTY activity is not a single clean stream. A user’s apparent terminal session can pass through SSH, shells, subprocesses, pseudoterminals, terminal echo, line discipline behavior, containers, namespaces, and reconnects. The instrument therefore preserves multiple identity axes, including process IDs, user IDs, cgroups, namespaces, TTY device numbers, inode context, and command/process names. No one field is treated as “the session” or “the user.” Streams or sessions may later be reconstructed from these facts, but the raw record should keep the original evidence intact.

Tracepoints, BPF, and Kernel Boundaries

The current implementation is tracepoint-based. Kernel tracepoints provide minimal, upstream-plausible attachment surfaces; BPF probes attach to those tracepoints, read bounded payload buffers, enrich records with available identity/timing/ordering context, and emit records through BPF ring buffers. The tracepoint is not the logging schema. The BPF observation record is the instrument-facing schema.

The current prototype includes kernel modifications because the existing kernel does not expose all of the stable attachment surfaces needed for this instrument. Those modifications add narrowly scoped TTY tracepoints so BPF programs can attach at the relevant read, write, and line-discipline receive paths. The goal is not to move logging policy into the kernel or to make the kernel produce the full instrument schema.

The kernel changes must remain minimal, reviewable, and justifiable as tracepoint support; the BPF probe and user-space components remain responsible for the instrument-specific observation record, enrichment, transport, and reconstruction behavior. This distinction matters because tracepoints need to be acceptable to upstream kernel maintainers as general kernel instrumentation, while the richer record schema is specific to this project’s BPF instrument.

Pipeline and Component Responsibilities

The TTY instrument is also important because it represents a characteristic high-pressure workload for the larger event pipeline. Human terminal input can produce many small payloads with high metadata overhead, while command output can produce sudden bursts of larger byte streams. Multi-user SSH workloads combine both patterns. This makes the instrument a useful benchmark for the event bus and downstream data store: it exercises throughput, buffering, ordering, loss reporting, payload preservation, and reconstruction under conditions that are demanding yet meaningful with a minimal number of tracepoints and probe sites. (Contrast with the filesystem, where roughly a dozen such sites are needed to capture the wider variety of filesystem operations.) The TTY instrument does not define the bus or storage policy, but its output should help reveal whether those downstream systems can handle realistic observational pressure without hiding loss or collapsing important context.

The basic pipeline is:

TTY kernel paths
    -> kernel tracepoints
    -> BPF probes
    -> primary observation channel + auxiliary/status channel
    -> user-space spool
    -> collator
    -> downstream storage, analysis, policy, or reporting

The probe’s job is bounded observation: copy raw bytes, attach factual context, assign ordering metadata, and emit records. The spool’s job is to drain kernel output quickly and forward it onward, relieving pressure on the kernel ring buffer channels. The collator’s job is reconstruction: reassemble fragments, check sequence continuity, place loss, resolve safe aggregations, and prepare externally useful event representations.

Core Fidelity Invariants

A central invariant is that captured bytes are preserved as bytes. The probe does not decode Unicode, detect commands, infer prompts, identify student intent, or decide where lines begin and end. Payload may contain terminal control sequences, partial multi-byte characters, shell output, pasted text, or fragments of larger observations - the data is handled the same at the instrument level regardless of its nature. Decoding and semantic interpretation happen downstream.

Another central invariant is that fragmentation must be explicit. A single observed TTY occurrence may require multiple emitted records if the payload is too large for the bounded BPF record size. Fragmentation is not loss; it is just the transport representation of one observation. If bytes cannot be emitted, the missing portion must remain visible through loss/status reporting rather than being hidden by truncation or best-effort concatenation.

Ordering is based on sequence accounting, not timestamps alone. CPU-local sequence numbers are useful now for validating local emission order, fragmentation behavior, and probe diagnostics. The target design adds monotonic per-stream sequence numbers for authoritative stream-level ordering and loss placement. Timestamps are still important, but they support wall-clock calibration and approximate correlation rather than serving as the primary proof of order.

Loss accounting is a first-class part of the instrument. The design explicitly prefers capturing less data with accurate loss reporting over capturing more data whose completeness cannot be trusted. Loss can occur in the kernel probe, spool, collator, or downstream ingestion path; where possible, loss reports should identify the origin, affected sequence range, affected identity context, and lost record or byte counts.

The auxiliary/status channel exists because status records are needed to interpret captured data, but must not contend for transport resources with observational records. Health/status records may describe loss, backpressure, saturation, calibration, drift, lifecycle transitions, verifier failures, spool pressure, collator pressure, or reconstruction ambiguity. These records describe the behavior and reliability of the instrument, not additional user activity.

Prototype Lineage

The instrument’s proof of concept is https://github.com/edurange/demo-bpf-tty-logger/tree/main, which demonstrates host-wide TTY capture using BPF kernel probes. That demo shows the basic idea: instead of attaching to one TTY device, it hooks kernel activity system-wide, captures raw buffers and context, and warns about feedback loops if the logger prints to a TTY it is also observing.

The newer prototype, reflected in https://github.com/edurange/prototype-tty-bpf-instrument, moves beyond the early BCC/kprobe sketch toward the current tracepoint/CO-RE direction. It contains a kernel patch for tracepoints, BPF probe code, shared observation ABI, constants, event maps, CPU-local sequence state, and a user-space dump tool. That archive should be treated as the current implementation reference, while the formal design document remains the higher-level specification.

Design Boundaries and Non-Goals

The instrument is not meant to be a simple keylogger, even though it necessarily captures raw terminal bytes. It is an instrument for producing trustworthy observational records from kernel TTY activity under realistic multi-user workloads. The important work is preserving factual byte activity, identity context, ordering evidence, timing calibration, fragmentation metadata, and loss visibility without letting the kernel probe become an analyzer. The probe observes; the spool drains; and the collator reconstructs. Interpretation is the domain of downstream systems that consume the instrument’s output.

To keep the components individually maintainable, it is crucial to preserve these boundaries. Tracepoints should stay minimal and justifiable to kernel maintainers. BPF code should stay bounded and observational. The spool should reduce pressure without inferring meaning. The collator should only aggregate when identity, sequence continuity, fragment completeness, and loss boundaries make aggregation safe. Storage policy, privacy enforcement, governance, and final analysis are outside the instrument itself unless a later design explicitly brings them in.

Related

Project Charter

TTY BPF Instrument Project Charter