EDURange - Recent changes [en]

Brief: EDURange is More Than a CRUD Architecture

Jwgranville — Fri, 19 Jun 2026 03:30:56 GMT

← Older revision		Revision as of 03:30, 19 June 2026
(2 intermediate revisions by the same user not shown)
Line 1:		Line 1:
	~~{{DISPLAYTITLE~~:EDURange is More Than a CRUD Architecture}}		= Brief: EDURange is More Than a CRUD Architecture =

	~~= Brief: EDURange~~ is ~~More Than~~ a ~~CRUD Architecture =~~		This is a brief explaining my position on adopting message-based, microservice architecture and the model-driven development process. This is the first page of its kind so I'm just guessing at the format. This top level "brief" section is in my own words but contributors should feel free to create other sections or add to the header content (above the "brief" section title) as needed. ~Joe G

	== The Up-Front Cost We Are Choosing to Pay ==		== The Up-Front Cost We Are Choosing to Pay ==
Line 51:		Line 51:
	A domain model is well suited to serve as the kernel of a complex unit of software because it depends on no outside code or system. It is not the database layer, the UI layer, the web framework, or the message bus. It is the part of the program that implements the logical rules we want the rest of the system to manifest. Its only job is to represent or calculate what is true or known under those logical rules. Once those rules are explicit, we can build layers around them which preserve their properties and truthfulness: persistence, messaging, APIs, user interfaces, tests, simulations, and analysis tools. Those outer layers may be influenced by outside systems, but the core model gives them something stable to serve as a reference for correct behavior.		A domain model is well suited to serve as the kernel of a complex unit of software because it depends on no outside code or system. It is not the database layer, the UI layer, the web framework, or the message bus. It is the part of the program that implements the logical rules we want the rest of the system to manifest. Its only job is to represent or calculate what is true or known under those logical rules. Once those rules are explicit, we can build layers around them which preserve their properties and truthfulness: persistence, messaging, APIs, user interfaces, tests, simulations, and analysis tools. Those outer layers may be influenced by outside systems, but the core model gives them something stable to serve as a reference for correct behavior.

	This matters especially for intelligent tutoring. ~~Domain models are~~ not only useful for organizing code. ~~They are~~ useful for observing, interpreting, and predicting the behavior of complex systems or relationships.		This matters especially for intelligent tutoring and other AI applications. Model-driven development is not only useful for organizing code. It is useful for observing, interpreting, and predicting the behavior of complex systems or relationships.

	What is the student doing? What is the computer system doing? What happened in the learning environment that the student might have observed? What evidence suggests the student understands the underlying concept? Why does a problem have a particular solution? Which actions are equivalent, misleading, partial, or strategically meaningful? Those are modeling questions before they are implementation questions. A desirable solution doesn’t just have implementation steps: there is a concrete and logical reason why that particular implementation is correct, and a definition of the scope or conditions under which the correctness holds.		What is the student doing? What is the computer system doing? What happened in the learning environment that the student might have observed? What evidence suggests the student understands the underlying concept? Why does a problem have a particular solution? Which actions are equivalent, misleading, partial, or strategically meaningful? Those are modeling questions before they are implementation questions. A desirable solution doesn’t just have implementation steps: there is a concrete and logical reason why that particular implementation is correct, and a definition of the scope or conditions under which the correctness holds.

Message Bus Service

Jwgranville — Fri, 19 Jun 2026 03:14:55 GMT

← Older revision		Revision as of 03:14, 19 June 2026
Line 3:		Line 3:
	== Related ==		== Related ==

	=== ~~Project Charter~~ ===		=== Design Documents: ===
	~~[[Message Bus Project Charter]]~~

	=== Examples, Tutorials~~, Supporting Documents~~ ===		* [[Message Bus Project Charter]]
	[[Message Bus Live Coding Exercise]]		* [[Brief: EDURange is More Than a CRUD Architecture]]

			=== Examples & Tutorials: ===

			* [[Message Bus Live Coding Exercise]]

New Version Planning

Jwgranville — Fri, 19 Jun 2026 03:13:40 GMT

Links to Subprojects/Related Topics

← Older revision		Revision as of 03:13, 19 June 2026
Line 92:		Line 92:
	This section is temporary until we settle for a multi-topic organization for new version planning.		This section is temporary until we settle for a multi-topic organization for new version planning.

			=== Services/Modules: ===
	* [[Data Store Project Charter]] - just the charter; pending a top-level topic page		* [[Data Store Project Charter]] - just the charter; pending a top-level topic page
	* [[Message Bus Service]] - mostly a placeholder topic page; links to charter and related docs		* [[Message Bus Service]] - mostly a placeholder topic page; links to charter and related docs
	* [[TTY BPF Instrument]] - also a placeholder page; links to repos/docs		* [[TTY BPF Instrument]] - also a placeholder page; links to repos/docs

			=== Briefs: ===

			* [[Brief: EDURange is More Than a CRUD Architecture\|EDURange is More Than a CRUD Architecture]] - reasoning behind message-oriented architecture and model-driven development

Brief: EDURange is More Than a CRUD Architecture

Jwgranville — Fri, 19 Jun 2026 03:09:41 GMT

← Older revision		Revision as of 03:09, 19 June 2026
Line 86:		Line 86:

	= Related =		= Related =
			[[New Version Planning]]

Brief: EDURange is More Than a CRUD Architecture

Jwgranville — Fri, 19 Jun 2026 03:08:49 GMT

Created page with "{{DISPLAYTITLE:EDURange is More Than a CRUD Architecture}} = Brief: EDURange is More Than a CRUD Architecture = == The Up-Front Cost We Are Choosing to Pay == The architecture I have been proposing for the next version of EDURange asks the team to pay some real up-front costs. A message-oriented service architecture, append-only observation, explicit domain modeling, and model-driven development are not free. They require shared vocabulary, new learning, more delibera..."

New page

{{DISPLAYTITLE:EDURange is More Than a CRUD Architecture}}

= Brief: EDURange is More Than a CRUD Architecture =

== The Up-Front Cost We Are Choosing to Pay ==

The architecture I have been proposing for the next version of EDURange asks the team to pay some real up-front costs. A message-oriented service architecture, append-only observation, explicit domain modeling, and model-driven development are not free. They require shared vocabulary, new learning, more deliberate interfaces, more testing discipline, and a willingness to plan and define software responsibilities before the immediate payoff is always obvious.

I want to acknowledge that directly. This is not the easiest path in the first sprint, and it may feel unfamiliar for our team whose shared experience is largely in cybersecurity contexts and classroom-facing teachable code rather than professional-scale software development techniques, lifecycle management, and architectural strategy. My argument is not that we should adopt heavier engineering practices because they are fashionable or inevitable. My argument is that EDURange specifically has grown to become the kind of project where not acknowledging this cost is likely to incur even greater debt.

== Why EDURange Is Not a CRUD Application ==

EDURange is not just a CRUD application. We are not simply making and mutating records when we manage scenarios, users, assignments, and scores. The code objects in EDURange need to accurately reflect the behavior of real-life systems outside the application. We are building an intelligent tutoring system whose learning environment is a cyber range. That means we are combining two mature application paradigms that are each complex and difficult to engineer individually.

A cyber range tries to recreate meaningful interaction with real computing systems, while mitigating actual lasting harm to the host system. Student behavior is exploratory, ambiguous, distributed over time, and a student’s cognitive state is only partially or indirectly observable - even under perfect conditions. An intelligent tutoring system tries to reason about learning: what the student understands, where they are stuck, what intervention might help, and how instruction should adapt. Neither side of this problem is well represented by a simple record-oriented database design.

The hard part is not just storing data. Another vital responsibility is preserving enough meaning that we can ask better questions later.

== What the Previous Architecture Taught Us ==

Earlier EDURange development has already shown the limits of ad hoc integration. Historically many key features have depended on specific legacy scripts, file formats, and implicit assumptions that are difficult to reproduce or explain to new contributors. Log processing has been constrained by hardcoded relationships among producers and consumers. Log data has existed in multiple places without an obvious canonical version. Data transformations have been destructive: once raw observations are cleaned, summarized, or reshaped for one purpose, later analysis inherits assumptions that may no longer be appropriate.

Adding a new observational capability, analysis method, or hinting strategy for EDURange today is less about the idea itself and more about finding a safe place to splice it into the existing stack.

That is the pain point I want us to take seriously. The previous architecture did not fail because people made wrong choices. It grew around real needs, limited time, and specific feature requests. But those local decisions were not considered in terms of long-term consequences. As a result, they accumulated into a system where change is expensive, historical knowledge is fragile, and developers need special guidance just to run the stack.

== Why Not Just Use Existing Middleware? ==

One reasonable question is: why not solve this with existing middleware?

In some cases, we should. And under the hood of several key modules, we will. For instance - I have no intention of implementing network messaging myself or asking our future devs to maintain it, so the message bus will have an adapter that itself connects to the ZeroMQ messaging middleware for actual transport. The project-facing message bus module is responsible for managing the boilerplate setup of ZeroMQ endpoints and wire formats, giving us a high-level “it just works” functionality while leaving open the option for low-level extension or migration to other transport APIs later.

I am not arguing for rebuilding standard tools out of pride or habit. But the append-only storage and messaging options I have evaluated tend to bring their own lifecycle assumptions, especially around message schemas, version numbers, and server/client reconciliation.

Those concerns are important in many systems, but generally beyond our scale. They assume systems distributed across multiple persistent servers, where it may not be possible to shut the whole system down when updating an individual component - and so components at all times have to be prepared for the other endpoints they work with to begin communicating with a different message schema version. Version management requires special care in data format design, code changes, and binary encoding. And they cost extra per-message space. They risk making the first hard problem for EDURange into tackling a middleware’s conceptual challenges, rather than addressing the design needs and intended uses of EDURange head on.

That matters because our team is largely composed of junior developers working independently with limited supervision. We could adopt a sophisticated middleware stack as our team model for messaging and then say, “message schema changes are senior-review only.” But that creates a brittle social rule around a technical dependency most of the team cannot easily inspect. A smaller wrapper, built around our conventions and MVP needs, gives us a more teachable path into message-oriented architecture and append-only data policy. It lets us adopt the important ideas now - durable observation, explicit message boundaries, replayability, narrow service responsibilities - without forcing the team to manage complexity we do not yet know we need.

This is not a permanent rejection of production middleware. It is a staging decision. If we later need stronger distribution, replication, schema negotiation, or broker-level guarantees, we will be in a better position to adopt those tools because we will already understand our own domain boundaries. With a version built with messages in mind, we would be well positioned to perform an apples-to-apples refactor from one message API to another, without having to restructure the logical organization of our existing parts or change expectations of correct behavior. By implementing the bus as a modular service instead of directly incorporating a middleware API, we have a clearly identified pivot point where all of our message-related code can be extended or changed when new needs emerge.

== Why Message-Oriented Architecture Fits the Problem ==

A message-oriented architecture itself helps promote this future adaptability because it treats activity as something services can observe and respond to without owning each other. Instead of one process consuming a log and deciding what every downstream feature is allowed to see, we can preserve observations and let multiple consumers subscribe to the same events. One service might classify commands. Another might track timing. Another might build a student model. Another might support replay or research analysis views.

These consumers should not need to preempt each other, overwrite each other, or force the capture layer to know every future use case. And because they are loosely connected in this way, they are easier to change, replace, and implement independently.

== Why Domain Models Matter ==

Domain-driven design and model-driven development address a different but related problem: meaning.

A domain model is well suited to serve as the kernel of a complex unit of software because it depends on no outside code or system. It is not the database layer, the UI layer, the web framework, or the message bus. It is the part of the program that implements the logical rules we want the rest of the system to manifest. Its only job is to represent or calculate what is true or known under those logical rules. Once those rules are explicit, we can build layers around them which preserve their properties and truthfulness: persistence, messaging, APIs, user interfaces, tests, simulations, and analysis tools. Those outer layers may be influenced by outside systems, but the core model gives them something stable to serve as a reference for correct behavior.

This matters especially for intelligent tutoring. Domain models are not only useful for organizing code. They are useful for observing, interpreting, and predicting the behavior of complex systems or relationships.

What is the student doing? What is the computer system doing? What happened in the learning environment that the student might have observed? What evidence suggests the student understands the underlying concept? Why does a problem have a particular solution? Which actions are equivalent, misleading, partial, or strategically meaningful? Those are modeling questions before they are implementation questions. A desirable solution doesn’t just have implementation steps: there is a concrete and logical reason why that particular implementation is correct, and a definition of the scope or conditions under which the correctness holds.

For EDURange, the relevant concepts may include observations, scenarios, learners, attempts, interventions, hints, policies, evidence, projections, and instructional state. Relevant teachable cybersecurity content changes. Scenario infrastructure dependencies change. Student modeling strategies change. AI tools and services change. Research questions change. UI needs change. If the important concepts only live unnamed as trace evidence in controller functions, database schemas, scripts, or frontend assumptions, then every change risks becoming a cross-stack archaeology project. If the concepts live in explicit models, then we have something stable to discuss, test, revise, and teach.

== Why Thin UIs and Plural Analysis Matter ==

This is also why thin UIs with narrow scopes are appealing. A UI should present a coherent experience, but it should not be where the system’s core reasoning lives. We want different views for different user roles - potentially including students, instructors, content authors, teaching assistants, developers, and perhaps even more. Those views should compose around shared definitions, services and models rather than each inventing its own interpretation of the system.

The same idea applies to data analysis. We should expect plural analysis, not one final pipeline. For a research platform, it is normal to compare models, rerun old data through new methods, test interventions, and ask questions we did not know to ask when the data was first captured. That requires preserved observations, explicit semantic contracts, and a clear distinction between raw evidence and downstream interpretation.

This also has practical value for grant-seeking work. Reproducibility and interpretability are not just academic ideals. They make it easier to explain and demonstrate what we measured, why we measured it, how an intervention was selected, and whether a result can be trusted. A system that preserves observations and exposes its models is easier to evaluate, easier to defend, and easier to extend into new research questions. Increasingly, features like reproducibility and explainability are not simply desired but expected.

== Questions the Architecture Should Help Us Ask ==

The goal is not architectural purity. The goal is to make future work easier to explain, easier to test, and safer to change. A good architecture should help us ask better questions:

* What observations must be preserved before we interpret them?
* Which transformations are irreversible, and what assumptions do they commit us to?
* Which parts of the system need transactional consistency?
* Which services should be allowed to change independently?
* Where do we expect multiple competing analysis strategies?
* Which domain concepts should new contributors learn first?
* What should remain small and local now, but have a path to generalization and distribution scale later?

These questions are not distractions from implementation. They are part of responsible implementation planning for a system whose purpose is scientific inquiry and adaptive instruction.

== A Modest Implementation Strategy ==

My hope is that we can take a modest, incremental approach: build small core services, keep their responsibilities narrow, test them rigorously, and use them to make the rest of the project easier rather than harder as complexity increases.

The up-front cost is real. But if we choose the boundaries well, that cost should buy us clearer reasoning, better data, more reproducible research, and greater and more sustainable developer productivity. Most of all, the goal is a codebase where the team can safely evolve, share, and try new ideas without irreversibly accumulating assumptions or undoing foundational progress along the way.

= Related =

Message Bus Service

Jwgranville — Wed, 17 Jun 2026 03:30:01 GMT

← Older revision		Revision as of 03:30, 17 June 2026
Line 1:		Line 1:
	Module repository and wiki documentation are pending. This is a placeholder topic page to index resources related to the development of the message bus service.		Module repository and wiki documentation are pending. This is a placeholder topic page to index resources related to the development of the message bus service.

	= Related =		== Related ==

	== Project Charter ==		=== Project Charter ===
	[[Message Bus Project Charter]]		[[Message Bus Project Charter]]

	== Examples, Tutorials, Supporting Documents ==		=== Examples, Tutorials, Supporting Documents ===
	[[Message Bus Live Coding Exercise]]		[[Message Bus Live Coding Exercise]]

Message Bus Live Coding Exercise

Jwgranville — Tue, 16 Jun 2026 16:50:09 GMT

Phase 3: Explore shell command data by hand

← Older revision		Revision as of 16:50, 16 June 2026
(2 intermediate revisions by the same user not shown)
Line 655:		Line 655:

	The dictionary contains the information, but it does not document the shape of the event. A typo is discovered only when the field is used. A named event class makes the processor code easier to read and easier to explain.		The dictionary contains the information, but it does not document the shape of the event. A typo is discovered only when the field is used. A named event class makes the processor code easier to read and easier to explain.

			Positional encoding like tuples and CSV-like records also suffer from this innate limitation. They are inherently detached from the structural contract of the record type; they describe only whatever data they carry, not any particular schema or type contract to which the record itself belongs.

	== Phase 4: Persist the event record classes ==		== Phase 4: Persist the event record classes ==

New Version Planning

Jwgranville — Tue, 16 Jun 2026 14:12:17 GMT

Links to Subprojects/Related Topics

← Older revision		Revision as of 14:12, 16 June 2026
Line 93:		Line 93:

	* [[Data Store Project Charter]] - just the charter; pending a top-level topic page		* [[Data Store Project Charter]] - just the charter; pending a top-level topic page
	* [[Message Bus ~~Project Charter~~]] - ~~also pending top-level~~ page		* [[Message Bus Service]] - mostly a placeholder topic page; links to charter and related docs
	* [[TTY BPF Instrument]] - ~~mostly~~ a placeholder ~~topic~~ page; links to repos/docs		* [[TTY BPF Instrument]] - also a placeholder page; links to repos/docs

Message Bus Live Coding Exercise

Jwgranville — Tue, 16 Jun 2026 05:31:58 GMT

Phase 14: Add a randomized processor scheduler

← Older revision		Revision as of 05:31, 16 June 2026
(5 intermediate revisions by the same user not shown)
Line 16:		Line 16:
	__init__.py		__init__.py
	</pre>		</pre>

			Pre-release copies of <code>mbus</code> and <code>aostore</code> are available on the Discord fileshare channel. Modules may be released under other package names; see also the EDURange GitHub.

	The exercise uses two problem families.		The exercise uses two problem families.
Line 31:		Line 33:
	** Simulate nondeterministic processor scheduling while keeping source input fixed.		** Simulate nondeterministic processor scheduling while keeping source input fixed.

	The shell examples draw from EDURange's institutional knowledge. The graph examples provide a ~~compact computer science~~ model for ~~safe~~ feedback loops.		The shell examples draw from EDURange's institutional knowledge. The graph examples provide a common model problem family for meaningful use of feedback loops.

	== Running the examples ==		== Running the examples ==
Line 52:		Line 54:
	touch demos/event_processors/__init__.py		touch demos/event_processors/__init__.py
	</pre>		</pre>


	== Important note about capture records ==		== Important note about capture records ==

Line 74:		Line 74:
	Estimated time: 1-3 hours (1-2 hours work, 0.5-1 hours setup)		Estimated time: 1-3 hours (1-2 hours work, 0.5-1 hours setup)

	; Session 2 - Shell ~~command~~ and ~~regex~~ milestones		; Session 2 - Shell commands and milestones

	* '''Phase 3:'''		* '''Phase 3:'''
Line 81:		Line 81:
	** Replace it with <code>ShellCommandEvent</code>.		** Replace it with <code>ShellCommandEvent</code>.
	* '''Phase 5:'''		* '''Phase 5:'''
	** Manually evaluate ~~regex~~ milestone expressions.		** Manually evaluate milestone expressions.
	* '''Phase 6:'''		* '''Phase 6:'''
	** Implement <code>RegexMilestoneProcessor</code>.		** Implement <code>RegexMilestoneProcessor</code>.
Line 92:		Line 92:
	* '''Phase 7:'''		* '''Phase 7:'''
	** Create raw TTY read, write, and line events.		** Create raw TTY read, write, and line events.
	** Show why ~~regex~~ milestones need reconstructed shell command events.		** Show why milestones need reconstructed shell command events.
	* '''Phase 8:'''		* '''Phase 8:'''
	** Implement keystroke timing.		** Implement keystroke timing.
Line 596:		Line 596:
	; Goal		; Goal

	Identify the high-level event shape needed by a ~~regex~~ milestone processor.		Identify the high-level event shape needed by a milestone processor.

	; Discussion points		; Discussion points
Line 622:		Line 622:
	</pre>		</pre>

	Inspect the fields used by ~~regex~~ milestone checks:		Inspect the fields used by milestone checks:

	<pre lang="python">		<pre lang="python">
Line 660:		Line 660:
	; Goal		; Goal

	Create reusable payload classes for the TTY, shell command, and ~~regex~~ milestone examples.		Create reusable payload classes for the TTY, shell command, and milestone examples.

	; File edit		; File edit
Line 797:		Line 797:
	The event shape is now explicit. Other code can depend on a named <code>ShellCommandEvent</code> instead of assuming that a loose dictionary has the expected fields.		The event shape is now explicit. Other code can depend on a named <code>ShellCommandEvent</code> instead of assuming that a loose dictionary has the expected fields.

	== Phase 5: Evaluate ~~regex~~ milestones manually ==		== Phase 5: Evaluate milestones manually ==

	; Goal		; Goal
Line 805:		Line 805:
	; Discussion points		; Discussion points

	* A ~~regex~~ milestone expression names one check against one shell command field.		* A milestone expression names one check against one shell command field.
	* The field may be command input, command output, working directory, or host context.		* The field may be command input, command output, working directory, or host context.
	* The result records whether the expression matched and, when available, the matched text.		* The result records whether the expression matched and, when available, the matched text.
Line 1,163:		Line 1,163:
	* TTY reads, TTY writes, and line-discipline lines are lower-level observations.		* TTY reads, TTY writes, and line-discipline lines are lower-level observations.
	* A shell command event is reconstructed by correlating those observations.		* A shell command event is reconstructed by correlating those observations.
	* Keeping reconstruction separate prevents the ~~regex~~ milestone processor from becoming a large TTY parser.		* Keeping reconstruction separate prevents the milestone processor from becoming a large TTY parser.

	; Interpreter exploration		; Interpreter exploration
Line 1,431:		Line 1,431:
	; Goal		; Goal

	Switch to a small graph problem that can demonstrate feedback loops ~~safely~~.		Switch to a small graph problem that can demonstrate safe feedback loops in an illustratable form.

	; Discussion points		; Discussion points
Line 2,212:		Line 2,212:
	* The scheduler varies which processor gets a chance to run first.		* The scheduler varies which processor gets a chance to run first.
	* This simulates uncertainty from processing time, scheduling priority, startup timing, or transmission delay.		* This simulates uncertainty from processing time, scheduling priority, startup timing, or transmission delay.
	* The graph example should converge to the same final path facts even when processor scheduling, and ~~hence~~ the order of intermediate derived processor events, differs.		* The graph example should converge to the same final path facts even when processor scheduling - and the order of intermediate derived processor events - differs.

	; File edit		; File edit
Line 2,557:		Line 2,557:
	; Goal		; Goal

	Connect the TTY examples back to the shell command event shape used by ~~regex~~ milestones.		Connect the TTY examples back to the shell command event shape used by milestones.

	; Discussion points		; Discussion points

	* <code>ShellCommandEvent</code> is the high-level event consumed by the ~~regex~~ milestone processor.		* <code>ShellCommandEvent</code> is the high-level event consumed by the milestone processor.
	* <code>TTYReadEvent</code>, <code>TTYWriteEvent</code>, and <code>LineDisciplineEvent</code> are lower-level observations.		* <code>TTYReadEvent</code>, <code>TTYWriteEvent</code>, and <code>LineDisciplineEvent</code> are lower-level observations.
	* A reconstructor can subscribe to line-discipline events and query recent related observations.		* A reconstructor can subscribe to line-discipline events and query recent related observations.
Line 2,576:		Line 2,576:
	</pre>		</pre>

	This is an appropriate follow-up exercise after the team understands both ~~regex~~ milestone processing and history queries.		This is an appropriate follow-up exercise after the team understands both milestone processing and history queries.

	= Related =		= Related =

TTY BPF Instrument

Jwgranville — Tue, 16 Jun 2026 04:10:31 GMT

Message Bus Service

Jwgranville — Tue, 16 Jun 2026 04:09:33 GMT

← Older revision		Revision as of 04:09, 16 June 2026
Line 1:		Line 1:
	Module repository and wiki documentation are pending. This is a placeholder topic page to index resources related to the development of the message bus service.		Module repository and wiki documentation are pending. This is a placeholder topic page to index resources related to the development of the message bus service.

	== Related ==		= Related =

	=== Project Charter ===		== Project Charter ==
	[[Message Bus Project Charter]]		[[Message Bus Project Charter]]

	=== Examples, Tutorials, Supporting Documents ===		== Examples, Tutorials, Supporting Documents ==
	[[Message Bus Live Coding Exercise]]		[[Message Bus Live Coding Exercise]]

Message Bus Live Coding Exercise

Jwgranville — Tue, 16 Jun 2026 04:09:15 GMT

← Older revision		Revision as of 04:09, 16 June 2026
(One intermediate revision by the same user not shown)
Line 2,577:		Line 2,577:

	This is an appropriate follow-up exercise after the team understands both regex milestone processing and history queries.		This is an appropriate follow-up exercise after the team understands both regex milestone processing and history queries.

			= Related =

			== Top-level topic page ==
			[[Message Bus Service]]

Message Bus Service

Jwgranville — Tue, 16 Jun 2026 04:07:59 GMT

Created page with "Module repository and wiki documentation are pending. This is a placeholder topic page to index resources related to the development of the message bus service. == Related == === Project Charter === Message Bus Project Charter === Examples, Tutorials, Supporting Documents === Message Bus Live Coding Exercise"

New page

Module repository and wiki documentation are pending. This is a placeholder topic page to index resources related to the development of the message bus service.

== Related ==

=== Project Charter ===
[[Message Bus Project Charter]]

=== Examples, Tutorials, Supporting Documents ===
[[Message Bus Live Coding Exercise]]

Message Bus Live Coding Exercise

Jwgranville — Tue, 16 Jun 2026 04:06:29 GMT

Created page with "= Introduction = == Audience and scope == This exercise is intended for EDURange developers learning how to write event processors with the message bus. The packaged <code>mbus</code> and <code>aostore</code> modules are treated as read-only for this exercise. All new code belongs under a demonstration directory such as: <pre lang="text"> workspace/ mbus/ aostore/ demos/ __init__.py event_processors/ __init__.py </pre> The exe..."

Show changes

TTY BPF Instrument

Jwgranville — Sat, 06 Jun 2026 01:31:01 GMT

← Older revision		Revision as of 01:31, 6 June 2026
Line 49:		Line 49:

	To keep the components individually maintainable, it is crucial to preserve these boundaries. Tracepoints should stay minimal and justifiable to kernel maintainers. BPF code should stay bounded and observational. The spool should reduce pressure without inferring meaning. The collator should only aggregate when identity, sequence continuity, fragment completeness, and loss boundaries make aggregation safe. Storage policy, privacy enforcement, governance, and final analysis are outside the instrument itself unless a later design explicitly brings them in.		To keep the components individually maintainable, it is crucial to preserve these boundaries. Tracepoints should stay minimal and justifiable to kernel maintainers. BPF code should stay bounded and observational. The spool should reduce pressure without inferring meaning. The collator should only aggregate when identity, sequence continuity, fragment completeness, and loss boundaries make aggregation safe. Storage policy, privacy enforcement, governance, and final analysis are outside the instrument itself unless a later design explicitly brings them in.

			== Related ==

			=== Project Charter ===
			[[TTY BPF Instrument Project Charter]]

TTY BPF Instrument Project Charter

Jwgranville — Sat, 06 Jun 2026 01:29:09 GMT

Created page with "== Scoping Outline: == === Challenges: === * '''What:''' ''"Create ways to..." / "Redesign the..."'' ** Host-level observation instrument that can capture terminal byte activity at the kernel boundary without proxying, replacing, or modifying individual TTY devices. ** Modular BPF-based instrumentation pattern that preserves raw observations, identity context, ordering evidence, fragmentation metadata, timing calibration, and visible loss/status information, while keep..."

New page

== Scoping Outline: ==

=== Challenges: ===

* '''What:''' ''"Create ways to..." / "Redesign the..."''
** Host-level observation instrument that can capture terminal byte activity at the kernel boundary without proxying, replacing, or modifying individual TTY devices.
** Modular BPF-based instrumentation pattern that preserves raw observations, identity context, ordering evidence, fragmentation metadata, timing calibration, and visible loss/status information, while keeping semantic interpretation outside the probe.
** Tracepoint strategy that is narrow enough to be plausible for upstream kernel acceptance, so long-term use of the instrument does not depend on a small team maintaining private kernel patches indefinitely.
* '''For Whom:''' ''"For <user>... (considering <other stakeholders>)..."''
** Developers, researchers, and infrastructure maintainers who need trustworthy records of terminal activity from multi-user Linux hosts, while preserving enough context for later reconstruction, validation, audit, or analysis.
** Small research and instructional infrastructure teams that can use BPF-based host instrumentation, but may not have enough durable kernel expertise to maintain private patch sets indefinitely.
** Future contributors and outside adopters who need the instrument to be understandable, modular, and useful outside its original project setting.
** Considering kernel maintainers as critical design stakeholders: the tracepoint surface must remain minimal, general-purpose, and reviewable as kernel instrumentation, rather than treating the kernel as a project-specific logging subsystem.
** Considering downstream consumers such as storage, analysis, privacy, governance, and reporting systems: the instrument should emit factual records with enough context to support those systems, without owning their policies or interpretations.
* '''Context:''' ''"In a world where..." / "Keeping in mind that..."''
** Terminal activity is transient: if TTY byte activity is not observed as it occurs, the exact bytes, timing, ordering context, and kernel-side identity context cannot be reconstructed later from ordinary application state.
** TTY activity is structurally ambiguous: an apparent terminal session may involve shells, subprocesses, pseudoterminals, process privilege changes, etc. The instrument therefore needs to preserve raw identity axes rather than prematurely deciding what counts as a user, session, stream, or command.
** Raw terminal bytes are sensitive: the API consumers should not treat capture as endorsement of unrestricted use. Privacy classification, redaction, retention, consent, access control, and governance belong to deployment policy and downstream systems - the instrument's role is to expose system state, so governance is an explicit responsibility of the user.
** Private kernel patch maintenance is costly for a small research/infrastructure team: the tracepoint design should remain narrow, reviewable, and plausible for upstream acceptance wherever possible.
** The module should remain application-neutral: EDURange motivates the first deployment and validation workload, but the instrument should be useful to outside users who need careful host-level TTY observation. As an open-source module, EDURange benefits from the instrument's broader circulation and adoption.

=== Considerations: ===

* '''Goals:''' ''"We aim to..."''
** Provide an application-neutral TTY observation instrument that can be used by both EDURange and by outside teams without embedding EDURange-specific assumptions into the core design.
** Preserve captured terminal activity as raw bytes, not decoded commands, prompts, lines, or student actions.
** Keep kernel-space behavior bounded, observational, and non-interpretive: the probe should copy bytes, attach factual context, assign ordering metadata, and emit records, not analyze terminal content.
** Make loss visible. Capturing less data with accurate loss/status reporting is preferable to capturing more data whose completeness cannot be trusted.
** Make sequence accounting, not timestamps alone, the primary basis for ordering and loss placement.
** Preserve identity as an orthogonal vector of observed facts, not collapsed into a single inferred session identity.
** Keep the architectural responsibilities clear: tracepoints expose attachment surfaces; BPF probes observe; the spool drains and forwards; and the collator reconstructs. Only downstream consumers interpret, store, redact, govern, or analyze.
** Make the TTY instrument a reusable model for future host-level instruments, while avoiding premature generalization before the TTY capture path is empirically validated.
* '''Crux:''' ''"We really need to..."''
** Produce trustworthy host-wide TTY observations without turning the kernel into a private logging subsystem.
** Determine whether the instrument can preserve enough byte, identity, ordering, timing, fragmentation, and loss context for downstream reconstruction while staying non-disruptive under realistic multi-user load.
** Find the smallest maintainable MVP that demonstrates the full path from kernel observation to user-space reconstruction, while keeping open the long-term path toward upstream tracepoints and broader adoption.

== Framing Checklist: ==
We should discuss these qualities:

* You tried on some behaviors (like empathy and rapid prototyping) on a current project before launching a new one.
* Project is a human, subjective challenge (understanding people is key to the project success).
* Project is geared toward discovery (not optimization).
* Challenge can be solved with a product, service, or event, not a strategy or system (for your first project).
* Those most affected by the work are acknowledged as actors with agency, not simply receivers of outcomes.
* Framing doesn’t embed a solution.
* Framing doesn’t (unintentionally) assume the form of the solution.
* Framing doesn’t (unintentionally) presume people’s needs.
* Goal of the project work is clear without dictating the specific solution outcome.
* You and your team actually care about this challenge. (If not, why are you doing it?)

New Version Planning

Jwgranville — Fri, 05 Jun 2026 23:55:32 GMT

Links to Subprojects/Related Topics

← Older revision		Revision as of 23:55, 5 June 2026
Line 93:		Line 93:

	* [[Data Store Project Charter]] - just the charter; pending a top-level topic page		* [[Data Store Project Charter]] - just the charter; pending a top-level topic page
			* [[Message Bus Project Charter]] - also pending top-level page
	* [[TTY BPF Instrument]] - mostly a placeholder topic page; links to repos/docs		* [[TTY BPF Instrument]] - mostly a placeholder topic page; links to repos/docs

Message Bus Project Charter

Jwgranville — Fri, 05 Jun 2026 23:54:48 GMT

Created page with "== Scoping Outline: == === Challenges: === * '''What:''' ''"Create ways to..." / "Redesign the..."'' ** Design a local message bus that allows independently developed software components to communicate through typed broadcast topics without tightly coupling producers to consumers. ** Create a communication substrate whose traffic can be captured, replayed, and inspected, so that system behavior can be reconstructed after the fact. ** Support early-stage teams in which..."

New page

== Scoping Outline: ==

=== Challenges: ===

* '''What:''' ''"Create ways to..." / "Redesign the..."''
** Design a local message bus that allows independently developed software components to communicate through typed broadcast topics without tightly coupling producers to consumers.
** Create a communication substrate whose traffic can be captured, replayed, and inspected, so that system behavior can be reconstructed after the fact.
** Support early-stage teams in which contributors may be junior developers, by making the ordinary publishing and subscribing interface simple while centralizing transport, provenance, and capture policy.
* '''For Whom:''' ''"For <user>... (considering <other stakeholders>)..."''
** For those building modular, event-driven experimental applications, especially teams where independently authored processors, interfaces, and services need to cooperate through stable contracts.
** For maintainers, researchers, and outside collaborators who need replayable records of component interaction sufficient for debugging, validation, and post hoc analysis.
* '''Context:''' ''"In a world where..." / "Keeping in mind that..."''
** Keeping in mind that small research and academic software teams often need many contributors to work in parallel before all production infrastructure is complete.
** Keeping in mind that junior contributors should be able to write useful processors and application components without personally implementing persistence, provenance, or transport policy.
** Keeping in mind that event streams and request/reply interactions between components may become important evidence when reconstructing system behavior.

=== Considerations: ===

* '''Goals:''' ''"We aim to..."''
** Provide a local publish-subscribe API with minimal friction for ordinary component authors.
** Provide transport-level message identity, topic identity, writer identity, sequence numbering, and bus ingress timing.
** Support broad capture of messages crossing component boundaries so runs can be replayed or inspected.
** Support request/reply interactions without requiring direct component coupling.
** Support fixture playback and mock services so processors can be developed before production instruments, stores, or external integrations are ready.
** Allow multiple processors or interfaces to operate in parallel for comparison, A/B testing, and fault isolation.
** Remain self-contained enough to be used outside the initial use case application that motivated it.
* '''Crux:''' ''"We really need to..."''
** Make component communication simple for contributors while making message flow accountable enough for replay, debugging, and later research use.
** Centralize transport discipline and capture-friendly metadata without forcing every component author to understand the full governance model.

== Framing Checklist: ==
We should discuss these qualities:

* You tried on some behaviors (like empathy and rapid prototyping) on a current project before launching a new one.
* Project is a human, subjective challenge (understanding people is key to the project success).
* Project is geared toward discovery (not optimization).
* Challenge can be solved with a product, service, or event, not a strategy or system (for your first project).
* Those most affected by the work are acknowledged as actors with agency, not simply receivers of outcomes.
* Framing doesn’t embed a solution.
* Framing doesn’t (unintentionally) assume the form of the solution.
* Framing doesn’t (unintentionally) presume people’s needs.
* Goal of the project work is clear without dictating the specific solution outcome.
* You and your team actually care about this challenge. (If not, why are you doing it?)

TTY BPF Instrument

Jwgranville — Thu, 04 Jun 2026 16:44:04 GMT

Prototype Lineage

← Older revision		Revision as of 16:44, 4 June 2026
(One intermediate revision by the same user not shown)
Line 38:		Line 38:
	Loss accounting is a first-class part of the instrument. The design explicitly prefers capturing less data with accurate loss reporting over capturing more data whose completeness cannot be trusted. Loss can occur in the kernel probe, spool, collator, or downstream ingestion path; where possible, loss reports should identify the origin, affected sequence range, affected identity context, and lost record or byte counts.		Loss accounting is a first-class part of the instrument. The design explicitly prefers capturing less data with accurate loss reporting over capturing more data whose completeness cannot be trusted. Loss can occur in the kernel probe, spool, collator, or downstream ingestion path; where possible, loss reports should identify the origin, affected sequence range, affected identity context, and lost record or byte counts.

	The auxiliary/status channel exists because status records are needed to interpret captured data. Health/status records may describe loss, backpressure, saturation, calibration, drift, lifecycle transitions, verifier failures, spool pressure, collator pressure, or reconstruction ambiguity. These records describe the behavior and reliability of the instrument, not additional user activity.		The auxiliary/status channel exists because status records are needed to interpret captured data, but must not contend for transport resources with observational records. Health/status records may describe loss, backpressure, saturation, calibration, drift, lifecycle transitions, verifier failures, spool pressure, collator pressure, or reconstruction ambiguity. These records describe the behavior and reliability of the instrument, not additional user activity.

	=== Prototype Lineage ===		=== Prototype Lineage ===
	The instrument’s proof of concept is ~~<nowiki>~~https://github.com/edurange/demo-bpf-tty-logger/tree/main~~</nowiki>~~, which demonstrates host-wide TTY capture using BPF kernel probes. That demo shows the basic idea: instead of attaching to one TTY device, it hooks kernel activity system-wide, captures raw buffers and context, and warns about feedback loops if the logger prints to a TTY it is also observing.		The instrument’s proof of concept is https://github.com/edurange/demo-bpf-tty-logger/tree/main, which demonstrates host-wide TTY capture using BPF kernel probes. That demo shows the basic idea: instead of attaching to one TTY device, it hooks kernel activity system-wide, captures raw buffers and context, and warns about feedback loops if the logger prints to a TTY it is also observing.

	The newer prototype, reflected in ~~<nowiki>~~https://github.com/edurange/prototype-tty-bpf-instrument~~</nowiki>~~, moves beyond the early BCC/kprobe sketch toward the current tracepoint/CO-RE direction. It contains a kernel patch for tracepoints, BPF probe code, shared observation ABI, constants, event maps, CPU-local sequence state, and a user-space dump tool. That archive should be treated as the current implementation reference, while the formal design document remains the higher-level specification.		The newer prototype, reflected in https://github.com/edurange/prototype-tty-bpf-instrument, moves beyond the early BCC/kprobe sketch toward the current tracepoint/CO-RE direction. It contains a kernel patch for tracepoints, BPF probe code, shared observation ABI, constants, event maps, CPU-local sequence state, and a user-space dump tool. That archive should be treated as the current implementation reference, while the formal design document remains the higher-level specification.

	=== Design Boundaries and Non-Goals ===		=== Design Boundaries and Non-Goals ===

TTY BPF Instrument

Jwgranville — Thu, 04 Jun 2026 16:17:09 GMT

Created page with "Demonstrations and preliminary documentation can be found at https://github.com/edurange/demo-bpf-tty-logger and https://github.com/edurange/prototype-tty-bpf-instrument. Wiki documentation is pending. Preliminary draft files are available in the Discord <code>#fileshare</code> channel and will be linked here once Wiki configuration is adjusted to support attached non-image documents. == Summary == The TTY instrument is a BPF-based kernel observation tool for capturing..."

New page

Demonstrations and preliminary documentation can be found at https://github.com/edurange/demo-bpf-tty-logger and https://github.com/edurange/prototype-tty-bpf-instrument. Wiki documentation is pending. Preliminary draft files are available in the Discord <code>#fileshare</code> channel and will be linked here once Wiki configuration is adjusted to support attached non-image documents.

== Summary ==
The TTY instrument is a BPF-based kernel observation tool for capturing terminal activity across a host system. Its purpose is to observe TTY byte activity at the kernel boundary and emit records that downstream components can reconstruct, validate, and analyze. The current design treats terminal activity as factual kernel observations first, and reserves higher-level ideas like sessions, users, commands, and learning activity for later interpretation rather than things the probe is responsible for determining.

Although the immediate target is TTY activity, the design is also meant to serve as a template for future host-level instruments. It separates kernel observation, bounded transport, user-space draining, reconstruction, and downstream interpretation in a way that can be reused for other event sources. The TTY instrument is intentionally demanding: it involves raw byte payloads, ambiguous identity, ordering challenges, fragmentation, loss accounting, and privacy-sensitive data. If this pattern works for TTY activity, it provides a strong example for building other instruments with similar fidelity and reliability requirements - for the kernel-level services like the filesystem or network interfaces, common user-level processes like `bash`, etc.

=== Why TTY Capture Is Difficult ===
This project is difficult because TTY activity is not a single clean stream. A user’s apparent terminal session can pass through SSH, shells, subprocesses, pseudoterminals, terminal echo, line discipline behavior, containers, namespaces, and reconnects. The instrument therefore preserves multiple identity axes, including process IDs, user IDs, cgroups, namespaces, TTY device numbers, inode context, and command/process names. No one field is treated as “the session” or “the user.” Streams or sessions may later be reconstructed from these facts, but the raw record should keep the original evidence intact.

=== Tracepoints, BPF, and Kernel Boundaries ===
The current implementation is tracepoint-based. Kernel tracepoints provide minimal, upstream-plausible attachment surfaces; BPF probes attach to those tracepoints, read bounded payload buffers, enrich records with available identity/timing/ordering context, and emit records through BPF ring buffers. The tracepoint is not the logging schema. The BPF observation record is the instrument-facing schema.

The current prototype includes kernel modifications because the existing kernel does not expose all of the stable attachment surfaces needed for this instrument. Those modifications add narrowly scoped TTY tracepoints so BPF programs can attach at the relevant read, write, and line-discipline receive paths. The goal is not to move logging policy into the kernel or to make the kernel produce the full instrument schema.

The kernel changes must remain minimal, reviewable, and justifiable as tracepoint support; the BPF probe and user-space components remain responsible for the instrument-specific observation record, enrichment, transport, and reconstruction behavior. This distinction matters because tracepoints need to be acceptable to upstream kernel maintainers as general kernel instrumentation, while the richer record schema is specific to this project’s BPF instrument.

=== Pipeline and Component Responsibilities ===
The TTY instrument is also important because it represents a characteristic high-pressure workload for the larger event pipeline. Human terminal input can produce many small payloads with high metadata overhead, while command output can produce sudden bursts of larger byte streams. Multi-user SSH workloads combine both patterns. This makes the instrument a useful benchmark for the event bus and downstream data store: it exercises throughput, buffering, ordering, loss reporting, payload preservation, and reconstruction under conditions that are demanding yet meaningful with a minimal number of tracepoints and probe sites. (Contrast with the filesystem, where roughly a dozen such sites are needed to capture the wider variety of filesystem operations.) The TTY instrument does not define the bus or storage policy, but its output should help reveal whether those downstream systems can handle realistic observational pressure without hiding loss or collapsing important context.

The basic pipeline is:<pre>
TTY kernel paths
-> kernel tracepoints
-> BPF probes
-> primary observation channel + auxiliary/status channel
-> user-space spool
-> collator
-> downstream storage, analysis, policy, or reporting
</pre>The probe’s job is bounded observation: copy raw bytes, attach factual context, assign ordering metadata, and emit records. The spool’s job is to drain kernel output quickly and forward it onward, relieving pressure on the kernel ring buffer channels. The collator’s job is reconstruction: reassemble fragments, check sequence continuity, place loss, resolve safe aggregations, and prepare externally useful event representations.

=== Core Fidelity Invariants ===
A central invariant is that captured bytes are preserved as bytes. The probe does not decode Unicode, detect commands, infer prompts, identify student intent, or decide where lines begin and end. Payload may contain terminal control sequences, partial multi-byte characters, shell output, pasted text, or fragments of larger observations - the data is handled the same at the instrument level regardless of its nature. Decoding and semantic interpretation happen downstream.

Another central invariant is that fragmentation must be explicit. A single observed TTY occurrence may require multiple emitted records if the payload is too large for the bounded BPF record size. Fragmentation is not loss; it is just the transport representation of one observation. If bytes cannot be emitted, the missing portion must remain visible through loss/status reporting rather than being hidden by truncation or best-effort concatenation.

Ordering is based on sequence accounting, not timestamps alone. CPU-local sequence numbers are useful now for validating local emission order, fragmentation behavior, and probe diagnostics. The target design adds monotonic per-stream sequence numbers for authoritative stream-level ordering and loss placement. Timestamps are still important, but they support wall-clock calibration and approximate correlation rather than serving as the primary proof of order.

Loss accounting is a first-class part of the instrument. The design explicitly prefers capturing less data with accurate loss reporting over capturing more data whose completeness cannot be trusted. Loss can occur in the kernel probe, spool, collator, or downstream ingestion path; where possible, loss reports should identify the origin, affected sequence range, affected identity context, and lost record or byte counts.

The auxiliary/status channel exists because status records are needed to interpret captured data. Health/status records may describe loss, backpressure, saturation, calibration, drift, lifecycle transitions, verifier failures, spool pressure, collator pressure, or reconstruction ambiguity. These records describe the behavior and reliability of the instrument, not additional user activity.

=== Prototype Lineage ===
The instrument’s proof of concept is <nowiki>https://github.com/edurange/demo-bpf-tty-logger/tree/main</nowiki>, which demonstrates host-wide TTY capture using BPF kernel probes. That demo shows the basic idea: instead of attaching to one TTY device, it hooks kernel activity system-wide, captures raw buffers and context, and warns about feedback loops if the logger prints to a TTY it is also observing.

The newer prototype, reflected in <nowiki>https://github.com/edurange/prototype-tty-bpf-instrument</nowiki>, moves beyond the early BCC/kprobe sketch toward the current tracepoint/CO-RE direction. It contains a kernel patch for tracepoints, BPF probe code, shared observation ABI, constants, event maps, CPU-local sequence state, and a user-space dump tool. That archive should be treated as the current implementation reference, while the formal design document remains the higher-level specification.

=== Design Boundaries and Non-Goals ===
The instrument is not meant to be a simple keylogger, even though it necessarily captures raw terminal bytes. It is an instrument for producing trustworthy observational records from kernel TTY activity under realistic multi-user workloads. The important work is preserving factual byte activity, identity context, ordering evidence, timing calibration, fragmentation metadata, and loss visibility without letting the kernel probe become an analyzer. The probe observes; the spool drains; and the collator reconstructs. Interpretation is the domain of downstream systems that consume the instrument’s output.

To keep the components individually maintainable, it is crucial to preserve these boundaries. Tracepoints should stay minimal and justifiable to kernel maintainers. BPF code should stay bounded and observational. The spool should reduce pressure without inferring meaning. The collator should only aggregate when identity, sequence continuity, fragment completeness, and loss boundaries make aggregation safe. Storage policy, privacy enforcement, governance, and final analysis are outside the instrument itself unless a later design explicitly brings them in.

New Version Planning

Jwgranville — Thu, 04 Jun 2026 15:56:44 GMT

← Older revision		Revision as of 15:56, 4 June 2026
Line 88:		Line 88:
	* The '''Learning Environment''' can offer '''Interventions''' based on '''Student''' performance		* The '''Learning Environment''' can offer '''Interventions''' based on '''Student''' performance
	* '''Students''' start and stop the '''Learning Environment''' of '''Scenarios''' made accessible to them (the '''Learning Environment''' does not consume resources when not in use)		* '''Students''' start and stop the '''Learning Environment''' of '''Scenarios''' made accessible to them (the '''Learning Environment''' does not consume resources when not in use)

			== Links to Subprojects/Related Topics ==
			This section is temporary until we settle for a multi-topic organization for new version planning.

			* [[Data Store Project Charter]] - just the charter; pending a top-level topic page
			* [[TTY BPF Instrument]] - mostly a placeholder topic page; links to repos/docs

Scenario 0

Jwgranville — Thu, 04 Jun 2026 15:51:27 GMT

Jwgranville deleted page Scenario 0 Orphaned. Content was: "This page is dedicated to holding information and notes about Scenario 0.", and the only contributor was "CLSimpson" (talk)

← Older revision		Revision as of 04:10, 16 June 2026
Line 1:		Line 1:
	Demonstrations and preliminary documentation can be found at https://github.com/edurange/demo-bpf-tty-logger and https://github.com/edurange/prototype-tty-bpf-instrument. Wiki documentation is pending. Preliminary draft files are available in the Discord <code>#fileshare</code> channel and will be linked here once Wiki configuration is adjusted to support attached non-image documents.		Demonstrations and preliminary documentation can be found at https://github.com/edurange/demo-bpf-tty-logger and https://github.com/edurange/prototype-tty-bpf-instrument. Wiki documentation is pending. Preliminary draft files are available in the Discord <code>#fileshare</code> channel and will be linked here once Wiki configuration is adjusted to support attached non-image documents.

	== Summary ==		= Summary =
	The TTY instrument is a BPF-based kernel observation tool for capturing terminal activity across a host system. Its purpose is to observe TTY byte activity at the kernel boundary and emit records that downstream components can reconstruct, validate, and analyze. The current design treats terminal activity as factual kernel observations first, and reserves higher-level ideas like sessions, users, commands, and learning activity for later interpretation rather than things the probe is responsible for determining.		The TTY instrument is a BPF-based kernel observation tool for capturing terminal activity across a host system. Its purpose is to observe TTY byte activity at the kernel boundary and emit records that downstream components can reconstruct, validate, and analyze. The current design treats terminal activity as factual kernel observations first, and reserves higher-level ideas like sessions, users, commands, and learning activity for later interpretation rather than things the probe is responsible for determining.

	Although the immediate target is TTY activity, the design is also meant to serve as a template for future host-level instruments. It separates kernel observation, bounded transport, user-space draining, reconstruction, and downstream interpretation in a way that can be reused for other event sources. The TTY instrument is intentionally demanding: it involves raw byte payloads, ambiguous identity, ordering challenges, fragmentation, loss accounting, and privacy-sensitive data. If this pattern works for TTY activity, it provides a strong example for building other instruments with similar fidelity and reliability requirements - for the kernel-level services like the filesystem or network interfaces, common user-level processes like `bash`, etc.		Although the immediate target is TTY activity, the design is also meant to serve as a template for future host-level instruments. It separates kernel observation, bounded transport, user-space draining, reconstruction, and downstream interpretation in a way that can be reused for other event sources. The TTY instrument is intentionally demanding: it involves raw byte payloads, ambiguous identity, ordering challenges, fragmentation, loss accounting, and privacy-sensitive data. If this pattern works for TTY activity, it provides a strong example for building other instruments with similar fidelity and reliability requirements - for the kernel-level services like the filesystem or network interfaces, common user-level processes like `bash`, etc.

	=== Why TTY Capture Is Difficult ===		== Why TTY Capture Is Difficult ==
	This project is difficult because TTY activity is not a single clean stream. A user’s apparent terminal session can pass through SSH, shells, subprocesses, pseudoterminals, terminal echo, line discipline behavior, containers, namespaces, and reconnects. The instrument therefore preserves multiple identity axes, including process IDs, user IDs, cgroups, namespaces, TTY device numbers, inode context, and command/process names. No one field is treated as “the session” or “the user.” Streams or sessions may later be reconstructed from these facts, but the raw record should keep the original evidence intact.		This project is difficult because TTY activity is not a single clean stream. A user’s apparent terminal session can pass through SSH, shells, subprocesses, pseudoterminals, terminal echo, line discipline behavior, containers, namespaces, and reconnects. The instrument therefore preserves multiple identity axes, including process IDs, user IDs, cgroups, namespaces, TTY device numbers, inode context, and command/process names. No one field is treated as “the session” or “the user.” Streams or sessions may later be reconstructed from these facts, but the raw record should keep the original evidence intact.

	=== Tracepoints, BPF, and Kernel Boundaries ===		== Tracepoints, BPF, and Kernel Boundaries ==
	The current implementation is tracepoint-based. Kernel tracepoints provide minimal, upstream-plausible attachment surfaces; BPF probes attach to those tracepoints, read bounded payload buffers, enrich records with available identity/timing/ordering context, and emit records through BPF ring buffers. The tracepoint is not the logging schema. The BPF observation record is the instrument-facing schema.		The current implementation is tracepoint-based. Kernel tracepoints provide minimal, upstream-plausible attachment surfaces; BPF probes attach to those tracepoints, read bounded payload buffers, enrich records with available identity/timing/ordering context, and emit records through BPF ring buffers. The tracepoint is not the logging schema. The BPF observation record is the instrument-facing schema.

Line 16:		Line 16:
	The kernel changes must remain minimal, reviewable, and justifiable as tracepoint support; the BPF probe and user-space components remain responsible for the instrument-specific observation record, enrichment, transport, and reconstruction behavior. This distinction matters because tracepoints need to be acceptable to upstream kernel maintainers as general kernel instrumentation, while the richer record schema is specific to this project’s BPF instrument.		The kernel changes must remain minimal, reviewable, and justifiable as tracepoint support; the BPF probe and user-space components remain responsible for the instrument-specific observation record, enrichment, transport, and reconstruction behavior. This distinction matters because tracepoints need to be acceptable to upstream kernel maintainers as general kernel instrumentation, while the richer record schema is specific to this project’s BPF instrument.

	=== Pipeline and Component Responsibilities ===		== Pipeline and Component Responsibilities ==
	The TTY instrument is also important because it represents a characteristic high-pressure workload for the larger event pipeline. Human terminal input can produce many small payloads with high metadata overhead, while command output can produce sudden bursts of larger byte streams. Multi-user SSH workloads combine both patterns. This makes the instrument a useful benchmark for the event bus and downstream data store: it exercises throughput, buffering, ordering, loss reporting, payload preservation, and reconstruction under conditions that are demanding yet meaningful with a minimal number of tracepoints and probe sites. (Contrast with the filesystem, where roughly a dozen such sites are needed to capture the wider variety of filesystem operations.) The TTY instrument does not define the bus or storage policy, but its output should help reveal whether those downstream systems can handle realistic observational pressure without hiding loss or collapsing important context.		The TTY instrument is also important because it represents a characteristic high-pressure workload for the larger event pipeline. Human terminal input can produce many small payloads with high metadata overhead, while command output can produce sudden bursts of larger byte streams. Multi-user SSH workloads combine both patterns. This makes the instrument a useful benchmark for the event bus and downstream data store: it exercises throughput, buffering, ordering, loss reporting, payload preservation, and reconstruction under conditions that are demanding yet meaningful with a minimal number of tracepoints and probe sites. (Contrast with the filesystem, where roughly a dozen such sites are needed to capture the wider variety of filesystem operations.) The TTY instrument does not define the bus or storage policy, but its output should help reveal whether those downstream systems can handle realistic observational pressure without hiding loss or collapsing important context.

Line 29:		Line 29:
	</pre>The probe’s job is bounded observation: copy raw bytes, attach factual context, assign ordering metadata, and emit records. The spool’s job is to drain kernel output quickly and forward it onward, relieving pressure on the kernel ring buffer channels. The collator’s job is reconstruction: reassemble fragments, check sequence continuity, place loss, resolve safe aggregations, and prepare externally useful event representations.		</pre>The probe’s job is bounded observation: copy raw bytes, attach factual context, assign ordering metadata, and emit records. The spool’s job is to drain kernel output quickly and forward it onward, relieving pressure on the kernel ring buffer channels. The collator’s job is reconstruction: reassemble fragments, check sequence continuity, place loss, resolve safe aggregations, and prepare externally useful event representations.

	=== Core Fidelity Invariants ===		== Core Fidelity Invariants ==
	A central invariant is that captured bytes are preserved as bytes. The probe does not decode Unicode, detect commands, infer prompts, identify student intent, or decide where lines begin and end. Payload may contain terminal control sequences, partial multi-byte characters, shell output, pasted text, or fragments of larger observations - the data is handled the same at the instrument level regardless of its nature. Decoding and semantic interpretation happen downstream.		A central invariant is that captured bytes are preserved as bytes. The probe does not decode Unicode, detect commands, infer prompts, identify student intent, or decide where lines begin and end. Payload may contain terminal control sequences, partial multi-byte characters, shell output, pasted text, or fragments of larger observations - the data is handled the same at the instrument level regardless of its nature. Decoding and semantic interpretation happen downstream.

Line 40:		Line 40:
	The auxiliary/status channel exists because status records are needed to interpret captured data, but must not contend for transport resources with observational records. Health/status records may describe loss, backpressure, saturation, calibration, drift, lifecycle transitions, verifier failures, spool pressure, collator pressure, or reconstruction ambiguity. These records describe the behavior and reliability of the instrument, not additional user activity.		The auxiliary/status channel exists because status records are needed to interpret captured data, but must not contend for transport resources with observational records. Health/status records may describe loss, backpressure, saturation, calibration, drift, lifecycle transitions, verifier failures, spool pressure, collator pressure, or reconstruction ambiguity. These records describe the behavior and reliability of the instrument, not additional user activity.

	=== Prototype Lineage ===		== Prototype Lineage ==
	The instrument’s proof of concept is https://github.com/edurange/demo-bpf-tty-logger/tree/main, which demonstrates host-wide TTY capture using BPF kernel probes. That demo shows the basic idea: instead of attaching to one TTY device, it hooks kernel activity system-wide, captures raw buffers and context, and warns about feedback loops if the logger prints to a TTY it is also observing.		The instrument’s proof of concept is https://github.com/edurange/demo-bpf-tty-logger/tree/main, which demonstrates host-wide TTY capture using BPF kernel probes. That demo shows the basic idea: instead of attaching to one TTY device, it hooks kernel activity system-wide, captures raw buffers and context, and warns about feedback loops if the logger prints to a TTY it is also observing.

	The newer prototype, reflected in https://github.com/edurange/prototype-tty-bpf-instrument, moves beyond the early BCC/kprobe sketch toward the current tracepoint/CO-RE direction. It contains a kernel patch for tracepoints, BPF probe code, shared observation ABI, constants, event maps, CPU-local sequence state, and a user-space dump tool. That archive should be treated as the current implementation reference, while the formal design document remains the higher-level specification.		The newer prototype, reflected in https://github.com/edurange/prototype-tty-bpf-instrument, moves beyond the early BCC/kprobe sketch toward the current tracepoint/CO-RE direction. It contains a kernel patch for tracepoints, BPF probe code, shared observation ABI, constants, event maps, CPU-local sequence state, and a user-space dump tool. That archive should be treated as the current implementation reference, while the formal design document remains the higher-level specification.

	=== Design Boundaries and Non-Goals ===		== Design Boundaries and Non-Goals ==
	The instrument is not meant to be a simple keylogger, even though it necessarily captures raw terminal bytes. It is an instrument for producing trustworthy observational records from kernel TTY activity under realistic multi-user workloads. The important work is preserving factual byte activity, identity context, ordering evidence, timing calibration, fragmentation metadata, and loss visibility without letting the kernel probe become an analyzer. The probe observes; the spool drains; and the collator reconstructs. Interpretation is the domain of downstream systems that consume the instrument’s output.		The instrument is not meant to be a simple keylogger, even though it necessarily captures raw terminal bytes. It is an instrument for producing trustworthy observational records from kernel TTY activity under realistic multi-user workloads. The important work is preserving factual byte activity, identity context, ordering evidence, timing calibration, fragmentation metadata, and loss visibility without letting the kernel probe become an analyzer. The probe observes; the spool drains; and the collator reconstructs. Interpretation is the domain of downstream systems that consume the instrument’s output.

	To keep the components individually maintainable, it is crucial to preserve these boundaries. Tracepoints should stay minimal and justifiable to kernel maintainers. BPF code should stay bounded and observational. The spool should reduce pressure without inferring meaning. The collator should only aggregate when identity, sequence continuity, fragment completeness, and loss boundaries make aggregation safe. Storage policy, privacy enforcement, governance, and final analysis are outside the instrument itself unless a later design explicitly brings them in.		To keep the components individually maintainable, it is crucial to preserve these boundaries. Tracepoints should stay minimal and justifiable to kernel maintainers. BPF code should stay bounded and observational. The spool should reduce pressure without inferring meaning. The collator should only aggregate when identity, sequence continuity, fragment completeness, and loss boundaries make aggregation safe. Storage policy, privacy enforcement, governance, and final analysis are outside the instrument itself unless a later design explicitly brings them in.

	== Related ==		= Related =

	=== Project Charter ===		== Project Charter ==
	[[TTY BPF Instrument Project Charter]]		[[TTY BPF Instrument Project Charter]]