EDURange - Recent changes [en]

Brief: EDURange is More Than a CRUD Architecture

Jwgranville — Fri, 19 Jun 2026 03:30:56 GMT

← Older revision		Revision as of 03:30, 19 June 2026
(2 intermediate revisions by the same user not shown)
Line 1:		Line 1:
	~~{{DISPLAYTITLE~~:EDURange is More Than a CRUD Architecture}}		= Brief: EDURange is More Than a CRUD Architecture =

	~~= Brief: EDURange~~ is ~~More Than~~ a ~~CRUD Architecture =~~		This is a brief explaining my position on adopting message-based, microservice architecture and the model-driven development process. This is the first page of its kind so I'm just guessing at the format. This top level "brief" section is in my own words but contributors should feel free to create other sections or add to the header content (above the "brief" section title) as needed. ~Joe G

	== The Up-Front Cost We Are Choosing to Pay ==		== The Up-Front Cost We Are Choosing to Pay ==
Line 51:		Line 51:
	A domain model is well suited to serve as the kernel of a complex unit of software because it depends on no outside code or system. It is not the database layer, the UI layer, the web framework, or the message bus. It is the part of the program that implements the logical rules we want the rest of the system to manifest. Its only job is to represent or calculate what is true or known under those logical rules. Once those rules are explicit, we can build layers around them which preserve their properties and truthfulness: persistence, messaging, APIs, user interfaces, tests, simulations, and analysis tools. Those outer layers may be influenced by outside systems, but the core model gives them something stable to serve as a reference for correct behavior.		A domain model is well suited to serve as the kernel of a complex unit of software because it depends on no outside code or system. It is not the database layer, the UI layer, the web framework, or the message bus. It is the part of the program that implements the logical rules we want the rest of the system to manifest. Its only job is to represent or calculate what is true or known under those logical rules. Once those rules are explicit, we can build layers around them which preserve their properties and truthfulness: persistence, messaging, APIs, user interfaces, tests, simulations, and analysis tools. Those outer layers may be influenced by outside systems, but the core model gives them something stable to serve as a reference for correct behavior.

	This matters especially for intelligent tutoring. ~~Domain models are~~ not only useful for organizing code. ~~They are~~ useful for observing, interpreting, and predicting the behavior of complex systems or relationships.		This matters especially for intelligent tutoring and other AI applications. Model-driven development is not only useful for organizing code. It is useful for observing, interpreting, and predicting the behavior of complex systems or relationships.

	What is the student doing? What is the computer system doing? What happened in the learning environment that the student might have observed? What evidence suggests the student understands the underlying concept? Why does a problem have a particular solution? Which actions are equivalent, misleading, partial, or strategically meaningful? Those are modeling questions before they are implementation questions. A desirable solution doesn’t just have implementation steps: there is a concrete and logical reason why that particular implementation is correct, and a definition of the scope or conditions under which the correctness holds.		What is the student doing? What is the computer system doing? What happened in the learning environment that the student might have observed? What evidence suggests the student understands the underlying concept? Why does a problem have a particular solution? Which actions are equivalent, misleading, partial, or strategically meaningful? Those are modeling questions before they are implementation questions. A desirable solution doesn’t just have implementation steps: there is a concrete and logical reason why that particular implementation is correct, and a definition of the scope or conditions under which the correctness holds.

Message Bus Service

Jwgranville — Fri, 19 Jun 2026 03:14:55 GMT

← Older revision		Revision as of 03:14, 19 June 2026
Line 3:		Line 3:
	== Related ==		== Related ==

	=== ~~Project Charter~~ ===		=== Design Documents: ===
	~~[[Message Bus Project Charter]]~~

	=== Examples, Tutorials~~, Supporting Documents~~ ===		* [[Message Bus Project Charter]]
	[[Message Bus Live Coding Exercise]]		* [[Brief: EDURange is More Than a CRUD Architecture]]

			=== Examples & Tutorials: ===

			* [[Message Bus Live Coding Exercise]]

New Version Planning

Jwgranville — Fri, 19 Jun 2026 03:13:40 GMT

Links to Subprojects/Related Topics

← Older revision		Revision as of 03:13, 19 June 2026
Line 92:		Line 92:
	This section is temporary until we settle for a multi-topic organization for new version planning.		This section is temporary until we settle for a multi-topic organization for new version planning.

			=== Services/Modules: ===
	* [[Data Store Project Charter]] - just the charter; pending a top-level topic page		* [[Data Store Project Charter]] - just the charter; pending a top-level topic page
	* [[Message Bus Service]] - mostly a placeholder topic page; links to charter and related docs		* [[Message Bus Service]] - mostly a placeholder topic page; links to charter and related docs
	* [[TTY BPF Instrument]] - also a placeholder page; links to repos/docs		* [[TTY BPF Instrument]] - also a placeholder page; links to repos/docs

			=== Briefs: ===

			* [[Brief: EDURange is More Than a CRUD Architecture\|EDURange is More Than a CRUD Architecture]] - reasoning behind message-oriented architecture and model-driven development

Brief: EDURange is More Than a CRUD Architecture

Jwgranville — Fri, 19 Jun 2026 03:09:41 GMT

← Older revision		Revision as of 03:09, 19 June 2026
Line 86:		Line 86:

	= Related =		= Related =
			[[New Version Planning]]

Brief: EDURange is More Than a CRUD Architecture

Jwgranville — Fri, 19 Jun 2026 03:08:49 GMT

Created page with "{{DISPLAYTITLE:EDURange is More Than a CRUD Architecture}} = Brief: EDURange is More Than a CRUD Architecture = == The Up-Front Cost We Are Choosing to Pay == The architecture I have been proposing for the next version of EDURange asks the team to pay some real up-front costs. A message-oriented service architecture, append-only observation, explicit domain modeling, and model-driven development are not free. They require shared vocabulary, new learning, more delibera..."

New page

{{DISPLAYTITLE:EDURange is More Than a CRUD Architecture}}

= Brief: EDURange is More Than a CRUD Architecture =

== The Up-Front Cost We Are Choosing to Pay ==

The architecture I have been proposing for the next version of EDURange asks the team to pay some real up-front costs. A message-oriented service architecture, append-only observation, explicit domain modeling, and model-driven development are not free. They require shared vocabulary, new learning, more deliberate interfaces, more testing discipline, and a willingness to plan and define software responsibilities before the immediate payoff is always obvious.

I want to acknowledge that directly. This is not the easiest path in the first sprint, and it may feel unfamiliar for our team whose shared experience is largely in cybersecurity contexts and classroom-facing teachable code rather than professional-scale software development techniques, lifecycle management, and architectural strategy. My argument is not that we should adopt heavier engineering practices because they are fashionable or inevitable. My argument is that EDURange specifically has grown to become the kind of project where not acknowledging this cost is likely to incur even greater debt.

== Why EDURange Is Not a CRUD Application ==

EDURange is not just a CRUD application. We are not simply making and mutating records when we manage scenarios, users, assignments, and scores. The code objects in EDURange need to accurately reflect the behavior of real-life systems outside the application. We are building an intelligent tutoring system whose learning environment is a cyber range. That means we are combining two mature application paradigms that are each complex and difficult to engineer individually.

A cyber range tries to recreate meaningful interaction with real computing systems, while mitigating actual lasting harm to the host system. Student behavior is exploratory, ambiguous, distributed over time, and a student’s cognitive state is only partially or indirectly observable - even under perfect conditions. An intelligent tutoring system tries to reason about learning: what the student understands, where they are stuck, what intervention might help, and how instruction should adapt. Neither side of this problem is well represented by a simple record-oriented database design.

The hard part is not just storing data. Another vital responsibility is preserving enough meaning that we can ask better questions later.

== What the Previous Architecture Taught Us ==

Earlier EDURange development has already shown the limits of ad hoc integration. Historically many key features have depended on specific legacy scripts, file formats, and implicit assumptions that are difficult to reproduce or explain to new contributors. Log processing has been constrained by hardcoded relationships among producers and consumers. Log data has existed in multiple places without an obvious canonical version. Data transformations have been destructive: once raw observations are cleaned, summarized, or reshaped for one purpose, later analysis inherits assumptions that may no longer be appropriate.

Adding a new observational capability, analysis method, or hinting strategy for EDURange today is less about the idea itself and more about finding a safe place to splice it into the existing stack.

That is the pain point I want us to take seriously. The previous architecture did not fail because people made wrong choices. It grew around real needs, limited time, and specific feature requests. But those local decisions were not considered in terms of long-term consequences. As a result, they accumulated into a system where change is expensive, historical knowledge is fragile, and developers need special guidance just to run the stack.

== Why Not Just Use Existing Middleware? ==

One reasonable question is: why not solve this with existing middleware?

In some cases, we should. And under the hood of several key modules, we will. For instance - I have no intention of implementing network messaging myself or asking our future devs to maintain it, so the message bus will have an adapter that itself connects to the ZeroMQ messaging middleware for actual transport. The project-facing message bus module is responsible for managing the boilerplate setup of ZeroMQ endpoints and wire formats, giving us a high-level “it just works” functionality while leaving open the option for low-level extension or migration to other transport APIs later.

I am not arguing for rebuilding standard tools out of pride or habit. But the append-only storage and messaging options I have evaluated tend to bring their own lifecycle assumptions, especially around message schemas, version numbers, and server/client reconciliation.

Those concerns are important in many systems, but generally beyond our scale. They assume systems distributed across multiple persistent servers, where it may not be possible to shut the whole system down when updating an individual component - and so components at all times have to be prepared for the other endpoints they work with to begin communicating with a different message schema version. Version management requires special care in data format design, code changes, and binary encoding. And they cost extra per-message space. They risk making the first hard problem for EDURange into tackling a middleware’s conceptual challenges, rather than addressing the design needs and intended uses of EDURange head on.

That matters because our team is largely composed of junior developers working independently with limited supervision. We could adopt a sophisticated middleware stack as our team model for messaging and then say, “message schema changes are senior-review only.” But that creates a brittle social rule around a technical dependency most of the team cannot easily inspect. A smaller wrapper, built around our conventions and MVP needs, gives us a more teachable path into message-oriented architecture and append-only data policy. It lets us adopt the important ideas now - durable observation, explicit message boundaries, replayability, narrow service responsibilities - without forcing the team to manage complexity we do not yet know we need.

This is not a permanent rejection of production middleware. It is a staging decision. If we later need stronger distribution, replication, schema negotiation, or broker-level guarantees, we will be in a better position to adopt those tools because we will already understand our own domain boundaries. With a version built with messages in mind, we would be well positioned to perform an apples-to-apples refactor from one message API to another, without having to restructure the logical organization of our existing parts or change expectations of correct behavior. By implementing the bus as a modular service instead of directly incorporating a middleware API, we have a clearly identified pivot point where all of our message-related code can be extended or changed when new needs emerge.

== Why Message-Oriented Architecture Fits the Problem ==

A message-oriented architecture itself helps promote this future adaptability because it treats activity as something services can observe and respond to without owning each other. Instead of one process consuming a log and deciding what every downstream feature is allowed to see, we can preserve observations and let multiple consumers subscribe to the same events. One service might classify commands. Another might track timing. Another might build a student model. Another might support replay or research analysis views.

These consumers should not need to preempt each other, overwrite each other, or force the capture layer to know every future use case. And because they are loosely connected in this way, they are easier to change, replace, and implement independently.

== Why Domain Models Matter ==

Domain-driven design and model-driven development address a different but related problem: meaning.

A domain model is well suited to serve as the kernel of a complex unit of software because it depends on no outside code or system. It is not the database layer, the UI layer, the web framework, or the message bus. It is the part of the program that implements the logical rules we want the rest of the system to manifest. Its only job is to represent or calculate what is true or known under those logical rules. Once those rules are explicit, we can build layers around them which preserve their properties and truthfulness: persistence, messaging, APIs, user interfaces, tests, simulations, and analysis tools. Those outer layers may be influenced by outside systems, but the core model gives them something stable to serve as a reference for correct behavior.

This matters especially for intelligent tutoring. Domain models are not only useful for organizing code. They are useful for observing, interpreting, and predicting the behavior of complex systems or relationships.

What is the student doing? What is the computer system doing? What happened in the learning environment that the student might have observed? What evidence suggests the student understands the underlying concept? Why does a problem have a particular solution? Which actions are equivalent, misleading, partial, or strategically meaningful? Those are modeling questions before they are implementation questions. A desirable solution doesn’t just have implementation steps: there is a concrete and logical reason why that particular implementation is correct, and a definition of the scope or conditions under which the correctness holds.

For EDURange, the relevant concepts may include observations, scenarios, learners, attempts, interventions, hints, policies, evidence, projections, and instructional state. Relevant teachable cybersecurity content changes. Scenario infrastructure dependencies change. Student modeling strategies change. AI tools and services change. Research questions change. UI needs change. If the important concepts only live unnamed as trace evidence in controller functions, database schemas, scripts, or frontend assumptions, then every change risks becoming a cross-stack archaeology project. If the concepts live in explicit models, then we have something stable to discuss, test, revise, and teach.

== Why Thin UIs and Plural Analysis Matter ==

This is also why thin UIs with narrow scopes are appealing. A UI should present a coherent experience, but it should not be where the system’s core reasoning lives. We want different views for different user roles - potentially including students, instructors, content authors, teaching assistants, developers, and perhaps even more. Those views should compose around shared definitions, services and models rather than each inventing its own interpretation of the system.

The same idea applies to data analysis. We should expect plural analysis, not one final pipeline. For a research platform, it is normal to compare models, rerun old data through new methods, test interventions, and ask questions we did not know to ask when the data was first captured. That requires preserved observations, explicit semantic contracts, and a clear distinction between raw evidence and downstream interpretation.

This also has practical value for grant-seeking work. Reproducibility and interpretability are not just academic ideals. They make it easier to explain and demonstrate what we measured, why we measured it, how an intervention was selected, and whether a result can be trusted. A system that preserves observations and exposes its models is easier to evaluate, easier to defend, and easier to extend into new research questions. Increasingly, features like reproducibility and explainability are not simply desired but expected.

== Questions the Architecture Should Help Us Ask ==

The goal is not architectural purity. The goal is to make future work easier to explain, easier to test, and safer to change. A good architecture should help us ask better questions:

* What observations must be preserved before we interpret them?
* Which transformations are irreversible, and what assumptions do they commit us to?
* Which parts of the system need transactional consistency?
* Which services should be allowed to change independently?
* Where do we expect multiple competing analysis strategies?
* Which domain concepts should new contributors learn first?
* What should remain small and local now, but have a path to generalization and distribution scale later?

These questions are not distractions from implementation. They are part of responsible implementation planning for a system whose purpose is scientific inquiry and adaptive instruction.

== A Modest Implementation Strategy ==

My hope is that we can take a modest, incremental approach: build small core services, keep their responsibilities narrow, test them rigorously, and use them to make the rest of the project easier rather than harder as complexity increases.

The up-front cost is real. But if we choose the boundaries well, that cost should buy us clearer reasoning, better data, more reproducible research, and greater and more sustainable developer productivity. Most of all, the goal is a codebase where the team can safely evolve, share, and try new ideas without irreversibly accumulating assumptions or undoing foundational progress along the way.

= Related =

Message Bus Service

Jwgranville — Wed, 17 Jun 2026 03:30:01 GMT

← Older revision		Revision as of 03:30, 17 June 2026
Line 1:		Line 1:
	Module repository and wiki documentation are pending. This is a placeholder topic page to index resources related to the development of the message bus service.		Module repository and wiki documentation are pending. This is a placeholder topic page to index resources related to the development of the message bus service.

	= Related =		== Related ==

	== Project Charter ==		=== Project Charter ===
	[[Message Bus Project Charter]]		[[Message Bus Project Charter]]

	== Examples, Tutorials, Supporting Documents ==		=== Examples, Tutorials, Supporting Documents ===
	[[Message Bus Live Coding Exercise]]		[[Message Bus Live Coding Exercise]]

Message Bus Live Coding Exercise

Jwgranville — Tue, 16 Jun 2026 16:50:09 GMT

Phase 3: Explore shell command data by hand

← Older revision		Revision as of 16:50, 16 June 2026
(2 intermediate revisions by the same user not shown)
Line 655:		Line 655:

	The dictionary contains the information, but it does not document the shape of the event. A typo is discovered only when the field is used. A named event class makes the processor code easier to read and easier to explain.		The dictionary contains the information, but it does not document the shape of the event. A typo is discovered only when the field is used. A named event class makes the processor code easier to read and easier to explain.

			Positional encoding like tuples and CSV-like records also suffer from this innate limitation. They are inherently detached from the structural contract of the record type; they describe only whatever data they carry, not any particular schema or type contract to which the record itself belongs.

	== Phase 4: Persist the event record classes ==		== Phase 4: Persist the event record classes ==

New Version Planning

Jwgranville — Tue, 16 Jun 2026 14:12:17 GMT

Links to Subprojects/Related Topics

← Older revision		Revision as of 14:12, 16 June 2026
Line 93:		Line 93:

	* [[Data Store Project Charter]] - just the charter; pending a top-level topic page		* [[Data Store Project Charter]] - just the charter; pending a top-level topic page
	* [[Message Bus ~~Project Charter~~]] - ~~also pending top-level~~ page		* [[Message Bus Service]] - mostly a placeholder topic page; links to charter and related docs
	* [[TTY BPF Instrument]] - ~~mostly~~ a placeholder ~~topic~~ page; links to repos/docs		* [[TTY BPF Instrument]] - also a placeholder page; links to repos/docs

Message Bus Live Coding Exercise

Jwgranville — Tue, 16 Jun 2026 05:31:58 GMT

Phase 14: Add a randomized processor scheduler

← Older revision		Revision as of 05:31, 16 June 2026
(5 intermediate revisions by the same user not shown)
Line 16:		Line 16:
	__init__.py		__init__.py
	</pre>		</pre>

			Pre-release copies of <code>mbus</code> and <code>aostore</code> are available on the Discord fileshare channel. Modules may be released under other package names; see also the EDURange GitHub.

	The exercise uses two problem families.		The exercise uses two problem families.
Line 31:		Line 33:
	** Simulate nondeterministic processor scheduling while keeping source input fixed.		** Simulate nondeterministic processor scheduling while keeping source input fixed.

	The shell examples draw from EDURange's institutional knowledge. The graph examples provide a ~~compact computer science~~ model for ~~safe~~ feedback loops.		The shell examples draw from EDURange's institutional knowledge. The graph examples provide a common model problem family for meaningful use of feedback loops.

	== Running the examples ==		== Running the examples ==
Line 52:		Line 54:
	touch demos/event_processors/__init__.py		touch demos/event_processors/__init__.py
	</pre>		</pre>


	== Important note about capture records ==		== Important note about capture records ==

Line 74:		Line 74:
	Estimated time: 1-3 hours (1-2 hours work, 0.5-1 hours setup)		Estimated time: 1-3 hours (1-2 hours work, 0.5-1 hours setup)

	; Session 2 - Shell ~~command~~ and ~~regex~~ milestones		; Session 2 - Shell commands and milestones

	* '''Phase 3:'''		* '''Phase 3:'''
Line 81:		Line 81:
	** Replace it with <code>ShellCommandEvent</code>.		** Replace it with <code>ShellCommandEvent</code>.
	* '''Phase 5:'''		* '''Phase 5:'''
	** Manually evaluate ~~regex~~ milestone expressions.		** Manually evaluate milestone expressions.
	* '''Phase 6:'''		* '''Phase 6:'''
	** Implement <code>RegexMilestoneProcessor</code>.		** Implement <code>RegexMilestoneProcessor</code>.
Line 92:		Line 92:
	* '''Phase 7:'''		* '''Phase 7:'''
	** Create raw TTY read, write, and line events.		** Create raw TTY read, write, and line events.
	** Show why ~~regex~~ milestones need reconstructed shell command events.		** Show why milestones need reconstructed shell command events.
	* '''Phase 8:'''		* '''Phase 8:'''
	** Implement keystroke timing.		** Implement keystroke timing.
Line 596:		Line 596:
	; Goal		; Goal

	Identify the high-level event shape needed by a ~~regex~~ milestone processor.		Identify the high-level event shape needed by a milestone processor.

	; Discussion points		; Discussion points
Line 622:		Line 622:
	</pre>		</pre>

	Inspect the fields used by ~~regex~~ milestone checks:		Inspect the fields used by milestone checks:

	<pre lang="python">		<pre lang="python">
Line 660:		Line 660:
	; Goal		; Goal

	Create reusable payload classes for the TTY, shell command, and ~~regex~~ milestone examples.		Create reusable payload classes for the TTY, shell command, and milestone examples.

	; File edit		; File edit
Line 797:		Line 797:
	The event shape is now explicit. Other code can depend on a named <code>ShellCommandEvent</code> instead of assuming that a loose dictionary has the expected fields.		The event shape is now explicit. Other code can depend on a named <code>ShellCommandEvent</code> instead of assuming that a loose dictionary has the expected fields.

	== Phase 5: Evaluate ~~regex~~ milestones manually ==		== Phase 5: Evaluate milestones manually ==

	; Goal		; Goal
Line 805:		Line 805:
	; Discussion points		; Discussion points

	* A ~~regex~~ milestone expression names one check against one shell command field.		* A milestone expression names one check against one shell command field.
	* The field may be command input, command output, working directory, or host context.		* The field may be command input, command output, working directory, or host context.
	* The result records whether the expression matched and, when available, the matched text.		* The result records whether the expression matched and, when available, the matched text.
Line 1,163:		Line 1,163:
	* TTY reads, TTY writes, and line-discipline lines are lower-level observations.		* TTY reads, TTY writes, and line-discipline lines are lower-level observations.
	* A shell command event is reconstructed by correlating those observations.		* A shell command event is reconstructed by correlating those observations.
	* Keeping reconstruction separate prevents the ~~regex~~ milestone processor from becoming a large TTY parser.		* Keeping reconstruction separate prevents the milestone processor from becoming a large TTY parser.

	; Interpreter exploration		; Interpreter exploration
Line 1,431:		Line 1,431:
	; Goal		; Goal

	Switch to a small graph problem that can demonstrate feedback loops ~~safely~~.		Switch to a small graph problem that can demonstrate safe feedback loops in an illustratable form.

	; Discussion points		; Discussion points
Line 2,212:		Line 2,212:
	* The scheduler varies which processor gets a chance to run first.		* The scheduler varies which processor gets a chance to run first.
	* This simulates uncertainty from processing time, scheduling priority, startup timing, or transmission delay.		* This simulates uncertainty from processing time, scheduling priority, startup timing, or transmission delay.
	* The graph example should converge to the same final path facts even when processor scheduling, and ~~hence~~ the order of intermediate derived processor events, differs.		* The graph example should converge to the same final path facts even when processor scheduling - and the order of intermediate derived processor events - differs.

	; File edit		; File edit
Line 2,557:		Line 2,557:
	; Goal		; Goal

	Connect the TTY examples back to the shell command event shape used by ~~regex~~ milestones.		Connect the TTY examples back to the shell command event shape used by milestones.

	; Discussion points		; Discussion points

	* <code>ShellCommandEvent</code> is the high-level event consumed by the ~~regex~~ milestone processor.		* <code>ShellCommandEvent</code> is the high-level event consumed by the milestone processor.
	* <code>TTYReadEvent</code>, <code>TTYWriteEvent</code>, and <code>LineDisciplineEvent</code> are lower-level observations.		* <code>TTYReadEvent</code>, <code>TTYWriteEvent</code>, and <code>LineDisciplineEvent</code> are lower-level observations.
	* A reconstructor can subscribe to line-discipline events and query recent related observations.		* A reconstructor can subscribe to line-discipline events and query recent related observations.
Line 2,576:		Line 2,576:
	</pre>		</pre>

	This is an appropriate follow-up exercise after the team understands both ~~regex~~ milestone processing and history queries.		This is an appropriate follow-up exercise after the team understands both milestone processing and history queries.

	= Related =		= Related =

TTY BPF Instrument

Jwgranville — Tue, 16 Jun 2026 04:10:31 GMT

Message Bus Service

Jwgranville — Tue, 16 Jun 2026 04:09:33 GMT

← Older revision		Revision as of 04:09, 16 June 2026
Line 1:		Line 1:
	Module repository and wiki documentation are pending. This is a placeholder topic page to index resources related to the development of the message bus service.		Module repository and wiki documentation are pending. This is a placeholder topic page to index resources related to the development of the message bus service.

	== Related ==		= Related =

	=== Project Charter ===		== Project Charter ==
	[[Message Bus Project Charter]]		[[Message Bus Project Charter]]

	=== Examples, Tutorials, Supporting Documents ===		== Examples, Tutorials, Supporting Documents ==
	[[Message Bus Live Coding Exercise]]		[[Message Bus Live Coding Exercise]]

Message Bus Live Coding Exercise

Jwgranville — Tue, 16 Jun 2026 04:09:15 GMT

← Older revision		Revision as of 04:09, 16 June 2026
(One intermediate revision by the same user not shown)
Line 2,577:		Line 2,577:

	This is an appropriate follow-up exercise after the team understands both regex milestone processing and history queries.		This is an appropriate follow-up exercise after the team understands both regex milestone processing and history queries.

			= Related =

			== Top-level topic page ==
			[[Message Bus Service]]

Message Bus Service

Jwgranville — Tue, 16 Jun 2026 04:07:59 GMT

Created page with "Module repository and wiki documentation are pending. This is a placeholder topic page to index resources related to the development of the message bus service. == Related == === Project Charter === Message Bus Project Charter === Examples, Tutorials, Supporting Documents === Message Bus Live Coding Exercise"

New page

Module repository and wiki documentation are pending. This is a placeholder topic page to index resources related to the development of the message bus service.

== Related ==

=== Project Charter ===
[[Message Bus Project Charter]]

=== Examples, Tutorials, Supporting Documents ===
[[Message Bus Live Coding Exercise]]

Message Bus Live Coding Exercise

Jwgranville — Tue, 16 Jun 2026 04:06:29 GMT

Created page with "= Introduction = == Audience and scope == This exercise is intended for EDURange developers learning how to write event processors with the message bus. The packaged <code>mbus</code> and <code>aostore</code> modules are treated as read-only for this exercise. All new code belongs under a demonstration directory such as: <pre lang="text"> workspace/ mbus/ aostore/ demos/ __init__.py event_processors/ __init__.py </pre> The exe..."

Show changes

← Older revision		Revision as of 04:10, 16 June 2026
Line 1:		Line 1:
	Demonstrations and preliminary documentation can be found at https://github.com/edurange/demo-bpf-tty-logger and https://github.com/edurange/prototype-tty-bpf-instrument. Wiki documentation is pending. Preliminary draft files are available in the Discord <code>#fileshare</code> channel and will be linked here once Wiki configuration is adjusted to support attached non-image documents.		Demonstrations and preliminary documentation can be found at https://github.com/edurange/demo-bpf-tty-logger and https://github.com/edurange/prototype-tty-bpf-instrument. Wiki documentation is pending. Preliminary draft files are available in the Discord <code>#fileshare</code> channel and will be linked here once Wiki configuration is adjusted to support attached non-image documents.

	== Summary ==		= Summary =
	The TTY instrument is a BPF-based kernel observation tool for capturing terminal activity across a host system. Its purpose is to observe TTY byte activity at the kernel boundary and emit records that downstream components can reconstruct, validate, and analyze. The current design treats terminal activity as factual kernel observations first, and reserves higher-level ideas like sessions, users, commands, and learning activity for later interpretation rather than things the probe is responsible for determining.		The TTY instrument is a BPF-based kernel observation tool for capturing terminal activity across a host system. Its purpose is to observe TTY byte activity at the kernel boundary and emit records that downstream components can reconstruct, validate, and analyze. The current design treats terminal activity as factual kernel observations first, and reserves higher-level ideas like sessions, users, commands, and learning activity for later interpretation rather than things the probe is responsible for determining.

	Although the immediate target is TTY activity, the design is also meant to serve as a template for future host-level instruments. It separates kernel observation, bounded transport, user-space draining, reconstruction, and downstream interpretation in a way that can be reused for other event sources. The TTY instrument is intentionally demanding: it involves raw byte payloads, ambiguous identity, ordering challenges, fragmentation, loss accounting, and privacy-sensitive data. If this pattern works for TTY activity, it provides a strong example for building other instruments with similar fidelity and reliability requirements - for the kernel-level services like the filesystem or network interfaces, common user-level processes like `bash`, etc.		Although the immediate target is TTY activity, the design is also meant to serve as a template for future host-level instruments. It separates kernel observation, bounded transport, user-space draining, reconstruction, and downstream interpretation in a way that can be reused for other event sources. The TTY instrument is intentionally demanding: it involves raw byte payloads, ambiguous identity, ordering challenges, fragmentation, loss accounting, and privacy-sensitive data. If this pattern works for TTY activity, it provides a strong example for building other instruments with similar fidelity and reliability requirements - for the kernel-level services like the filesystem or network interfaces, common user-level processes like `bash`, etc.

	=== Why TTY Capture Is Difficult ===		== Why TTY Capture Is Difficult ==
	This project is difficult because TTY activity is not a single clean stream. A user’s apparent terminal session can pass through SSH, shells, subprocesses, pseudoterminals, terminal echo, line discipline behavior, containers, namespaces, and reconnects. The instrument therefore preserves multiple identity axes, including process IDs, user IDs, cgroups, namespaces, TTY device numbers, inode context, and command/process names. No one field is treated as “the session” or “the user.” Streams or sessions may later be reconstructed from these facts, but the raw record should keep the original evidence intact.		This project is difficult because TTY activity is not a single clean stream. A user’s apparent terminal session can pass through SSH, shells, subprocesses, pseudoterminals, terminal echo, line discipline behavior, containers, namespaces, and reconnects. The instrument therefore preserves multiple identity axes, including process IDs, user IDs, cgroups, namespaces, TTY device numbers, inode context, and command/process names. No one field is treated as “the session” or “the user.” Streams or sessions may later be reconstructed from these facts, but the raw record should keep the original evidence intact.

	=== Tracepoints, BPF, and Kernel Boundaries ===		== Tracepoints, BPF, and Kernel Boundaries ==
	The current implementation is tracepoint-based. Kernel tracepoints provide minimal, upstream-plausible attachment surfaces; BPF probes attach to those tracepoints, read bounded payload buffers, enrich records with available identity/timing/ordering context, and emit records through BPF ring buffers. The tracepoint is not the logging schema. The BPF observation record is the instrument-facing schema.		The current implementation is tracepoint-based. Kernel tracepoints provide minimal, upstream-plausible attachment surfaces; BPF probes attach to those tracepoints, read bounded payload buffers, enrich records with available identity/timing/ordering context, and emit records through BPF ring buffers. The tracepoint is not the logging schema. The BPF observation record is the instrument-facing schema.

Line 16:		Line 16:
	The kernel changes must remain minimal, reviewable, and justifiable as tracepoint support; the BPF probe and user-space components remain responsible for the instrument-specific observation record, enrichment, transport, and reconstruction behavior. This distinction matters because tracepoints need to be acceptable to upstream kernel maintainers as general kernel instrumentation, while the richer record schema is specific to this project’s BPF instrument.		The kernel changes must remain minimal, reviewable, and justifiable as tracepoint support; the BPF probe and user-space components remain responsible for the instrument-specific observation record, enrichment, transport, and reconstruction behavior. This distinction matters because tracepoints need to be acceptable to upstream kernel maintainers as general kernel instrumentation, while the richer record schema is specific to this project’s BPF instrument.

	=== Pipeline and Component Responsibilities ===		== Pipeline and Component Responsibilities ==
	The TTY instrument is also important because it represents a characteristic high-pressure workload for the larger event pipeline. Human terminal input can produce many small payloads with high metadata overhead, while command output can produce sudden bursts of larger byte streams. Multi-user SSH workloads combine both patterns. This makes the instrument a useful benchmark for the event bus and downstream data store: it exercises throughput, buffering, ordering, loss reporting, payload preservation, and reconstruction under conditions that are demanding yet meaningful with a minimal number of tracepoints and probe sites. (Contrast with the filesystem, where roughly a dozen such sites are needed to capture the wider variety of filesystem operations.) The TTY instrument does not define the bus or storage policy, but its output should help reveal whether those downstream systems can handle realistic observational pressure without hiding loss or collapsing important context.		The TTY instrument is also important because it represents a characteristic high-pressure workload for the larger event pipeline. Human terminal input can produce many small payloads with high metadata overhead, while command output can produce sudden bursts of larger byte streams. Multi-user SSH workloads combine both patterns. This makes the instrument a useful benchmark for the event bus and downstream data store: it exercises throughput, buffering, ordering, loss reporting, payload preservation, and reconstruction under conditions that are demanding yet meaningful with a minimal number of tracepoints and probe sites. (Contrast with the filesystem, where roughly a dozen such sites are needed to capture the wider variety of filesystem operations.) The TTY instrument does not define the bus or storage policy, but its output should help reveal whether those downstream systems can handle realistic observational pressure without hiding loss or collapsing important context.

Line 29:		Line 29:
	</pre>The probe’s job is bounded observation: copy raw bytes, attach factual context, assign ordering metadata, and emit records. The spool’s job is to drain kernel output quickly and forward it onward, relieving pressure on the kernel ring buffer channels. The collator’s job is reconstruction: reassemble fragments, check sequence continuity, place loss, resolve safe aggregations, and prepare externally useful event representations.		</pre>The probe’s job is bounded observation: copy raw bytes, attach factual context, assign ordering metadata, and emit records. The spool’s job is to drain kernel output quickly and forward it onward, relieving pressure on the kernel ring buffer channels. The collator’s job is reconstruction: reassemble fragments, check sequence continuity, place loss, resolve safe aggregations, and prepare externally useful event representations.

	=== Core Fidelity Invariants ===		== Core Fidelity Invariants ==
	A central invariant is that captured bytes are preserved as bytes. The probe does not decode Unicode, detect commands, infer prompts, identify student intent, or decide where lines begin and end. Payload may contain terminal control sequences, partial multi-byte characters, shell output, pasted text, or fragments of larger observations - the data is handled the same at the instrument level regardless of its nature. Decoding and semantic interpretation happen downstream.		A central invariant is that captured bytes are preserved as bytes. The probe does not decode Unicode, detect commands, infer prompts, identify student intent, or decide where lines begin and end. Payload may contain terminal control sequences, partial multi-byte characters, shell output, pasted text, or fragments of larger observations - the data is handled the same at the instrument level regardless of its nature. Decoding and semantic interpretation happen downstream.

Line 40:		Line 40:
	The auxiliary/status channel exists because status records are needed to interpret captured data, but must not contend for transport resources with observational records. Health/status records may describe loss, backpressure, saturation, calibration, drift, lifecycle transitions, verifier failures, spool pressure, collator pressure, or reconstruction ambiguity. These records describe the behavior and reliability of the instrument, not additional user activity.		The auxiliary/status channel exists because status records are needed to interpret captured data, but must not contend for transport resources with observational records. Health/status records may describe loss, backpressure, saturation, calibration, drift, lifecycle transitions, verifier failures, spool pressure, collator pressure, or reconstruction ambiguity. These records describe the behavior and reliability of the instrument, not additional user activity.

	=== Prototype Lineage ===		== Prototype Lineage ==
	The instrument’s proof of concept is https://github.com/edurange/demo-bpf-tty-logger/tree/main, which demonstrates host-wide TTY capture using BPF kernel probes. That demo shows the basic idea: instead of attaching to one TTY device, it hooks kernel activity system-wide, captures raw buffers and context, and warns about feedback loops if the logger prints to a TTY it is also observing.		The instrument’s proof of concept is https://github.com/edurange/demo-bpf-tty-logger/tree/main, which demonstrates host-wide TTY capture using BPF kernel probes. That demo shows the basic idea: instead of attaching to one TTY device, it hooks kernel activity system-wide, captures raw buffers and context, and warns about feedback loops if the logger prints to a TTY it is also observing.

	The newer prototype, reflected in https://github.com/edurange/prototype-tty-bpf-instrument, moves beyond the early BCC/kprobe sketch toward the current tracepoint/CO-RE direction. It contains a kernel patch for tracepoints, BPF probe code, shared observation ABI, constants, event maps, CPU-local sequence state, and a user-space dump tool. That archive should be treated as the current implementation reference, while the formal design document remains the higher-level specification.		The newer prototype, reflected in https://github.com/edurange/prototype-tty-bpf-instrument, moves beyond the early BCC/kprobe sketch toward the current tracepoint/CO-RE direction. It contains a kernel patch for tracepoints, BPF probe code, shared observation ABI, constants, event maps, CPU-local sequence state, and a user-space dump tool. That archive should be treated as the current implementation reference, while the formal design document remains the higher-level specification.

	=== Design Boundaries and Non-Goals ===		== Design Boundaries and Non-Goals ==
	The instrument is not meant to be a simple keylogger, even though it necessarily captures raw terminal bytes. It is an instrument for producing trustworthy observational records from kernel TTY activity under realistic multi-user workloads. The important work is preserving factual byte activity, identity context, ordering evidence, timing calibration, fragmentation metadata, and loss visibility without letting the kernel probe become an analyzer. The probe observes; the spool drains; and the collator reconstructs. Interpretation is the domain of downstream systems that consume the instrument’s output.		The instrument is not meant to be a simple keylogger, even though it necessarily captures raw terminal bytes. It is an instrument for producing trustworthy observational records from kernel TTY activity under realistic multi-user workloads. The important work is preserving factual byte activity, identity context, ordering evidence, timing calibration, fragmentation metadata, and loss visibility without letting the kernel probe become an analyzer. The probe observes; the spool drains; and the collator reconstructs. Interpretation is the domain of downstream systems that consume the instrument’s output.

	To keep the components individually maintainable, it is crucial to preserve these boundaries. Tracepoints should stay minimal and justifiable to kernel maintainers. BPF code should stay bounded and observational. The spool should reduce pressure without inferring meaning. The collator should only aggregate when identity, sequence continuity, fragment completeness, and loss boundaries make aggregation safe. Storage policy, privacy enforcement, governance, and final analysis are outside the instrument itself unless a later design explicitly brings them in.		To keep the components individually maintainable, it is crucial to preserve these boundaries. Tracepoints should stay minimal and justifiable to kernel maintainers. BPF code should stay bounded and observational. The spool should reduce pressure without inferring meaning. The collator should only aggregate when identity, sequence continuity, fragment completeness, and loss boundaries make aggregation safe. Storage policy, privacy enforcement, governance, and final analysis are outside the instrument itself unless a later design explicitly brings them in.

	== Related ==		= Related =

	=== Project Charter ===		== Project Charter ==
	[[TTY BPF Instrument Project Charter]]		[[TTY BPF Instrument Project Charter]]