163 lines
7.3 KiB
Markdown
163 lines
7.3 KiB
Markdown
# memabra Progress
|
|
|
|
## Current status
|
|
|
|
Project status: safe self-improving alpha, benchmark-gated online learning loop complete
|
|
Date: 2026-04-15
|
|
Project: memabra
|
|
Subtitle: An intuition-driven control plane for agent memory and action selection.
|
|
|
|
## What exists now
|
|
|
|
memabra now has a complete safe self-improving alpha control-plane loop:
|
|
- candidate retrieval
|
|
- routing decisions
|
|
- memory / skill / tool execution
|
|
- telemetry events
|
|
- trajectory construction
|
|
- runtime validation
|
|
- artifact persistence
|
|
- replay and analytics
|
|
- artifact indexing and dataset slicing
|
|
- lightweight learning router training
|
|
- A/B evaluation
|
|
- router weight versioning and rollback
|
|
- benchmark-gated promotion with explicit policy thresholds
|
|
- auditable training reports
|
|
- exception-safe online learning coordinator
|
|
- configurable CLI entrypoint
|
|
- persisted seen-trajectory tracking across restarts (safe for cron jobs)
|
|
- dry-run mode for training/evaluation without promotion risk
|
|
- baseline version selection for challenger evaluation
|
|
- task case index (`CaseIndex`) for episodic retrieval: maps normalized inputs to the best past trajectory ID
|
|
- `CaseIndex` integration into `MemabraApp` (build, save, load, lookup) and `MemabraRunner` (injects episodic candidate on matching inputs)
|
|
- CLI flags `--case-index` and `--rebuild-case-index` for operator-managed episodic retrieval
|
|
- `OnlineLearningCoordinator` auto-rebuilds case index after each cycle when `case_index_path` is provided, ensuring benchmark-generated trajectories are indexed
|
|
- `TrajectorySummarizer` generates human-readable trajectory summaries from task input, decisions, outcome, and reward
|
|
- `MemabraRunner` enriches episodic memory candidate summaries using `TrajectorySummarizer` when `persistence_store` is available
|
|
- CLI `--status` flag prints current system state (active router version, counts, latest report) without triggering a learning cycle
|
|
- CLI is now subcommand-driven (`run`, `status`, `version list`, `version rollback`) with a dedicated packaged `memabra` entrypoint
|
|
- CLI `--format text` mode provides operator-friendly summaries for status checks, version listings, rollbacks, and workflow runs, including latest report details, current-version highlighting, sectioned workflow summaries, normalized yes/no flags, and fixed-precision benchmark/promotion metrics
|
|
|
|
## Major completed capabilities
|
|
|
|
### Foundations
|
|
- project naming, architecture, roadmap, decisions, reward spec
|
|
- candidate / event / trajectory / memory schemas
|
|
- prototype package structure under `src/memabra/`
|
|
|
|
### Runtime path
|
|
- `retrieval.py`: typed candidate retrieval
|
|
- `router.py`: heuristic router, feature-scoring router, learning router
|
|
- `execution.py`: memory, skill, tool executors and adapters
|
|
- `runner.py`: end-to-end task -> trajectory orchestration
|
|
- `persistence.py`: trajectory and memory artifact storage
|
|
- `replay.py`: replay summaries over examples and persisted runs
|
|
- `memory_store.py`: typed memory records with verify/revoke support
|
|
|
|
### Adapters and evaluation
|
|
- real tool adapters:
|
|
- `LocalFunctionToolAdapter`
|
|
- `SubprocessToolAdapter`
|
|
- `ToolRegistry`
|
|
- real skill loading:
|
|
- `FileSystemSkillBackend`
|
|
- richer evaluation path:
|
|
- `OutcomeEngine`
|
|
- `RewardEngine`
|
|
- `ArtifactIndex`
|
|
- `DatasetBuilder`
|
|
- `Evaluator`
|
|
- `RouterVersionStore`
|
|
- Alpha Iteration 1 — online learning loop:
|
|
- `PromotionPolicy` with benchmark-gated promotion rules
|
|
- `BenchmarkSuite` persistence (JSON load/save + default seed)
|
|
- `OnlineLearningCoordinator` for retrain/evaluate/promote cycles
|
|
- exception-safe coordinator: training/evaluation failures emit auditable error reports instead of crashing
|
|
- `TrainingReportStore.get_report()` for by-id report lookup
|
|
|
|
### Product/demo surface
|
|
- `app.py`: `MemabraApp`, demo builders, artifact index access, training hooks, `run_online_learning_cycle`
|
|
- `cli.py`: wrap-up workflow and `run_online_learning_workflow` with benchmark-gated promotion
|
|
- `cli.py`: argument parsing (`--base-dir`, `--min-new-trajectories`) and clean `python -m src.memabra.cli` execution
|
|
- `DEMO.md`: runnable walkthrough with CLI options
|
|
|
|
## Current test status
|
|
|
|
Command:
|
|
`source venv/bin/activate && python -m pytest tests/memabra -q`
|
|
|
|
Latest result:
|
|
`118 passed`
|
|
|
|
All alpha iteration 1 source, tests, and documentation have been committed to the repository (commit `34cf507c`).
|
|
|
|
## Most important current files
|
|
|
|
### Core package
|
|
- `src/memabra/app.py`
|
|
- `src/memabra/cli.py`
|
|
- `src/memabra/router.py`
|
|
- `src/memabra/runner.py`
|
|
- `src/memabra/execution.py`
|
|
- `src/memabra/evaluator.py`
|
|
- `src/memabra/router_versioning.py`
|
|
- `src/memabra/promotion.py`
|
|
- `src/memabra/online_learning.py`
|
|
- `src/memabra/training_reports.py`
|
|
- `src/memabra/benchmarks.py`
|
|
- `src/memabra/case_index.py`
|
|
|
|
### Tests
|
|
- `tests/memabra/test_app.py`
|
|
- `tests/memabra/test_cli_workflow.py`
|
|
- `tests/memabra/test_package_exports.py`
|
|
- `tests/memabra/test_promotion.py`
|
|
- `tests/memabra/test_online_learning.py`
|
|
- `tests/memabra/test_training_reports.py`
|
|
- `tests/memabra/test_benchmarks.py`
|
|
- `tests/memabra/test_router_versioning.py`
|
|
- `tests/memabra/test_evaluator.py`
|
|
- `tests/memabra/test_router_protocol.py`
|
|
- `tests/memabra/test_execution_persistence.py`
|
|
|
|
## Wrap-up status
|
|
|
|
The project is now in a safe self-improving alpha state.
|
|
It can:
|
|
- run realistic demo tasks
|
|
- persist trajectories
|
|
- replay and inspect results
|
|
- train a lightweight router from saved artifacts
|
|
- compare baseline vs challenger routers
|
|
- apply a promotion policy with explicit thresholds
|
|
- save and reload router versions with metadata
|
|
- emit auditable training reports
|
|
- run an online-learning cycle from the CLI
|
|
- leave the active router unchanged when challenger fails
|
|
- survive training/evaluation failures gracefully and emit error reports
|
|
- accept CLI overrides for artifact directory and trajectory thresholds
|
|
- persist seen-trajectory state across restarts so cron jobs don't retrain on the same data
|
|
- default CLI `main()` persists seen trajectories to `<base-dir>/seen-trajectories.json`
|
|
- run in dry-run mode to evaluate a challenger without promoting it
|
|
- run in baseline-version mode to compare a challenger against a specific saved version instead of the currently active router
|
|
- index successful task cases by normalized input for episodic retrieval (`CaseIndex`)
|
|
- build/save/load a case index from `MemabraApp`
|
|
- inject episodic memory candidates during runner retrieval when a similar past task exists
|
|
- use `--case-index` and `--rebuild-case-index` CLI flags to manage episodic retrieval
|
|
- online-learning cycles automatically refresh the case index after training/evaluation when a case-index path is configured
|
|
- episodic memory candidates now include rich human-readable summaries when the past trajectory is available via `persistence_store`
|
|
- CLI `--status` flag provides a quick read-only snapshot of the active router, versions, trajectories, and reports
|
|
- CLI `--rollback` and `--list-versions` flags enable operator-safe router version management without touching code
|
|
|
|
## Next sensible frontier
|
|
|
|
1. tighter integration with real Hermes trajectories
|
|
2. multi-turn conversation state and working-memory updates
|
|
3. richer real-world tool ecosystem integration (MCP, web, git, files)
|
|
4. stronger storage/index backend beyond plain JSON files
|
|
|
|
## One-line summary
|
|
|
|
memabra is now a runnable, test-covered safe self-improving alpha for agent memory/action routing, with online learning, benchmark-gated promotion, and auditable reports.
|