Resource broker
Why
One machine, several competing workloads.
Tactic
Measure memory first, then allow, queue, or deny work before it can destabilize the host.
Cerbi Lab
Build notes from theengineering bench.
Measured experiments in local AI, agent evaluation, reliability, and the systems behind Cerbi. Short on hype, long on outcomes.
What this is
A working lab, not an AI demo reel.
The goal is to make local agents measurably better at useful work while keeping the machine recoverable, the experiments disposable, and sensitive data local.
Public-safe engineering notes only
Exploratory by design
These are experiments and measured build notes, not product commitments or roadmap promises.
Fail closed
Uncertain privacy, bad validation, missing evidence, or resource pressure means the experiment stops rather than improvises.
Evidence earns promotion
Models, prompts, and agent policies advance only when builds, tests, benchmarks, and held-out evaluations say they should.
Learning loop
Improve from outcomes, not self-opinion.
A model does not get to declare itself better. Real tasks generate evidence. Verified evidence becomes curated learning data. Challengers then have to beat the current champion on frozen evaluations before anything changes.
Frontier teacher + local student
Hard cases can be escalated to a stronger teacher when policy allows, but teacher output still has to survive local verification before it is eligible for training.
01
Observe
Capture real work and evidence
02
Verify
Builds, tests, security, outcomes
03
Curate
Promote only useful examples
04
Adapt
Prompt, policy, LoRA or training
05
Challenge
Run frozen evaluations
06
Promote
Only when evidence wins
Learning data
Medallion architecture, adapted for agent experience.
Raw trajectories are useful, but they are not automatically good training data. We separate collection, verification, curation, evaluation-only data, and uncertain material so provenance survives the entire learning cycle.
Bronze
Raw experience
Prompts, tool calls, patches, failures, tests, timings, and provenance. Append-only and local when sensitive.
Silver
Verified + normalized
Schema-valid evidence, deduplicated tasks, normalized outcomes, sensitivity tags, and classified failures.
Gold
Training eligible
High-value examples with verification evidence. Gold means eligible for training, never automatic inclusion.
Holdout stays sealed
Frozen regression and adversarial tasks are evaluation-only. Dataset generation should fail if holdout material leaks into training.
Quarantine uncertain data
Possible secrets, malformed trajectories, unverified teacher output, and uncertain provenance stay out of the learning pipeline until resolved.
Tactics
Different problems need different controls.
We are deliberately testing improvements to tools, memory, prompts, model weights, evaluation, and orchestration separately so a score increase has a traceable cause.
Shadow challengers
Why
A new agent can look better while quietly regressing.
Tactic
Run challengers beside the champion in isolated worktrees and compare outcomes before promotion.
Failure memory
Why
A failed run is wasted if the system can repeat it tomorrow.
Tactic
Turn verified failures into reproducible regression candidates and permanent evaluation cases.
Local-first privacy
Why
Some code and data should never leave the machine.
Tactic
Separate offline, cloud-allowed, and adversarial workloads with explicit policy boundaries.
Daily log
What changed, and what actually happened.
Entries are intentionally compact. Private infrastructure details, credentials, personal information, and confidential source material are excluded.
August 22, 2026
Learning control plane is now in place
The local AI lab moved from architecture sketches to a tested, shadow-only learning control plane: provenance, Medallion promotion, evaluation, resource planning, experiment manifests, model governance, and public-safe outcome tracking now exist before any training workload is allowed to run.
Control-plane tests
26 passed
Training runs
0 — by design
Production disruption
None
Runtime mutation
None
Built / changed
- Completed the first shadow-only learning control plane using standard-library Python and SQLite-backed metadata.
- Implemented Bronze, Silver, Gold, Holdout, and Quarantine data states with deterministic provenance and content hashing.
- Added hard training protections so Holdout, Quarantine, unverified, duplicate, or explicitly non-trainable artifacts cannot enter generated datasets.
- Added immutable experiment manifests, a metadata-only job queue, and a fail-closed resource broker that can inspect the DGX without mutating runtime state.
- Added model-neutral L1–L7 evaluation scaffolding, failure-to-regression candidates, and conservative champion/challenger promotion rules.
- Added dry-run routing for local, specialist, frontier-cloud, and human-review paths with sensitivity-aware restrictions.
- Added initial synthetic Cerbi scenarios covering PII exposure, cardinality pressure, and OpenTelemetry backpressure.
- Expanded the local control CLI with registry, experiment, dataset, broker, memory, health, and dashboard inspection commands.
Outcomes
- Twenty-six unit and integrity tests passed, including holdout-contamination, experiment-overwrite, registry-integrity, and provenance protections.
- Static validation passed across Python, shell, JSON, YAML, Compose overlays, SQLite integrity, secret scanning, and read-only resource inspection.
- No production container, service, model allocation, network, listener, package set, or production data changed during this phase.
- The lab can now record evidence and reject unsafe training inputs before a model-training runtime is introduced.
- The next safe runtime experiment is containment-only: a short-lived, no-GPU, network-isolated job that proves evidence capture and cleanup without model loading.
What we learned
- The durable asset is the evaluation and provenance system, not any single base model.
- Gold should mean eligible for training, never automatically selected for training.
- Prompt and tool-policy improvements can be evaluated independently from model-weight changes.
- A resource broker should fail closed until memory reserves are measured rather than guessed.
- Teacher output is still untrusted data until independently verified; it belongs in Quarantine first.
Next
- Run the isolated no-GPU containment experiment and verify immutable evidence capture.
- Benchmark unified-memory headroom before enabling authoritative concurrency or profile budgets.
- Add the frozen benchmark runner and schema-aware creation commands.
- Begin collecting verified trajectories and regression cases before enabling any training runtime.