feat: add observability readiness gate

This commit is contained in:
Affaan Mustafa 2026-05-11 18:23:53 -04:00 committed by Affaan Mustafa
parent ab6e998383
commit 8aa8c32d2a
9 changed files with 624 additions and 3 deletions

View File

@ -0,0 +1,66 @@
# ECC 2.0 Observability Readiness
ECC 2.0 should be observable before it becomes more autonomous. The local
default is an opt-in, repo-owned readiness gate that checks whether the core
signals are present without sending telemetry anywhere.
Run:
```bash
npm run observability:ready
node scripts/observability-readiness.js --format json
```
The gate is deterministic and safe to run in CI. It only checks repository
files and reports whether the release surface can expose the signals an
operator needs.
## Signal Model
- Live status: `scripts/loop-status.js` can emit JSON, watch active loops, and
write snapshots for dashboards or handoffs.
- Session traces: `scripts/session-inspect.js` can inspect Claude, dmux, and
adapter-backed sessions, then write canonical snapshots.
- Harness baseline: `scripts/harness-audit.js` provides a repeatable scorecard
for tool coverage, context efficiency, quality gates, memory persistence,
eval coverage, security guardrails, and cost efficiency.
- Tool activity: `scripts/hooks/session-activity-tracker.js` records local
`tool-usage.jsonl` events that ECC2 can sync.
- Risk ledger: `ecc2/src/observability/mod.rs` scores tool calls and stores a
paginated ledger for review.
## Reference Pressure
The current agent-tooling ecosystem is converging on the same operating needs:
- dmux, Orca, and Superset emphasize isolated worktrees plus one place to see
agent state and merge/review work.
- Claude HUD makes context, tool activity, agent activity, and todo progress
visible inside the coding loop.
- Autocontext records every run as durable traces, reports, artifacts, and
reusable improvements.
- Meta-Harness treats the harness itself as something to evaluate and improve,
which requires clean logs of proposer behavior and outcomes.
- Zed and OpenCode emphasize agent control surfaces, reviewable changes, and
harness-specific configuration that should still preserve portable project
knowledge.
ECC's answer is not a hosted analytics dependency by default. The first
release-candidate gate is local and file-backed. Hosted telemetry can come
later, but only after the local event model is useful enough to trust.
## Operator Workflow
1. Run `npm run observability:ready`.
2. Run `npm run harness:audit -- --format json` for the broader harness
scorecard.
3. Run `node scripts/loop-status.js --json --write-dir .ecc/loop-status`
during longer autonomous batches.
4. Run `node scripts/session-inspect.js --list-adapters` to confirm which
session surfaces are available.
5. Use ECC2 tool logs for risky operations, conflict analysis, and handoff
review before increasing autonomy.
The end-state is practical: before asking ECC to run larger multi-agent loops,
the operator can prove the system has live status, durable session traces,
baseline scorecards, and a local risk ledger.

View File

@ -31,6 +31,15 @@ Expected result: every test passes with zero failures. For release-specific drif
node tests/docs/ecc2-release-surface.test.js
```
Then check the local observability surface:
```bash
npm run observability:ready
```
This runs the [observability readiness gate](../../architecture/observability-readiness.md)
for loop status, session traces, harness audit, and ECC2 tool-risk logs.
## First Skill
Read `skills/hermes-imports/SKILL.md` first.

View File

@ -13,6 +13,7 @@ Claude Code remains a core target. Codex, OpenCode, Cursor, Gemini, and other ha
- Clarified the split between ECC as the reusable substrate and Hermes as the operator shell.
- Documented the cross-harness portability model for skills, hooks, MCPs, rules, and instructions.
- Added a Hermes import playbook for turning local operator patterns into publishable ECC skills.
- Added a local [observability readiness gate](../../architecture/observability-readiness.md) for loop status, session traces, harness audit, and ECC2 tool-risk logs.
## Why This Matters
@ -50,6 +51,7 @@ What stays local:
1. Follow the [rc.1 quickstart](quickstart.md).
2. Read the [Hermes setup guide](../../HERMES-SETUP.md).
3. Review the [cross-harness architecture](../../architecture/cross-harness.md).
4. Start with one workflow lane: engineering, research, content, or outreach.
5. Import only sanitized operator patterns into ECC skills.
6. Treat `ecc2/` as an alpha control plane until release packaging and installer behavior are finalized.
4. Run the [observability readiness gate](../../architecture/observability-readiness.md).
5. Start with one workflow lane: engineering, research, content, or outreach.
6. Import only sanitized operator patterns into ECC skills.
7. Treat `ecc2/` as an alpha control plane until release packaging and installer behavior are finalized.

View File

@ -72,6 +72,7 @@
"scripts/ecc.js",
"scripts/gemini-adapt-agents.js",
"scripts/harness-audit.js",
"scripts/observability-readiness.js",
"scripts/hooks/",
"scripts/install-apply.js",
"scripts/install-plan.js",
@ -265,6 +266,7 @@
"catalog:sync": "node scripts/ci/catalog.js --write --text",
"lint": "eslint . && markdownlint '**/*.md' --ignore node_modules",
"harness:audit": "node scripts/harness-audit.js",
"observability:ready": "node scripts/observability-readiness.js",
"claw": "node scripts/claw.js",
"orchestrate:status": "node scripts/orchestration-status.js",
"orchestrate:worker": "bash scripts/orchestrate-codex-worker.sh",

View File

@ -14,7 +14,9 @@ const ignoredDirs = new Set([
'node_modules',
'.dmux',
'.next',
'.venv',
'coverage',
'venv',
]);
const textExtensions = new Set([

View File

@ -0,0 +1,309 @@
#!/usr/bin/env node
'use strict';
const fs = require('fs');
const path = require('path');
const RUBRIC_VERSION = '2026-05-11';
function usage() {
console.log([
'Usage: node scripts/observability-readiness.js [--format <text|json>] [--root <dir>]',
'',
'Deterministic ECC 2.0 observability readiness gate.',
'',
'Options:',
' --format <text|json> Output format (default: text)',
' --root <dir> Repository root to inspect (default: cwd)',
' --help, -h Show this help'
].join('\n'));
}
function readValue(args, index, flagName) {
const value = args[index + 1];
if (!value || value.startsWith('--')) {
throw new Error(`${flagName} requires a value`);
}
return value;
}
function parseArgs(argv) {
const args = argv.slice(2);
const parsed = {
format: 'text',
help: false,
root: path.resolve(process.cwd())
};
for (let index = 0; index < args.length; index += 1) {
const arg = args[index];
if (arg === '--help' || arg === '-h') {
parsed.help = true;
continue;
}
if (arg === '--format') {
parsed.format = readValue(args, index, arg).toLowerCase();
index += 1;
continue;
}
if (arg.startsWith('--format=')) {
parsed.format = arg.slice('--format='.length).toLowerCase();
continue;
}
if (arg === '--root') {
parsed.root = path.resolve(readValue(args, index, arg));
index += 1;
continue;
}
if (arg.startsWith('--root=')) {
parsed.root = path.resolve(arg.slice('--root='.length));
continue;
}
throw new Error(`Unknown argument: ${arg}`);
}
if (!['text', 'json'].includes(parsed.format)) {
throw new Error(`Invalid format: ${parsed.format}. Use text or json.`);
}
return parsed;
}
function fileExists(rootDir, relativePath) {
return fs.existsSync(path.join(rootDir, relativePath));
}
function readText(rootDir, relativePath) {
try {
return fs.readFileSync(path.join(rootDir, relativePath), 'utf8');
} catch (_error) {
return '';
}
}
function safeParseJson(text) {
if (!text || !text.trim()) {
return null;
}
try {
return JSON.parse(text);
} catch (_error) {
return null;
}
}
function includesAll(text, needles) {
return needles.every(needle => text.includes(needle));
}
function buildChecks(rootDir) {
const packageJsonText = readText(rootDir, 'package.json');
const packageJson = safeParseJson(packageJsonText) || {};
const packageFiles = Array.isArray(packageJson.files) ? packageJson.files : [];
const packageScripts = packageJson.scripts || {};
const loopStatus = readText(rootDir, 'scripts/loop-status.js');
const sessionInspect = readText(rootDir, 'scripts/session-inspect.js');
const harnessAudit = readText(rootDir, 'scripts/harness-audit.js');
const activityTracker = readText(rootDir, 'scripts/hooks/session-activity-tracker.js');
const observabilityRust = readText(rootDir, 'ecc2/src/observability/mod.rs');
const sessionStoreRust = readText(rootDir, 'ecc2/src/session/store.rs');
const sessionManagerRust = readText(rootDir, 'ecc2/src/session/manager.rs');
const readinessDoc = readText(rootDir, 'docs/architecture/observability-readiness.md');
const quickstart = readText(rootDir, 'docs/releases/2.0.0-rc.1/quickstart.md');
const releaseNotes = readText(rootDir, 'docs/releases/2.0.0-rc.1/release-notes.md');
return [
{
id: 'loop-status-live-signal',
category: 'Live Status',
points: 2,
path: 'scripts/loop-status.js',
description: 'Loop status supports JSON output, watch mode, and snapshot writes',
pass: fileExists(rootDir, 'scripts/loop-status.js')
&& includesAll(loopStatus, ['--json', '--watch', '--write-dir']),
fix: 'Restore loop-status JSON/watch/write-dir support.'
},
{
id: 'session-inspect-adapter-registry',
category: 'Session Trace',
points: 2,
path: 'scripts/session-inspect.js',
description: 'Session inspection exposes registered adapters and writable snapshots',
pass: fileExists(rootDir, 'scripts/session-inspect.js')
&& fileExists(rootDir, 'scripts/lib/session-adapters/registry.js')
&& includesAll(sessionInspect, ['--list-adapters', '--write', 'inspectSessionTarget']),
fix: 'Restore session-inspect adapter registry, list-adapters, and write support.'
},
{
id: 'harness-audit-scorecard',
category: 'Harness Baseline',
points: 2,
path: 'scripts/harness-audit.js',
description: 'Harness audit emits deterministic text/JSON scorecards',
pass: fileExists(rootDir, 'scripts/harness-audit.js')
&& packageScripts['harness:audit'] === 'node scripts/harness-audit.js'
&& includesAll(harnessAudit, ['Deterministic harness audit', '--format', 'overall_score']),
fix: 'Restore the harness:audit package script and deterministic scorecard output.'
},
{
id: 'hook-activity-jsonl',
category: 'Tool Activity',
points: 2,
path: 'scripts/hooks/session-activity-tracker.js',
description: 'Hook activity tracker writes tool usage JSONL for later sync',
pass: fileExists(rootDir, 'scripts/hooks/session-activity-tracker.js')
&& includesAll(activityTracker, ['tool-usage.jsonl', 'session_id', 'tool_name']),
fix: 'Restore hook-side tool activity recording to metrics/tool-usage.jsonl.'
},
{
id: 'ecc2-tool-risk-ledger',
category: 'Tool Activity',
points: 3,
path: 'ecc2/src/observability/mod.rs',
description: 'ECC2 records tool calls with risk scoring and paginated queries',
pass: fileExists(rootDir, 'ecc2/src/observability/mod.rs')
&& includesAll(observabilityRust, ['ToolCallEvent', 'RiskAssessment', 'ToolLogger'])
&& includesAll(sessionStoreRust, ['insert_tool_log', 'query_tool_logs'])
&& includesAll(sessionManagerRust, ['sync_tool_activity_metrics', 'tool-usage.jsonl']),
fix: 'Restore ECC2 tool logging, risk scoring, store queries, and metrics sync.'
},
{
id: 'release-observability-onramp',
category: 'Operator Onramp',
points: 2,
path: 'docs/architecture/observability-readiness.md',
description: 'Release docs explain the local observability readiness workflow',
pass: readinessDoc.includes('node scripts/observability-readiness.js --format json')
&& quickstart.includes('observability-readiness.md')
&& releaseNotes.includes('observability-readiness.md'),
fix: 'Add the observability readiness doc and link it from rc.1 release docs.'
},
{
id: 'package-exposes-readiness-gate',
category: 'Packaging',
points: 1,
path: 'package.json',
description: 'Package exposes the observability readiness gate',
pass: packageScripts['observability:ready'] === 'node scripts/observability-readiness.js'
&& packageFiles.includes('scripts/observability-readiness.js'),
fix: 'Add scripts/observability-readiness.js to package files and observability:ready.'
}
];
}
function buildReport(rootDir) {
const checks = buildChecks(rootDir);
const categories = {};
for (const check of checks) {
if (!categories[check.category]) {
categories[check.category] = {
score: 0,
max_score: 0,
passed: 0,
total: 0
};
}
categories[check.category].max_score += check.points;
categories[check.category].total += 1;
if (check.pass) {
categories[check.category].score += check.points;
categories[check.category].passed += 1;
}
}
const overallScore = checks
.filter(check => check.pass)
.reduce((sum, check) => sum + check.points, 0);
const maxScore = checks.reduce((sum, check) => sum + check.points, 0);
const failingChecks = checks.filter(check => !check.pass);
return {
schema_version: 'ecc.observability-readiness.v1',
rubric_version: RUBRIC_VERSION,
deterministic: true,
root_dir: fs.realpathSync(rootDir),
overall_score: overallScore,
max_score: maxScore,
ready: overallScore === maxScore,
categories,
checks,
top_actions: failingChecks
.sort((left, right) => right.points - left.points || left.id.localeCompare(right.id))
.slice(0, 3)
.map(check => ({
id: check.id,
path: check.path,
fix: check.fix
}))
};
}
function renderText(report) {
const lines = [
`Observability Readiness: ${report.overall_score}/${report.max_score}`,
`Ready: ${report.ready ? 'yes' : 'no'}`,
'',
'Categories:'
];
for (const [name, category] of Object.entries(report.categories)) {
lines.push(`- ${name}: ${category.score}/${category.max_score} (${category.passed}/${category.total})`);
}
lines.push('', 'Checks:');
for (const check of report.checks) {
lines.push(`- ${check.pass ? 'PASS' : 'FAIL'} ${check.id}: ${check.description}`);
}
if (report.top_actions.length > 0) {
lines.push('', 'Top Actions:');
for (const action of report.top_actions) {
lines.push(`- ${action.path}: ${action.fix}`);
}
}
return `${lines.join('\n')}\n`;
}
function main() {
const args = parseArgs(process.argv);
if (args.help) {
usage();
return;
}
const report = buildReport(args.root);
if (args.format === 'json') {
console.log(JSON.stringify(report, null, 2));
} else {
process.stdout.write(renderText(report));
}
}
if (require.main === module) {
try {
main();
} catch (error) {
console.error(`Error: ${error.message}`);
process.exit(1);
}
}
module.exports = {
buildChecks,
buildReport,
parseArgs,
renderText
};

View File

@ -109,6 +109,27 @@ if (
passed++;
else failed++;
if (
test('skips Python virtual environments', () => {
const root = makeTempRoot('ecc-unicode-venv-');
fs.mkdirSync(path.join(root, '.venv', 'lib', 'python3.12', 'site-packages'), { recursive: true });
fs.mkdirSync(path.join(root, 'venv', 'lib', 'python3.12', 'site-packages'), { recursive: true });
fs.writeFileSync(
path.join(root, '.venv', 'lib', 'python3.12', 'site-packages', 'package.py'),
`message = "hello ${rocketEmoji}"\n`
);
fs.writeFileSync(
path.join(root, 'venv', 'lib', 'python3.12', 'site-packages', 'package.py'),
`message = "hello ${rocketEmoji}"\n`
);
const result = runCheck(root);
assert.strictEqual(result.status, 0, result.stdout + result.stderr);
})
)
passed++;
else failed++;
console.log(`\nPassed: ${passed}`);
console.log(`Failed: ${failed}`);
process.exit(failed > 0 ? 1 : 0);

View File

@ -53,6 +53,7 @@ function buildExpectedPublishPaths(repoRoot) {
"scripts/install-plan.js",
"scripts/list-installed.js",
"scripts/loop-status.js",
"scripts/observability-readiness.js",
"scripts/skill-create-output.js",
"scripts/repair.js",
"scripts/harness-audit.js",

View File

@ -0,0 +1,209 @@
/**
* Tests for scripts/observability-readiness.js
*/
const assert = require('assert');
const fs = require('fs');
const os = require('os');
const path = require('path');
const { execFileSync, spawnSync } = require('child_process');
const SCRIPT = path.join(__dirname, '..', '..', 'scripts', 'observability-readiness.js');
const { buildReport, parseArgs } = require(SCRIPT);
function createTempDir(prefix) {
return fs.mkdtempSync(path.join(os.tmpdir(), prefix));
}
function cleanup(dirPath) {
fs.rmSync(dirPath, { recursive: true, force: true });
}
function writeFile(rootDir, relativePath, content) {
const targetPath = path.join(rootDir, relativePath);
fs.mkdirSync(path.dirname(targetPath), { recursive: true });
fs.writeFileSync(targetPath, content);
}
function run(args = [], options = {}) {
return execFileSync('node', [SCRIPT, ...args], {
cwd: options.cwd || path.join(__dirname, '..', '..'),
encoding: 'utf8',
stdio: ['pipe', 'pipe', 'pipe'],
timeout: 10000
});
}
function runProcess(args = [], options = {}) {
return spawnSync('node', [SCRIPT, ...args], {
cwd: options.cwd || path.join(__dirname, '..', '..'),
encoding: 'utf8',
stdio: ['pipe', 'pipe', 'pipe'],
timeout: 10000
});
}
function seedMinimalRepo(rootDir, overrides = {}) {
const files = {
'package.json': JSON.stringify({
name: 'everything-claude-code',
files: ['scripts/observability-readiness.js'],
scripts: {
'harness:audit': 'node scripts/harness-audit.js',
'observability:ready': 'node scripts/observability-readiness.js'
}
}, null, 2),
'scripts/loop-status.js': '--json --watch --write-dir',
'scripts/session-inspect.js': '--list-adapters --write inspectSessionTarget',
'scripts/lib/session-adapters/registry.js': 'module.exports = {};',
'scripts/harness-audit.js': 'Deterministic harness audit --format overall_score',
'scripts/hooks/session-activity-tracker.js': 'tool-usage.jsonl session_id tool_name',
'ecc2/src/observability/mod.rs': 'ToolCallEvent RiskAssessment ToolLogger',
'ecc2/src/session/store.rs': 'insert_tool_log query_tool_logs',
'ecc2/src/session/manager.rs': 'sync_tool_activity_metrics tool-usage.jsonl',
'docs/architecture/observability-readiness.md': 'node scripts/observability-readiness.js --format json',
'docs/releases/2.0.0-rc.1/quickstart.md': 'observability-readiness.md',
'docs/releases/2.0.0-rc.1/release-notes.md': 'observability-readiness.md'
};
for (const [relativePath, content] of Object.entries({ ...files, ...overrides })) {
if (content === null) {
continue;
}
writeFile(rootDir, relativePath, content);
}
}
function test(name, fn) {
try {
fn();
console.log(`${name}`);
return true;
} catch (error) {
console.log(`${name}`);
console.log(` Error: ${error.message}`);
return false;
}
}
function runTests() {
console.log('\n=== Testing observability-readiness.js ===\n');
let passed = 0;
let failed = 0;
if (test('parseArgs accepts supported forms and rejects invalid input', () => {
const rootDir = createTempDir('observability-readiness-args-');
try {
assert.strictEqual(parseArgs(['node', 'script', '--help']).help, true);
assert.strictEqual(parseArgs(['node', 'script', '-h']).help, true);
const spaced = parseArgs(['node', 'script', '--format', 'json', '--root', rootDir]);
assert.strictEqual(spaced.format, 'json');
assert.strictEqual(spaced.root, path.resolve(rootDir));
const equals = parseArgs(['node', 'script', '--format=json', `--root=${rootDir}`]);
assert.strictEqual(equals.format, 'json');
assert.strictEqual(equals.root, path.resolve(rootDir));
assert.throws(() => parseArgs(['node', 'script', '--format', 'xml']), /Invalid format: xml/);
assert.throws(() => parseArgs(['node', 'script', '--root']), /--root requires a value/);
assert.throws(() => parseArgs(['node', 'script', '--unknown']), /Unknown argument: --unknown/);
} finally {
cleanup(rootDir);
}
})) passed++; else failed++;
if (test('cli help exits cleanly and invalid cli args exit with stderr', () => {
const help = runProcess(['--help']);
assert.strictEqual(help.status, 0);
assert.strictEqual(help.stderr, '');
assert.ok(help.stdout.includes('Usage: node scripts/observability-readiness.js'));
const invalid = runProcess(['--format', 'xml']);
assert.strictEqual(invalid.status, 1);
assert.strictEqual(invalid.stdout, '');
assert.ok(invalid.stderr.includes('Error: Invalid format: xml. Use text or json.'));
})) passed++; else failed++;
if (test('current repo reports a complete readiness score', () => {
const parsed = JSON.parse(run(['--format=json']));
assert.strictEqual(parsed.schema_version, 'ecc.observability-readiness.v1');
assert.strictEqual(parsed.deterministic, true);
assert.strictEqual(parsed.ready, true);
assert.strictEqual(parsed.overall_score, parsed.max_score);
assert.strictEqual(parsed.top_actions.length, 0);
})) passed++; else failed++;
if (test('text output includes summary, categories, and checks', () => {
const output = run();
assert.ok(output.includes('Observability Readiness:'));
assert.ok(output.includes('Categories:'));
assert.ok(output.includes('Checks:'));
assert.ok(output.includes('PASS loop-status-live-signal'));
})) passed++; else failed++;
if (test('minimal seeded repo passes all checks', () => {
const projectRoot = createTempDir('observability-readiness-pass-');
try {
seedMinimalRepo(projectRoot);
const report = buildReport(projectRoot);
assert.strictEqual(report.ready, true);
assert.strictEqual(report.overall_score, report.max_score);
assert.deepStrictEqual(report.top_actions, []);
} finally {
cleanup(projectRoot);
}
})) passed++; else failed++;
if (test('missing tool logger surfaces become prioritized top actions', () => {
const projectRoot = createTempDir('observability-readiness-fail-');
try {
seedMinimalRepo(projectRoot, {
'ecc2/src/observability/mod.rs': 'ToolCallEvent only'
});
const report = buildReport(projectRoot);
assert.strictEqual(report.ready, false);
assert.ok(report.top_actions.some(action => action.id === 'ecc2-tool-risk-ledger'));
assert.ok(report.checks.some(check => check.id === 'ecc2-tool-risk-ledger' && !check.pass));
} finally {
cleanup(projectRoot);
}
})) passed++; else failed++;
if (test('missing release onramp fails without disturbing core tool checks', () => {
const projectRoot = createTempDir('observability-readiness-doc-fail-');
try {
seedMinimalRepo(projectRoot, {
'docs/releases/2.0.0-rc.1/quickstart.md': 'quickstart without link'
});
const report = buildReport(projectRoot);
assert.strictEqual(report.ready, false);
assert.ok(report.checks.some(check => check.id === 'release-observability-onramp' && !check.pass));
assert.ok(report.checks.some(check => check.id === 'loop-status-live-signal' && check.pass));
} finally {
cleanup(projectRoot);
}
})) passed++; else failed++;
console.log('\nResults:');
console.log(` Passed: ${passed}`);
console.log(` Failed: ${failed}`);
if (failed > 0) {
process.exit(1);
}
}
if (require.main === module) {
runTests();
}