mirror of
https://github.com/affaan-m/everything-claude-code.git
synced 2026-05-14 02:10:07 +08:00
feat: add observability readiness gate
This commit is contained in:
parent
ab6e998383
commit
8aa8c32d2a
66
docs/architecture/observability-readiness.md
Normal file
66
docs/architecture/observability-readiness.md
Normal file
@ -0,0 +1,66 @@
|
||||
# ECC 2.0 Observability Readiness
|
||||
|
||||
ECC 2.0 should be observable before it becomes more autonomous. The local
|
||||
default is an opt-in, repo-owned readiness gate that checks whether the core
|
||||
signals are present without sending telemetry anywhere.
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
npm run observability:ready
|
||||
node scripts/observability-readiness.js --format json
|
||||
```
|
||||
|
||||
The gate is deterministic and safe to run in CI. It only checks repository
|
||||
files and reports whether the release surface can expose the signals an
|
||||
operator needs.
|
||||
|
||||
## Signal Model
|
||||
|
||||
- Live status: `scripts/loop-status.js` can emit JSON, watch active loops, and
|
||||
write snapshots for dashboards or handoffs.
|
||||
- Session traces: `scripts/session-inspect.js` can inspect Claude, dmux, and
|
||||
adapter-backed sessions, then write canonical snapshots.
|
||||
- Harness baseline: `scripts/harness-audit.js` provides a repeatable scorecard
|
||||
for tool coverage, context efficiency, quality gates, memory persistence,
|
||||
eval coverage, security guardrails, and cost efficiency.
|
||||
- Tool activity: `scripts/hooks/session-activity-tracker.js` records local
|
||||
`tool-usage.jsonl` events that ECC2 can sync.
|
||||
- Risk ledger: `ecc2/src/observability/mod.rs` scores tool calls and stores a
|
||||
paginated ledger for review.
|
||||
|
||||
## Reference Pressure
|
||||
|
||||
The current agent-tooling ecosystem is converging on the same operating needs:
|
||||
|
||||
- dmux, Orca, and Superset emphasize isolated worktrees plus one place to see
|
||||
agent state and merge/review work.
|
||||
- Claude HUD makes context, tool activity, agent activity, and todo progress
|
||||
visible inside the coding loop.
|
||||
- Autocontext records every run as durable traces, reports, artifacts, and
|
||||
reusable improvements.
|
||||
- Meta-Harness treats the harness itself as something to evaluate and improve,
|
||||
which requires clean logs of proposer behavior and outcomes.
|
||||
- Zed and OpenCode emphasize agent control surfaces, reviewable changes, and
|
||||
harness-specific configuration that should still preserve portable project
|
||||
knowledge.
|
||||
|
||||
ECC's answer is not a hosted analytics dependency by default. The first
|
||||
release-candidate gate is local and file-backed. Hosted telemetry can come
|
||||
later, but only after the local event model is useful enough to trust.
|
||||
|
||||
## Operator Workflow
|
||||
|
||||
1. Run `npm run observability:ready`.
|
||||
2. Run `npm run harness:audit -- --format json` for the broader harness
|
||||
scorecard.
|
||||
3. Run `node scripts/loop-status.js --json --write-dir .ecc/loop-status`
|
||||
during longer autonomous batches.
|
||||
4. Run `node scripts/session-inspect.js --list-adapters` to confirm which
|
||||
session surfaces are available.
|
||||
5. Use ECC2 tool logs for risky operations, conflict analysis, and handoff
|
||||
review before increasing autonomy.
|
||||
|
||||
The end-state is practical: before asking ECC to run larger multi-agent loops,
|
||||
the operator can prove the system has live status, durable session traces,
|
||||
baseline scorecards, and a local risk ledger.
|
||||
@ -31,6 +31,15 @@ Expected result: every test passes with zero failures. For release-specific drif
|
||||
node tests/docs/ecc2-release-surface.test.js
|
||||
```
|
||||
|
||||
Then check the local observability surface:
|
||||
|
||||
```bash
|
||||
npm run observability:ready
|
||||
```
|
||||
|
||||
This runs the [observability readiness gate](../../architecture/observability-readiness.md)
|
||||
for loop status, session traces, harness audit, and ECC2 tool-risk logs.
|
||||
|
||||
## First Skill
|
||||
|
||||
Read `skills/hermes-imports/SKILL.md` first.
|
||||
|
||||
@ -13,6 +13,7 @@ Claude Code remains a core target. Codex, OpenCode, Cursor, Gemini, and other ha
|
||||
- Clarified the split between ECC as the reusable substrate and Hermes as the operator shell.
|
||||
- Documented the cross-harness portability model for skills, hooks, MCPs, rules, and instructions.
|
||||
- Added a Hermes import playbook for turning local operator patterns into publishable ECC skills.
|
||||
- Added a local [observability readiness gate](../../architecture/observability-readiness.md) for loop status, session traces, harness audit, and ECC2 tool-risk logs.
|
||||
|
||||
## Why This Matters
|
||||
|
||||
@ -50,6 +51,7 @@ What stays local:
|
||||
1. Follow the [rc.1 quickstart](quickstart.md).
|
||||
2. Read the [Hermes setup guide](../../HERMES-SETUP.md).
|
||||
3. Review the [cross-harness architecture](../../architecture/cross-harness.md).
|
||||
4. Start with one workflow lane: engineering, research, content, or outreach.
|
||||
5. Import only sanitized operator patterns into ECC skills.
|
||||
6. Treat `ecc2/` as an alpha control plane until release packaging and installer behavior are finalized.
|
||||
4. Run the [observability readiness gate](../../architecture/observability-readiness.md).
|
||||
5. Start with one workflow lane: engineering, research, content, or outreach.
|
||||
6. Import only sanitized operator patterns into ECC skills.
|
||||
7. Treat `ecc2/` as an alpha control plane until release packaging and installer behavior are finalized.
|
||||
|
||||
@ -72,6 +72,7 @@
|
||||
"scripts/ecc.js",
|
||||
"scripts/gemini-adapt-agents.js",
|
||||
"scripts/harness-audit.js",
|
||||
"scripts/observability-readiness.js",
|
||||
"scripts/hooks/",
|
||||
"scripts/install-apply.js",
|
||||
"scripts/install-plan.js",
|
||||
@ -265,6 +266,7 @@
|
||||
"catalog:sync": "node scripts/ci/catalog.js --write --text",
|
||||
"lint": "eslint . && markdownlint '**/*.md' --ignore node_modules",
|
||||
"harness:audit": "node scripts/harness-audit.js",
|
||||
"observability:ready": "node scripts/observability-readiness.js",
|
||||
"claw": "node scripts/claw.js",
|
||||
"orchestrate:status": "node scripts/orchestration-status.js",
|
||||
"orchestrate:worker": "bash scripts/orchestrate-codex-worker.sh",
|
||||
|
||||
@ -14,7 +14,9 @@ const ignoredDirs = new Set([
|
||||
'node_modules',
|
||||
'.dmux',
|
||||
'.next',
|
||||
'.venv',
|
||||
'coverage',
|
||||
'venv',
|
||||
]);
|
||||
|
||||
const textExtensions = new Set([
|
||||
|
||||
309
scripts/observability-readiness.js
Normal file
309
scripts/observability-readiness.js
Normal file
@ -0,0 +1,309 @@
|
||||
#!/usr/bin/env node
|
||||
'use strict';
|
||||
|
||||
const fs = require('fs');
|
||||
const path = require('path');
|
||||
|
||||
const RUBRIC_VERSION = '2026-05-11';
|
||||
|
||||
function usage() {
|
||||
console.log([
|
||||
'Usage: node scripts/observability-readiness.js [--format <text|json>] [--root <dir>]',
|
||||
'',
|
||||
'Deterministic ECC 2.0 observability readiness gate.',
|
||||
'',
|
||||
'Options:',
|
||||
' --format <text|json> Output format (default: text)',
|
||||
' --root <dir> Repository root to inspect (default: cwd)',
|
||||
' --help, -h Show this help'
|
||||
].join('\n'));
|
||||
}
|
||||
|
||||
function readValue(args, index, flagName) {
|
||||
const value = args[index + 1];
|
||||
if (!value || value.startsWith('--')) {
|
||||
throw new Error(`${flagName} requires a value`);
|
||||
}
|
||||
return value;
|
||||
}
|
||||
|
||||
function parseArgs(argv) {
|
||||
const args = argv.slice(2);
|
||||
const parsed = {
|
||||
format: 'text',
|
||||
help: false,
|
||||
root: path.resolve(process.cwd())
|
||||
};
|
||||
|
||||
for (let index = 0; index < args.length; index += 1) {
|
||||
const arg = args[index];
|
||||
|
||||
if (arg === '--help' || arg === '-h') {
|
||||
parsed.help = true;
|
||||
continue;
|
||||
}
|
||||
|
||||
if (arg === '--format') {
|
||||
parsed.format = readValue(args, index, arg).toLowerCase();
|
||||
index += 1;
|
||||
continue;
|
||||
}
|
||||
|
||||
if (arg.startsWith('--format=')) {
|
||||
parsed.format = arg.slice('--format='.length).toLowerCase();
|
||||
continue;
|
||||
}
|
||||
|
||||
if (arg === '--root') {
|
||||
parsed.root = path.resolve(readValue(args, index, arg));
|
||||
index += 1;
|
||||
continue;
|
||||
}
|
||||
|
||||
if (arg.startsWith('--root=')) {
|
||||
parsed.root = path.resolve(arg.slice('--root='.length));
|
||||
continue;
|
||||
}
|
||||
|
||||
throw new Error(`Unknown argument: ${arg}`);
|
||||
}
|
||||
|
||||
if (!['text', 'json'].includes(parsed.format)) {
|
||||
throw new Error(`Invalid format: ${parsed.format}. Use text or json.`);
|
||||
}
|
||||
|
||||
return parsed;
|
||||
}
|
||||
|
||||
function fileExists(rootDir, relativePath) {
|
||||
return fs.existsSync(path.join(rootDir, relativePath));
|
||||
}
|
||||
|
||||
function readText(rootDir, relativePath) {
|
||||
try {
|
||||
return fs.readFileSync(path.join(rootDir, relativePath), 'utf8');
|
||||
} catch (_error) {
|
||||
return '';
|
||||
}
|
||||
}
|
||||
|
||||
function safeParseJson(text) {
|
||||
if (!text || !text.trim()) {
|
||||
return null;
|
||||
}
|
||||
|
||||
try {
|
||||
return JSON.parse(text);
|
||||
} catch (_error) {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
function includesAll(text, needles) {
|
||||
return needles.every(needle => text.includes(needle));
|
||||
}
|
||||
|
||||
function buildChecks(rootDir) {
|
||||
const packageJsonText = readText(rootDir, 'package.json');
|
||||
const packageJson = safeParseJson(packageJsonText) || {};
|
||||
const packageFiles = Array.isArray(packageJson.files) ? packageJson.files : [];
|
||||
const packageScripts = packageJson.scripts || {};
|
||||
const loopStatus = readText(rootDir, 'scripts/loop-status.js');
|
||||
const sessionInspect = readText(rootDir, 'scripts/session-inspect.js');
|
||||
const harnessAudit = readText(rootDir, 'scripts/harness-audit.js');
|
||||
const activityTracker = readText(rootDir, 'scripts/hooks/session-activity-tracker.js');
|
||||
const observabilityRust = readText(rootDir, 'ecc2/src/observability/mod.rs');
|
||||
const sessionStoreRust = readText(rootDir, 'ecc2/src/session/store.rs');
|
||||
const sessionManagerRust = readText(rootDir, 'ecc2/src/session/manager.rs');
|
||||
const readinessDoc = readText(rootDir, 'docs/architecture/observability-readiness.md');
|
||||
const quickstart = readText(rootDir, 'docs/releases/2.0.0-rc.1/quickstart.md');
|
||||
const releaseNotes = readText(rootDir, 'docs/releases/2.0.0-rc.1/release-notes.md');
|
||||
|
||||
return [
|
||||
{
|
||||
id: 'loop-status-live-signal',
|
||||
category: 'Live Status',
|
||||
points: 2,
|
||||
path: 'scripts/loop-status.js',
|
||||
description: 'Loop status supports JSON output, watch mode, and snapshot writes',
|
||||
pass: fileExists(rootDir, 'scripts/loop-status.js')
|
||||
&& includesAll(loopStatus, ['--json', '--watch', '--write-dir']),
|
||||
fix: 'Restore loop-status JSON/watch/write-dir support.'
|
||||
},
|
||||
{
|
||||
id: 'session-inspect-adapter-registry',
|
||||
category: 'Session Trace',
|
||||
points: 2,
|
||||
path: 'scripts/session-inspect.js',
|
||||
description: 'Session inspection exposes registered adapters and writable snapshots',
|
||||
pass: fileExists(rootDir, 'scripts/session-inspect.js')
|
||||
&& fileExists(rootDir, 'scripts/lib/session-adapters/registry.js')
|
||||
&& includesAll(sessionInspect, ['--list-adapters', '--write', 'inspectSessionTarget']),
|
||||
fix: 'Restore session-inspect adapter registry, list-adapters, and write support.'
|
||||
},
|
||||
{
|
||||
id: 'harness-audit-scorecard',
|
||||
category: 'Harness Baseline',
|
||||
points: 2,
|
||||
path: 'scripts/harness-audit.js',
|
||||
description: 'Harness audit emits deterministic text/JSON scorecards',
|
||||
pass: fileExists(rootDir, 'scripts/harness-audit.js')
|
||||
&& packageScripts['harness:audit'] === 'node scripts/harness-audit.js'
|
||||
&& includesAll(harnessAudit, ['Deterministic harness audit', '--format', 'overall_score']),
|
||||
fix: 'Restore the harness:audit package script and deterministic scorecard output.'
|
||||
},
|
||||
{
|
||||
id: 'hook-activity-jsonl',
|
||||
category: 'Tool Activity',
|
||||
points: 2,
|
||||
path: 'scripts/hooks/session-activity-tracker.js',
|
||||
description: 'Hook activity tracker writes tool usage JSONL for later sync',
|
||||
pass: fileExists(rootDir, 'scripts/hooks/session-activity-tracker.js')
|
||||
&& includesAll(activityTracker, ['tool-usage.jsonl', 'session_id', 'tool_name']),
|
||||
fix: 'Restore hook-side tool activity recording to metrics/tool-usage.jsonl.'
|
||||
},
|
||||
{
|
||||
id: 'ecc2-tool-risk-ledger',
|
||||
category: 'Tool Activity',
|
||||
points: 3,
|
||||
path: 'ecc2/src/observability/mod.rs',
|
||||
description: 'ECC2 records tool calls with risk scoring and paginated queries',
|
||||
pass: fileExists(rootDir, 'ecc2/src/observability/mod.rs')
|
||||
&& includesAll(observabilityRust, ['ToolCallEvent', 'RiskAssessment', 'ToolLogger'])
|
||||
&& includesAll(sessionStoreRust, ['insert_tool_log', 'query_tool_logs'])
|
||||
&& includesAll(sessionManagerRust, ['sync_tool_activity_metrics', 'tool-usage.jsonl']),
|
||||
fix: 'Restore ECC2 tool logging, risk scoring, store queries, and metrics sync.'
|
||||
},
|
||||
{
|
||||
id: 'release-observability-onramp',
|
||||
category: 'Operator Onramp',
|
||||
points: 2,
|
||||
path: 'docs/architecture/observability-readiness.md',
|
||||
description: 'Release docs explain the local observability readiness workflow',
|
||||
pass: readinessDoc.includes('node scripts/observability-readiness.js --format json')
|
||||
&& quickstart.includes('observability-readiness.md')
|
||||
&& releaseNotes.includes('observability-readiness.md'),
|
||||
fix: 'Add the observability readiness doc and link it from rc.1 release docs.'
|
||||
},
|
||||
{
|
||||
id: 'package-exposes-readiness-gate',
|
||||
category: 'Packaging',
|
||||
points: 1,
|
||||
path: 'package.json',
|
||||
description: 'Package exposes the observability readiness gate',
|
||||
pass: packageScripts['observability:ready'] === 'node scripts/observability-readiness.js'
|
||||
&& packageFiles.includes('scripts/observability-readiness.js'),
|
||||
fix: 'Add scripts/observability-readiness.js to package files and observability:ready.'
|
||||
}
|
||||
];
|
||||
}
|
||||
|
||||
function buildReport(rootDir) {
|
||||
const checks = buildChecks(rootDir);
|
||||
const categories = {};
|
||||
|
||||
for (const check of checks) {
|
||||
if (!categories[check.category]) {
|
||||
categories[check.category] = {
|
||||
score: 0,
|
||||
max_score: 0,
|
||||
passed: 0,
|
||||
total: 0
|
||||
};
|
||||
}
|
||||
|
||||
categories[check.category].max_score += check.points;
|
||||
categories[check.category].total += 1;
|
||||
|
||||
if (check.pass) {
|
||||
categories[check.category].score += check.points;
|
||||
categories[check.category].passed += 1;
|
||||
}
|
||||
}
|
||||
|
||||
const overallScore = checks
|
||||
.filter(check => check.pass)
|
||||
.reduce((sum, check) => sum + check.points, 0);
|
||||
const maxScore = checks.reduce((sum, check) => sum + check.points, 0);
|
||||
const failingChecks = checks.filter(check => !check.pass);
|
||||
|
||||
return {
|
||||
schema_version: 'ecc.observability-readiness.v1',
|
||||
rubric_version: RUBRIC_VERSION,
|
||||
deterministic: true,
|
||||
root_dir: fs.realpathSync(rootDir),
|
||||
overall_score: overallScore,
|
||||
max_score: maxScore,
|
||||
ready: overallScore === maxScore,
|
||||
categories,
|
||||
checks,
|
||||
top_actions: failingChecks
|
||||
.sort((left, right) => right.points - left.points || left.id.localeCompare(right.id))
|
||||
.slice(0, 3)
|
||||
.map(check => ({
|
||||
id: check.id,
|
||||
path: check.path,
|
||||
fix: check.fix
|
||||
}))
|
||||
};
|
||||
}
|
||||
|
||||
function renderText(report) {
|
||||
const lines = [
|
||||
`Observability Readiness: ${report.overall_score}/${report.max_score}`,
|
||||
`Ready: ${report.ready ? 'yes' : 'no'}`,
|
||||
'',
|
||||
'Categories:'
|
||||
];
|
||||
|
||||
for (const [name, category] of Object.entries(report.categories)) {
|
||||
lines.push(`- ${name}: ${category.score}/${category.max_score} (${category.passed}/${category.total})`);
|
||||
}
|
||||
|
||||
lines.push('', 'Checks:');
|
||||
for (const check of report.checks) {
|
||||
lines.push(`- ${check.pass ? 'PASS' : 'FAIL'} ${check.id}: ${check.description}`);
|
||||
}
|
||||
|
||||
if (report.top_actions.length > 0) {
|
||||
lines.push('', 'Top Actions:');
|
||||
for (const action of report.top_actions) {
|
||||
lines.push(`- ${action.path}: ${action.fix}`);
|
||||
}
|
||||
}
|
||||
|
||||
return `${lines.join('\n')}\n`;
|
||||
}
|
||||
|
||||
function main() {
|
||||
const args = parseArgs(process.argv);
|
||||
|
||||
if (args.help) {
|
||||
usage();
|
||||
return;
|
||||
}
|
||||
|
||||
const report = buildReport(args.root);
|
||||
|
||||
if (args.format === 'json') {
|
||||
console.log(JSON.stringify(report, null, 2));
|
||||
} else {
|
||||
process.stdout.write(renderText(report));
|
||||
}
|
||||
}
|
||||
|
||||
if (require.main === module) {
|
||||
try {
|
||||
main();
|
||||
} catch (error) {
|
||||
console.error(`Error: ${error.message}`);
|
||||
process.exit(1);
|
||||
}
|
||||
}
|
||||
|
||||
module.exports = {
|
||||
buildChecks,
|
||||
buildReport,
|
||||
parseArgs,
|
||||
renderText
|
||||
};
|
||||
@ -109,6 +109,27 @@ if (
|
||||
passed++;
|
||||
else failed++;
|
||||
|
||||
if (
|
||||
test('skips Python virtual environments', () => {
|
||||
const root = makeTempRoot('ecc-unicode-venv-');
|
||||
fs.mkdirSync(path.join(root, '.venv', 'lib', 'python3.12', 'site-packages'), { recursive: true });
|
||||
fs.mkdirSync(path.join(root, 'venv', 'lib', 'python3.12', 'site-packages'), { recursive: true });
|
||||
fs.writeFileSync(
|
||||
path.join(root, '.venv', 'lib', 'python3.12', 'site-packages', 'package.py'),
|
||||
`message = "hello ${rocketEmoji}"\n`
|
||||
);
|
||||
fs.writeFileSync(
|
||||
path.join(root, 'venv', 'lib', 'python3.12', 'site-packages', 'package.py'),
|
||||
`message = "hello ${rocketEmoji}"\n`
|
||||
);
|
||||
|
||||
const result = runCheck(root);
|
||||
assert.strictEqual(result.status, 0, result.stdout + result.stderr);
|
||||
})
|
||||
)
|
||||
passed++;
|
||||
else failed++;
|
||||
|
||||
console.log(`\nPassed: ${passed}`);
|
||||
console.log(`Failed: ${failed}`);
|
||||
process.exit(failed > 0 ? 1 : 0);
|
||||
|
||||
@ -53,6 +53,7 @@ function buildExpectedPublishPaths(repoRoot) {
|
||||
"scripts/install-plan.js",
|
||||
"scripts/list-installed.js",
|
||||
"scripts/loop-status.js",
|
||||
"scripts/observability-readiness.js",
|
||||
"scripts/skill-create-output.js",
|
||||
"scripts/repair.js",
|
||||
"scripts/harness-audit.js",
|
||||
|
||||
209
tests/scripts/observability-readiness.test.js
Normal file
209
tests/scripts/observability-readiness.test.js
Normal file
@ -0,0 +1,209 @@
|
||||
/**
|
||||
* Tests for scripts/observability-readiness.js
|
||||
*/
|
||||
|
||||
const assert = require('assert');
|
||||
const fs = require('fs');
|
||||
const os = require('os');
|
||||
const path = require('path');
|
||||
const { execFileSync, spawnSync } = require('child_process');
|
||||
|
||||
const SCRIPT = path.join(__dirname, '..', '..', 'scripts', 'observability-readiness.js');
|
||||
const { buildReport, parseArgs } = require(SCRIPT);
|
||||
|
||||
function createTempDir(prefix) {
|
||||
return fs.mkdtempSync(path.join(os.tmpdir(), prefix));
|
||||
}
|
||||
|
||||
function cleanup(dirPath) {
|
||||
fs.rmSync(dirPath, { recursive: true, force: true });
|
||||
}
|
||||
|
||||
function writeFile(rootDir, relativePath, content) {
|
||||
const targetPath = path.join(rootDir, relativePath);
|
||||
fs.mkdirSync(path.dirname(targetPath), { recursive: true });
|
||||
fs.writeFileSync(targetPath, content);
|
||||
}
|
||||
|
||||
function run(args = [], options = {}) {
|
||||
return execFileSync('node', [SCRIPT, ...args], {
|
||||
cwd: options.cwd || path.join(__dirname, '..', '..'),
|
||||
encoding: 'utf8',
|
||||
stdio: ['pipe', 'pipe', 'pipe'],
|
||||
timeout: 10000
|
||||
});
|
||||
}
|
||||
|
||||
function runProcess(args = [], options = {}) {
|
||||
return spawnSync('node', [SCRIPT, ...args], {
|
||||
cwd: options.cwd || path.join(__dirname, '..', '..'),
|
||||
encoding: 'utf8',
|
||||
stdio: ['pipe', 'pipe', 'pipe'],
|
||||
timeout: 10000
|
||||
});
|
||||
}
|
||||
|
||||
function seedMinimalRepo(rootDir, overrides = {}) {
|
||||
const files = {
|
||||
'package.json': JSON.stringify({
|
||||
name: 'everything-claude-code',
|
||||
files: ['scripts/observability-readiness.js'],
|
||||
scripts: {
|
||||
'harness:audit': 'node scripts/harness-audit.js',
|
||||
'observability:ready': 'node scripts/observability-readiness.js'
|
||||
}
|
||||
}, null, 2),
|
||||
'scripts/loop-status.js': '--json --watch --write-dir',
|
||||
'scripts/session-inspect.js': '--list-adapters --write inspectSessionTarget',
|
||||
'scripts/lib/session-adapters/registry.js': 'module.exports = {};',
|
||||
'scripts/harness-audit.js': 'Deterministic harness audit --format overall_score',
|
||||
'scripts/hooks/session-activity-tracker.js': 'tool-usage.jsonl session_id tool_name',
|
||||
'ecc2/src/observability/mod.rs': 'ToolCallEvent RiskAssessment ToolLogger',
|
||||
'ecc2/src/session/store.rs': 'insert_tool_log query_tool_logs',
|
||||
'ecc2/src/session/manager.rs': 'sync_tool_activity_metrics tool-usage.jsonl',
|
||||
'docs/architecture/observability-readiness.md': 'node scripts/observability-readiness.js --format json',
|
||||
'docs/releases/2.0.0-rc.1/quickstart.md': 'observability-readiness.md',
|
||||
'docs/releases/2.0.0-rc.1/release-notes.md': 'observability-readiness.md'
|
||||
};
|
||||
|
||||
for (const [relativePath, content] of Object.entries({ ...files, ...overrides })) {
|
||||
if (content === null) {
|
||||
continue;
|
||||
}
|
||||
writeFile(rootDir, relativePath, content);
|
||||
}
|
||||
}
|
||||
|
||||
function test(name, fn) {
|
||||
try {
|
||||
fn();
|
||||
console.log(` ✓ ${name}`);
|
||||
return true;
|
||||
} catch (error) {
|
||||
console.log(` ✗ ${name}`);
|
||||
console.log(` Error: ${error.message}`);
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
function runTests() {
|
||||
console.log('\n=== Testing observability-readiness.js ===\n');
|
||||
|
||||
let passed = 0;
|
||||
let failed = 0;
|
||||
|
||||
if (test('parseArgs accepts supported forms and rejects invalid input', () => {
|
||||
const rootDir = createTempDir('observability-readiness-args-');
|
||||
|
||||
try {
|
||||
assert.strictEqual(parseArgs(['node', 'script', '--help']).help, true);
|
||||
assert.strictEqual(parseArgs(['node', 'script', '-h']).help, true);
|
||||
|
||||
const spaced = parseArgs(['node', 'script', '--format', 'json', '--root', rootDir]);
|
||||
assert.strictEqual(spaced.format, 'json');
|
||||
assert.strictEqual(spaced.root, path.resolve(rootDir));
|
||||
|
||||
const equals = parseArgs(['node', 'script', '--format=json', `--root=${rootDir}`]);
|
||||
assert.strictEqual(equals.format, 'json');
|
||||
assert.strictEqual(equals.root, path.resolve(rootDir));
|
||||
|
||||
assert.throws(() => parseArgs(['node', 'script', '--format', 'xml']), /Invalid format: xml/);
|
||||
assert.throws(() => parseArgs(['node', 'script', '--root']), /--root requires a value/);
|
||||
assert.throws(() => parseArgs(['node', 'script', '--unknown']), /Unknown argument: --unknown/);
|
||||
} finally {
|
||||
cleanup(rootDir);
|
||||
}
|
||||
})) passed++; else failed++;
|
||||
|
||||
if (test('cli help exits cleanly and invalid cli args exit with stderr', () => {
|
||||
const help = runProcess(['--help']);
|
||||
assert.strictEqual(help.status, 0);
|
||||
assert.strictEqual(help.stderr, '');
|
||||
assert.ok(help.stdout.includes('Usage: node scripts/observability-readiness.js'));
|
||||
|
||||
const invalid = runProcess(['--format', 'xml']);
|
||||
assert.strictEqual(invalid.status, 1);
|
||||
assert.strictEqual(invalid.stdout, '');
|
||||
assert.ok(invalid.stderr.includes('Error: Invalid format: xml. Use text or json.'));
|
||||
})) passed++; else failed++;
|
||||
|
||||
if (test('current repo reports a complete readiness score', () => {
|
||||
const parsed = JSON.parse(run(['--format=json']));
|
||||
|
||||
assert.strictEqual(parsed.schema_version, 'ecc.observability-readiness.v1');
|
||||
assert.strictEqual(parsed.deterministic, true);
|
||||
assert.strictEqual(parsed.ready, true);
|
||||
assert.strictEqual(parsed.overall_score, parsed.max_score);
|
||||
assert.strictEqual(parsed.top_actions.length, 0);
|
||||
})) passed++; else failed++;
|
||||
|
||||
if (test('text output includes summary, categories, and checks', () => {
|
||||
const output = run();
|
||||
|
||||
assert.ok(output.includes('Observability Readiness:'));
|
||||
assert.ok(output.includes('Categories:'));
|
||||
assert.ok(output.includes('Checks:'));
|
||||
assert.ok(output.includes('PASS loop-status-live-signal'));
|
||||
})) passed++; else failed++;
|
||||
|
||||
if (test('minimal seeded repo passes all checks', () => {
|
||||
const projectRoot = createTempDir('observability-readiness-pass-');
|
||||
|
||||
try {
|
||||
seedMinimalRepo(projectRoot);
|
||||
const report = buildReport(projectRoot);
|
||||
|
||||
assert.strictEqual(report.ready, true);
|
||||
assert.strictEqual(report.overall_score, report.max_score);
|
||||
assert.deepStrictEqual(report.top_actions, []);
|
||||
} finally {
|
||||
cleanup(projectRoot);
|
||||
}
|
||||
})) passed++; else failed++;
|
||||
|
||||
if (test('missing tool logger surfaces become prioritized top actions', () => {
|
||||
const projectRoot = createTempDir('observability-readiness-fail-');
|
||||
|
||||
try {
|
||||
seedMinimalRepo(projectRoot, {
|
||||
'ecc2/src/observability/mod.rs': 'ToolCallEvent only'
|
||||
});
|
||||
const report = buildReport(projectRoot);
|
||||
|
||||
assert.strictEqual(report.ready, false);
|
||||
assert.ok(report.top_actions.some(action => action.id === 'ecc2-tool-risk-ledger'));
|
||||
assert.ok(report.checks.some(check => check.id === 'ecc2-tool-risk-ledger' && !check.pass));
|
||||
} finally {
|
||||
cleanup(projectRoot);
|
||||
}
|
||||
})) passed++; else failed++;
|
||||
|
||||
if (test('missing release onramp fails without disturbing core tool checks', () => {
|
||||
const projectRoot = createTempDir('observability-readiness-doc-fail-');
|
||||
|
||||
try {
|
||||
seedMinimalRepo(projectRoot, {
|
||||
'docs/releases/2.0.0-rc.1/quickstart.md': 'quickstart without link'
|
||||
});
|
||||
const report = buildReport(projectRoot);
|
||||
|
||||
assert.strictEqual(report.ready, false);
|
||||
assert.ok(report.checks.some(check => check.id === 'release-observability-onramp' && !check.pass));
|
||||
assert.ok(report.checks.some(check => check.id === 'loop-status-live-signal' && check.pass));
|
||||
} finally {
|
||||
cleanup(projectRoot);
|
||||
}
|
||||
})) passed++; else failed++;
|
||||
|
||||
console.log('\nResults:');
|
||||
console.log(` Passed: ${passed}`);
|
||||
console.log(` Failed: ${failed}`);
|
||||
|
||||
if (failed > 0) {
|
||||
process.exit(1);
|
||||
}
|
||||
}
|
||||
|
||||
if (require.main === module) {
|
||||
runTests();
|
||||
}
|
||||
Loading…
x
Reference in New Issue
Block a user