ci: gate observability on release safety evidence

Add release-safety evidence coverage to observability readiness and refresh rc.1 publication gate docs.
This commit is contained in:
Affaan Mustafa 2026-05-13 04:14:47 -04:00 committed by GitHub
parent d4728a0d80
commit 42f04edc03
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
6 changed files with 129 additions and 6 deletions

View File

@ -35,6 +35,10 @@ operator needs.
- Progress sync: `docs/architecture/progress-sync-contract.md` defines how - Progress sync: `docs/architecture/progress-sync-contract.md` defines how
GitHub, Linear, local handoffs, the repo roadmap, and `scripts/work-items.js` GitHub, Linear, local handoffs, the repo roadmap, and `scripts/work-items.js`
stay aligned during merge batches and release-gate reviews. stay aligned during merge batches and release-gate reviews.
- Release safety: `docs/releases/2.0.0-rc.1/publication-readiness.md`,
post-hardening evidence, supply-chain incident response, workflow-security
validation, npm pack checks, and release-surface tests must be present before
any public tag, package publish, plugin submission, or announcement action.
## Reference Pressure ## Reference Pressure
@ -71,6 +75,9 @@ later, but only after the local event model is useful enough to trust.
relying on local work-item status for a tracked repository. relying on local work-item status for a tracked repository.
7. Use ECC2 tool logs for risky operations, conflict analysis, and handoff 7. Use ECC2 tool logs for risky operations, conflict analysis, and handoff
review before increasing autonomy. review before increasing autonomy.
8. Re-run the release-safety evidence checks before any public release action:
publication readiness, supply-chain incident response, workflow-security
validation, package surface, and release-surface tests.
The end-state is practical: before asking ECC to run larger multi-agent loops, The end-state is practical: before asking ECC to run larger multi-agent loops,
the operator can prove the system has live status, durable session traces, the operator can prove the system has live status, durable session traces,

View File

@ -43,14 +43,15 @@ The prior post-#42 local checkout handoff recorded both ECC-Tools repos at
| --- | --- | --- | | --- | --- | --- |
| Harness audit | `npm run harness:audit -- --format json` | `overall_score: 70`, `max_score: 70`, no top actions | | Harness audit | `npm run harness:audit -- --format json` | `overall_score: 70`, `max_score: 70`, no top actions |
| Adapter scorecard | `npm run harness:adapters -- --check` | `Harness Adapter Compliance: PASS`; 11 adapters | | Adapter scorecard | `npm run harness:adapters -- --check` | `Harness Adapter Compliance: PASS`; 11 adapters |
| Observability readiness | `npm run observability:ready -- --format json` | `overall_score: 18`, `max_score: 18`, `ready: true`, no top actions | | Observability readiness | `npm run observability:ready -- --format json` | `overall_score: 21`, `max_score: 21`, `ready: true`, no top actions; includes Release Safety 3/3 |
| Workflow security validator | `node scripts/ci/validate-workflow-security.js` | Validated 7 workflow files | | Workflow security validator | `node scripts/ci/validate-workflow-security.js` | Validated 7 workflow files |
| Workflow validator tests | `node tests/ci/validate-workflow-security.test.js` | Passed 14/14 | | Workflow validator tests | `node tests/ci/validate-workflow-security.test.js` | Passed 14/14 |
| Release surface | `node tests/docs/ecc2-release-surface.test.js` | Passed 18/18 | | Release surface | `node tests/docs/ecc2-release-surface.test.js` | Passed 18/18 |
| Package surface | `node tests/scripts/npm-publish-surface.test.js` | Passed 2/2 | | Package surface | `node tests/scripts/npm-publish-surface.test.js` | Passed 2/2 |
| Root suite | `node tests/run-all.js` | Passed 2380/2380, 0 failed | | Root suite | `node tests/run-all.js` | Passed 2381/2381, 0 failed |
| Markdown lint | `npx markdownlint-cli '**/*.md' --ignore node_modules --ignore docs/drafts` | Passed | | Markdown lint | `npx markdownlint-cli '**/*.md' --ignore node_modules --ignore docs/drafts` | Passed |
| Rust surface | `cd ecc2 && cargo test` | Passed 462/462; warnings only for unused functions/fields | | Rust surface | `cd ecc2 && cargo test` | Passed 462/462; warnings only for unused functions/fields |
| GitGuardian Security Checks | GitHub check on post-hardening security PRs | Passed before merge |
## Supply-Chain Evidence ## Supply-Chain Evidence
@ -60,6 +61,7 @@ The prior post-#42 local checkout handoff recorded both ECC-Tools repos at
| Local npm signature audit | `npm audit signatures` | 241 verified registry signatures and 30 verified attestations | | Local npm signature audit | `npm audit signatures` | 241 verified registry signatures and 30 verified attestations |
| Rust advisory audit | `cd ecc2 && cargo audit -q` | Passed silently | | Rust advisory audit | `cd ecc2 && cargo audit -q` | Passed silently |
| TanStack / Mini Shai-Hulud IOC check | Grep for affected package namespaces, payload filenames, and known commit marker | No runtime or lockfile dependency on affected packages; no worm IOC matches | | TanStack / Mini Shai-Hulud IOC check | Grep for affected package namespaces, payload filenames, and known commit marker | No runtime or lockfile dependency on affected packages; no worm IOC matches |
| GitGuardian Security Checks | GitHub check on post-hardening security PRs | Passed before merge |
## External Advisory Mapping ## External Advisory Mapping

View File

@ -52,8 +52,10 @@ Record the exact commit SHA and command output before any publication action:
| Clean release branch | `git status --short --branch` | On intended release commit; no unrelated files | Pending final clean-checkout release pass; May 13 evidence branch still had unrelated untracked `docs/drafts/` | | Clean release branch | `git status --short --branch` | On intended release commit; no unrelated files | Pending final clean-checkout release pass; May 13 evidence branch still had unrelated untracked `docs/drafts/` |
| Harness audit | `npm run harness:audit -- --format json` | 70/70 passing | `publication-evidence-2026-05-13.md`: 70/70 | | Harness audit | `npm run harness:audit -- --format json` | 70/70 passing | `publication-evidence-2026-05-13.md`: 70/70 |
| Adapter scorecard | `npm run harness:adapters -- --check` | PASS | `publication-evidence-2026-05-13.md`: PASS, 11 adapters | | Adapter scorecard | `npm run harness:adapters -- --check` | PASS | `publication-evidence-2026-05-13.md`: PASS, 11 adapters |
| Observability readiness | `npm run observability:ready` | 18/18 passing | `publication-evidence-2026-05-13-post-hardening.md`: 18/18, ready true | | Observability readiness | `npm run observability:ready` | 21/21 passing | `publication-evidence-2026-05-13-post-hardening.md`: 21/21, ready true after release-safety gate refresh |
| Root suite | `node tests/run-all.js` | 0 failures | `publication-evidence-2026-05-13-post-hardening.md`: 2380 passed, 0 failed | | Release safety gate | `npm run observability:ready -- --format json` | Release Safety category passing with publication readiness, supply-chain, workflow security, package surface, and release-surface evidence | `publication-evidence-2026-05-13-post-hardening.md`: Release Safety 3/3 |
| Supply-chain verification | `npm audit --json`; `npm audit signatures`; `cd ecc2 && cargo audit -q`; Dependabot alerts; GitGuardian Security Checks | 0 vulnerabilities/alerts, registry signatures verified, GitGuardian clean | `publication-evidence-2026-05-13-post-hardening.md`: npm, cargo, Dependabot, TanStack/Mini Shai-Hulud, and GitGuardian evidence |
| Root suite | `node tests/run-all.js` | 0 failures | `publication-evidence-2026-05-13-post-hardening.md`: 2381 passed, 0 failed |
| Markdown lint | `npx markdownlint-cli '**/*.md' --ignore node_modules` | 0 failures | `publication-evidence-2026-05-13.md`: passed after zh-CN CLAUDE list-marker normalization | | Markdown lint | `npx markdownlint-cli '**/*.md' --ignore node_modules` | 0 failures | `publication-evidence-2026-05-13.md`: passed after zh-CN CLAUDE list-marker normalization |
| Package surface | `node tests/scripts/npm-publish-surface.test.js` | 0 failures; no Python bytecode in npm tarball | `2/2` passed in May 12 evidence pass | | Package surface | `node tests/scripts/npm-publish-surface.test.js` | 0 failures; no Python bytecode in npm tarball | `2/2` passed in May 12 evidence pass |
| Release surface | `node tests/docs/ecc2-release-surface.test.js` | 0 failures | `publication-evidence-2026-05-13.md`: 18/18 passed | | Release surface | `node tests/docs/ecc2-release-surface.test.js` | 0 failures | `publication-evidence-2026-05-13.md`: 18/18 passed |

View File

@ -8,7 +8,9 @@ they do not prove that the workflow executed the intended code path.
## Current External Trigger ## Current External Trigger
As of 2026-05-13, the active incident class is the May 2026 TanStack npm As of 2026-05-13, the active incident class is the May 2026 TanStack npm
supply-chain compromise: supply-chain compromise. ECC also keeps Mini Shai-Hulud-style npm worm IOCs in
the same release-safety sweep because both incident classes target package
install/publish paths and developer credentials:
- TanStack reported 84 malicious versions across 42 `@tanstack/*` packages, - TanStack reported 84 malicious versions across 42 `@tanstack/*` packages,
published on 2026-05-11 between 19:20 and 19:26 UTC. published on 2026-05-11 between 19:20 and 19:26 UTC.

View File

@ -127,6 +127,13 @@ function buildChecks(rootDir) {
const progressSyncContract = readText(rootDir, 'docs/architecture/progress-sync-contract.md'); const progressSyncContract = readText(rootDir, 'docs/architecture/progress-sync-contract.md');
const gaRoadmap = readText(rootDir, 'docs/ECC-2.0-GA-ROADMAP.md'); const gaRoadmap = readText(rootDir, 'docs/ECC-2.0-GA-ROADMAP.md');
const workItems = readText(rootDir, 'scripts/work-items.js'); const workItems = readText(rootDir, 'scripts/work-items.js');
const publicationReadiness = readText(rootDir, 'docs/releases/2.0.0-rc.1/publication-readiness.md');
const postHardeningEvidence = readText(rootDir, 'docs/releases/2.0.0-rc.1/publication-evidence-2026-05-13-post-hardening.md');
const supplyChainIncidentResponse = readText(rootDir, 'docs/security/supply-chain-incident-response.md');
const workflowSecurityValidator = readText(rootDir, 'scripts/ci/validate-workflow-security.js');
const workflowSecurityValidatorTests = readText(rootDir, 'tests/ci/validate-workflow-security.test.js');
const publishSurfaceTest = readText(rootDir, 'tests/scripts/npm-publish-surface.test.js');
const releaseSurfaceTest = readText(rootDir, 'tests/docs/ecc2-release-surface.test.js');
const hudStatusFixture = safeParseJson(readText(rootDir, 'examples/hud-status-contract.json')) || {}; const hudStatusFixture = safeParseJson(readText(rootDir, 'examples/hud-status-contract.json')) || {};
const quickstart = readText(rootDir, 'docs/releases/2.0.0-rc.1/quickstart.md'); const quickstart = readText(rootDir, 'docs/releases/2.0.0-rc.1/quickstart.md');
const releaseNotes = readText(rootDir, 'docs/releases/2.0.0-rc.1/release-notes.md'); const releaseNotes = readText(rootDir, 'docs/releases/2.0.0-rc.1/release-notes.md');
@ -275,6 +282,56 @@ function buildChecks(rootDir) {
]), ]),
fix: 'Add the progress sync contract, link it from the GA roadmap, and preserve work-items GitHub sync.' fix: 'Add the progress sync contract, link it from the GA roadmap, and preserve work-items GitHub sync.'
}, },
{
id: 'release-safety-evidence',
category: 'Release Safety',
points: 3,
path: 'docs/releases/2.0.0-rc.1/publication-readiness.md',
description: 'Release readiness includes package, workflow, and supply-chain evidence before publication',
pass: fileExists(rootDir, 'docs/releases/2.0.0-rc.1/publication-readiness.md')
&& fileExists(rootDir, 'docs/releases/2.0.0-rc.1/publication-evidence-2026-05-13-post-hardening.md')
&& fileExists(rootDir, 'docs/security/supply-chain-incident-response.md')
&& fileExists(rootDir, 'scripts/ci/validate-workflow-security.js')
&& fileExists(rootDir, 'tests/ci/validate-workflow-security.test.js')
&& fileExists(rootDir, 'tests/scripts/npm-publish-surface.test.js')
&& fileExists(rootDir, 'tests/docs/ecc2-release-surface.test.js')
&& includesAll(publicationReadiness, [
'Publication Gates',
'Required Command Evidence',
'Do Not Publish If',
'npm dist-tag',
'GitGuardian',
'Dependabot alerts',
'npm audit signatures'
])
&& includesAll(postHardeningEvidence, [
'npm audit --json',
'npm audit signatures',
'cargo audit',
'Dependabot alert API',
'TanStack',
'Mini Shai-Hulud',
'GitGuardian Security Checks'
])
&& includesAll(supplyChainIncidentResponse, [
'TanStack',
'Mini Shai-Hulud',
'npm audit signatures',
'trusted publishing',
'pull_request_target',
'id-token: write'
])
&& includesAll(workflowSecurityValidator, [
'persist-credentials: false',
'npm audit signatures',
'pull_request_target',
'id-token: write'
])
&& includesAll(workflowSecurityValidatorTests, ['npm audit signatures', 'persist-credentials: false'])
&& includesAll(publishSurfaceTest, ['npm pack', 'Python bytecode'])
&& includesAll(releaseSurfaceTest, ['publication-readiness.md']),
fix: 'Refresh publication readiness, post-hardening evidence, supply-chain response docs, workflow-security validator coverage, and package/release surface tests.'
},
{ {
id: 'package-exposes-readiness-gate', id: 'package-exposes-readiness-gate',
category: 'Packaging', category: 'Packaging',

View File

@ -92,7 +92,43 @@ function seedMinimalRepo(rootDir, overrides = {}) {
sync: {} sync: {}
}, null, 2), }, null, 2),
'docs/releases/2.0.0-rc.1/quickstart.md': 'observability-readiness.md', 'docs/releases/2.0.0-rc.1/quickstart.md': 'observability-readiness.md',
'docs/releases/2.0.0-rc.1/release-notes.md': 'observability-readiness.md' 'docs/releases/2.0.0-rc.1/release-notes.md': 'observability-readiness.md',
'docs/releases/2.0.0-rc.1/publication-readiness.md': [
'Publication Gates',
'Required Command Evidence',
'Do Not Publish If',
'npm dist-tag',
'GitGuardian',
'Dependabot alerts',
'npm audit signatures'
].join('\n'),
'docs/releases/2.0.0-rc.1/publication-evidence-2026-05-13-post-hardening.md': [
'npm audit --json',
'npm audit signatures',
'cargo audit',
'Dependabot alert API',
'TanStack',
'Mini Shai-Hulud',
'GitGuardian Security Checks'
].join('\n'),
'docs/security/supply-chain-incident-response.md': [
'TanStack',
'Mini Shai-Hulud',
'npm audit signatures',
'trusted publishing',
'pull_request_target',
'id-token: write'
].join('\n'),
'scripts/ci/validate-workflow-security.js': [
'persist-credentials: false',
'npm audit signatures',
'pull_request_target',
'id-token: write',
'shared cache'
].join('\n'),
'tests/ci/validate-workflow-security.test.js': 'npm audit signatures persist-credentials: false',
'tests/scripts/npm-publish-surface.test.js': 'npm pack --dry-run Python bytecode',
'tests/docs/ecc2-release-surface.test.js': 'publication-readiness.md',
}; };
for (const [relativePath, content] of Object.entries({ ...files, ...overrides })) { for (const [relativePath, content] of Object.entries({ ...files, ...overrides })) {
@ -258,6 +294,23 @@ function runTests() {
} }
})) passed++; else failed++; })) passed++; else failed++;
if (test('missing release safety evidence fails without disturbing live status checks', () => {
const projectRoot = createTempDir('observability-readiness-release-safety-fail-');
try {
seedMinimalRepo(projectRoot, {
'docs/releases/2.0.0-rc.1/publication-evidence-2026-05-13-post-hardening.md': 'npm audit --json only'
});
const report = buildReport(projectRoot);
assert.strictEqual(report.ready, false);
assert.ok(report.checks.some(check => check.id === 'release-safety-evidence' && !check.pass));
assert.ok(report.checks.some(check => check.id === 'loop-status-live-signal' && check.pass));
} finally {
cleanup(projectRoot);
}
})) passed++; else failed++;
console.log('\nResults:'); console.log('\nResults:');
console.log(` Passed: ${passed}`); console.log(` Passed: ${passed}`);
console.log(` Failed: ${failed}`); console.log(` Failed: ${failed}`);