docs: record ECC Tools analyzer corpus evidence (#1808)

This commit is contained in:
Affaan Mustafa 2026-05-12 11:39:59 -04:00 committed by GitHub
parent adc97769be
commit 86a529b3da
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194

View File

@ -95,6 +95,9 @@ As of 2026-05-12:
- ECC-Tools PR #36 added deep-analyzer predictive follow-ups, a Deep Analyzer - ECC-Tools PR #36 added deep-analyzer predictive follow-ups, a Deep Analyzer
Evidence PR-risk bucket, and a Linear-ready project sync backlog table for Evidence PR-risk bucket, and a Linear-ready project sync backlog table for
deferred follow-up work. deferred follow-up work.
- ECC-Tools PR #37 added a maintained analyzer corpus fixture, corpus validation
tests, and co-located analyzer reference-set evidence recognition for future
predictive follow-ups and PR-risk taxonomy checks.
- ECC PR #1803 landed the contributor Quarkus handling branch after maintainer - ECC PR #1803 landed the contributor Quarkus handling branch after maintainer
cleanup, current-`main` alignment, full local validation, and preservation of cleanup, current-`main` alignment, full local validation, and preservation of
the author's removal of incomplete ja-JP and zh-CN Quarkus translations. the author's removal of incomplete ja-JP and zh-CN Quarkus translations.
@ -130,8 +133,8 @@ is not complete unless the evidence column exists and has been freshly verified.
| Claude and Codex plugin publication | Contact/submission path with required artifacts and status | Publication readiness gate exists | Not complete | | Claude and Codex plugin publication | Contact/submission path with required artifacts and status | Publication readiness gate exists | Not complete |
| Articles, tweets, and announcements | X thread, LinkedIn copy, GitHub release copy, push checklist | Draft launch collateral exists under rc.1 release docs | Needs URL-backed refresh | | Articles, tweets, and announcements | X thread, LinkedIn copy, GitHub release copy, push checklist | Draft launch collateral exists under rc.1 release docs | Needs URL-backed refresh |
| AgentShield enterprise iteration | Policy gates, SARIF, packs, provenance, corpus, HTML reports | PRs #53, #55-#60 landed with test evidence | Needs next value decision | | AgentShield enterprise iteration | Policy gates, SARIF, packs, provenance, corpus, HTML reports | PRs #53, #55-#60 landed with test evidence | Needs next value decision |
| ECC Tools next-level app | Billing audit, PR checks, deep analyzer, sync backlog | PRs #26-#36 landed with test evidence | Needs native Linear API sync / deeper corpus coverage | | ECC Tools next-level app | Billing audit, PR checks, deep analyzer, sync backlog | PRs #26-#37 landed with test evidence | Needs native Linear API sync / broader evaluator corpus |
| GitGuardian/Dependabot/CodeRabbit-style checks | Non-blocking taxonomy and deterministic follow-up checks | ECC-Tools risk taxonomy check plus follow-up signals landed, including Skill Quality, Deep Analyzer Evidence, and RAG/Evaluator Evidence | Partially complete | | GitGuardian/Dependabot/CodeRabbit-style checks | Non-blocking taxonomy and deterministic follow-up checks | ECC-Tools risk taxonomy check plus follow-up signals landed, including Skill Quality, Deep Analyzer Evidence, Analyzer Corpus Evidence, and RAG/Evaluator Evidence | Partially complete |
| Harness-agnostic learning system | Audit, adapter matrix, observability, traces, promotion loop | Audit/adapters/observability gates exist | Needs evaluation/RAG prototype | | Harness-agnostic learning system | Audit, adapter matrix, observability, traces, promotion loop | Audit/adapters/observability gates exist | Needs evaluation/RAG prototype |
| Linear roadmap is detailed | Linear project status plus repo mirror | Repo mirror exists; issue creation is blocked by workspace limit | Needs recurring status updates | | Linear roadmap is detailed | Linear project status plus repo mirror | Repo mirror exists; issue creation is blocked by workspace limit | Needs recurring status updates |
| Flow separation and progress tracking | Flow lanes with owner artifacts and update cadence | This roadmap defines lanes below | Active | | Flow separation and progress tracking | Flow lanes with owner artifacts and update cadence | This roadmap defines lanes below | Active |
@ -153,7 +156,7 @@ back to the repo evidence and merge commits.
| Harness OS core | Audit, adapter matrix, observability docs, `ecc2/` | HUD/session-control acceptance spec | Weekly until GA | | Harness OS core | Audit, adapter matrix, observability docs, `ecc2/` | HUD/session-control acceptance spec | Weekly until GA |
| Evaluation and RAG | Reference-set validation, harness audit, traces | Read-only evaluator/RAG prototype design | Before deep analyzer expansion | | Evaluation and RAG | Reference-set validation, harness audit, traces | Read-only evaluator/RAG prototype design | Before deep analyzer expansion |
| AgentShield enterprise | AgentShield PR evidence and roadmap notes | PDF-export decision or next enterprise signal | After value decision | | AgentShield enterprise | AgentShield PR evidence and roadmap notes | PDF-export decision or next enterprise signal | After value decision |
| ECC Tools app | ECC-Tools PR evidence, billing audit, risk taxonomy | Native Linear sync or deeper analyzer corpus slice | Next implementation batch | | ECC Tools app | ECC-Tools PR evidence, billing audit, risk taxonomy | Native Linear sync or broader evaluator corpus slice | Next implementation batch |
| Linear progress | Linear project status updates and this mirror | Status update with queue/evidence/missing gates | Every significant merge batch | | Linear progress | Linear project status updates and this mirror | Status update with queue/evidence/missing gates | Every significant merge batch |
The project status update should always include: The project status update should always include:
@ -280,8 +283,8 @@ Acceptance:
failure modes. failure modes.
- Deep analyzer covers diff patterns, CI/CD workflows, dependency/security - Deep analyzer covers diff patterns, CI/CD workflows, dependency/security
surface, PR review behavior, failure history, harness config, skill quality, surface, PR review behavior, failure history, harness config, skill quality,
dedicated analyzer corpus evidence, RAG/evaluator comparison, and dedicated analyzer corpus evidence, co-located analyzer reference sets,
reference-set validation. RAG/evaluator comparison, and reference-set validation.
- PR check suite taxonomy includes Security Evidence, Harness Drift, Install - PR check suite taxonomy includes Security Evidence, Harness Drift, Install
Manifest Integrity, CI/CD Recommendation, Cost/Token Risk, Reference Set Manifest Integrity, CI/CD Recommendation, Cost/Token Risk, Reference Set
Validation, Deep Analyzer Evidence, RAG/Evaluator Evidence, Skill Quality, Validation, Deep Analyzer Evidence, RAG/Evaluator Evidence, Skill Quality,
@ -294,6 +297,10 @@ Acceptance:
- Deep-analyzer follow-ups flag repository, commit, architecture, pattern, and - Deep-analyzer follow-ups flag repository, commit, architecture, pattern, and
analysis-pipeline changes that lack analyzer corpus, snapshot, fixture, or analysis-pipeline changes that lack analyzer corpus, snapshot, fixture, or
benchmark evidence. benchmark evidence.
- Analyzer corpus evidence includes maintained fixtures and tests for current
architecture and commit analyzer outputs, plus co-located
`src/analyzers/{fixtures,goldens,reference-sets,benchmarks,evals}/` evidence
paths.
- RAG/evaluator follow-ups flag retrieval, embedding, ranking, and evaluator - RAG/evaluator follow-ups flag retrieval, embedding, ranking, and evaluator
changes that lack reference-set comparison, golden trace, benchmark, fixture, changes that lack reference-set comparison, golden trace, benchmark, fixture,
or eval-run evidence. or eval-run evidence.
@ -334,3 +341,5 @@ Acceptance:
executive report and corpus benchmark output. executive report and corpus benchmark output.
2. Add native Linear API sync for ECC Tools backlog items after workspace issue 2. Add native Linear API sync for ECC Tools backlog items after workspace issue
capacity clears. capacity clears.
3. Expand the analyzer/evaluator corpus with real stale-salvage and PR-review
cases as future cleanup batches produce safe maintainer-owned examples.