Add deterministic AI reporter qualification gates

[Sean Larkin (TheLarkInn)]

Summary

• add a deterministic, network-free corpus for bootstrap/version, configuration, input, dependency-tool, operation, cache, network/auth, plugin, cancellation, and internal failures, plus successful controls
• compare AI, detailed plaintext, legacy, and full-detail file output with machine-readable blocking results and actionable per-case failures
• preserve stable diagnostic codes, root-cause ordering, classified context, remediation, payload-only stdout, warning suppression, and correlated absolute full-log references
• expose a qualification decision helper without wiring environment-based automatic reporter selection

Part of #5981.

Stack

• Parent/base: Add direct Rush reporter demo path #5997 (copilot/reporter-r5b-demo-reporters)
• Privacy prerequisite: Harden reporter telemetry privacy projection #5990
• Sibling exclusion: does not include the R7 branch

This PR does not copy or modify the telemetry projection from #5990. It applies the same producer-identity privacy boundary to presented reporter output and artifacts.

Corpus and gates

Gate	Blocking threshold	Result
Coverage	at least 10 failure cases and 2 successful controls	10 failures, 2 controls
Actionability	100% stable code, category, context, and remediation on failures	100%
Absolute size	at most 64 KiB AI output per case	976 bytes maximum
Context size	aggregate AI output at most 50% of legacy and plaintext	19.55% of legacy; 20.08% of plaintext
Determinism	identical normalized output across 3 runs	100%
Privacy	no classified secret or private producer/plugin identity leakage	100%
Full log	absolute, existing, owner-only where supported, complete, and failure-correlated	100%
Machine contracts	payload-only NDJSON plus warning suppression/detail behavior	100%

Corpus cases:

bootstrap-unsupported-node, configuration-invalid-json, input-unknown-project, dependency-package-manager, operation-build-failure, cache-restore-failure, network-auth-unauthorized, plugin-api-incompatible, logical-cancellation, internal-unexpected-error, success-no-warning, and success-warning-only.

External-service-shaped failures use stable canonical event fixtures. Returned results contain only case metadata, byte counts, booleans, and normalized hashes; absolute paths, private identities, and classified values are not stored.

Reproduce

cd libraries/reporter
rushx build
node scripts/runAiReporterQualification.js

The script prints the versioned machine-readable result and exits nonzero when any blocking gate fails.

Selection boundary

getQualifiedAiReporterDecision() validates built-in COPILOT_CLI detection and configured agentEnvironmentVariables, rejects absent or failed qualification results, and keeps RUSH_REPORTER=legacy authoritative.

It is intentionally not consumed by the Rush frontend in this PR. No environment variable automatically selects the AI reporter; the pre-major live path remains explicit --reporter or repository-experiment opt-in, and no-opt-in behavior remains legacy.

Validation

• rush build --to @microsoft/rush
• rushx _phase:test in libraries/reporter (314 tests)
• deterministic corpus, including 3 repeated normalized runs
• rushx _phase:test in libraries/rush-lib
• rushx _phase:test in apps/rush
• direct AI invalid-project failure: exit 1, payload-only NDJSON, actionable parser diagnostic
• no-opt-in COPILOT_CLI=1 legacy behavior and RUSH_REPORTER=legacy override
• rush check
• rush change --verify --no-fetch

Non-goals

• enabling environment-based automatic reporter selection
• changing Rush 6 or daemon-aligned defaults
• Heft child-reporter work
• bootstrap handoff integration
• telemetry projection changes

[Sean Larkin (TheLarkInn)]

Independent /pr-agentmerge review completed; fixes are in ac3fd75.

Findings fixed

• Full-log qualification previously inspected only the file reporter artifact. It now validates the AI-emitted absolute path, exact artifact correlation, format/completeness, existence, owner-only mode, required lifecycle content, external evidence, and failure/session/diagnostic correlation.
• Actionability now checks exact public context values and required privacy markers instead of key presence.
• The uniform 4 KiB synthetic blob was replaced by scenario-specific deterministic evidence. Aggregate ratios are supplemented by compact-case and per-case baseline gates so one oversized fixture cannot hide a regression.
• Local-sensitive producer identity is retained in the owner-only full log, while secret producer identity/values remain redacted. Secret diagnostics preserve honest aggregate counts and truncation without exposing code/category/detail or suppressing fallback errors.
• Future eligibility now requires both a passing qualification result and an explicit accepted privacy prerequisite. COPILOT_CLI/configured detection alone stays ineligible; RUSH_REPORTER=legacy remains authoritative. No telemetry implementation or Rush 6/default selection behavior was added.
• Added an adversarial reporter regression test: dropping artifactAvailable drives full-log to 0% with all affected case names in the CI diagnostic.

Corpus evidence

• 12 cases: 10 failures plus success and warning-only controls covering bootstrap/environment, configuration, input, dependency tool, operation/build, cache, network/auth, plugin, cancellation, internal, and success paths.
• Three independent standalone corpus executions were byte-identical across every normalized SHA-256.
• Gates: actionability/privacy/full-log/stdout/warnings/determinism all 100%; max AI case 976 B; compact max 969/2048 B; worst comparable per-case 68.47% of legacy and 72.00% of plaintext; aggregate 43.77% and 46.51%.
• Machine results were scanned for temp paths, fixture secrets, secret producer identity, and private plugin identity; none were present.

Validation

• Reporter build/lint/API Extractor and 319 tests passed.
• rush test --to @microsoft/rush passed the reporter, rush-lib, apps/rush, and dependency chain.
• rush check and stacked rush change --verify --target-branch origin/copilot/reporter-r5b-demo-reporters --no-fetch passed.

State / blockers

• PR remains based on Add direct Rush reporter demo path #5997, mergeable/clean, and auto-merge is disabled.
• No review threads existed to resolve.
• GitHub currently reports only the CLA check and no 操作 run for this stacked head.
• Harden reporter telemetry privacy projection #5990 and Add direct Rush reporter demo path #5997 remain open; [rush reporter][R8] Qualify AI reporter auto-selection #5981 must remain open until both prerequisites merge, the qualification passes on main, and any approved activation remains gated by both prerequisites.

[Sean Larkin (TheLarkInn)]

Combined deep review of current head ac3fd75530.

1. HIGH local-sensitive messageEmitted text can be copied verbatim into AI stdout fallback output. Only public text may enter AI output. Use a redacted fallback and protected log reference otherwise.
2. LOW the oversized-record marker relabels non-public envelopes as public while retaining source and scope. Preserve privacy or remove those fields.

[Sean Larkin (TheLarkInn)]

Fixed both combined review findings in c46a4ea8036805e70de1143f25d206b4c653e42c.

• AI fallback text is emitted only for public envelopes. Local-sensitive and secret fallback errors are count-only, marked truncated, and retain the protected full-log reference.
• Non-public JSON oversized markers keep their original privacy and remove source and scope. Public markers retain scope and child-session correlation.
• The corpus now covers mixed-privacy fallback messages, oversized local-sensitive payload/source/scope sentinels, secret producer identities, safe machine output/results, and useful owner-only logs.

The three R8 commits are rebased directly onto 97f807ed6549387b6aab2eccec197123196b725d. All 13 cases passed across three identical runs. Reporter, rush-lib, and apps/rush tests passed, along with API/lint, Rush check, and stacked change verification.

Head is pending review. Auto-merge remains disabled.

[Sean Larkin (TheLarkInn)]

Four-review gate on c46a4ea803 confirms the AI fallback and oversized-marker fixes. One HIGH privacy issue remains: normal-sized secret JSON events still retain scope and parentOperationId, which can expose private project, phase, or operation names on machine stdout.

Required fix: use an allowlisted secret-envelope projection that removes contextual metadata not guaranteed opaque, while preserving protocol and ordering fields. Extend the corpus with sentinel scope values.

Dismissible low follow-on: decide whether local file:// and json:// sidecars should retain local-sensitive message text or match stdout redaction.

[Sean Larkin (TheLarkInn)]

Fixed the HIGH secret scope metadata finding in 6dac2ffdcdb761232829e9160ec13031d3fb65a6.

• Shared redaction now projects secret envelopes through a strict allowlist. Parent, command, operation, project, phase, scope, and producer component metadata are removed.
• JSON normal and originally oversized secret messages and diagnostics use the same redacted contract.
• AI ignores secret command and operation lifecycle context while preserving safe diagnostic, fallback count, and command completion behavior.
• Corpus probes now use sentinel command, operation, project, phase, parent session, parent operation, source, message, and diagnostic values. None reach machine stdout or qualification results.

Reporter build, lint, API Extractor, and 336 tests passed. All 13 corpus cases passed across three identical runs. Reporter, rush-lib, apps/rush, Rush check, and stacked change verification passed.

Local-sensitive and full-log behavior are unchanged. The low file-sidecar policy follow-on remains separate. Head is pending review and auto-merge remains disabled.

[Sean Larkin (TheLarkInn)]

Final combined review of 6dac2ffdcd found no critical, high, or medium actionable issue. Secret JSON and AI records now remove scope, parent, command, operation, project, phase, and producer component metadata while preserving safe protocol and count fields. Qualification gates remain deterministic and automatic selection stays disabled.

Dismissible low follow-ons:

• Decide whether owner-only file grouping should suppress secret operation headings and raw secret output beyond the documented raw-child limitation.
• Decide whether local file sidecars retain local-sensitive message text or match stdout redaction.

This head is low-only.