Appendix: Thesis 4 Software Substrate Benchmarks
Version: 0.3 (2026-06-05) — adds the execution-deterministic principle (compile-once-run-many, model out of the execution loop, accuracy gate against silent failures, bounded runtime invocation), per Compiled AI: Deterministic Code Generation for LLM-Based Workflow Automation, arXiv:2604.05150. Prior: 0.2 (2026-06-05) — verifier-hardened principle per Let the Barbarians In: How AI Can Accelerate Systems Performance Research, arXiv:2512.14806; 0.1 (2026-04-24).
This appendix specifies benchmark families for Thesis 4, A Self-Modifying Software Substrate With Acceptance Gates. It is a benchmark design contract, not an implemented software-repair benchmark suite. Capability Status: specified/proposed. Evidence Status: Documented/Proposed.
The goal is to make software-substrate claims testable without implying that Consullo has a complete autonomous coding or deployment system. A software change earns stronger status only when it passes specified evidence classes: specification, generation or repair, compilation, tests, semantic checks, security, provenance, cost/benefit, deployment discipline, permission, and observation.
Benchmark reports produced under this appendix should populate benchmark_result records in appendix-evidence-ledger-schema.md; the report fields below define the benchmark-specific evidence_payload structure for those records.
Benchmark Principles
- Specification-first: every patch candidate should begin with a scope, non-goals, validation plan, and rollback expectation.
- Multi-gate: compilation is necessary but not sufficient.
- Reference-backed: validator evolution should be tested against held-out known-good, known-bad, and near-miss cases.
- Security-aware: privileged, validator-affecting, externally consequential, and alignment-infrastructure changes require stricter gates.
- Provenance-preserving: generated code, prompts, tools, model outputs, human edits, and test artifacts should be traceable.
- Non-deployment by default: benchmark success does not imply production deployment permission.
- Verifier-hardened: candidates should be confined to the target code and forbidden from editing the evaluator, workload, scoring code, or tests; scoring functions should be smooth and deterministic; and evaluation should use diverse held-out workloads — controls that prevent an evolutionary or LLM optimizer from reward-hacking the benchmark (Let the Barbarians In: How AI Can Accelerate Systems Performance Research, arXiv:2512.14806).
- Execution-deterministic: where the task permits, a generated artifact should run as static code with the model out of the execution loop, and the benchmark should check execution determinism (identical inputs produce identical outputs) rather than re-invoking a model per transaction. Critically, the validation suite must include an accuracy gate that compares outputs to golden expectations, because an artifact can pass the execution gate (100% run success) while producing wrong outputs — a silent failure that execution success alone would deploy; compile-time accuracy checking converts that silent runtime failure into a detectable pre-deployment one. Any irreducible runtime LLM call should be a bounded invocation under a fixed schema with fallback and human escalation, not an open-ended per-transaction prompt (Compiled AI: Deterministic Code Generation for LLM-Based Workflow Automation, arXiv:2604.05150).
Required Benchmark Report Fields
| Field | Meaning |
|---|---|
benchmark_id | Stable identifier and version. |
repair_domain | Compilation, dependency, JSON/schema, semantic behavior, security, deployment, validator, or provenance task class. |
specification_record | Scope, expected files, non-goals, validation plan, and rollback expectation. |
baseline_failure | Reproducible failure, regression, or missing capability. |
candidate_patch | Patch source, tool/model/method memory, and changed artifacts. |
validation_matrix | Compilation, tests, semantic checks, security, provenance, cost/benefit, and permission checks. |
reference_suite | Known-good, known-bad, adversarial, and historical false-accept/false-reject cases. |
ValidatorStrength_rule | Composite scoring rule or pending calibration note. |
deployment_state | Sandbox, simulated canary, canary, production-equivalent, or no deployment. |
observation_record | Post-change behavior, incidents, rollback, or supersession. |
ledger_links | Benchmark, provenance, improvement, incident, and trust records. |
status_result | What claim status, if any, the benchmark can strengthen. |
Suite A: Deterministic Local Repair
Purpose: test narrow repair machinery with reproducible expected outcomes.
Representative tasks:
- fix missing imports or package names
- repair dependency ordering
- correct malformed JSON or method signatures
- repair generated descriptions or agent cards
- insert missing required check-in methods
Suggested dimensions:
- failure reproduction
- patch minimality
- compilation success
- test pass rate
- provenance completeness
- cost and latency
Negative controls:
- patch that deletes failing tests
- patch that changes unrelated files
- patch whose provenance is missing or ambiguous
This suite supports component-level repair claims, not full self-modifying substrate validation.
Suite B: Semantic Near-Miss Detection
Purpose: test whether the substrate rejects patches that pass surface checks while weakening behavior.
Representative tasks:
- patch compiles but removes a validation check
- generated test asserts wrong behavior
- repair changes policy semantics while preserving API shape
- patch handles the happy path but breaks edge cases
- code change changes trust or permission behavior unintentionally
Suggested dimensions:
- semantic-failure detection
- edge-case coverage
- reviewer burden
- false-accept rate
- explanation quality
- reference-suite update
Negative controls:
- syntactically correct but semantically wrong patch
- test suite that passes because it encodes the bug
- validator prompt that rewards plausibility over behavior
Semantic validation is the main boundary between code generation and safe software modification.
Suite C: Validator Non-Regression
Purpose: test whether validators and generated-test pipelines improve without weakening gates.
Representative tasks:
- run validator before/after change on held-out reference suite
- measure known-good acceptance and known-bad rejection
- detect historical false accepts
- detect security failures and provenance gaps
- reject validator change that improves throughput by reducing scrutiny
Suggested dimensions:
- known-good acceptance
- known-bad rejection
- semantic-wrongness detection
- security-failure detection
- provenance-gap detection
- risk-severity calibration
Negative controls:
- validator that rejects nearly everything
- validator that accepts generated tests sharing the same model-family blind spot
- reference suite modified without preserved rationale
ValidatorStrength should reward discrimination, not permissiveness or blanket conservatism.
Suite D: Security And Privilege Boundary
Purpose: test whether software changes touching sensitive areas route through stricter controls.
Representative tasks:
- patch authentication or authorization logic
- modify A2A ingress policy
- alter evidence-ledger writes
- change validator or deployment scripts
- request broader tool privileges
Suggested dimensions:
- security-check coverage
- privilege-boundary preservation
- Thesis 5 routing correctness
- provenance and review completeness
- rollback readiness
- incident-prevention quality
Negative controls:
- change that silently broadens access
- patch that weakens logging or audit
- validator-affecting change treated as ordinary repair
Security-sensitive repair cannot inherit evidence from low-risk compilation repair.
Suite E: Staged Deployment And Rollback
Purpose: test whether accepted patches move through explicit exposure states and retain recovery paths.
Representative tasks:
- accept patch into sandbox only
- simulate canary behavior
- observe post-change regression
- narrow or rollback a candidate after observation
- preserve rollback record and postmortem evidence
Suggested dimensions:
- stage-entry correctness
- monitoring coverage
- rollback or supersession success
- incident classification
- evidence-ledger completeness
- cost of recovery
Negative controls:
- patch marked accepted without stage
- rollback path missing or untested
- observation omitted after deployment-equivalent exposure
Deployment discipline is part of the acceptance gate, not an operations afterthought.
Minimal Demonstration Package
The first Thesis 4 demonstration should use one bounded low-risk repository-local task. It should include one accepted repair, one rejected semantic near miss, one validator-reference-suite case, and one staged exposure record.
Required contents:
- specification record
- reproducible baseline failure
- candidate patch and provenance
- validation matrix
- reference-suite result
- cost/benefit table
- Thesis 1 acceptance linkage
- Thesis 5 permission linkage where applicable
- staged exposure or no-deployment rationale
- rollback or supersession path
- evidence-ledger records
Non-Claims
This appendix does not claim that Consullo has implemented a complete autonomous software engineer, a secure deployment pipeline, full semantic validation, validator non-regression, or production self-modification. It specifies what benchmark evidence would be needed before Thesis 4 claims can strengthen beyond component-level implementation and specified/proposed architecture.