doc(simulator): add simulator RFC - #596
Conversation
|
|
7d098ac to
0c230d1
比较
0c230d1 to
335162c
比较
|
|
||
| The simulator models SubmitQueue and Runway, which together form a closed pre-merge loop. Runway is stateless — no request store, no database, idempotency derived from the VCS — so the harness substitutes storage for one domain only. | ||
|
|
||
| Stovepipe is out of scope by design rather than by convenience. It is a post-merge trunk-health poller with no coupling to SubmitQueue in either direction, and its only outputs are hooks, whose contract states that hook outcomes never write pipeline state. Nothing Stovepipe learns can reach the queue. Escaped conflicts, where changes pass individually and break together, surface inside SubmitQueue's own build stage, so measuring them needs no post-merge modeling. |
There was a problem hiding this comment.
Stovepipe is out of scope by design rather than by convenience.
i think we also have need eventually at some point to say if some relaxation algorithm make main broken more than the others? would we want stovepipe to be in scope
|
|
||
| Stovepipe is out of scope by design rather than by convenience. It is a post-merge trunk-health poller with no coupling to SubmitQueue in either direction, and its only outputs are hooks, whose contract states that hook outcomes never write pipeline state. Nothing Stovepipe learns can reach the queue. Escaped conflicts, where changes pass individually and break together, surface inside SubmitQueue's own build stage, so measuring them needs no post-merge modeling. | ||
|
|
||
| One hazard does live in the gap between the two. Runway performs transforming merges: rebase and squash-rebase rewrite commits, so the tree that lands is not always the tree that was validated. A break introduced by the transform itself is invisible to pre-merge validation, and therefore to any simulation of it. That is a limit on what the safety metric can claim, not a component to model. |
There was a problem hiding this comment.
Runway performs transforming merges: rebase and squash-rebase rewrite commits, so the tree that lands is not always the tree that was validated. A break introduced by the transform itself is invisible to pre-merge validation, and therefore to any simulation of it. That is a limit on what the safety metric can claim, not a component to model.
can we simply the wording here? i think i get the gist for needs to be direct
|
|
||
| One hazard does live in the gap between the two. Runway performs transforming merges: rebase and squash-rebase rewrite commits, so the tree that lands is not always the tree that was validated. A break introduced by the transform itself is invisible to pre-merge validation, and therefore to any simulation of it. That is a limit on what the safety metric can claim, not a component to model. | ||
|
|
||
| Concurrency-correctness work is also out of scope — invariant assertions, fault injection, and controlled interleaving exercise the delivery contract rather than orchestrator policy, need no corpus and no build oracle, and belong with end-to-end testing. |
There was a problem hiding this comment.
invariant assertions, fault injection, and controlled interleaving exercise the delivery contract rather than orchestrator policy, need no corpus and no build oracle, and belong with end-to-end testing.
explain? we may want to be able to inject faults and test things, maybe a separate thing to think about
|
|
||
| A variant may differ in as many seams as the question needs. Pinning all but one is what makes a difference *attributable*: comparing two Generators under the same Allocator and Scorer is the only way to read the ranking-quality and starvation [diagnostics](#diagnostics) apart. The seam list belongs to the extension contract rather than to this document, and grows with it. | ||
|
|
||
| A seam must be substitutable on its own. Nothing can supply a `Generator`'s `Iterator` without supplying the Generator, so the Generator is the unit; construction parameters such as the path projection are knobs rather than seams. |
There was a problem hiding this comment.
Nothing can supply a Generator’s Iterator without supplying the Generator, so the Generator is the unit; construction parameters such as the path projection are knobs rather than seams.
what if i want to try between 2 scorers or regression models under the hood? should that be a full generator wire-up or just scorer/model
|
|
||
| **Layer 2 — controller replay.** Finalization, refutation, output validation, and the merge gate are controller-owned, so Layer 1 cannot reach them and Layer 3 is too heavy to run on every change. Layer 2 feeds a recorded queue snapshot to a controller and diffs the actions and writes it emits. A speculate run is a pure function of a single read, so a snapshot is a complete input; other snapshot-determined controllers can use the same mechanism. | ||
|
|
||
| **Layer 3 — system simulation.** Some questions are about consequences rather than verdicts: does a looser `conflict.Analyzer` reduce land time once batching and speculation react to the changed dependency graph? That needs the real pipeline running forward — gateway, orchestrator, and Runway in one process, in-memory storage and message queues in place of MySQL, and fakes only at the boundaries reaching source control, CI, and git. |
There was a problem hiding this comment.
fakes only at the boundaries reaching source control, CI, and git.
do we need to consider this as well? #209
|
|
||
| **Layer 2 — controller replay.** Finalization, refutation, output validation, and the merge gate are controller-owned, so Layer 1 cannot reach them and Layer 3 is too heavy to run on every change. Layer 2 feeds a recorded queue snapshot to a controller and diffs the actions and writes it emits. A speculate run is a pure function of a single read, so a snapshot is a complete input; other snapshot-determined controllers can use the same mechanism. | ||
|
|
||
| **Layer 3 — system simulation.** Some questions are about consequences rather than verdicts: does a looser `conflict.Analyzer` reduce land time once batching and speculation react to the changed dependency graph? That needs the real pipeline running forward — gateway, orchestrator, and Runway in one process, in-memory storage and message queues in place of MySQL, and fakes only at the boundaries reaching source control, CI, and git. |
There was a problem hiding this comment.
in-memory storage and message queues in place of MySQL
would this emulate fakes? so say if we wanted to test a failure of Ack or Nack? can we simulate with it
| | Corpus store | identities, recorded inputs and outputs, provenance | all | | ||
| | Recorder | mines history, records at the seam, or shadows in production | all | | ||
| | Replayer | drives one extension from corpus identities | 1 | | ||
| | Snapshot replayer | drives one controller from a recorded queue snapshot | 2 | |
There was a problem hiding this comment.
Snapshot replayer
unsure of this vs just replaer? how they are different
| A replay produces one of three outcomes, and collapsing them into pass and fail destroys the instrument. | ||
|
|
||
| - **Harness fault.** The run did not complete: a fake failed, a resolver timed out, a candidate panicked. Says nothing about either implementation. | ||
| - **Corpus fault.** The run completed, but the baseline no longer reproduces its own recorded output. Every candidate number derived from that entry is void. |
There was a problem hiding this comment.
Harness fault. The run did not complete: a fake failed, a resolver timed out, a candidate panicked. Says nothing about either implementation.
Corpus fault. The run completed, but the baseline no longer reproduces its own recorded output. Every candidate number derived from that entry is void.
we need simplified terminologies? i am unsure of what harness means here?
|
|
||
| An extension reproducible from a pinned identity can be replayed against any candidate, including on inputs history never saw. One that is not can only be compared against the single answer history recorded. That decides how its corpus is collected (see [Corpus collection](#corpus-collection)). | ||
|
|
||
| | Extension | Reproducible? | Why | |
There was a problem hiding this comment.
Reproducible?
this is what currently implemented but not what might be eventually, for example conflicts would be coming from Tango, scorer will be ML based?
add simulator RFC