Skip to content

feat(runtime): ADR 0012 burst + preemption (staged on the #44/#50 design track) - #53

Closed
ThomasK33 wants to merge 9 commits into
mainfrom
feat/adr-0012-preempt-burst-draft
Closed

feat(runtime): ADR 0012 burst + preemption (staged on the #44/#50 design track)#53
ThomasK33 wants to merge 9 commits into
mainfrom
feat/adr-0012-preempt-burst-draft

Conversation

@ThomasK33

@ThomasK33 ThomasK33 commented Aug 27, 2026

Copy link
Copy Markdown
Member

Important

Draft — do not merge. Staged pending the deferred-dispatch admission-bound design (#44) and the fenced cross-instance coordination design (#50), per the maintainer's convergence decision on #38.

Carries the remaining ADR 0012 surface split out of #38: ConcurrencyBurst, the OnLockConflict force/steerability preemption hook, and the LockForcer State capability (ForceReleaseLock on memory/redis/postgres/NATS + conformance coverage), with all fixes from nine codex review rounds (pre-acquire ownership reservations, atomic pending-validation+preemption, per-member burst budgets, NATS revision-gated force with renewal retry, victim-await handoff, burst cause telemetry).

Why staged

Review kept producing legitimate P1 races in exactly these two areas — the latest being non-atomic remote-force validation vs. supersession, a stallable burst refresh call, and unfenced sync-mode holders. Root cause: key-only ForceReleaseLock cannot bind the force to the lease observed at conflict time (the State contract hides holder identity), and burst/preemption lifecycles need the admission-bound model to be settled first. Both get designed once on the #44/#50 track; the outstanding round-9/10 findings on #38 apply to this branch and should be folded into that design pass:

  1. Remote force validation must be atomic with supersession (or token-bound via the runtime: cross-instance coalescing for queue/debounce supersession (ADR 0012 wait/coalesce State extension) #50 fenced contract).
  2. The burst refresh loop must be stoppable while a member runs (bound the ExtendLock call).
  3. Sync-dispatch holders on other instances need fencing before a preemptor proceeds (or the mix must be rejected).

This branch is the full pre-split state of #38 (0892821) and will need a rebase over the reduced #38 once it merges.

Refs #7.

Generated with mux • Model: anthropic:claude-fable-5 • Thinking: xhigh

@ThomasK33
feat(runtime): expand concurrency strategies per ADR 0012
34a5d13 
Implement the remaining ADR 0012 surface: ConcurrencyDebounce,
ConcurrencyBurst, ConcurrencyConcurrent, a configurable LockScope
(thread default, channel opt-in), and handler preemption/steerability
via RuntimeOptions.OnLockConflict with force-release-lock coordination
through the new chat.LockForcer optional State capability
(ForceReleaseLock on memory/redis/postgres/nats states + conformance).

Drop stays the default; existing drop/queue behavior is unchanged and
the public API is additive.

Closes #7
@ThomasK33
fix(runtime): address codex review findings on ADR 0012 strategies
01a4da3 
- preempt: verify the waiter is still pending before destroying in-flight
  work; wait (ctx-bounded) for the local victim to finish before force
  releasing so drop/queue serialization holds locally; bound the force
  release by the Detached Work Context instead of WithoutCancel
- outcome telemetry: record OutcomePreempted by cancellation cause even
  when the handler returns nil, and close acquire-failure waits (queue/
  debounce/preempt/burst) as OutcomeError instead of ignored
- validation: reject DetachTimeout <= DebounceInterval under debounce
  and burst, which would abandon every event before dispatch
- lock scope: length-prefix channel key fields so the mapping is
  injective across adapter/tenant/channel combinations
- debounce: superseded waiters exit promptly on a displacement signal
  instead of parking through the full interval holding their payloads
@ThomasK33
fix(runtime): close round-2 codex races on preemption and lock waits
a88c3c0 
- preempt: re-validate pending ownership after waiting for the local
  victim, so a displaced preemptor cannot force-release a lease a newer
  waiter already holds; residual instruction-scale races stay fenced by
  lease refresh cancellation
- lock waits: re-check pending ownership after AcquireLock returns held,
  releasing the fresh lease when the waiter was displaced mid-acquire
  (debounce/queue newest-waiter-only guarantee)
- lock scope: namespace channel-scope fallback thread keys
  (thread-scope/<len>:<id>) so an opaque Thread ID can never collide
  with a synthesized channel key
- burst: document join-order (not arrival-order) batch semantics
@ThomasK33
fix(runtime): force-release only when no local victim exists
bee3256 
A local preemption victim releases its own lease after cancellation, so
the preemptor now simply waits for it and acquires normally - a lease
acquired by a fresh unrelated handler after the victim's release can no
longer be destroyed. Force release through LockForcer is reserved for
leases with no locally registered holder (remote instance or orphaned),
where lease-refresh cancellation remains the documented fence. The
preemption log/observation now carries forced=true/false.
@ThomasK33
fix(runtime): round-4 codex findings - burst budget, NATS force gate
022d1f8 
- burst: the batch dispatch now runs under a fresh DetachTimeout that
  starts when the collection window closes, so collection time can no
  longer consume accepted batch members' execution budget
- nats: gate ForceReleaseLock's delete by the observed revision so only
  the lease seen in-call is invalidated (parity with the atomic Redis
  DEL / Postgres DELETE), treating a mid-call handover as no-op
- docs: state explicitly that queue/debounce supersession and burst
  windows coalesce per runtime instance (cross-instance coalescing is
  the ADR 0012 wait/coalesce State extension, tracked in #50)
@ThomasK33
fix(runtime): round-5 codex findings - ownership reservation, NATS re…
b1a49c3 
…newal retry

- preempt: local lease ownership is now reserved the moment a lease is
  acquired (prelude and lock-wait acquisitions), not when the handler
  starts, so a preemptor arriving while the owner is still routing or
  deduping can never mistake the fresh local lease for a remote one and
  force-release it; a reservation preempted before the handler starts
  prevents the handler from running at all (span outcome preempted,
  started=false)
- nats: a revision mismatch during ForceReleaseLock re-reads the entry
  and retries while the token is unchanged, so an ordinary same-token
  ExtendLock renewal is no longer mistaken for a lease handover
@ThomasK33
fix(runtime): round-6 codex findings - pre-acquire reservations, memb…
544eda0 
…er budgets, universal lease-loss cancel

- preempt: ownership reservations are registered BEFORE AcquireLock (a
  lease can never exist locally without a visible reservation) and the
  registry is multi-entry per scope, so concurrent acquirers and the
  holder coexist; the preemptor cancels and awaits every reservation,
  treats a formerly-held one as a clean local handoff, and loops on
  non-holding acquirers before concluding the lease is remote
- runtime: every deferred lock holder is now cancelled (ErrPreempted)
  when its lease is lost, regardless of the local hook configuration,
  so a force release from another instance actually stops the victim
- burst: each accepted batch member runs under its own DetachTimeout
  budget; earlier members and collection time never consume it, and
  lease loss cancels the remaining members
@ThomasK33
fix(runtime): round-7 codex findings - atomic preemption validation, …
ebe1ab8 
…burst cause telemetry

- preempt: pending-slot validation and local cancellation are now one
  atomic operation under the pending registry lock, so a preemptor
  displaced by a newer registration can never cancel the active holder
  or poison the newer waiter's reservation on behalf of a dead event
- burst: a member stopped by lease loss is classified by its
  cancellation cause (OutcomePreempted + preemption log), matching
  runLockedTail, instead of leaking handled/error outcomes
@ThomasK33
fix(runtime): cancel handlers when the lease refresh loop errors out
0892821 
An ExtendLock error exits the refresh loop, after which the lease
expires at TTL while the handler (or remaining burst members) would
keep running unserialized. Treat a refresh error like a lost lease:
mark it lost (benign release) and cancel the handler with ErrPreempted
so no holder outlives a lease the runtime can no longer maintain.
This was referenced Aug 27, 2026
Closed
Closed
Merged
ThomasK33 added a commit that referenced this pull request Aug 27, 2026
@ThomasK33
docs(adr): decision-only re-scope per maintainer ruling — decisions/i…
aef222e 
…nvariants/non-goals; all mechanism prose cut; burst admission deferred to #53 revival
ThomasK33 added a commit that referenced this pull request Aug 27, 2026
@ThomasK33
docs(adr): ADR 0015 — deferred-dispatch admission bound; cross-instan…
984d094 
…ce coalescing rejected for now (#54)

* docs(adr): ADR 0015 — deferred-dispatch admission bound and fenced cross-instance coalescing (design for #44 + #50)

* docs(adr): honest preemption-overlap contract (cooperative cancellation) and explicit sync-dispatch serving-layer non-goal

* docs(adr): close coalescing protocol holes — global fence sequence, register-all-routed, deferred-only fencing, abandonment vs degradation

* docs(adr): commit-point degradation rule, admission-anchored register TTL, NATS uniform-TTL provision, reject preemption under concurrent

* docs(adr): bounded allocation-to-registration window with TTL arithmetic covering it

* docs(adr): concrete fence timing bounds, uniform-fleet caveat, LockForcer commit-point rule, bounded refresh calls, burst requires deferred

* docs(adr): full-park check horizon, fence as single ordering source, idempotent takeover reconciliation, shape-aware admission rejection

* docs(adr): bounded idempotent takeover reconciliation; fleet-uniform timing for TTL arithmetic across all backends

* docs(adr): scope-wide degradation ordering, pre-RPC window anchor, expired register-entry purge

* docs(adr): universal SHA-256 holder identity; reject burst + preemption hook

* docs(adr): uniform pre/post-mark degradation rule; concrete refresh RPC bound (one refresh interval)

* docs(adr): takeover + reconciliation execute in the detached tail under the DetachTimeout budget

* docs(adr): extend-as-reconciliation-probe with TTL horizon; ack-budget bounds on prelude fence calls

* docs(adr): conflict-time observation binding, hook in tail, extend-only probe, read-completion validity, allocation before all prelude State reads, spec-level scoping

* docs(adr): descope preemption to rejected shapes + binding requirements + deferred protocol design (non-converging surface, per the #38 precedent)

* docs(adr): re-scope per maintainer directive — admission bound kept, cross-instance coalescing rejected for now with formal-design reopening bar

* docs(adr): prelude-return slot release, ADR 0003 acceptance qualification, explicit ADR 0012 supersessions for burst gating

* docs(adr): slot release at tail-goroutine return, duplicate fast path at the cap, optional per-tenant sublimit

* docs(adr): drop unimplementable duplicate fast path (qualify ADR 0002, prohibit mark-as-probe), composite tenant key + empty-tenant bucket + validation, burst final-member slot through cleanup

* docs(adr): consistent slot-lifetime wording; burst lock sequencing per shipped debounce precedent with explicit ADR 0002 qualification

* docs(adr): batch FIFO seal-order dispatch, batch-derived member retention bound (ADR 0002 qualification), budgeted busy-response for ack-separated interactions

* docs(adr): burst window = DebounceInterval, bounded batch coordination with observable abandonment, lease-loss batch disposition

* docs(adr): cap-reaching member ownership, cooperative-bound residual, no-response-url fallback, no-undocumented-loss invariant

* docs(adr): fixed burst window anchor, OutcomeSkippedLeaseLoss closed-set value, member reference clearing before slot release

* docs(adr): shutdown drains open windows, handler-error continuation, detached busy-post context, index ADR 0015 in explanation.md

* docs(adr): ADR 0004 burst lock qualification, atomic shutdown admission close, tenant counter cleanup

* docs(adr): idle scope-coordinator removal; busy-post drain before adapter cleanup

* docs(adr): decision-only re-scope per maintainer ruling — decisions/invariants/non-goals; all mechanism prose cut; burst admission deferred to #53 revival
@ThomasK33

ThomasK33 commented Aug 27, 2026

Copy link
Copy Markdown
Member Author

Closing per ADR 0015 verdict (merged, #54).

The branch stays referenced here for the burst revival. Thanks for the staging discipline — the split gave the design track exactly the evidence it needed.

Generated with mux • Model: anthropic:claude-fable-5 • Thinking: xhigh

@ThomasK33 ThomasK33 closed this Aug 27, 2026
This was referenced Aug 27, 2026
Closed
Merged
ThomasK33 added a commit that referenced this pull request Aug 27, 2026
@ThomasK33
feat(runtime): revive ConcurrencyBurst per-instance under ADR 0015 in…
7e020e1 
…variants (#57)

Adds the burst Concurrency Strategy per ADR 0015's PR #53 revival verdict:
per-scope collection windows (BurstWindow, anchored at the first member),
optional MaxBurstBatch cap-sealing, FIFO dispatch of rolled batches under a
single Thread Lock hold, per-member DetachTimeout budgets, and full Admission
Bound integration (each parked member occupies a MaxDetached slot until its
terminal disposition).

Every burst-lifecycle finding from the ADR 0015 review history (PR #54) is
proven by a dedicated hardening test. Completes ADR 0012's accepted surface;
force/steerability stays reserved.

Closes #55
注册 for free to join this conversation on GitHub. Already have an account? 登录 to comment

Reviewers

No reviews

Assignees

No one assigned

标签

None yet

项目

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@ThomasK33