fix(compiler): fix stack overflow when checking assignability of recursive types

[JoshLove-msft]

isTypeAssignableToInternal created a brand new relation cache for every nested call instead of forwarding the one it was given, so the "in progress" entry seeded by areModelsRelated only survived a single level. Two mutually recursive models therefore recursed forever. Unions were never seeded at all, so any union reaching itself did the same.

This reproduces on main today through a regular template constraint, so it can be pasted directly into the playground:

model SourceA {
  b: SourceB;
}
model SourceB {
  a: SourceA;
}
model TargetA {
  b: TargetB;
}
model TargetB {
  a: TargetA;
}

model Constrained<T extends TargetA> {}
alias Result = Constrained<SourceA>;

The compiler crashes with RangeError: Maximum call stack size exceeded.

What changed

• isTypeAssignableToInternal now forwards the relationCache it was given instead of allocating a fresh one.
• The cache stores [Related, errors] instead of just Related. Forwarding the cache alone was not safe: the old code returned [cached, []] and areModelsRelated turns a result with no errors back into Related.true, so a cached false would flip to true.
• The source-union branch of isTypeAssignableToWorker and isAssignableToUnion now seed the cache before walking variants, the same way areModelsRelated already did.

Semantics of a cyclic union

A purely cyclic union such as union Loop { self: Loop } has no way to produce a value, so it describes the empty set and is vacuously assignable to anything, exactly like never. That is why the source side is seeded with Related.maybe.

The target side is the dual: being assignable to a union only requires one variant to match, so a cycle brings no new information and is seeded with Related.false.

Validation

• tsc -p tsconfig.build.json --noEmit clean
• 4,121 compiler tests pass, including a playground-style regression test verified to fail with the exact stack overflow on the pre-fix checker
• @typespec/openapi3 2,578 tests pass, plus http, json-schema, versioning, rest, events, sse, streams, and xml
• pnpm lint clean

--generated by Copilot

[pkg-pr-new]

Open in StackBlitz

npm i https://pkg.pr.new/@typespec/compiler@11779

commit: 0c4eb4f

[Copilot]

Pull request overview

This PR fixes a compiler stack overflow in assignability checks involving recursive types by making the relation cache effective across nested assignability calls and by correctly seeding union relations to terminate cycles.

Changes:

• Forward a single relation cache through nested isTypeAssignableToInternal calls and cache both relation results and associated errors.
• Seed relation cache entries for source unions (and for assignability-to-union checks) before walking variants to ensure cyclic unions terminate.
• Add regression tests covering self-recursive and mutually recursive models/unions, plus mismatch cases deep in cycles.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.

File	Description
packages/compiler/src/core/type-relation-checker.ts	Reworks relation caching to include errors and ensures recursion termination for cyclic unions/models.
packages/compiler/test/checker/relation.test.ts	Adds regression tests for recursive models/unions and cyclic assignability scenarios.
.chronus/changes/fix-relation-checker-recursion-2026-8-26.md	Adds a compiler changelog entry documenting the stack overflow fix and affected cases.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[github-actions]

All changed packages have been documented.

• ✅ @typespec/compiler

Show changes

@typespec/compiler - fix ✏️

Fix a stack overflow when checking assignability of mutually recursive types,> ,> Checking whether a type was assignable to another one could recurse forever and crash the compiler with RangeError: Maximum call stack size exceeded. Two cases were affected:,> ,> - mutually recursive models, such as model A { b: B } / model B { a: A },> - any union reaching itself, such as union Foo { self: Foo },> ,> The relation cache is now shared for the whole check instead of being recreated at every level, and unions seed it before walking their variants, so a cycle coming back to the same pair of types resolves instead of recursing.

[azure-sdk-automation]

You can try these changes here

🛝 Playground	🌐 Website	🛝 VSCode Extension

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

packages/compiler/src/core/type-relation-checker.ts:1120

• isAssignableToUnion creates the same unassignable diagnostic twice (once for the seeded cache entry and again on the final return). This does extra work on the common failure path and can be avoided by reusing a single errors array for both the seed and the return value.

    relationCache.set(
      [source, target],
      [Related.false, [createUnassignableDiagnostic(source, target, diagnosticTarget)]],
    );