Skip to content

gh-153946: Add API to defer automatic garbage collection - #154188

Open
pablogsal wants to merge 6 commits into
python:mainfrom
pablogsal:agent/defer-automatic-gc
Open

gh-153946: Add API to defer automatic garbage collection#154188
pablogsal wants to merge 6 commits into
python:mainfrom
pablogsal:agent/defer-automatic-gc

Conversation

@pablogsal

@pablogsal pablogsal commented Jul 19, 2026

Copy link
Copy Markdown
Member

Copy link
Copy Markdown
Member Author

I ran this a few times on release builds pinned to one CPU, comparing aeedae8 with this branch. These timings are the median of the medians from three independent runs, with 15-20 samples and 10 iterations per sample.

Workload main This PR Change Automatic collections
Parse eight large stdlib files into AST objects 96.22 ms 78.20 ms 18.7% faster 8 per batch -> 0
marshal.loads(), 50,000 records 12.73 ms 10.49 ms 17.6% faster 1 per call -> 0

I also ran perf stat -r 5 on the same CPython workloads. The AST case went from 26.25B to 24.14B instructions, 8.1% fewer, and task clock dropped 18.5%. The marshal case went from 4.25B to 3.57B instructions, 15.9% fewer, and task clock dropped 17.8%.

I tried scheduling a collection directly from _PyGC_ResumeAutomaticCollection() as well. That made AST conversion about 3% slower than main, so this version keeps counting and waits for the next normal scheduling point. I think this is the right behavior here.

Comment on lines +2074 to +2080
// Close the race with a deferral that started before the world stopped.
if (state->reason == _Py_GC_REASON_HEAP &&
_Py_atomic_load_int(
&state->gcstate->automatic_collection_pause_count)) {
_PyEval_StartTheWorld(interp);
return false;
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need a similar check before this?

cpython/Python/gc.c

Lines 1486 to 1497 in fe0cc2b

/* update collection and allocation counters */
if (generation+1 < NUM_GENERATIONS) {
gcstate->generations[generation+1].count += 1;
}
for (i = 0; i <= generation; i++) {
gcstate->generations[i].count = 0;
}
/* merge younger generations with one we are currently collecting */
for (i = 0; i < generation; i++) {
gc_list_merge(GEN_HEAD(gcstate, i), GEN_HEAD(gcstate, generation));
}

This is a very shy question, so please treat it with a huge grain of salt. My hunch is that GC callbacks may still cause race, since they're just regular Python code and may release GIL:

cpython/Python/gc.c

Lines 1467 to 1472 in fe0cc2b

GC_STAT_ADD(generation, collections, 1);
struct gc_generation_stats stats = { 0 };
if (reason != _Py_GC_REASON_SHUTDOWN) {
invoke_gc_callback(tstate, "start", generation, &stats);
}

I interrogated Claude Opus to show me what is the actual place for invoking and returning from the callbacks, and that's how I arrived at 1486.

Comment thread Include/internal/pycore_interp_structs.h
@pablogsal
pablogsal force-pushed the agent/defer-automatic-gc branch from fe0cc2b to 920df6a 比较 August 25, 2026 23:52
@pablogsal
pablogsal marked this pull request as ready for review August 27, 2026 23:34
注册 for free to join this conversation on GitHub. Already have an account? 登录 to comment

项目

None yet

Development

Successfully merging this pull request may close these issues.

2 participants