Skip to content

fix(evaluation): reject num_samples=0 in JudgeModelOptions at construction time - #6939

Open
gaurav-gandhi-2411 wants to merge 1 commit into
google:mainfrom
gaurav-gandhi-2411:fix/eval-metrics-num-samples-ge1
Open

fix(evaluation): reject num_samples=0 in JudgeModelOptions at construction time#6939
gaurav-gandhi-2411 wants to merge 1 commit into
google:mainfrom
gaurav-gandhi-2411:fix/eval-metrics-num-samples-ge1

Conversation

@gaurav-gandhi-2411

Copy link
Copy Markdown
Contributor

🔴 Required Information

Describe the Bug:
JudgeModelOptions.parallelism_limit has ge=1 but its sibling field
num_samples has no lower-bound constraint at all. ge=1 was added to
parallelism_limit by commit 0dbb88c ("feat: parallelize LLM-as-judge
evaluation using asyncio.gather()", 2026-08-19), but that commit did not touch
num_samples. Across this file's entire visible history, num_samples has
never carried a lower bound.

num_samples=0 is never an intentional value anywhere in this codebase — a
repo-wide grep of every num_samples usage in src/ and tests/ finds only
1, 3, or the default 5; nothing ever sets or derives 0. It is, however, a real
live bug surface: LlmAsJudge.evaluate_invocations silently drops any
invocation with zero samples from the result set entirely, with no error and
no NOT_EVALUATED marker. (The "judge model yields no response" exception path
is a separate, already-fixed case on main; this PR is narrower and
orthogonal, targeting only the num_samples=0 misconfiguration itself.)

Steps to Reproduce:
JudgeModelOptions(num_samples=0) # succeeds silently today

Expected Behavior:
Same as JudgeModelOptions(parallelism_limit=0) today — a
pydantic.ValidationError at construction time.

Observed Behavior:
Constructs successfully, then silently drops every invocation evaluated by it
from the aggregated result.

Changes

  • src/google/adk/evaluation/eval_metrics.py: add ge=1 to
    JudgeModelOptions.num_samples, matching the existing convention for
    count-like fields in this package — two other examples: run_config.py: max_workers and context_cache_config.py:cache_intervals, both ge=1.
  • tests/unittests/evaluation/test_eval_config.py: add
    test_judge_model_options_rejects_zero_num_samples, asserting
    JudgeModelOptions(num_samples=0) raises pydantic.ValidationError.

Note for reviewers: this makes a defensive if not samples: return NOT_EVALUATED branch in simulation/per_turn_user_simulator_quality_v1.py
(~lines 365-369) unreachable via the public constructor — that branch was
never exercised by an intentional num_samples=0, so removing it is optional
cleanup, not required here; left untouched to keep this diff minimal.

Testing

pytest tests/unittests/evaluation/test_eval_config.py -q   → 21 passed
pytest tests/unittests/evaluation/test_llm_as_judge.py -q  → 10 passed

No regressions in either file.

Risk & rollback

Additive validation only — narrows accepted input, does not change behavior
for any value ever actually used (1, 3, 5 across every existing caller/test).
Revert is a single-line removal of ge=1.

…ction time

parallelism_limit already has ge=1; num_samples never did, despite being the
same kind of count-like field in the same class. num_samples=0 is never an
intentional value anywhere in this codebase, and silently causes
LlmAsJudge.evaluate_invocations to drop the affected invocation from results
entirely with no error and no NOT_EVALUATED marker.
注册 for free to join this conversation on GitHub. Already have an account? 登录 to comment

标签

None yet

项目

None yet

Development

Successfully merging this pull request may close these issues.

2 participants