[Feature] Add COMA multi-agent objective

[Iliamsou]

What

Adds a dedicated COMALoss objective for multi-agent reinforcement learning. The loss implements the COMA counterfactual baseline with a decentralised actor and a centralised Q-value network, including n-step Q-value targets and optional advantage normalisation.

Contents

• torchrl/objectives/multiagent/coma.py: COMALoss implementation and helpers to construct the joint observation, joint action without the current agent, and masked joint action inputs used by the centralised critic.
• torchrl/objectives/multiagent/__init__.py and torchrl/objectives/__init__.py: expose COMALoss.
• test/objectives/test_coma.py: tests for the counterfactual baseline, Q-value targets, n-step returns, input construction helpers, advantage normalisation, and diagnostics.

Tests

Ran:

python -m pytest test/objectives/test_coma.py

8 passed.

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/rl/4151

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 4 新建 Failures, 1 Unrelated Failure

As of commit d167d23 with merge base b0eab87 ():

NEW FAILURES - The following jobs have failed:

• Lint / lint-done (gh)
• Lint / python-source-and-configs / linux-job (gh)
  RuntimeError: Command docker exec -t 51516157762efc0e797725120fb5360c959575ce6b798bdce1d91c43bce49766 /exec failed with exit code 1
• SOTA Tests on Linux / tests (3.10, 13.0, 1) / linux-job (gh)
  RuntimeError: Command docker exec -t 5c10826a2e312aa5aeb3cf2d061bc2e30b18bc130ac918227e26db154865e6fa /exec failed with exit code 1
• SOTA Tests on Linux / tests (3.10, 13.0, 2) / linux-job (gh)
  RuntimeError: Command docker exec -t b4952d6f1e30d268e3846188549fa4e6e41bffe296838fb9c565d8adfee92248 /exec failed with exit code 1

BROKEN TRUNK - The following job failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• Libs Tests on Linux / unittests-gym (3.10, 12.8) / linux-job (gh) (trunk failure)
  test/libs/test_gym.py::TestGym::test_gym[True-True-1-HalfCheetah-v4]

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[meta-cla]

Hi @Iliamsou!

Thank you for your pull request and welcome to our community.

Action Required

In order to merge any pull request (code, docs, etc.), we require contributors to sign our Contributor License Agreement, and we don't seem to have one on file for you.

Process

In order for us to review and merge your suggested changes, please sign at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need to sign the corporate CLA.

Once the CLA is signed, our tooling will perform checks and validations. Afterwards, the pull request will be tagged with CLA signed. The tagging process may take up to 1 hour after signing. Please give it that time before contacting us about it.

If you have received this in error or have any questions, please contact us at cla@meta.com. Thanks!

[meta-cla]

Thank you for signing our Contributor License Agreement. We can now accept your code for this (and any) Meta Open Source project. Thanks!

[vmoens]

Thanks for adding a dedicated COMA objective. I found several issues that need to be addressed before this can merge:

1. [P1] Shift the actual time dimension. In compute_value_target, next_value[:, :-1] = value_target[:, 1:] only shifts time for exactly [B, T, ...] inputs. A standard collector over an unbatched environment returns [T, n_agents, 1], in which case this shifts the agent dimension. Multiple environment batch dimensions also shift the wrong axis. Please resolve the named/final TensorDict time dimension and add a [T] regression case.

2. [P1] Bootstrap rollout boundaries and truncations correctly. Zero-filling shifted values makes the last n_step transitions of every non-terminal fixed-length rollout use incomplete returns. Masking with done also suppresses a valid bootstrap at truncation; TorchRL return semantics distinguish this with terminated. The target critic needs to evaluate the transition next observation at boundaries rather than treating the end of the sampled batch as terminal.

3. [P1] Honor collector validity masks. Direct .mean() and F.mse_loss(...) reductions bypass LossModule._reduce_loss, so ("collector", "mask") and shifted_valid do not exclude padded transitions. COMA commonly consumes episodic sequences, so this can train both actor and critic on padding. Please keep losses elementwise and reduce through _reduce_loss(..., tensordict=tensordict).

4. [P2] Preserve flat NestedKey values. ("next",) + tuple(self.tensor_keys.reward) works for tuple keys but turns reward="reward" into ("next", "r", "e", "w", "a", "r", "d"). Please use the repository nested-key normalization pattern for both reward and termination keys.

The test volume is proportionate, but the current target tests only use synthetic [1, T] batches, which hides the time-axis failure. Please cover [T], a truncated boundary, automatic loss masking, and both flat and nested set_keys() values.

The public API integration also needs the repository-required documentation: a complete Sphinx-style class docstring with runnable example and paper reference, an entry in docs/source/reference/objectives_multiagent.rst, and the applicable tutorial/SOTA recipe. Since this is a new test file, please also add the executable if __name__ == "__main__": pytest.main(...) block.

[gtnv]

done=True, terminated=False still shifts a value across the reset. changing the next episode's first action moved the previous target from 0.5 to 2.0. can we invalidate this bootstrap at every done boundary?

[gtnv]

one valid actor value returns nan here. masked padding also changes the scale. can we use the effective loss mask with correction=0?