Skip to content

Algorithm Notes

This page is a practical reading of the Melting Pot evaluation reports for this native PettingZoo port. It is not a leaderboard claim. The goal is to identify training and evaluation ideas that are likely to be useful for src/mp environments, and to flag approaches that can produce high scores while missing the cooperative behavior we actually care about.

Source Context

Useful references:

Evaluation Lesson

Melting Pot evaluates a multi-agent population learning algorithm rather than one isolated policy. The important generalization test is social: trained focal agents are placed with unfamiliar background populations in scenarios that may differ from the exact training setup.

For meltingpot-pz, this means a good training benchmark should eventually include:

  • native mp.* substrates for training;
  • a native scenario layer with focal and background populations;
  • multiple seeds and held-out partner mixes;
  • metrics for focal return, background return, total welfare, inequality, and task-specific events;
  • evaluation on unshaped environment rewards, even if shaped rewards were used during training.

Plain self-play score on a single substrate is useful for smoke testing, but it is not enough to measure cooperative generalization.

Useful Algorithms

PPO or A2C as baseline learners. These remain good first baselines because they are easy to run on PettingZoo-style parallel environments and are debuggable. They are likely enough for Coins, Real Matrix, small cooking layouts, and early regression tests. They are less likely to solve the larger pixel-control maps without substantial tuning.

IMPALA-style distributed actor-learner training. The contest appendix and many Melting Pot baselines point toward scalable actor-learner methods for vision-heavy substrates. This is useful when many actors can generate varied social encounters, especially for Clean Up, Commons Harvest, Paintball, Predator Prey, Territory, and Fruit Market.

PopArt or value normalization. Rewards differ wildly across substrates: Collaborative Cooking soup rewards, Territory stochastic territory income, Predator Prey acorn rewards, and matrix payoffs all live on different scales. PopArt-style value normalization is worth keeping in the toolbox for cross-substrate agents and curricula.

Policy sharing with role conditioning. Many substrates have symmetric agents, but several have roles: Daycare child/parent, Hidden Agenda crewmate and impostor, Predator Prey predator/prey, Paintball teams, Fruit Market farmers, and Allelopathic preferences. Shared weights with explicit role, team, or preference inputs can improve sample efficiency without pretending that all agents are identical.

Population-based training and partner diversity. The Melting Pot evaluation protocol rewards robustness to new partners. Training should include mixtures: self-play partners, scripted bots, older checkpoints, noisy versions of the policy, prosocial partners, selfish partners, and specialists. This is more important than squeezing one more point from a homogeneous self-play pool.

Prosocial reward mixing. The technical report and evaluation notebook include prosocial variants, and contest teams also used average-population reward signals. A practical wrapper can train on:

train_reward_i = own_reward_i + alpha * mean_reward_all

or replace mean_reward_all with team reward, Nash welfare, Rawlsian minimum, or substrate-specific public-good terms. Keep this as a training objective, not an evaluation metric. Sweep alpha; pure collective reward can erase useful role incentives in mixed-motive games.

Targeted reward shaping for sparse public goods. Clean Up is the obvious example: apple reward is sparse until agents learn to clean the river. Shaping dirty-tile cleaning during training can bootstrap the public good, as long as the final evaluation returns to native rewards. Similar temporary shaping can help Territory exploration, Hidden Agenda gem delivery, Chemistry reaction chains, and Collaborative Cooking intermediate steps.

Curriculum and staged training. Some contest approaches used staged training: teach exploration or a subskill first, then add other agents and conflict. This is useful for Territory, Paintball, Factory Commons, Boat Race, and any layout where agents must first discover an affordance before social learning has a signal.

Centralized critics for training only. A centralized value function or object-state critic can stabilize learning while actors still receive only their local observations. The contest report notes that full-observation value networks were not always worth the speed hit, so this should be optional and measured rather than assumed.

Structured observations where available. Pixel policies are parity-correct, but object observations can dramatically speed up algorithm development. For this repo, object observations should be treated as a configurable research knob: useful for learning and diagnostics, but RGB observations should remain available for upstream-compatible evaluation.

What Is Less Useful

Pure single-substrate self-play. It often overfits to a small set of partners and conventions. It is fine for unit tests and early learning curves, but it does not exercise Melting Pot's core social-generalization claim.

Hard-coded map policies as headline agents. Rule policies are valuable as debugging oracles and background bots. They are much less useful as evidence of general cooperative intelligence. The contest literature explicitly warns that high focal return can come from brittle or exploitative policies.

Optimizing focal return alone. A policy can improve focal-agent score while lowering background-agent welfare. Report background return and joint welfare alongside focal return whenever possible.

Pure prosocial reward everywhere. Shared reward is natural for cooking, but it can be wrong for deception, market exchange, predator-prey, voting, or mixed-motive commons. Prosocial objectives should be parameterized and compared against individual-reward baselines.

Large pixel networks before simple baselines work. Before running expensive visual RL, validate the environment with scripted policies, random rollouts, object observations if present, and short-horizon curricula. Many failures are mechanics or reward-shaping bugs, not representation bottlenecks.

LLM-only control from pixels. Language agents may be interesting for high-level planning or commentary, but current Melting Pot-style tasks demand fast low-level spatial control. Use LLMs as planners, curriculum generators, or policy inspectors only after a competent controller exists.

State of Public Results

As of May 2026, there is no single accepted "SOTA Melting Pot agent" that solves the whole suite in a way that clearly generalizes across all social demands. The strongest public signals are:

  • Melting Pot 2.0 baselines: useful reference points, not saturated solutions.
  • NeurIPS 2023 Melting Pot Contest: 8 of 23 final-phase teams beat the best reported 2.0 baseline on the full held-out scenario set, but approaches varied widely, including hard-coded policies and RL systems with IMPALA/PPO/PopArt and reward shaping.
  • Later analysis of contest submissions argues that higher aggregate score is not always higher prosocial ability; some high-scoring submissions appear strongest where explicit prosocial demands are absent.

For this repo, a credible "good agent" result should therefore report more than one number:

  • focal return;
  • background return;
  • total welfare;
  • inequality or minimum-agent return;
  • per-substrate and per-scenario breakdowns;
  • robustness across partner populations;
  • ablations for reward shaping, prosocial mixing, and object observations.
  1. Add native evaluation helpers that mirror the Melting Pot population/scenario protocol but operate on mp.* environments.
  2. Add reward-wrapper experiments for individual, collective, mixed, Nash, and Rawlsian objectives.
  3. Add simple scripted background populations for commons, cooking, matrix, paintball, and predator-prey scenarios.
  4. Add reference training recipes for PPO and IMPALA-style learners with optional PopArt and role conditioning.
  5. Treat any high score without partner-generalization and welfare metrics as a debugging milestone, not as evidence of cooperative intelligence.