Structured observations¶

The environment render_mode selects the observation representation:

• render_mode=None (the default) returns tile-resolution categorical grids without constructing RGB frames during reset() or step(). render() returns None.
• render_mode="rgb_array" and render_mode="human" retain the Melting Pot RGB observation format. render() returns the global frame in "rgb_array" mode; human mode displays that frame in a pygame window.

state() remains an explicit, on-demand global RGB render in every mode. Real Matrix is nonspatial: its default observations contain PAYOFFS and LAST_ACTIONS, with RGB added only in visual modes.

Observation views¶

Every spatial substrate supports three observation_mode values, independently of render_mode. RGB and structured observations use the same view window.

Focal observations cover the map from any agent position and orientation. The agent's center cell is (L // 2, L // 2); RGB has shape (L * sprite_size, L * sprite_size, 3). Cells outside a bounded map are zero; wrapped maps can appear more than once in the window. The larger window costs more memory than a local or global view. For example, default Coins focal grids are (33, 33), and Daycare focal grids are (39, 39).

Global and focal views expand spatial coverage while preserving each viewer's role-dependent appearance and visibility rules. An inactive agent can still receive blank grids. Global structured observations add SELF to mark the viewer's avatar; focal views identify it by the center cell. Real Matrix supports only "local" and "global" payoff views.

from mp.commons_harvest import parallel_env

env = parallel_env("open", render_mode=None)
observations, infos = env.reset(seed=0)
print(observations["player_0"].keys())
print(env.object_observation_ids["APPLES"])
env.close()

For a whole-map view in the agent's frame:

from mp.coins.coins import parallel_env

env = parallel_env(render_mode=None, observation_mode="focal")
observations, infos = env.reset(seed=0)
print(observations["player_0"]["COINS"].shape)  # (33, 33)
env.close()

All spatial substrates provide TERRAIN, AVATARS, and AVATAR_ORIENTATIONS. Zero means empty, unseen, or out of bounds, except where a layer uses zero for a visible zero value such as pot progress. Avatar IDs are one-based player indices and orientations are north=1, east=2, south=3, west=4 in world coordinates, even when the grid is egocentric. Grids use uint8, or uint16 if a configured category count requires it. Existing nonvisual observations such as stamina, readiness, inventories, offers, voting, and hunger are retained.

Substrate grids¶

The table lists layers in addition to the three common grids and the SELF grid added in global mode.

Each environment exposes the exact, immutable category mapping through env.object_observation_ids. Categories represent only information visible in the corresponding RGB observation. For example, Daycare applies its role-dependent fruit appearance and Hidden Agenda reveals the impostor marker only to the impostor observing itself.

Visible state and readiness¶

• BEAMS encodes the beam kinds drawn by each substrate. Most are one-step cues; Daycare also marks held fruit, and Predator Prey retains the beam during a predator's post-kill freeze.
• AVATAR_STAMINA matches the visible stamina bar: level_0 through level_12 have IDs 1 through 13. Zero means no visible bar, including when stamina is above the displayed range. The agent's own STAMINA scalar remains a separate observation.
• Coins records absolute coin colors in both COINS and AVATAR_STATES. Factory Commons distinguishes free, held, and in-flight objects and the two dispenser kinds.
• Daycare distinguishes attached and picked fruit, keeps tree sites visible after picking, and adds HOLDING (0 nothing, 1 apple, 2 banana) alongside HUNGER. Fruit labels follow the viewer's role-dependent perception.
• Cooking distinguishes one and two tomatoes, shows cooking progress, and adds scalar READY_TO_INTERACT (1 ready, 0 cooling down).
• Paintball shows avatar health and control indicators, Territory distinguishes wet and dry claims, and Matrix exposes interaction-result colors and roles.

HOLDING and READY_TO_INTERACT are structured-mode additions. Hidden simulator timers, such as Cooperative Mining's partial-gold countdown, stay hidden.