Simulator API Documentation

Entry

Date: 2024-11-11 05:20:17 LastEditors: muzhancun muzhancun@stu.pku.edu.cn LastEditTime: 2025-05-26 22:10:06 FilePath: /MineStudio/minestudio/simulator/entry.py

class minestudio.simulator.entry.CameraConfig(camera_binsize: int = 2, camera_maxval: int = 10, camera_mu: float = 10.0, camera_quantization_scheme: str = 'mu_law')

Configuration for camera quantization and binning settings.

Parameters:

• camera_binsize – The size of each bin for camera quantization, default is 2.

• camera_maxval – The maximum value for camera quantization, default is 10.

• camera_mu – The mu parameter for mu-law quantization, default is 10.0.

• camera_quantization_scheme – The quantization scheme to use, either “mu_law” or “linear”, default is “mu_law”.

property action_transformer_kwargs

Dictionary of camera settings used by an action transformer.

property n_camera_bins

The bin number of the setting.

Returns:

The number of camera bins.

class minestudio.simulator.entry.MinecraftSim(*args: Any, **kwargs: Any)

MineStudio Minecraft Simulator.

Parameters:

• action_type – The type of the action space, can be ‘env’ or ‘agent’.

• obs_size – The resolution of the observation, default is (224, 224).

• render_size – The original resolution of the game, default is (640, 360).

• seed – The seed of the minecraft world, default is 0.

• inventory – The initial inventory of the agent, default is an empty dict.

• preferred_spawn_biome – The preferred spawn biome when calling reset, default is None.

• num_empty_frames – The number of empty frames to skip when calling reset, default is 20.

• callbacks – A list of callbacks to be called before and after each basic calling.

• camera_config – The configuration for camera quantization and binning settings.

Keyword Arguments:

kwargs – Additional keyword arguments.

property action_space: gymnasium.spaces.Dict

The action space of the environment.

agent_action_to_env_action(action: Dict[str, Any])

Converts an agent action to an environment action.

Parameters:

action – The agent action.

Returns:

The environment action.

close() → None

Performs any necessary cleanup.

Returns:

The close status from the underlying environment.

env_action_to_agent_action(action: Dict[str, Any])

Converts an environment action to an agent action.

Parameters:

action – The environment action.

Returns:

The agent action.

noop_action() → Dict[str, Any]

Returns a no-op action for the current action type.

Returns:

A no-op action.

property observation_space: gymnasium.spaces.Dict

The observation space of the environment.

render() → None

Renders the environment.

Returns:

The rendered image.

reset() → Tuple[ndarray, Dict]

Resets the environment to an initial state and returns the initial observation and info.

Returns:

A tuple containing the initial observation and info dictionary.

step(action: Dict[str, Any]) → Tuple[ndarray, float, bool, bool, Dict[str, Any]]

Runs one timestep of the environment’s dynamics.

Parameters:

action – The action to take.

Returns:

A tuple containing the observation, reward, terminated flag, truncated flag, and info dictionary.

minestudio.simulator.entry.check_engine(skip_confirmation=False)

Checks if the simulator engine exists and downloads it if not.

Parameters:

skip_confirmation – If True, skips the confirmation prompt before downloading.

minestudio.simulator.entry.download_engine()

Downloads the simulator engine from Hugging Face Hub and extracts it.

Callbacks

BaseCallback

Date: 2025-01-06 17:32:04 LastEditors: muzhancun muzhancun@stu.pku.edu.cn LastEditTime: 2025-06-15 17:02:18 FilePath: /MineStudio/minestudio/simulator/callbacks/callback.py

class minestudio.simulator.callbacks.callback.Compose(callbacks: list, options: int = -1)

A callback that composes multiple callbacks into a single callback. It allows for applying a sequence of callbacks or a random subset of them.

after_close(sim)

Called after the simulator is closed. Calls the after_close method of all activated callbacks.

Parameters:

sim – The simulator instance.

after_render(sim, image)

Called after the simulator renders an image. Calls the after_render method of all activated callbacks. Note: This method currently calls before_render on the callbacks, which might be an error. It should likely call after_render.

Parameters:

• sim – The simulator instance.

• image – The rendered image.

Returns:

The (potentially modified) image, after processing by active callbacks.

after_reset(sim, obs, info)

Called after the simulator resets the environment. Calls the after_reset method of all activated callbacks.

Parameters:

• sim – The simulator instance.

• obs – The initial observation after reset.

• info – The initial info dictionary after reset.

Returns:

The (potentially modified) observation and info, after processing by active callbacks.

Return type:

Tuple[Any, Dict]

after_step(sim, obs, reward, terminated, truncated, info)

Called after the simulator takes a step. Calls the after_step method of all activated callbacks.

Parameters:

• sim – The simulator instance.

• obs – The observation after the step.

• reward – The reward received after the step.

• terminated – A boolean flag indicating if the episode has terminated.

• truncated – A boolean flag indicating if the episode has been truncated.

• info – The info dictionary after the step.

Returns:

The (potentially modified) observation, reward, terminated flag, truncated flag, and info, after processing by active callbacks.

Return type:

Tuple[Any, float, bool, bool, Dict]

before_close(sim)

Called before the simulator is closed. Calls the before_close method of all activated callbacks.

Parameters:

sim – The simulator instance.

before_render(sim, image)

Called before the simulator renders an image. Calls the before_render method of all activated callbacks.

Parameters:

• sim – The simulator instance.

• image – The image to be rendered.

Returns:

The (potentially modified) image, after processing by active callbacks.

before_reset(sim, reset_flag: bool) → bool

Called before the simulator resets the environment. Activates a subset of callbacks based on the options attribute and calls their before_reset methods.

Parameters:

• sim – The simulator instance.

• reset_flag (bool) – A boolean flag indicating whether a reset is currently planned.

Returns:

A boolean flag indicating whether the environment should be reset, after processing by active callbacks.

Return type:

bool

before_step(sim, action)

Called before the simulator takes a step. Calls the before_step method of all activated callbacks.

Parameters:

• sim – The simulator instance.

• action – The action to be executed.

Returns:

The (potentially modified) action, after processing by active callbacks.

class minestudio.simulator.callbacks.callback.MinecraftCallback

Base class for creating callbacks that can be used to customize the behavior of the Minecraft simulator. Callbacks can be used to modify actions, observations, rewards, and other aspects of the simulation.

after_close(sim)

Called after the simulator is closed. This method can be used to perform final cleanup operations.

Parameters:

sim (any) – The simulator instance.

after_render(sim, image)

Called after the simulator renders an image. This method can be used to modify the image after it is rendered.

Parameters:

• sim – The simulator instance.

• image – The rendered image.

Returns:

The (potentially modified) image.

after_reset(sim, obs, info)

Called after the simulator resets the environment. This method can be used to modify the initial observation and info.

Parameters:

• sim (any) – The simulator instance.

• obs (any) – The initial observation after reset.

• info (Dict) – The initial info dictionary after reset.

Returns:

The (potentially modified) observation and info.

Return type:

Tuple[Any, Dict]

after_step(sim, obs, reward, terminated, truncated, info)

Called after the simulator takes a step. This method can be used to modify the observation, reward, done flags, and info.

Parameters:

• sim (any) – The simulator instance.

• obs (any) – The observation after the step.

• reward (float) – The reward received after the step.

• terminated (bool) – A boolean flag indicating if the episode has terminated.

• truncated (bool) – A boolean flag indicating if the episode has been truncated.

• info (Dict) – The info dictionary after the step.

Returns:

The (potentially modified) observation, reward, terminated flag, truncated flag, and info.

Return type:

Tuple[Any, float, bool, bool, Dict]

before_close(sim)

Called before the simulator is closed. This method can be used to perform any cleanup tasks.

Parameters:

sim (any) – The simulator instance.

before_render(sim, image)

Called before the simulator renders an image. This method can be used to modify the image before it is rendered.

Parameters:

• sim – The simulator instance.

• image – The image to be rendered.

Returns:

The (potentially modified) image.

before_reset(sim, reset_flag: bool) → bool

Called before the simulator resets the environment. This method can be used to determine if the environment needs to be reset.

Parameters:

• sim (any) – The simulator instance.

• reset_flag (bool) – A boolean flag indicating whether a reset is currently planned.

Returns:

A boolean flag indicating whether the environment should be reset.

Return type:

bool

before_step(sim, action)

Called before the simulator takes a step. This method can be used to modify the action before it is executed.

Parameters:

• sim (any) – The simulator instance.

• action (any) – The action to be executed.

Returns:

The (potentially modified) action.

Return type:

any

create_from_conf()

Creates a callback instance from a YAML file or a dictionary. This method is intended to be overridden by subclasses.

Parameters:

yaml_file (Union[str, Dict]) – The path to the YAML file or a dictionary containing the configuration.

Returns:

A callback instance, or None if not implemented.

Return type:

Optional[MinecraftCallback]

load_data_from_conf() → Dict

Loads data from a YAML file or a dictionary.

Parameters:

source (Union[str, Dict]) – The path to the YAML file or a dictionary containing the data.

Raises:

AssertionError – if the file specified by source does not exist.

Returns:

A dictionary containing the loaded data.

Return type:

Dict

CommandsCallback

Date: 2024-11-11 19:31:53 LastEditors: caishaofei-mus1 1744260356@qq.com LastEditTime: 2025-01-16 18:09:12 FilePath: /ROCKET-2/var/nfs-shared/shaofei/nfs-workspace/MineStudio/minestudio/simulator/callbacks/commands.py

class minestudio.simulator.callbacks.commands.CommandsCallback(commands: List[str])

Executes a list of Minecraft commands at specific lifecycle events.

This callback is typically used to run setup commands after an environment reset.

after_reset(sim, obs: Dict, info: Dict) → Tuple[Dict, Dict]

Executes the configured commands after the environment resets.

Each command is run sequentially, and the observation and info dictionaries are updated with the results from each command execution.

Parameters:

• sim – The simulator instance.

• obs – The current observation dictionary.

• info – The current info dictionary.

Returns:

Updated observation and info dictionaries.

Return type:

Tuple[Dict, Dict]

create_from_conf()

Creates a CommandsCallback instance from a configuration.

The configuration can be a path to a YAML file or a dictionary. It should contain a list of command strings under the key ‘custom_init_commands’ or ‘commands’.

Parameters:

source (Union[str, Dict]) – Configuration source (file path or dict).

Returns:

A CommandsCallback instance or None if no commands are specified.

Return type:

Optional[CommandsCallback]

DemonstrationCallback

Date: 2025-01-07 05:58:26 LastEditors: caishaofei-mus1 1744260356@qq.com LastEditTime: 2025-01-15 20:40:34 FilePath: /ROCKET-2/var/nfs-shared/shaofei/nfs-workspace/MineStudio/minestudio/simulator/callbacks/demonstration.py

class minestudio.simulator.callbacks.demonstration.DemonstrationCallback(task)

Provides demonstration data, primarily for GROOT.

Manages access to task-specific reference videos, including downloading them if absent.

Parameters:

task (str) – Name of the task for demonstration data.

after_reset(sim, obs, info)

Adds the reference video path to the observation dictionary after a reset.

This method ensures obs[‘ref_video_path’] is set with the path to the selected demonstration video.

Parameters:

• sim – The simulation environment.

• obs – The observation dictionary.

• info – Additional information dictionary.

Returns:

The modified obs and info.

Return type:

tuple[dict, dict]

after_step(sim, obs, reward, terminated, truncated, info)

Adds the reference video path to the observation dictionary after each step.

This method ensures obs[‘ref_video_path’] is set with the path to the selected demonstration video.

Parameters:

• sim – The simulation environment.

• obs – The observation dictionary.

• reward – The reward from the current step.

• terminated – Whether the episode has terminated.

• truncated – Whether the episode has been truncated.

• info – Additional information dictionary.

Returns:

The modified obs, reward, terminated, truncated, and info.

Return type:

tuple[dict, float, bool, bool, dict]

create_from_conf()

Creates a DemonstrationCallback from a configuration.

Loads data from the source (file path or dict). Initializes DemonstrationCallback if ‘reference_video’ is present.

Parameters:

source (any) – Configuration source.

Returns:

DemonstrationCallback instance or None.

Return type:

Optional[DemonstrationCallback]

minestudio.simulator.callbacks.demonstration.download_reference_videos()

Downloads reference videos from Hugging Face.

Retrieves the Minecraft reference videos dataset (CraftJarvis/MinecraftReferenceVideos) and saves them to “reference_videos” in the MineStudio root directory.

FastResetCallback

Date: 2024-11-11 16:15:32 LastEditors: muzhancun muzhancun@stu.pku.edu.cn LastEditTime: 2025-06-12 19:47:39 FilePath: /MineStudio/minestudio/simulator/callbacks/fast_reset.py

class minestudio.simulator.callbacks.fast_reset.FastResetCallback(biomes, random_tp_range, start_time=0, start_weather='clear')

Implements a fast reset mechanism for the Minecraft simulator.

This callback speeds up the reset process by executing a series of commands (e.g., /kill, /time set, /weather, /tp) instead of fully reinitializing the environment, if the environment has already been reset once.

Parameters:

• biomes (list[str]) – A list of biomes to randomly teleport to.

• random_tp_range (int) – The range for random teleportation coordinates (x, z).

• start_time (int, optional) – The in-game time to set at reset, defaults to 0.

• start_weather (str, optional) – The weather to set at reset, defaults to ‘clear’.

before_reset(sim, reset_flag)

Performs a fast reset if the simulator has been reset before.

If sim.already_reset is False (first reset), it allows the standard reset. Otherwise, it executes a sequence of commands to quickly reset the state: kills entities, sets time and weather, and teleports the player to a random biome and location.

Parameters:

• sim – The simulator instance.

• reset_flag – The current reset flag status.

Returns:

False if a fast reset was performed, otherwise reset_flag.

Return type:

bool

create_from_conf()

Creates a FastReset from a configuration.

Loads data from the source (file path or dict).

Parameters:

source (Dict) – Configuration source.

Returns:

FastResetCallback instance or None if no valid configuration is found.

Return type:

Optional[FastResetCallback]

HardResetCallback

Date: 2024-11-11 16:15:32 LastEditors: caishaofei-mus1 1744260356@qq.com LastEditTime: 2025-01-16 23:45:32 FilePath: /MineStudio/minestudio/simulator/callbacks/hard_reset.py

class minestudio.simulator.callbacks.hard_reset.HardResetCallback(spawn_positions: List)

Performs a hard reset of the Minecraft environment.

This callback forces a full environment reset by setting a specific seed and teleporting the player to a predefined spawn position. It is used when a complete and predictable reset is required.

Parameters:

spawn_positions (List[dict]) – A list of dictionaries, each specifying a “seed” (int) and a “position” ([x, z, y] list) for spawning.

after_reset(sim, obs, info)

Teleports the player and allows the environment to settle after reset.

After the environment resets, this method teleports the player to the selected x, z, y coordinates and then executes a number of no-op actions to allow the game world to stabilize.

Parameters:

• sim – The simulator instance.

• obs – The initial observation after reset.

• info – The initial info dictionary after reset.

Returns:

The modified observation and info.

Return type:

tuple[dict, dict]

before_reset(sim, reset_flag)

Selects a spawn position and sets the environment seed before reset.

Randomly chooses one of the provided spawn_positions, sets the environment’s seed to the chosen seed, and forces a reset.

Parameters:

• sim – The simulator instance.

• reset_flag – The current reset flag status.

Returns:

True, to indicate that a reset should occur.

Return type:

bool

create_from_conf()

Creates a HardResetCallback instance from a configuration source.

Loads data from the given configuration (file path or dict) and initializes a HardResetCallback if ‘spawn_positions’ is present.

Parameters:

source (any) – The configuration source.

Returns:

A HardResetCallback instance or None.

Return type:

Optional[HardResetCallback]

InitInventoryCallback

Date: 2025-01-05 22:26:22 LastEditors: caishaofei-mus1 1744260356@qq.com LastEditTime: 2025-01-16 23:43:49 FilePath: /MineStudio/minestudio/simulator/callbacks/init_inventory.py

class minestudio.simulator.callbacks.init_inventory.InitInventoryCallback(init_inventory: List[Dict], distraction_level: List[int] | str = [0])

Initializes the player’s inventory at the start of an episode.

This callback allows for precise control over the items the player starts with, including specifying items for exact slots, random slots, and adding distraction items. Quantities can be exact or defined by conditions.

Parameters:

• init_inventory (List[Dict]) – A list of dictionaries, each defining an item to initialize. Each dictionary can specify “slot”, “type”, and “quantity”. Example: [{“slot”: 0, “type”: “oak_planks”, “quantity”: 64}]

• distraction_level (Union[List[int], str], optional) – Defines the number of random distraction items. Can be a string key (e.g., “easy”, “hard”) or a list of numbers. Defaults to [0] (no distraction items).

after_reset(sim, obs: Dict, info: Dict) → Tuple[Dict, Dict]

Sets up the player’s inventory after the environment resets.

This method processes the init_inventory configuration: 1. Places items in specified slots. 2. Assigns items with “random” slots to available empty slots. 3. Adds distraction items based on distraction_level. 4. Uses Minecraft commands (/replaceitem) to modify the inventory. 5. Performs no-op actions and checks to ensure inventory setup is complete.

Parameters:

• sim – The simulator instance.

• obs – The current observation dictionary.

• info – The current info dictionary.

Returns:

The modified observation and info dictionaries.

Return type:

Tuple[Dict, Dict]

create_from_conf()

Creates an InitInventoryCallback instance from a configuration source.

Loads data from the given configuration (file path or dict) and initializes an InitInventoryCallback if ‘init_inventory’ is present.

Parameters:

source (Union[str, Dict]) – The configuration source.

Returns:

An InitInventoryCallback instance or None.

Return type:

Optional[InitInventoryCallback]

JudgeResetCallback

Date: 2025-06-12 19:46:03 LastEditors: muzhancun muzhancun@stu.pku.edu.cn LastEditTime: 2025-06-12 19:47:07 FilePath: /MineStudio/minestudio/simulator/callbacks/judgereset.py

class minestudio.simulator.callbacks.judgereset.JudgeResetCallback(time_limit: int = 600)

Resets the environment if a time limit is reached or episode terminates.

This callback monitors the number of steps taken in an episode. If the episode terminates naturally or if the step count exceeds time_limit, it forces a reset for the next step.

Parameters:

time_limit (int, optional) – The maximum number of steps per episode before forcing a reset. Defaults to 600.

after_reset(sim, obs: Dict, info: Dict) → Tuple[Dict, Dict]

Resets the internal step counter after an environment reset.

Parameters:

• sim – The simulator instance.

• obs – The initial observation after reset.

• info – The initial info dictionary after reset.

Returns:

The passed obs and info.

Return type:

Tuple[Dict, Dict]

after_step(sim, obs: Dict, reward: float, terminated: bool, truncated: bool, info: Dict) → Tuple[Dict, float, bool, bool, Dict]

Checks for termination or time limit and flags for reset if needed.

Increments the step counter. If terminated is true or self.time_step exceeds self.time_limit, it sets terminated to True to signal a reset and resets self.time_step.

Parameters:

• sim – The simulator instance.

• obs – The observation after the step.

• reward – The reward received.

• terminated – Whether the episode has terminated.

• truncated – Whether the episode has been truncated.

• info – The info dictionary.

Returns:

The (potentially modified) obs, reward, terminated, truncated, and info.

Return type:

Tuple[Dict, float, bool, bool, Dict]

create_from_conf() → JudgeResetCallback | None

Creates a JudgeResetCallback from a configuration.

Parameters:

source (Dict) – Configuration source.

Returns:

JudgeResetCallback instance or None if no valid configuration is found.

Return type:

Optional[JudgeResetCallback]

MaskActionsCallback

Date: 2024-11-11 17:37:06 LastEditors: muzhancun muzhancun@stu.pku.edu.cn LastEditTime: 2025-05-26 21:20:51 FilePath: /MineStudio/minestudio/simulator/callbacks/mask_actions.py

class minestudio.simulator.callbacks.mask_actions.MaskActionsCallback(**action_settings)

Overrides specific action components with fixed values before each step.

This callback can be used to constrain the agent’s actions by forcing certain parts of the action dictionary to predefined values.

Parameters:

action_settings – Keyword arguments where keys are action names (str) and values are the fixed values for those actions. Example: MaskActionsCallback(attack=0)

before_step(sim, action: dict) → dict

Modifies the action dictionary before the simulator takes a step.

Iterates through the action_settings provided during initialization and updates the corresponding keys in the action dictionary with the fixed values.

Parameters:

• sim – The simulator instance.

• action (dict) – The action dictionary to be modified.

Returns:

The modified action dictionary.

Return type:

dict

PlayCallback

Date: 2024-11-14 20:10:54 LastEditors: muzhancun muzhancun@stu.pku.edu.cn LastEditTime: 2024-11-20 01:06:35 FilePath: /MineStudio/minestudio/simulator/callbacks/play.py

class minestudio.simulator.callbacks.play.PlayCallback(agent_generator: Callable | None = None, extra_draw_call: List[Callable] | None = None)

Enables interactive play and/or agent-driven gameplay in a GUI window.

This callback provides a graphical interface for human players to interact with the Minecraft environment. It can also run a pre-trained agent and allow switching control between human and agent.

Key functionalities: - Renders the game view in a separate window. - Captures keyboard and mouse input for human control. - Can load and run a specified agent model. - Allows switching between human and agent control (default: ‘L’ key). - Displays game information (FPS, player position, current mode) in the GUI. - Handles custom key bindings for actions like mouse capture and closing.

Parameters:

• agent_generator (Callable, optional) – A callable that returns an agent instance. If None, only human play is enabled. Defaults to None.

• extra_draw_call (Optional[List[Callable]], optional) – A list of additional callable functions to be executed during the GUI drawing phase. Defaults to None.

after_reset(sim, obs: Dict, info: Dict) → Tuple[Dict, Dict]

Handles tasks after the environment resets.

Resets termination flag, agent state, updates GUI, and resets timestep.

Parameters:

• sim – The simulator instance.

• obs – The initial observation.

• info – The initial info dictionary.

Returns:

The passed obs and info.

Return type:

Tuple[Dict, Dict]

after_step(sim, obs: Dict, reward: float, terminated: bool, truncated: bool, info: Dict) → Tuple[Dict, float, bool, bool, Dict]

Handles tasks after the environment takes a step.

Updates GUI with new observation and info, calculates FPS, processes key releases (like switching control or mouse capture), and handles termination.

Parameters:

• sim – The simulator instance.

• obs – The observation after the step.

• reward – The reward received.

• terminated – Whether the episode has terminated.

• truncated – Whether the episode has been truncated.

• info – The info dictionary.

Returns:

The (potentially modified) obs, reward, terminated, truncated, and info.

Return type:

Tuple[Dict, float, bool, bool, Dict]

before_close(sim)

Closes the GUI window before the simulator closes.

Parameters:

sim – The simulator instance.

before_reset(sim, reset_flag: bool) → bool

Resets the GUI before the environment resets.

Parameters:

• sim – The simulator instance.

• reset_flag – The current reset flag status.

Returns:

The passed reset_flag.

Return type:

bool

before_step(sim, action: Any) → Dict

Determines and processes the action before the environment steps.

Handles input from human (keyboard/mouse via GUI) or agent based on the current control switch (self.switch). Also processes chat messages.

The executed action will be added to the info dict as “taken_action”.

Parameters:

• sim – The simulator instance.

• action (Any) – The proposed action (can be None, a string like ‘human’, or an action dict).

Returns:

The action dictionary to be executed by the environment.

Return type:

Dict

process_keys(sim, released_keys: set) → set

Processes special key releases for GUI and simulation control.

Handles: - ‘C’: Toggle mouse capture (exclusive mouse mode). - Ctrl+’C’: Close the window and terminate the simulation. - ‘ESCAPE’: Enter/exit command mode (currently exits by clearing keys).

Parameters:

• sim – The simulator instance.

• released_keys (set) – A set of keys that were released in this frame.

Returns:

The set of released_keys after processing (potentially modified).

Return type:

set

reset_agent()

Resets the agent’s internal state (e.g., memory).

Called when switching to agent control or at the start of a new episode.

PointCallback

Date: 2024-11-18 20:37:50 LastEditors: caishaofei caishaofei@stu.pku.edu.cn LastEditTime: 2024-11-24 08:23:45 FilePath: /MineStudio/minestudio/simulator/callbacks/point.py

class minestudio.simulator.callbacks.point.PlaySegmentCallback(sam_path: str, sam_choice: str = 'base')

Integrates Segment Anything Model (SAM) for interactive object segmentation.

This callback allows a human player to provide positive and negative point prompts on the game’s POV to segment objects using a SAM2 model. It then tracks the segmented object across subsequent frames.

Note: This callback should typically be placed before the PlayCallback in the callback list to ensure its GUI interactions are handled correctly.

Key Features: - Loads a specified SAM2 model checkpoint. - Provides a GUI for adding positive/negative point prompts. - Generates an initial segmentation based on prompts. - Tracks the segmented object in subsequent frames. - Adds the segmentation mask to obs[‘segment’][‘obj_mask’].

Activation: - Press ‘S’ to start/stop the segmentation process.

GUI Controls (during segmentation): - Left Mouse Click: Add a positive point prompt. - Right Mouse Click: Add a negative point prompt. - ‘C’: Clear all current points. - ‘Enter’: Start tracking the current segmentation. - ‘ESCAPE’: Exit segmentation mode.

Parameters:

• sam_path (str) – Path to the directory containing SAM2 model checkpoints and configs.

• sam_choice (str, optional) – Which SAM2 model to load (e.g., ‘base’, ‘large’, ‘small’, ‘tiny’). Defaults to ‘base’.

after_reset(sim, obs: Dict, info: Dict) → Tuple[Dict, Dict]

Clears segmentation state and adds a GUI message after environment reset.

Parameters:

• sim – The simulator instance.

• obs – The initial observation.

• info – The initial info dictionary.

Returns:

The passed obs and info.

Return type:

Tuple[Dict, Dict]

after_step(sim, obs: Dict, reward: float, terminated: bool, truncated: bool, info: Dict) → Tuple[Dict, float, bool, bool, Dict]

Manages the segmentation lifecycle based on user input (‘S’ key) and GUI interaction.

Handles: - Starting segmentation GUI when ‘S’ is pressed and not already tracking. - Stopping tracking when ‘S’ is pressed while tracking. - Updating the segmentation mask in obs if tracking is active.

Parameters:

• sim – The simulator instance.

• obs – The observation after the step.

• reward – The reward received.

• terminated – Whether the episode has terminated.

• truncated – Whether the episode has been truncated.

• info – The info dictionary.

Returns:

The (potentially modified) obs, reward, terminated, truncated, and info.

Return type:

Tuple[Dict, float, bool, bool, Dict]

before_step(sim, action: Any) → Any

Prevents actions if segmentation GUI is active but not yet tracking.

If the ‘S’ key was pressed to start segmentation but tracking hasn’t begun (i.e., user is still providing prompts), this returns a no-op action to pause game progression.

Parameters:

• sim – The simulator instance.

• action (Any) – The proposed action.

Returns:

A no-op action if segmenting GUI is active, else the original action.

Return type:

Any

class minestudio.simulator.callbacks.point.PointCallback

Allows the player to select a point on the screen using the mouse.

When activated (default: by pressing ‘P’), this callback opens a GUI window displaying the current game view. The player can click on this window to select a 2D point. The selected point’s coordinates are stored in the info[‘point’] dictionary.

after_reset(sim, obs: Dict, info: Dict) → Tuple[Dict, Dict]

Adds a message to the simulator to inform the user about the activation key.

Parameters:

• sim – The simulator instance.

• obs – The initial observation.

• info – The initial info dictionary.

Returns:

The passed obs and info.

Return type:

Tuple[Dict, Dict]

after_step(sim, obs: Dict, reward: float, terminated: bool, truncated: bool, info: Dict) → Tuple[Dict, float, bool, bool, Dict]

Handles the point selection process if activated.

If the ‘P’ key is pressed (i.e., info.get(‘P’, False) is True): 1. Opens a new GUI window. 2. Enters a loop to capture mouse clicks for point selection. 3. Updates info[‘point’] with the clicked coordinates. 4. Closes the GUI when ‘ESCAPE’ is pressed.

Parameters:

• sim – The simulator instance.

• obs – The observation after the step.

• reward – The reward received.

• terminated – Whether the episode has terminated.

• truncated – Whether the episode has been truncated.

• info – The info dictionary.

Returns:

The (potentially modified) obs, reward, terminated, truncated, and info.

Return type:

Tuple[Dict, float, bool, bool, Dict]

PrevActionCallback

Date: 2024-11-11 19:31:53 LastEditors: muzhancun muzhancun@stu.pku.edu.cn LastEditTime: 2025-06-12 19:48:23 FilePath: /MineStudio/minestudio/simulator/callbacks/prev_action.py

class minestudio.simulator.callbacks.prev_action.PrevActionCallback

A callback that stores the previous action and adds it to the observation.

This callback is useful for tasks where the agent needs to know its previous action to make a decision.

after_reset(sim, obs, info)

Adds a default previous action to the observation after a reset.

Parameters:

• sim – The Minecraft simulator.

• obs – The observation from the simulator.

• info – Additional information from the simulator.

Returns:

The modified observation and info.

after_step(sim, obs, reward, terminated, truncated, info)

Adds the previous action to the observation.

Parameters:

• sim – The Minecraft simulator.

• obs – The observation from the simulator.

• reward – The reward from the simulator.

• terminated – Whether the episode has terminated.

• truncated – Whether the episode has been truncated.

• info – Additional information from the simulator.

Returns:

The modified observation, reward, terminated, truncated, and info.

before_step(sim, action)

Stores the action before it is executed.

Parameters:

• sim – The Minecraft simulator.

• action – The action to be executed.

Returns:

The action.

create_from_conf()

Creates a PrevActionCallback from a configuration.

Loads data from the source (file path or dict).

Parameters:

source (Dict) – Configuration source.

Returns:

PrevActionCallback or None if no valid configuration is found.

Return type:

Optional[PrevActionCallback]

RecordCallback

Date: 2024-11-11 16:40:57 LastEditors: muzhancun muzhancun@stu.pku.edu.cn LastEditTime: 2024-11-20 00:30:51 FilePath: /MineStudio/minestudio/simulator/callbacks/record.py

class minestudio.simulator.callbacks.record.RecordCallback(record_path: str, fps: int = 20, frame_type: Literal['pov', 'obs'] = 'pov', recording: bool = True, show_actions=False, record_actions=False, record_infos=False, record_origin_observation=False, **kwargs)

A callback for recording episodes of Minecraft gameplay.

This callback can record video of the gameplay, as well as actions, information, and original observations.

after_reset(sim, obs, info)

Handles the after_reset event.

Adds a message to the GUI and starts recording if enabled.

Parameters:

• sim – The Minecraft simulator.

• obs – The observation from the simulator.

• info – Additional information from the simulator.

Returns:

The observation and info.

after_step(sim, obs, reward, terminated, truncated, info)

Handles the after_step event.

Toggles recording based on user input (‘R’ key). Records frames and info if enabled.

Parameters:

• sim – The Minecraft simulator.

• obs – The observation from the simulator.

• reward – The reward from the simulator.

• terminated – Whether the episode has terminated.

• truncated – Whether the episode has been truncated.

• info – Additional information from the simulator.

Returns:

The modified observation, reward, terminated, truncated, and info.

before_close(sim)

Saves the current episode before closing the simulator.

Parameters:

sim – The Minecraft simulator.

before_reset(sim, reset_flag: bool) → bool

Saves the current episode before resetting.

Parameters:

• sim – The Minecraft simulator.

• reset_flag – The reset flag.

Returns:

The reset flag.

before_step(sim, action)

Records the action before it is executed.

Parameters:

• sim – The Minecraft simulator.

• action – The action to be executed.

Returns:

The action.

RewardGateCallback

class minestudio.simulator.callbacks.reward_gate.GateRewardsCallback

A callback for calculating rewards based on the formation of a Nether portal.

This callback rewards the agent for building a valid Nether portal structure using obsidian blocks.

after_reset(sim, obs, info)

Resets the current step count and previous reward.

Parameters:

• sim – The Minecraft simulator.

• obs – The observation from the simulator.

• info – Additional information from the simulator.

Returns:

The observation and info.

after_step(sim, obs, reward, terminated, truncated, info)

Calculates the gate reward for the current step.

The reward is the difference between the current gate reward and the previous gate reward (delta reward).

Parameters:

• sim – The Minecraft simulator.

• obs – The observation from the simulator.

• reward – The original reward from the simulator.

• terminated – Whether the episode has terminated.

• truncated – Whether the episode has been truncated.

• info – Additional information from the simulator.

Returns:

The modified observation, overridden reward, terminated, truncated, and info.

gate_reward(info, obs={})

Calculates the gate reward based on the current voxel information.

It iterates through all obsidian blocks and finds the maximum possible portal reward.

Parameters:

• info – The info dictionary containing voxel data.

• obs – The observation dictionary (optional).

Returns:

The maximum gate reward.

reward_as_smlest_pos(obsidian_position, obsidian_positions)

Calculates the reward for a potential portal frame based on a starting obsidian block.

It checks for both X-fixed and Z-fixed portal orientations.

Parameters:

• obsidian_position – The (x, y, z) coordinates of a starting obsidian block.

• obsidian_positions – A list of (x, y, z) coordinates of all obsidian blocks.

Returns:

The calculated reward for the best portal frame found from this starting block.

RewardsCallback

Date: 2024-11-11 17:44:15 LastEditors: muzhancun muzhancun@stu.pku.edu.cn LastEditTime: 2024-11-14 20:09:56 FilePath: /Minestudio/minestudio/simulator/callbacks/rewards.py

class minestudio.simulator.callbacks.rewards.RewardsCallback(reward_cfg)

A callback for calculating rewards based on in-game events.

This callback allows defining custom rewards for various events, such as killing entities.

after_reset(sim, obs, info)

Resets the reward memory and current step count.

Parameters:

• sim – The Minecraft simulator.

• obs – The observation from the simulator.

• info – Additional information from the simulator.

Returns:

The observation and info.

after_step(sim, obs, reward, terminated, truncated, info)

Calculates and overrides the reward based on the reward configuration.

Parameters:

• sim – The Minecraft simulator.

• obs – The observation from the simulator.

• reward – The original reward from the simulator.

• terminated – Whether the episode has terminated.

• truncated – Whether the episode has been truncated.

• info – Additional information from the simulator.

Returns:

The modified observation, overridden reward, terminated, truncated, and info.

SpeedTestCallback

Date: 2024-11-11 15:59:38 LastEditors: muzhancun muzhancun@stu.pku.edu.cn LastEditTime: 2024-11-17 21:43:39 FilePath: /Minestudio/minestudio/simulator/callbacks/speed_test.py

class minestudio.simulator.callbacks.speed_test.SpeedTestCallback(interval: int = 100)

A callback for testing the speed of the simulator.

This callback measures the average time per step and the average FPS over a specified interval.

after_step(sim, obs, reward, terminated, truncated, info)

Calculates and prints the speed test status if the interval is reached.

Parameters:

• sim – The Minecraft simulator.

• obs – The observation from the simulator.

• reward – The reward from the simulator.

• terminated – Whether the episode has terminated.

• truncated – Whether the episode has been truncated.

• info – Additional information from the simulator.

Returns:

The observation, reward, terminated, truncated, and info.

before_step(sim, action)

Records the start time before executing a step.

Parameters:

• sim – The Minecraft simulator.

• action – The action to be executed.

Returns:

The action.

SummonMobsCallback

Date: 2024-11-11 17:26:22 LastEditors: caishaofei-mus1 1744260356@qq.com LastEditTime: 2025-04-19 15:42:35 FilePath: /MineStudio/var/minestudio/simulator/callbacks/summon_mobs.py

class minestudio.simulator.callbacks.summon_mobs.SummonMobsCallback(mobs)

A callback for summoning mobs in the Minecraft world.

This callback allows specifying the types, numbers, and spawn ranges of mobs to be summoned after each reset.

after_reset(sim, obs, info)

Summons the specified mobs after the environment is reset.

Parameters:

• sim – The Minecraft simulator.

• obs – The observation from the simulator.

• info – Additional information from the simulator.

Returns:

The modified observation and info after summoning mobs.

create_from_conf()

Creates a SummonMobsCallback instance from a configuration source.

Parameters:

source – The configuration source (e.g., file path or dictionary).

Returns:

A SummonMobsCallback instance or None if ‘summon_mobs’ is not in the config.

mobs

Examples: mobs = [{

‘name’: ‘cow’, ‘number’: 10, ‘range_x’: [-5, 5], ‘range_z’: [-5, 5],

}]

TaskCallback

Date: 2024-11-11 19:29:45 LastEditors: caishaofei-mus1 1744260356@qq.com LastEditTime: 2024-11-12 00:12:11 FilePath: /MineStudio/minestudio/simulator/callbacks/task.py

class minestudio.simulator.callbacks.task.TaskCallback(task_cfg)

A callback for managing and assigning tasks in the Minecraft environment.

This callback randomly selects a task from a predefined list of tasks after each environment reset and adds it to the observation.

after_reset(sim, obs, info)

Selects a random task and adds it to the observation after a reset.

Parameters:

• sim – The Minecraft simulator.

• obs – The observation from the simulator.

• info – Additional information from the simulator.

Returns:

The modified observation and info.

VoxelsCallback

class minestudio.simulator.callbacks.voxels.VoxelsCallback(voxels_ins=[-7, 7, -7, 7, -7, 7])

A callback for specifying the voxel observation range.

This callback adds a “voxels” instruction to the action, defining the volume from which voxel data should be collected.

before_step(sim, action)

Adds the voxel instruction to the action before a step is taken.

Parameters:

• sim – The Minecraft simulator.

• action – The action dictionary.

Returns:

The modified action dictionary with the “voxels” instruction.

Utils

Constants

Date: 2024-11-15 14:57:39 LastEditors: muzhancun muzhancun@stu.pku.edu.cn LastEditTime: 2024-11-17 22:32:08 FilePath: /Minestudio/minestudio/simulator/utils/constants.py

class minestudio.simulator.utils.constants.GUIConstants

Defines constants used for the Minecraft GUI.

This includes keyboard and mouse mappings for actions, display dimensions, FPS settings, and grid layout parameters for the information panel.

GUI

Date: 2024-11-15 15:15:22 LastEditors: muzhancun muzhancun@stu.pku.edu.cn LastEditTime: 2024-11-20 01:02:18 FilePath: /MineStudio/minestudio/simulator/utils/gui.py

minestudio.simulator.utils.gui.CommandModeDrawCall(info, **kwargs)

Draws a command mode indicator on the POV display.

If command mode is active (indicated by the ‘ESCAPE’ flag in info), the POV image is converted to grayscale, and “Command Mode” text is drawn on the top-left corner.

Parameters:

• info – A dictionary containing the ‘pov’ image and ‘ESCAPE’ (command mode) flag.

• kwargs – Additional keyword arguments (not used).

Returns:

The modified info dictionary with the command mode indicator.

class minestudio.simulator.utils.gui.MinecraftGUI(extra_draw_call: List[Callable] | None = None, show_info=True, **kwargs)

Manages the Pyglet-based GUI for the Minecraft simulator.

Handles window creation, event processing (keyboard, mouse), rendering of the POV display, and displaying informational messages. It also supports custom draw calls for additional visual elements.

close_gui()

Closes the Pyglet window and exits the Pyglet application.

create_window()

Creates the Pyglet window and sets up event handlers and ImGui integration.

reset_gui()

Resets the GUI state, clears the window, and shows a “Resetting” message.

minestudio.simulator.utils.gui.MultiPointDrawCall(info, **kwargs)

Draws multiple point indicators (positive and negative) on the POV display.

Positive points are drawn as green circles, and negative points are drawn as red circles. Point coordinates can be remapped using ‘remap_points’ in kwargs.

Parameters:

• info – A dictionary containing ‘positive_points’ and ‘negative_points’ lists, and the ‘pov’ image.

• kwargs – Additional keyword arguments, including ‘remap_points’.

Returns:

The modified info dictionary with points drawn on the ‘pov’ image.

minestudio.simulator.utils.gui.PointDrawCall(info, **kwargs)

Draws a point indicator on the POV display.

If a ‘point’ is present in the info dictionary, a red circle is drawn at the specified coordinates on the POV image. Text indicating the point’s coordinates is also displayed.

Parameters:

• info – A dictionary containing the ‘pov’ image and ‘point’ coordinates.

• kwargs – Additional keyword arguments (not used).

Returns:

The modified info dictionary with the point drawn on the ‘pov’ image.

minestudio.simulator.utils.gui.RecordDrawCall(info, **kwargs)

Draws a recording indicator on the POV display.

A red or green circle and “Rec” text are drawn on the top-left corner of the POV image if recording is active. The color of the circle alternates based on the current time.

Parameters:

• info – A dictionary containing the ‘pov’ image and ‘R’ (recording status) flag.

• kwargs – Additional keyword arguments (not used).

Returns:

The modified info dictionary with the recording indicator drawn on the ‘pov’ image.

minestudio.simulator.utils.gui.SegmentDrawCall(info, **kwargs)

Draws a segmentation mask overlay on the POV display.

If a ‘segment’ mask is present in the info dictionary, it’s resized to the POV image dimensions and overlaid with a green color.

Parameters:

• info – A dictionary containing the ‘segment’ mask and ‘pov’ image.

• kwargs – Additional keyword arguments (not used).

Returns:

The modified info dictionary with the segmentation mask overlay.