Data

We design a trajectory structure for storing Minecraft data. Based on this data structure, users are able to store and retrieve arbitray trajectory segment in an efficient way.

MineStudio Data

• Raw Dataset
  • Basic Information
  • Loading Segment-level Data
  • Loading Episode-level Data
  • Using Lightning to Simplify the Data Loading Process
• Event Dataset
  • Basic Information
  • Loading All the Kill Entity Events
  • Using Lightning to Simplify Data Loading Process
  • Additional Design Details About EventDataset
• Visualization
  • Visualize Dataloader
  • Visualize Continuous Batches
  • Visualize Batches with Special Events
• Convertion
  • Prepare Raw Trajectories
  • Convert Raw Trajectories to MineStudio format
  • Check the MineStudio-Format Dataset

Quick Start

Here is a minimal example to show how we load a trajectory from the dataset.

from minestudio.data import load_dataset

dataset = load_dataset(
    mode='raw', 
    dataset_dirs=['6xx', '7xx', '8xx', '9xx', '10xx'], 
    enable_video=True,
    enable_action=True,
    frame_width=224, 
    frame_height=224,
    win_len=128, 
    split='train', 
    split_ratio=0.9, 
    verbose=True
)
item = dataset[0]
print(item.keys())

You may see the output like this:

[08:14:15] [Kernel] Driver video load 15738 episodes.  
[08:14:15] [Kernel] Driver action load 15823 episodes. 
[08:14:15] [Kernel] episodes: 15655, frames: 160495936. 
dict_keys(['text', 'timestamp', 'episode', 'progress', 'env_action', 'agent_action', 'env_prev_action', 'agent_prev_action', 'image', 'mask'])

Hint

Please note that the dataset_dirs parameter here is a list that can contain multiple dataset directories. In this example, we have loaded five dataset directories.

If an element in the list is one of 6xx, 7xx, 8xx, 9xx, or 10xx, the program will automatically download it from Hugging Face, so please ensure your network connection is stable and you have enough storage space.

If an element in the list is a directory like /nfs-shared/data/contractors/dataset_6xx, the program will load data directly from that directory.

You can also mix the two types of elements in the list.

Learn more about Raw Dataset

Alternatively, you can also load trajectories that have specific events, for example, loading all trajectories that contain the kill entity event.

from minestudio.data import load_dataset

dataset = load_dataset(
    mode='event', 
    dataset_dirs=['7xx'], 
    enable_video=True,
    enable_action=True,
    frame_width=224, 
    frame_height=224,
    win_len=128, 
    split='train', 
    split_ratio=0.9, 
    verbose=True,
    event_regex='minecraft.kill_entity:.*'
)
item = dataset[0]
print(item.keys())

You may see the output like this:

[08:19:14] [Kernel] Driver video load 4617 episodes.
[08:19:14] [Kernel] Driver action load 4681 episodes. 
[08:19:14] [Kernel] episodes: 4568, frames: 65291168. 
[08:19:14] [Event Kernel] Number of loaded events: 58. 
[08:19:14] [Event Dataset] Regex: minecraft.kill_entity:.*, Number of events: 58, number of items: 19652
dict_keys(['text', 'env_action', 'agent_action', 'env_prev_action', 'agent_prev_action', 'image', 'mask'])

Learn more about Event Dataset

Data Structure

We classify and save the data according to its corresponding modality, with each modality’s data being a sequence over time. Sequences from different modalities can be aligned in chronological order. For example, the “action” modality data stores the mouse and keyboard actions taken at each time step of the trajectory; the “video” modality data stores the observations returned by the environment at each time step of the trajectory.

Note

The data of different modalities is stored independently. The benefits are: (1) Users can selectively read data from different modalities according to their requirements; (2) Users are easily able to add new modalities to the dataset without affecting the existing data.

For the sequence data of each modality, we store it in segments, with each segment having a fixed length (e.g., 32), which facilitates the reading and storage of the data.

Note

For video data, the efficiency of random access is usually low because decoding is required during the reading process. An extreme case would be to save it as individual images, which would allow for high read efficiency but take up a large amount of storage space.

We adopt a compromise solution by saving the video data in video segments, which allows for relatively high read efficiency while not occupying too much storage space. When user wants to read a sequence of continuous frames, we only need to retrieve the corresponding segments and decode them.

Learn more about the details

Segmented sequence data is stored in individual lmdb files, each of which contains the following metadata:

{
    "__num_episodes__": int,     # the total number of episodes in this lmdb file
    "__num_total_frames__": int, # the total number of frames in this lmdb file
    "__chunk_size__": int,       # the length of each segment (e.g. 32)
    "__chunk_infos__": dict      # save the information of the episode part in this lmdb file, e.g. the start and end index, episode name. 
}

Once you know the episode name and which segment you want to read, you can identify the corresponding segment bytes in the lmdb file and decode it to get the data.

with lmdb_handler.begin() as txn:
    key = str((episode_idx, chunk_id)).encode()
    chunk_bytes = txn.get(key)

Hint

In fact, you don’t need to worry about these at all, as we have packaged these operations for you. You just need to call corresponding API. The class that is responsible for managing these details is minestudio.data.minecraft.core.LMDBDriver.

With LMDBDriver, you can do these operations to a lmdb file:

• Get the trajectory list:

  trajectory_list = lmdb_driver.get_trajectory_list()
  

• Get the total frames of several trajectories:

  lmdb_driver.get_total_frames([
      "trajectory_1",
      "trajectory_2",
      "trajectory_3"
  ])
  

• Read a sequence of frames from a trajectory:

  frames, mask = lmdb_driver.read_frames(
      eps="trajectory_1",
      start_frame=11,
      win_len=33, 
      merge_fn=merge_fn, 
      extract_fn=extract_fn, 
      padding_fn=padding_fn, 
  )
  

  Note

  merge_fn, extract_fn, and padding_fn are functions that are used to process the data and are specific to the data modality.

Built-in Modalities

We provide the following built-in modalities for users to store data:

Modality	Description	Data Format
video	Observations returned by the environment	np.ndarry
action	Mouse and keyboard actions	Dict
contractor info	Information of the contractor	Dict
segment info	Information of the segment	Dict

Video and Segmentation Visualization