Skip to content

create_zarr_from_hdf5

create_zarr_from_hdf5

Creates a Zarr-based replay buffer dataset from HDF5 files (e.g., LIBERO).

create_replay_buffer_from_hdf5 

create_replay_buffer_from_hdf5(schema, image_frames_per_shard=DEFAULT_IMAGE_FRAMES_PER_SHARD)

Creates a Zarr-based replay buffer from multiple HDF5 files.

Parameters:

Name Type Description Default
schema Hdf5DatasetSchema

Hdf5DatasetSchema instance with HDF5 paths and zarr path configured

 required
image_frames_per_shard int | None

Number of image frames stored in each shard, or None to disable sharding.

 DEFAULT_IMAGE_FRAMES_PER_SHARD

Raises:

Type Description
TypeError

If image_frames_per_shard is not an integer.
ValueError

If image_frames_per_shard is not a positive multiple of the image chunk length.
Source code in src/versatil/data/preprocessing/create_zarr_from_hdf5.py 
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
def create_replay_buffer_from_hdf5(
    schema: Hdf5DatasetSchema,
    image_frames_per_shard: int | None = DEFAULT_IMAGE_FRAMES_PER_SHARD,
) -> None:
    """Creates a Zarr-based replay buffer from multiple HDF5 files.

    Args:
        schema: Hdf5DatasetSchema instance with HDF5 paths and zarr path configured
        image_frames_per_shard: Number of image frames stored in each shard, or
            None to disable sharding.

    Raises:
        TypeError: If image_frames_per_shard is not an integer.
        ValueError: If image_frames_per_shard is not a positive multiple of the
            image chunk length.
    """
    total_episodes = _count_hdf5_episodes(schema=schema)
    episodes = _iter_hdf5_episodes(schema=schema)

    create_zarr_replay_buffer(
        schema=schema,
        episodes=episodes,
        total_episodes=total_episodes,
        image_frames_per_shard=image_frames_per_shard,
    )