Lazy datasets

Provide lazy, array-like access to datasets without loading full arrays into memory.

This module provides lightweight wrappers that expose array-like, lazy access to datasets without loading full arrays into memory.

The interface is defined by Dataset and mimics that of h5py datasets, allowing users to slice and access data on demand. Specific implementations load the necessary data and apply transformations, such as stacking, scaling, summing, etc., only for the requested slices.

xyz = LazyStackedDataset([ds1, ds2, ds3])

# Loaded and computed only for this slice
xyz_slice = xyz[100:200]

# Writing is supported for LazyScaledDataset and LazyStackedDataset
xyz[100:200] = new_data
scaled = LazyScaledDataset(ds, scale_factor=2.0)
scaled[50:100] = values  # Applies inverse scaling before writing

combined_signal = LazySumDataset([ds1, ds2])
combined_flags = LazyBooleanOrDataset([flags1, flags2])

# Loaded and computed only for this slice (read-only)
signal_slice = combined_signal[0:1024]
flags_slice = combined_flags[0:1024]

Some operations also supports writing. For example, LazyScaledDataset applies the inverse scaling operation before writing to the underlying dataset.

xyz = LazyScaledDataset(ds, scale_factor=2.0)
xyz[100:200] = new_values  # Writes new_values / 2.0 to ds[100:200]

Dataset interface

class mojito.dataset.Dataset(*args, **kwargs)[source]

Protocol for (potentially lazy) datasets with NumPy-style access.

Implementations must expose shape, dimensionality, dtype and support slicing through __getitem__ similarly to h5py.Dataset. Writing via __setitem__ is optional and may not be supported by all implementations.

__getitem__(*args, **kwargs)[source]

Return a sliced array.

Return type:

Any

__len__()[source]

Return the length of the first axis.

Raises:

TypeError – If the dataset is 0-dimensional.

Return type:

int

__repr__()[source]

Return a concise representation showing class name, shape and dtype.

Return type:

str

__setitem__(*args, **kwargs)[source]

Write to the dataset.

Raises:

NotImplementedError – If writing is not supported by the implementation.

Return type:

None

property dtype: dtype[Any]

Data type of the dataset.

property ndim: int

Number of dimensions.

property shape: tuple[int, ...]

Shape of the dataset.

Operations on datasets

class mojito.dataset.LazyMappedDataset(dataset, func, inverse_func=None, *, dtype=None, is_multivariate=None)[source]

Lazy-loaded dataset that applies a function to values on slicing.

This is a generic helper for dataset wrappers that transform a single dataset by applying a callable to the sliced values. Writing is supported by providing an inverse_func that maps values back before writing to the underlying dataset; if omitted, writing raises NotImplementedError.

Since func can change the data type of the values (e.g. casting or thresholding into booleans), dtype is inferred by applying func to a zero-sized probe array unless explicitly provided via dtype.

Supports both scalar-valued and multivariate map functions, with two constraints: func must not mix elements across the first (time) dimension, since the dataset is lazily sliced along it before func is applied; and func must preserve the exact shape of its input, as functions that change the output shape are not supported. In other words, func and its inverse (if defined) must accept an array-like broadcastable to dataset.shape[1:] and return an array of the exact same shape as its input.

Parameters:

  • dataset (Dataset) – Dataset to map.

  • func (Any) – Callable applied to sliced values when reading.

  • inverse_func (Any, default: None) – Callable applied to values before writing, effectively the inverse of func. If None, writing is not supported.

  • dtype (TypeAliasType | None, default: None) – Data type of the mapped values. If None, it is inferred by applying func to a zero-sized probe array.

  • is_multivariate (bool | None, default: None) – Whether func is multivariate. This is used for writing to the dataset. If None, this property is inferred by subjecting func to a series of simple tests. These tests might in very rare cases fail at identifying a multivariate function, so it is recommended to always explicitly provide it.

__getitem__(key)[source]

Slice and map the dataset lazily.

In case the mapping function is multivariate, it needs elements from different dimensions to compute the output. Therefore, the dataset is only sliced along the first dimension (time) before applying the mapping function. The output is then sliced with the original key to return the required slice.

Parameters:

key – Slicing key (integers, slices, lists, ellipsis).

Returns:

Result of func applied to the sliced array from the dataset.

Return type:

NDArray

__setitem__(key, value)[source]

Write mapped values to the dataset lazily.

inverse_func is applied to value before writing to the underlying dataset.

For multivariate map functions, partial writes along dimensions other than axis 0 are not supported: either value must be scalar or value.shape[1:] must be broadcastable to self.dataset.shape[1:]. This restriction does not apply to scalar-valued map functions.

Parameters:

  • key (Any) – Slicing key (integers, slices, lists, ellipsis).

  • value (Any) – Data to write. Will be passed through inverse_func before writing.

Raises:

NotImplementedError – If no inverse_func was provided.

Return type:

None

property dtype: dtype

Data type of the dataset.

property ndim: int

Number of dimensions.

property shape: tuple[int, ...]

Shape of the dataset.

class mojito.dataset.LazyScaledDataset(dataset, scale_factor=1.0, *, dtype=None)[source]

Lazy-loaded dataset that applies scaling on slicing.

This is a thin wrapper around LazyMappedDataset with a scaling function and its inverse.

Parameters:

  • dataset (Dataset) – HDF5 dataset to scale.

  • scale_factor (float, default: 1.0) – Scaling factor to apply to all values.

  • dtype (TypeAliasType | None, default: None) – Data type of the scaled values. If None, it is inferred automatically.

class mojito.dataset.LazyReducedDataset(datasets, reducer, *, dtype=None, shape=None)[source]

Lazy-loaded dataset that applies a NumPy reduction across datasets.

This is a generic helper for dataset wrappers that combine multiple datasets by loading the corresponding slice from each dataset and reducing the resulting arrays with a callable such as np.sum or np.logical_or.reduce.

Since reducer can change the shape (e.g. collapsing an axis) and data type (e.g. summing booleans into integers) of the values, shape and dtype are inferred by applying reducer to zero-sized probe arrays unless explicitly provided via shape and dtype.

Parameters:

  • datasets (Sequence[Dataset]) – Sequence of datasets to reduce.

  • reducer (Any) – Callable that accepts a list or array-like of sliced values and returns a reduced result. The function can optionally accept an axis argument to specify the axis along which to reduce.

  • dtype (TypeAliasType | None, default: None) – Data type of the reduced values. If None, it is inferred by applying reducer to zero-sized probe arrays.

  • shape (tuple[int, ...] | None, default: None) – Shape of the reduced dataset. If None, it is inferred the same way.

Raises:

ValueError – If datasets list is empty.

__getitem__(key)[source]

Slice and reduce datasets lazily.

Parameters:

key – Slicing key (integers, slices, lists, ellipsis).

Returns:

Reduced array from the sliced datasets.

Return type:

NDArray

Raises:

ValueError – If the reduced result does not match the expected shape or dtype.

property dtype: dtype

Data type of the dataset.

property ndim: int

Number of dimensions.

property shape: tuple[int, ...]

Shape of the dataset.

class mojito.dataset.LazySumDataset(datasets, *, dtype=None, shape=None)[source]

Lazy-loaded dataset that sums values across multiple datasets on slicing.

This is a thin wrapper around LazyReducedDataset with a sum reducer.

Parameters:

  • datasets (Sequence[Dataset]) – Sequence of HDF5 datasets to sum.

  • dtype (TypeAliasType | None, default: None) – Data type of the summed values. If None, it is inferred automatically.

  • shape (tuple[int, ...] | None, default: None) – Shape of the summed dataset. If None, it is inferred automatically.

Raises:

ValueError – If datasets list is empty.

class mojito.dataset.LazyBooleanOrDataset(datasets, *, dtype=None, shape=None)[source]

Lazy-loaded dataset that applies Boolean OR across multiple datasets.

This is a thin wrapper around LazyReducedDataset with a boolean OR reducer.

Parameters:

  • datasets (Sequence[Dataset]) – Sequence of HDF5 datasets to OR.

  • dtype (TypeAliasType | None, default: None) – Data type of the combined values. If None, it is inferred automatically.

  • shape (tuple[int, ...] | None, default: None) – Shape of the combined dataset. If None, it is inferred automatically.

Raises:

ValueError – If datasets list is empty.

Stacking datasets

class mojito.dataset.LazyStackedDataset(datasets, axis=-1)[source]

Lazy-loaded stacked dataset that does not load all data into memory.

This is useful to provide access to stacked and normalized datasets without loading all data into memory at once, similar to h5py datasets.

>>> stacked = LazyStackedDataset([ds1, ds2, ds3])
>>> print(stacked.shape)
(1000, 3)
>>> data_slice = stacked[100:200]  # Loads only the requested slice
>>> data_slice2 = stacked[100:200, ..., 0]  # Slicing along stacked axis
Parameters:

  • datasets (Sequence[Dataset]) – Sequence of HDF5 datasets to stack.

  • axis (int, default: -1) – Axis along which to stack the datasets.

Raises:

ValueError – If axis is out of bounds for stacking.

__getitem__(key)[source]

Slice the stacked datasets on demand without loading all data.

Supports slicing along all dimensions including the stacked axis. The stacked axis can be sliced using integers, slices, or lists of indices.

Note that fancy indexing (i.e., using arrays of indices for multiple axes) for non-stacked axes is not supported due to a limitation in h5py. However, it is supported for a single non-stacked axis at a time.

Insertion of None or np.newaxis is not supported.

Parameters:

key – Slicing key, can include integers (Python int or Numpy integer scalars), slices, lists of indices, and ellipsis.

Returns:

Sliced and stacked array.

Return type:

Any

Raises:

  • IndexError – If None or np.newaxis is used in the key.

  • IndexError – If too many indices are provided for the array.

__setitem__(key, value)[source]

Write to stacked datasets on demand.

The value is unstacked along the stacked axis and distributed to the appropriate underlying datasets.

Parameters:

  • key (Any) – Slicing key, can include integers, slices, or lists of indices.

  • value (Any) – Data to write. Must be compatible with the sliced shape.

Raises:

IndexError – If None or np.newaxis is used in the key, or if too many indices are provided.

Return type:

None

property dtype: dtype

Data type of the stacked array.

property ndim: int

Number of dimensions of the stacked array.

property normalized_axis: int

Normalize axis to a positive value.

property shape: tuple[int, ...]

Shape of the stacked array.