Skip to content

brainhops.io.base.hdf5

Shared plumbing for formats stored in HDF5 files.

Several transformation formats are HDF5 containers -- ITK's binary transforms (.h5) and BIDS X5 (.x5) among them -- and they cannot be told apart by their container: both are HDF5 files, and both even keep their transforms under a root group named TransformGroup. What differs is their content, so each format only says how to recognise and decode an open [h5py.File][], and the mixins here deal with everything else: paths, open binary streams, bytes held in memory, and lazily read datasets.

  • Hdf5Parser turns every input that brainhops accepts into an open h5py.File, and hands it to the format's sniff_h5 and from_h5.
  • Hdf5ParserWriter does the same for writing, through the format's to_h5.
  • DelayedH5Array is a dataset that can still be read after its file was closed, by reopening it on demand.

h5py is an optional dependency, so this module is imported only by the formats that need it, and only when it is installed.

Attributes

H5Like module-attribute 

H5Like = tx.BinaryIO | PathLike | str | h5py.File

Anything an HDF5 format can be read from or written to.

Classes

Hdf5Parser

Bases: BinaryFileParser

Reads a format stored in an HDF5 file.

A concrete format implements two class methods, both of which take an open h5py.File:

  • sniff_h5(h5file, error=False) -> float
  • from_h5(h5file, keep_open=False, load=True, **kwargs) -> Self

and this mixin routes paths, streams and bytes to them. A path is handed to h5py by name rather than as a stream, so that a dataset read lazily (load=False) can reopen the file long after it was parsed.

Attributes

EXTENSIONS class-attribute 
EXTENSIONS: tuple[str, ...] = ()

File extensions handled by this parser, e.g. (".nii", ".nii.gz").

Used as a first, cheap dispatch pass. When several parsers match, the longest matching extension wins, so a parser declaring ".nii.gz" takes precedence over one declaring ".gz".

PREFIXES class-attribute 
PREFIXES: tuple[str, ...] = ()

Filename prefixes required by this parser, e.g. ("y_", "iy_").

An empty tuple means "no constraint". A parser that constrains the prefix is more specific than one that does not, and wins ties.

Declaring EXTENSIONS and PREFIXES separately states the cross-product implicitly, which is how these conventions actually work: SPM's four names are {y_, iy_} x {.nii, .nii.gz}.

PRIORITY class-attribute 
PRIORITY: int = 0

Explicit tie-breaker, consulted only when specificity cannot decide. Higher wins. Leave at 0 unless two parsers genuinely collide.

Methods:

sniff_h5 classmethod 
sniff_h5(
    h5file: File, error: bool | Type[Exception] = False
) -> float

Score how confident the format is that an open HDF5 file is one of its files.

from_h5 classmethod 
from_h5(h5file: File, **kwargs) -> Self

Build an object from an open HDF5 file.

sniff_file classmethod 
sniff_file(
    file: H5Like,
    error: bool | Type[Exception] = False,
    **kwargs,
) -> float

Score a path, open HDF5 file, or binary stream.

sniff_filename classmethod 
sniff_filename(
    filename: str | PathLike,
    error: bool | Type[Exception] = False,
    **kwargs,
) -> float

Score the HDF5 file found at a path.

sniff_fileobj classmethod 
sniff_fileobj(
    file: IO,
    error: bool | Type[Exception] = False,
    **kwargs,
) -> float

Score an open, seekable binary stream.

sniff_bytes classmethod 
sniff_bytes(
    content: bytes,
    error: bool | Type[Exception] = False,
    **kwargs,
) -> float

Score the bytes of an HDF5 file.

from_file classmethod 
from_file(
    file: H5Like,
    keep_open: bool = False,
    load: bool = True,
    **kwargs,
) -> Self

Build an object from a file (path, file-like object, or HDF5 file).

Parameters:

Name Type Description Default
file str | PathLike | IO | File

Input file.

 required
keep_open bool

If True, keep the HDF5 file open after loading. If False, close the file after loading. If load=False and keep_open=False, the file is reopened every time a lazily read dataset is accessed.

 False
load bool

If True, read large datasets into memory. If False, keep them on disk.

 True
from_filename classmethod 
from_filename(
    filename: str | PathLike,
    keep_open: bool = False,
    load: bool = True,
    **kwargs,
) -> Self

Build an object from the HDF5 file found at a path.

from_fileobj classmethod 
from_fileobj(
    file: IO,
    keep_open: bool = False,
    load: bool = True,
    **kwargs,
) -> Self

Build an object from an open, seekable binary stream.

sniff classmethod 
sniff(
    file: FileOrContentLike,
    error: bool | Type[Exception] = False,
    **kwargs,
) -> float

Determine if the given file is of the type that this parser can handle.

Parameters:

Name Type Description Default
file FileOrContentLike

The file to sniff.

 required
error bool | type[Exception]

If not False, raise an error if the file cannot be sniffed.

 False
**kwargs

Parser-specific options.

 {}

Returns:

Type Description
float

Confidence that the file is of this type, in [0, 1].
sniff_content classmethod 
sniff_content(
    content: ContentLike,
    error: bool | Type[Exception] = False,
    **kwargs,
) -> float

Determine if the given content is of the type that this parser can handle.

Parameters:

Name Type Description Default
content ContentLike

The content to sniff.

 required
error bool | type[Exception]

If not False, raise an error if the content cannot be sniffed.

 False
**kwargs

Parser-specific options.

 {}

Returns:

Type Description
float

Confidence that the content is of this type, in [0, 1].
sniff_text classmethod 
sniff_text(
    text: str,
    error: bool | Type[Exception] = False,
    **kwargs,
) -> float

Determine if the given text is of the type that this parser can handle.

Parameters:

Name Type Description Default
text str

The text to sniff.

 required
error bool | type[Exception]

If not False, raise an error if the content cannot be sniffed.

 False
**kwargs

Parser-specific options.

 {}

Returns:

Type Description
float

Confidence that the text is of this type, in [0, 1].
sniff_lines classmethod 
sniff_lines(
    lines: Iterable[str],
    error: bool | Type[Exception] = False,
    **kwargs,
) -> float

Determine if the given lines are of the type that this parser can handle.

Parameters:

Name Type Description Default
lines Iterable[str]

The lines to sniff.

 required
error bool | type[Exception]

If not False, raise an error if the content cannot be sniffed.

 False
**kwargs

Parser-specific options.

 {}

Returns:

Type Description
float

Confidence that the lines is of this type, in [0, 1].
sniff_line classmethod 
sniff_line(
    line: str,
    error: bool | Type[Exception] = False,
    **kwargs,
) -> float

Determine if the given line is of the type that this parser can handle.

Parameters:

Name Type Description Default
line str

The line to sniff.

 required
error bool | type[Exception]

If not False, raise an error if the content cannot be sniffed.

 False
**kwargs

Parser-specific options.

 {}

Returns:

Type Description
float

Confidence that the line is of this type, in [0, 1].
load classmethod 
load(other: FileOrContentLike, **kwargs) -> Self

Build an object from a file (path, file-like object or iterable of lines).

This is the generic front door to the from_* family: it looks at what it was handed and calls the right one.

A str is always a path, whether or not the file exists, so a missing file raises FileNotFoundError whichever way its path was spelled. Text held in memory is read with from_text or from_content.

Parameters:

Name Type Description Default
other FileOrContentLike

Input file, or its content.

 required
**kwargs

Parser-specific options.

 {}

Returns:

Type Description
obj

The parsed object.

Raises:

Type Description
ParserExistsError

If other is a path to a file that does not exist. It is a FileNotFoundError.
from_spec classmethod 
from_spec(spec: SourceSpec, **kwargs) -> Self

Build an object from an unqualified structured source.

from_content classmethod 
from_content(content: ContentLike, **kwargs) -> Self

Build an object from a file content (bytes, str, or iterable of lines).

Parameters:

Name Type Description Default
content ContentLike

The content to parse.

 required
**kwargs

Parser-specific options.

 {}

Returns:

Type Description
obj

The parsed object.
from_bytes classmethod 
from_bytes(content: BinaryContentLike, **kwargs) -> Self

Build an object from a binary representation of a file.

If the class implements from_fileobj itself, the bytes are wrapped in an io.BytesIO stream and handed to from_fileobj. Otherwise, this raises ParserNotImplementedError: the default from_fileobj delegates to from_bytes, so falling back to it would recurse. A from_fileobj override that only forwards to super().from_fileobj should be decorated with _passthrough_from_fileobj; if it is not, the loop is still detected when from_bytes is re-entered for the same class, and ParserNotImplementedError is raised.

Parameters:

Name Type Description Default
content BinaryContentLike

The content to parse.

 required
**kwargs

Parser-specific options.

 {}

Returns:

Type Description
obj

The parsed object.

Raises:

Type Description
ParserNotImplementedError

If neither from_bytes nor from_fileobj is implemented.
from_text classmethod 
from_text(text: str, **kwargs) -> Self

Build an object from a text representation of a file.

Parameters:

Name Type Description Default
text str

The text to parse.

 required
**kwargs

Parser-specific options.

 {}

Returns:

Type Description
obj

The parsed object.
from_lines classmethod 
from_lines(lines: Iterable[str], **kwargs) -> Self

Build an object from an iterable of lines (e.g., the content of a file).

Parameters:

Name Type Description Default
lines Iterable[str]

The lines to sniff.

 required
**kwargs

Parser-specific options.

 {}

Returns:

Type Description
obj

The parsed object.
from_line classmethod 
from_line(line: str, **kwargs) -> Self

Build an object from a single line of text.

Parameters:

Name Type Description Default
line str

The line to parse.

 required
**kwargs

Parser-specific options.

 {}

Returns:

Type Description
obj

The parsed object.

Hdf5ParserWriter

Bases: Hdf5Parser, BinaryFileParserWriter

Reads and writes a format stored in an HDF5 file.

On top of what Hdf5Parser asks for, a concrete format implements to_h5(self, h5file, **kwargs), which fills a new, empty HDF5 file open for writing.

Attributes

EXTENSIONS class-attribute 
EXTENSIONS: tuple[str, ...] = ()

File extensions handled by this parser, e.g. (".nii", ".nii.gz").

Used as a first, cheap dispatch pass. When several parsers match, the longest matching extension wins, so a parser declaring ".nii.gz" takes precedence over one declaring ".gz".

PREFIXES class-attribute 
PREFIXES: tuple[str, ...] = ()

Filename prefixes required by this parser, e.g. ("y_", "iy_").

An empty tuple means "no constraint". A parser that constrains the prefix is more specific than one that does not, and wins ties.

Declaring EXTENSIONS and PREFIXES separately states the cross-product implicitly, which is how these conventions actually work: SPM's four names are {y_, iy_} x {.nii, .nii.gz}.

PRIORITY class-attribute 
PRIORITY: int = 0

Explicit tie-breaker, consulted only when specificity cannot decide. Higher wins. Leave at 0 unless two parsers genuinely collide.

Methods:

to_h5 
to_h5(h5file: File, **kwargs) -> None

Write this object into an empty HDF5 file open for writing.

to_file 
to_file(file: H5Like, **kwargs) -> None

Write to a path, an open binary stream, or an HDF5 file.

to_filename 
to_filename(filename: str | PathLike, **kwargs) -> None

Write to the file found at a path, replacing it.

to_fileobj 
to_fileobj(file: IO, **kwargs) -> None

Write to a binary stream open for writing.

to_bytes 
to_bytes(**kwargs) -> bytes

The bytes of the HDF5 file that encodes this object.

sniff classmethod 
sniff(
    file: FileOrContentLike,
    error: bool | Type[Exception] = False,
    **kwargs,
) -> float

Determine if the given file is of the type that this parser can handle.

Parameters:

Name Type Description Default
file FileOrContentLike

The file to sniff.

 required
error bool | type[Exception]

If not False, raise an error if the file cannot be sniffed.

 False
**kwargs

Parser-specific options.

 {}

Returns:

Type Description
float

Confidence that the file is of this type, in [0, 1].
sniff_file classmethod 
sniff_file(
    file: H5Like,
    error: bool | Type[Exception] = False,
    **kwargs,
) -> float

Score a path, open HDF5 file, or binary stream.

sniff_filename classmethod 
sniff_filename(
    filename: str | PathLike,
    error: bool | Type[Exception] = False,
    **kwargs,
) -> float

Score the HDF5 file found at a path.

sniff_fileobj classmethod 
sniff_fileobj(
    file: IO,
    error: bool | Type[Exception] = False,
    **kwargs,
) -> float

Score an open, seekable binary stream.

sniff_content classmethod 
sniff_content(
    content: ContentLike,
    error: bool | Type[Exception] = False,
    **kwargs,
) -> float

Determine if the given content is of the type that this parser can handle.

Parameters:

Name Type Description Default
content ContentLike

The content to sniff.

 required
error bool | type[Exception]

If not False, raise an error if the content cannot be sniffed.

 False
**kwargs

Parser-specific options.

 {}

Returns:

Type Description
float

Confidence that the content is of this type, in [0, 1].
sniff_bytes classmethod 
sniff_bytes(
    content: bytes,
    error: bool | Type[Exception] = False,
    **kwargs,
) -> float

Score the bytes of an HDF5 file.

sniff_text classmethod 
sniff_text(
    text: str,
    error: bool | Type[Exception] = False,
    **kwargs,
) -> float

Determine if the given text is of the type that this parser can handle.

Parameters:

Name Type Description Default
text str

The text to sniff.

 required
error bool | type[Exception]

If not False, raise an error if the content cannot be sniffed.

 False
**kwargs

Parser-specific options.

 {}

Returns:

Type Description
float

Confidence that the text is of this type, in [0, 1].
sniff_lines classmethod 
sniff_lines(
    lines: Iterable[str],
    error: bool | Type[Exception] = False,
    **kwargs,
) -> float

Determine if the given lines are of the type that this parser can handle.

Parameters:

Name Type Description Default
lines Iterable[str]

The lines to sniff.

 required
error bool | type[Exception]

If not False, raise an error if the content cannot be sniffed.

 False
**kwargs

Parser-specific options.

 {}

Returns:

Type Description
float

Confidence that the lines is of this type, in [0, 1].
sniff_line classmethod 
sniff_line(
    line: str,
    error: bool | Type[Exception] = False,
    **kwargs,
) -> float

Determine if the given line is of the type that this parser can handle.

Parameters:

Name Type Description Default
line str

The line to sniff.

 required
error bool | type[Exception]

If not False, raise an error if the content cannot be sniffed.

 False
**kwargs

Parser-specific options.

 {}

Returns:

Type Description
float

Confidence that the line is of this type, in [0, 1].
load classmethod 
load(other: FileOrContentLike, **kwargs) -> Self

Build an object from a file (path, file-like object or iterable of lines).

This is the generic front door to the from_* family: it looks at what it was handed and calls the right one.

A str is always a path, whether or not the file exists, so a missing file raises FileNotFoundError whichever way its path was spelled. Text held in memory is read with from_text or from_content.

Parameters:

Name Type Description Default
other FileOrContentLike

Input file, or its content.

 required
**kwargs

Parser-specific options.

 {}

Returns:

Type Description
obj

The parsed object.

Raises:

Type Description
ParserExistsError

If other is a path to a file that does not exist. It is a FileNotFoundError.
from_spec classmethod 
from_spec(spec: SourceSpec, **kwargs) -> Self

Build an object from an unqualified structured source.

from_file classmethod 
from_file(
    file: H5Like,
    keep_open: bool = False,
    load: bool = True,
    **kwargs,
) -> Self

Build an object from a file (path, file-like object, or HDF5 file).

Parameters:

Name Type Description Default
file str | PathLike | IO | File

Input file.

 required
keep_open bool

If True, keep the HDF5 file open after loading. If False, close the file after loading. If load=False and keep_open=False, the file is reopened every time a lazily read dataset is accessed.

 False
load bool

If True, read large datasets into memory. If False, keep them on disk.

 True
from_filename classmethod 
from_filename(
    filename: str | PathLike,
    keep_open: bool = False,
    load: bool = True,
    **kwargs,
) -> Self

Build an object from the HDF5 file found at a path.

from_fileobj classmethod 
from_fileobj(
    file: IO,
    keep_open: bool = False,
    load: bool = True,
    **kwargs,
) -> Self

Build an object from an open, seekable binary stream.

from_content classmethod 
from_content(content: ContentLike, **kwargs) -> Self

Build an object from a file content (bytes, str, or iterable of lines).

Parameters:

Name Type Description Default
content ContentLike

The content to parse.

 required
**kwargs

Parser-specific options.

 {}

Returns:

Type Description
obj

The parsed object.
from_bytes classmethod 
from_bytes(content: BinaryContentLike, **kwargs) -> Self

Build an object from a binary representation of a file.

If the class implements from_fileobj itself, the bytes are wrapped in an io.BytesIO stream and handed to from_fileobj. Otherwise, this raises ParserNotImplementedError: the default from_fileobj delegates to from_bytes, so falling back to it would recurse. A from_fileobj override that only forwards to super().from_fileobj should be decorated with _passthrough_from_fileobj; if it is not, the loop is still detected when from_bytes is re-entered for the same class, and ParserNotImplementedError is raised.

Parameters:

Name Type Description Default
content BinaryContentLike

The content to parse.

 required
**kwargs

Parser-specific options.

 {}

Returns:

Type Description
obj

The parsed object.

Raises:

Type Description
ParserNotImplementedError

If neither from_bytes nor from_fileobj is implemented.
from_text classmethod 
from_text(text: str, **kwargs) -> Self

Build an object from a text representation of a file.

Parameters:

Name Type Description Default
text str

The text to parse.

 required
**kwargs

Parser-specific options.

 {}

Returns:

Type Description
obj

The parsed object.
from_lines classmethod 
from_lines(lines: Iterable[str], **kwargs) -> Self

Build an object from an iterable of lines (e.g., the content of a file).

Parameters:

Name Type Description Default
lines Iterable[str]

The lines to sniff.

 required
**kwargs

Parser-specific options.

 {}

Returns:

Type Description
obj

The parsed object.
from_line classmethod 
from_line(line: str, **kwargs) -> Self

Build an object from a single line of text.

Parameters:

Name Type Description Default
line str

The line to parse.

 required
**kwargs

Parser-specific options.

 {}

Returns:

Type Description
obj

The parsed object.
save 
save(file: FileLike, **kwargs) -> None

Write the object to a file (path or file-like object).

This is the generic front door to the to_* family. It is named save rather than to because to already means something else on the data models these parsers are mixed into: Transformation.to converts an object to another type. A writer's to was shadowed by it on every writable transformation.

Parameters:

Name Type Description Default
file FileLike

The file to write to.

 required
**kwargs

Parser-specific options.

 {}
to_text 
to_text(**kwargs) -> str

Return a text version of the file.

Parameters:

Name Type Description Default
**kwargs

Parser-specific options.

 {}

Returns:

Type Description
str

A text version of the file.
to_lines 
to_lines(**kwargs) -> Iterator[str]

Return a text version of the file as an iterable of lines.

Parameters:

Name Type Description Default
**kwargs

Parser-specific options.

 {}

Returns:

Type Description
Iterator[str]

An iterable of lines representing the object.
to_line 
to_line(**kwargs) -> str

Return a line representing the object.

Parameters:

Name Type Description Default
**kwargs

Parser-specific options.

 {}

Returns:

Type Description
str

A line representing the object.
sniff_h5 classmethod 
sniff_h5(
    h5file: File, error: bool | Type[Exception] = False
) -> float

Score how confident the format is that an open HDF5 file is one of its files.

from_h5 classmethod 
from_h5(h5file: File, **kwargs) -> Self

Build an object from an open HDF5 file.

DelayedH5Array 

DelayedH5Array(file: H5Like, path: str)

An HDF5 dataset that can be read even after its file was closed, by reopening the file when needed.

Attributes

shape property 
shape: tuple

The shape of the dataset.

dtype property 
dtype: dtype

The data type of the dataset.

chunks property 
chunks: tuple

The chunk shape of the dataset, or None if it is not chunked.

ndim property 
ndim: int

The number of dimensions of the dataset.

size property 
size: int

The total number of elements in the dataset.

nbytes property 
nbytes: int

The size of the dataset in bytes.

Methods:

open 
open() -> File

Open (or reuse) the underlying HDF5 file and return it.

close 
close() -> None

Close the underlying HDF5 file, if this array opened it.

__del__ 
__del__() -> None

Close the underlying HDF5 file, if this array opened it.

to_dataset 
to_dataset(
    file: H5Like | None = None, keep_open: bool = False
) -> Dataset

Return the underlying h5py.Dataset, opening the file if needed.

to_array 
to_array(**kwargs) -> ndarray

Read the whole dataset into a numpy array.

to_dask 
to_dask(
    *, keep_open: bool = False, **kwargs
) -> ArrayProtocol

Wrap the dataset as a dask array that reads chunks lazily.

__getitem__ 
__getitem__(index: Any) -> Any

Read the indexed chunk of the dataset, opening the file if needed.

__array__ 
__array__(dtype: dtype = None, copy: Any = None) -> ndarray

Read the whole dataset into a numpy array.

Functions:

read_string 

read_string(value: Any) -> str | None

Decode a string stored in HDF5, whatever its storage.

HDF5 strings come back from h5py as str (variable-length UTF-8 attributes), as bytes (fixed-length or ASCII ones), as numpy scalars, or wrapped in a one-element array or dataset.

delayed_dataset 

delayed_dataset(
    h5file: File, key: str, keep_open: bool
) -> DelayedH5Array | ArrayProtocol

A dataset of an open file, to be read later rather than now.

With keep_open, the open file is held; otherwise the file is reopened by name when the data is read. The result is wrapped in a dask array when dask is installed.