Why Your HDF5 Dataset Can't Extend—and the H5CPP Way

Rudresh shared that attempts to extend a dataset were failing with crashes or no effect—despite calling .extend(...) or H5Dextend(). Their code was creating a 2D dataset of variable‑length strings, but had not enabled chunking or specified unlimited max dimensions, leaving them stuck.

The Insight

If you don’t create a dataset with chunked layout and max dimensions set to unlimited (H5S_UNLIMITED), HDF5 creates it with contiguous storage and a fixed size that cannot be extended later. That's the crux: without chunking and unlimited dims, calling .extend will fail or crash.

A Possible H5CPP Solution

Here’s how you can cleanly create an extendible dataset and append to it—using H5CPP:

#include <h5cpp/all>

int main() {
  // Create a new file and an extendible dataset of scalar values
  h5::fd_t fd = h5::create("example.h5", H5F_ACC_TRUNC);

  // Create a packet-table (extendible) dataset of scalars
  h5::pt_t pt = h5::create<size_t>(
    fd, 
    "stream of scalars", 
    h5::max_dims{H5S_UNLIMITED}
  );

  // Append values dynamically
  for (auto value : {1, 2, 3, 4, 5}) {
    h5::append(pt, value);
  }
}

For a frame‑by‑frame stream (e.g., matrices or images):

#include <h5cpp/all>
#include <armadillo>

int main() {
  h5::fd_t fd = h5::create("example.h5", H5F_ACC_TRUNC);
  size_t nrows = 2, ncols = 5, nframes = 3;

  auto pt = h5::create<double>(
    fd,
    "stream of matrices",
    h5::max_dims{H5S_UNLIMITED, nrows, ncols},
    h5::chunk{1, nrows, ncols}
  );

  arma::mat M(nrows, ncols);
  for (int i = 0; i < nframes; ++i) {
    h5::append(pt, M);
  }
}

Why This Works

Aspect	H5CPP Approach
Extendible storage	h5::max_dims{H5S_UNLIMITED}
Chunked layout setup	Automatic via h5::chunk{...}
Appending data	One-liners with h5::append(...)
Clean C++ modern API	Templates, RAII, zero boilerplate

TLDR

If your HDF5 dataset “can’t extend,” the culprit is almost always that it's not chunked with unlimited dimensions. Fix that creation pattern—chunking is mandatory. With H5CPP, appending becomes elegant and idiomatic:

• Create: h5::create with unlimited dims
• Append: h5::append(...) in plain C++ style

Let me know if you'd like this turned into a tutorial or compared to the raw HDF5 C++ API equivalent.