Skip to content

Index

  • in hdf5, c++
  • 7 min read

Zero-Cost C++ Structs to HDF5 Compound Types with H5CPP

🧬 The Setup: From HPC POD Structs to HDF5 COMPOUNDs

You're handed an HDF5 file with compound data like this:

DATASET "g_data" {
  DATATYPE  H5T_COMPOUND {
    H5T_IEEE_F64LE "temp";
    H5T_IEEE_F64LE "density";
    H5T_ARRAY { [3] H5T_IEEE_F64LE } "B";
    H5T_ARRAY { [3] H5T_IEEE_F64LE } "V";
    H5T_ARRAY { [20] H5T_IEEE_F64LE } "dm";
    H5T_ARRAY { [9] H5T_IEEE_F64LE } "jkq";
  }
  DATASPACE SIMPLE { ( 70, 3, 3 ) / ( 70, 3, 3 ) }
}
With H5CPP, you don't need to touch the C API. Just describe this with a plain-old C++ struct:

namespace sn {
    struct record_t {
        double temp;
        double density;
        double B[3];
        double V[3];
        double dm[20];
        double jkq[9];
    };
}

That’s it.

🧰 Write Code as If You Had a Magic HDF5 Interface

No manual H5Tinsert, no dataspace juggling:

#include <vector>
#include "struct.h"
#include <h5cpp/core>
#include "generated.h"  // <-- auto-generated with h5cpp compiler
#include <h5cpp/io>

int main(){
    h5::fd_t fd = h5::create("test.h5", H5F_ACC_TRUNC);
    h5::create<sn::record_t>(fd, "/Module/g_data", h5::max_dims{70, 3, 3});

    auto dataset = h5::read<std::vector<sn::record_t>>(fd, "/Module/g_data");
    for (const auto& rec : dataset)
        std::cerr << rec.temp << " ";
    std::cerr << std::endl;
}

Compile and run:

h5cpp struct.cpp -- -std=c++17 -I/usr/include -Dgenerated.h
g++ struct.o -lhdf5 -lz -ldl -lm -o struct
./struct

🧠 Under the Hood: Codegen Output

H5CPP automatically generates a minimal type descriptor that registers your struct with the HDF5 type system:

template<> hid_t inline register_struct<sn::record_t>(){
    // array declarations omitted for brevity
    hid_t ct = H5Tcreate(H5T_COMPOUND, sizeof(sn::record_t));
    H5Tinsert(ct, "temp", HOFFSET(sn::record_t, temp), H5T_NATIVE_DOUBLE);
    ...
    return ct;
}

Everything is cleaned up (H5Tclose(...)) to avoid resource leaks.

📚 Can I Extract One Field From a Compound?

"Can I create a virtual dataset from just the 'density' field in g_data?"

Short answer: yes, but not easily.

HDF5 virtual datasets (VDS) work best with entire datasets, not fields of compound types. However, a workaround is to:

  • Use hyperslab selections to read only the density field in software.
  • Or, repack the data using H5CPP into a derived dataset containing only the fields you want.

Future H5CPP support might wrap this more cleanly, but currently, you'll want to do the extraction in-memory.

✅ Summary

  • Describe your data in C++ once.
  • H5CPP generates the boilerplate and builds your type descriptors.
  • You can now read, write, and manipulate rich compound datasets without touching the C API.
  • Clean syntax, high performance, zero leaks.

Ready to ditch boilerplate and focus on real work?

  • in hdf5, c++
  • 11 min read

Extracting Insight from HDF5 Compound Datasets Using H5CPP

"What if you could treat HDF5 compound datasets like simple C++ structs and just… write your simulation code? Forget the messy HDF5 C API. Write clean, modern C++. Let the compiler do the rest."

🚀 The Problem

One of our HPC applications writes simulation results to HDF5 files using compound types—essentially structured records with fields like temperature, density, and multidimensional vectors such as velocity and magnetic fields. A simplified dump of the file looks like this:

HDF5 "test.h5" {
  GROUP "/" {
    ATTRIBUTE "GRID_DIMENSIONS" { H5T_STD_I32LE (3) }
    ATTRIBUTE "X_AXIS"          { H5T_IEEE_F64LE (3) }
    ATTRIBUTE "Z_AXIS"          { H5T_IEEE_F64LE (70) }

    GROUP "Module" {
      DATASET "g_data" {
        DATATYPE H5T_COMPOUND {
          H5T_IEEE_F64LE "temp";
          H5T_IEEE_F64LE "density";
          H5T_ARRAY { [3] H5T_IEEE_F64LE } "B";
          H5T_ARRAY { [3] H5T_IEEE_F64LE } "V";
          H5T_ARRAY { [20] H5T_IEEE_F64LE } "dm";
          H5T_ARRAY { [9] H5T_IEEE_F64LE } "jkq";
        }
        DATASPACE SIMPLE { (70, 3, 3) }
      }
    }
  }
}

Now imagine wanting to selectively extract the "density" field into its own dataset—for visualization, analysis, or postprocessing. Can we filter compound dataset fields like this? Better yet, can we work with this structure using modern C++?

Let’s take a practical dive.

🔧 Step 1: Define the POD Struct

Start by expressing your compound layout as a C++ POD type (we recommend using nested namespaces for organization):

#ifndef  H5TEST_STRUCT_01 
#define  H5TEST_STRUCT_01

namespace sn {
    struct record_t {
        double temp;
        double density;
        double B[3];
        double V[3];
        double dm[20];
        double jkq[9];
    };
}
#endif

You’ll use this structure as the basis for both reading and writing your data.

✨ Step 2: Write the Code Like You Mean It

With H5CPP, you don’t need to write any boilerplate for type descriptors or deal with the HDF5 C API. Just:

#include <iostream>
#include <vector>
#include "struct.h"
#include <h5cpp/core>
  #include "generated.h" // generated by the h5cpp toolchain
#include <h5cpp/io>

int main() {
    h5::fd_t fd = h5::create("test.h5", H5F_ACC_TRUNC);

    // Create the dataset
    h5::create<sn::record_t>(fd, "/Module/g_data", h5::max_dims{70,3,3});

    // Read entire dataset into memory
    auto dataset = h5::read<std::vector<sn::record_t>>(fd, "/Module/g_data");

    // Loop over records and print temperature
    for (const auto& rec : dataset)
        std::cerr << rec.temp << " ";
    std::cerr << std::endl;
}

No type registration? It’s handled for you.

🧠 Step 3: Generate the Type Descriptor

Use the H5CPP compiler to generate the type description (generated.h):

h5cpp struct.cpp -- -std=c++17 -I/usr/include

It emits something like:

template<> hid_t inline register_struct<sn::record_t>(){
    // define compound type layout with H5Tinsert() calls
    // ...
    H5Tinsert(ct_00, "density", HOFFSET(sn::record_t, density), H5T_NATIVE_DOUBLE);
    // ...
    return ct_00;
}

This magic glues your record_t to an HDF5 COMPOUND type with nested arrays.

🧪 Bonus: Can We Slice Fields From Compound Datasets?

Yes, with HDF5 1.10+ you can use virtual datasets (VDS) to reference subsets of existing datasets, but there are some caveats:

  • You cannot directly extract a field from a compound dataset as a virtual dataset without writing a filter or preprocessor.
  • H5CPP doesn't (yet) offer field projections, but you can write a thin wrapper to convert std::vector<record_t> into std::vector<double> by extracting the field manually.

A future enhancement could be writing VDS definitions that reference memory offsets within compound datasets, but for now, you can batch process like this:

std::vector<double> densities;
for (auto& rec : dataset)
    densities.push_back(rec.density);

Simple, fast, cache-friendly.

📈 Output and Verification

Once compiled and run, you’ll get:

./struct
h5dump -pH test.h5

Showing the full compound structure, chunk layout, and memory offsets exactly as designed.

🧩 Conclusion

Working with compound HDF5 types doesn’t have to mean writing hundreds of lines of C code. With H5CPP:

  • You declare a C++ struct that mirrors your HDF5 layout.
  • The compiler auto-generates the type description.
  • Your code stays clean, idiomatic, and testable.

And when you want to extract a field like "density"? You don’t need virtual datasets—you just write C++.

“If you can write a struct, you can store structured data.”

  • in hdf5, c++
  • 7 min read

Best Way to Store Tick-by-Tick Stock Data in HDF5

The Problem (OP asks)

I'm storing stock tick-by-tick data—what’s the best structure in HDF5 to handle this efficiently and flexibly?

Perspective from H5CPP (Steven Varga, May 19, 2018)

If you’re working with high-frequency time series (HFT-style), H5CPP is tailor-made for event-based datasets. It plays well with std::vector<> of your POD structs and most linear algebra stacks (e.g., Armadillo).

Two Design Patterns to Consider

​​ Extendable Multi-Dimensional Cube

You can model your tick data as an extendable cube (or even higher dimensions), then slice what you need:

h5::create(fd, {n_minutes_in_day, n_instruments, H5_UNLIMITED}); // creates extendable dataset

Armadillo can pull in cubes directly. H5CPP supports partial read/write, so you can map arbitrary hyperslabs into:

  • arma::cube (3D)
  • arma::mat (2D)
  • std::vector<YourPOD> (1D)

This gives you powerful flexibility if you want to access time-instrument windows efficiently. A single multi-dimensional dataset can model all your data—but its harder to manage day boundaries or rolling windows.

So instead, keep things simple:

  • Use one daily stream for your high-frequency irregular time series (IRTS)—packed as a 1D sequence.
  • Use daily matrices for regular time series (RTS) when appropriate.
  • Then stitch these together using a circular buffer to model your clock window (e.g., last N days).

That gives you clean separation for day-level operations, easy rolling windows across days, and better control inside your streaming logic.

Avoid Tables for Tick Data

HDF5 tables (i.e., compound heterogeneous datasets) are cumbersome:

  • Hard to use in linear algebra contexts
  • Not easily mapped to vectors or matrices
  • Often less performant for numerical processing

Stick with floats or doubles in a numeric dataset, and keep your tick schema POD-friendly.

Summary of Approaches

Pattern Use Case H5CPP-Enabled Benefits
Extendable Cube Multi-dimensional, flexible slicing Easy high-D slicing, integrates with arma/Stl
Daily Stream + Circular Day-partitioned, rolling windows Simpler structure, rolling window stitching
HDF5 Tables (compound) Rich schema, less focus on math Clunky for numeric analysis, avoid if possible

TL;DR

H5CPP shines for stock tick data. Model your data as:

  • A hybrid of daily streams and rolling buffers, or
  • A flexible extendable cube if you want high-D slices.

Either way, avoid compound table datasets for numeric speed and maintain clean interop with your analysis stacks.

  • in hdf5, c++
  • 5 min read

Fixing Hyperslab Slices: Simplify with H5CPP Column Reads

The Issue (Bert Bril, Apr 26, 2018)

Bert was trying to extract an entire column from a really large 3D dataset—perhaps terabyte scale—into a 1D array using neat hyperslab coordinate selection. But the results were all wrong: Two out of every three values ended up zero, even though his start/count looked correct. He wasn’t even requesting broad slicing—just first column, full depth—yet the read was skipping data.

Steven Varga’s Insight (Apr 27, 2018)

Steve led with a reality check and a practical recommendation: Bang it into a simpler problem: If the dataset fits in memory, load the whole cube into an arma::cube and slice from there. If it doesn’t fit: Switch to a chunked column reader with H5CPP:

double* pd = static_cast<double*>(calloc(ROW_SIZE, sizeof(double)));
hid_t fd = h5::open("your_datafile.h5", H5F_ACC_RDONLY);
hid_t ds = h5::open(fd, "your_dataset");

for (int i = 0; i < data_rows; ++i) {
    h5::read(ds, pd, {0, i, 0}, {COL_SIZE, 1, 1});
    // 'start': {row, col, slice}; 'count': {row_count, 1, 1}
    // Apply your own stride/sieve logic here if needed
}

H5Dclose(ds);
H5Fclose(fd);
He noted that stride support is coming—but often manual subsetting is easier and just as fast. In short: don’t fight the API; control your data fetch with a clean loop and let H5CPP handle the heavy lifting.

Why This Fixes It

Step Reason
Simplify the problem Verifies hyperslab logic without third-party complexity
Use H5CPP for reads Offers clear, chunked access that's easy to reason about and debug
Manual sieve control Keeps logic explicit—no hidden behavior or unexpected flattening
  • in hdf5, c++
  • 5 min read

Simplifying Hyperslab Reads with H5CPP

The Problem (Bert Bril, Apr 26, 2018)

Bert was trying to read a single column from a massive 3D dataset:

short data[251];
hsize_t count[3] = {1, 1, 251};
hsize_t offset[3] = {0, 0, 0};
hsize_t stride[3] = {1, 1, 1};

filedataset.selectHyperslab(H5S_SELECT_SET, count, offset, stride);
H5::DataSpace input = filedataset.getSpace();
H5::DataSpace output(1, &nrtoread);
dataset_->read(data, H5::PredType::STD_I16LE, output, input);
But weirdly, the data read back had two out of every three values missing: 3786 0 0 3555 0 0 3820 … Even simpler 1D reads using a 3D hyperslab weren’t behaving as expected—and Bert couldn’t figure out why.

Steven Varga to the Rescue (Apr 26, 2018)

Steven’s suggestion? Toss the boilerplate and go with H5CPP—a clean, MIT-licensed, header-only C++ template library for HDF5 that makes multidimensional, Armadillo-style reads so much simpler:

“Reading a cube into an arma::cube is one-step—see the Armadillo example—and use arma::Cube instead of arma::Mat.” H5CPP is well-documented, functional, with examples, and integrates with major linear-algebra libraries. — Steve

H5CPP lets you sidestep manual DataSpace manipulations by offering high-level read() calls that feel like native matrix slicing.

Why This Helps

Approach Complexity Behavior Clarity Extensibility
Manual Hyperslab (HDF5 C++) High Error-prone and subtle Hard to debug
H5CPP (template-based) Low Clear intent and results Easy integration with arma, eigen, etc.

What to Do Next

  1. Try H5CPP—especially for reading cubes, columns, or slices with Armadillo support.

  2. Skip the complex hyperslab boilerplate and reach for a higher-level read() that feels like:

arma::cube C;
h5::read(ds, C); // simplified multi-dimensional read
  1. If you’re still seeing missing elements, compare dimension ordering or endian assumptions—but often H5CPP's clean interface surfaces logic over noise.
  • in hdf5, c++
  • 6 min read

Fixing h5repack Test Failures During make check

The Problem (by leow155, March 19, 2018)

leow155 ran into a snag when building HDF5 (v1.10.1) on Ubuntu 16.04:

./configure --prefix=/usr/local/hdf5 --enable-build-mode=production
make
make check

Everything builds fine, but make check throws an error during the h5repack tests:

h5repack tests failed with 1 errors.  
Command exited with non-zero status 1  
...
Makefile:1449: recipe for target ‘h5repack.sh.chkexe_’ failed
...

Clearly, h5repack.sh.chkexe_ fails and bubbles up through the test suite.

Steven Varga’s Reply (March 19, 2018)

Greetings!

I’m running the same HDF5 version (1.10.1) on Linux Mint 18 (Ubuntu 16.04 under the hood) with zero errors—and I didn’t even specify --enable-build-mode=production, because it’s already the default.

Did a quick make check moments ago, and it all passed cleanly on my setup.

Best, Steve

What to Make of This

Issue Steven’s Observation Suggested Action
h5repack test failure Works fine on his Ubuntu 16.04 setup Retry clean build without production flag
Possible environment mismatch Steven's env passed—identical build config Align compiler, tool versions, locale, etc.

Next Steps to Try

  1. Drop the --enable-build-mode=production flag—Steven didn't use it, and it may subtly change the build behavior.
  2. Re-clone and rebuild cleanly—use Git tag hdf5-1_10_1 to match his exact reference (avoid tarballs that may introduce unintended patches).
  3. Audit your toolchain—ensure your gcc, automake, make versions align with Steven’s Ubuntu/Mint environment, as small differences can trip specific tests.

TL;DR

You’re not hallucinating—h5repack should pass make check. Steven Varga verified it does, on a near-identical system (Ubuntu 16.04 via Linux Mint 18). Your best bet: align your build environment and try a clean rebuild without forcing production mode. Let me know if you'd like help comparing your tool versions or automating a clean rebuild.

  • in hdf5, c++
  • 7 min read

Parallelization Patterns for HDF5 I/O in C++

The Question (Stefano Salvadè, Feb 19, 2018)

Goal: write analysis results in parallel to multiple HDF5 files—one per stream/process. The application is in C#, calling into a C/C++ API with HDF5 and MPI.

Current thought: typically one uses mpiexec -n x Program.exe, but spawning processes at runtime via MPI_Spawn() seems clunky.

Is there a more elegant way to spawn parallel I/O functions within the same program? Also, do I need one process per write action (whether to separate files or a single shared file)?

Steven Varga’s Take—Less PHDF5, More Pragmatism

Parallel HDF5 (PHDF5) shines in setups with parallel file systems and true distributed environments—think HPC clusters with coordinated I/O capabilities.
But in simpler contexts—e.g., a single machine or cloud instance—PHDF5 often imposes unused complexity and file-system limitations (filters unsupported, extra boilerplate, etc.).
Instead, Stefano could:
1. Use separate HDF5 files per process, even in RAM or temp storage
2. Aggregate later, e.g. via:
- copying into one file, or
- using HDF5’s external file driver to compose them into a single logical container
The aggregation step could run as a separate batch job after the main MPI job finishes.
If you do have real parallel I/O infrastructure, then yes—PHDF5 gives benefits. But often, simple is better.
— Steve

Summary Table

Scenario Recommended Approach Reasoning
N streams → N separate files (no shared file) Serial HDF5 per process Simplicity, no PHDF5 overhead, independent files
Need to combine results later Aggregate post-run (external file driver or merge scripts) Keeps initial write simple; flexible downstream processing
True parallel I/O on a parallel file system Use PHDF5 with MPI Efficient coordinated I/O, but more complexity and system requirements

When to Use What?

  • Use PHDF5 when:
  • You're in a high-performance cluster environment
  • The file system supports parallel write throughput

  • You benefit from collective operations and synchronized metadata handling

  • Stick with serial HDF5 per process when:

  • You're on a single system or cloud VM
  • You want to avoid complexity in your write path
  • You can afford a merge or collector step after the run

Wrap-Up Thoughts

Stefano’s “elegant parallel output within a single program” goal doesn’t necessarily require MPI-spawned processes or PHDF5. Often the simplest is best: spawn OS-level processes writing to their own files, then merge or link them later.
This keeps performance high, complexity low, and coordination overhead manageable.

  • in hdf5, c++
  • 7 min read

Templatized Mapping of HDF5 Native Types in Modern C++

The Problem (Posted by Mark C. Miller, Feb 5, 2018)

Mark needed a simple way to map std::vector<T> to the correct H5T_NATIVE_* type in a templated dataset writer:

template <class T>
static void WriteVecToHDF5(hid_t fid, const char* name,
                           const std::vector<T>& vec, int d2size)
{
    hsize_t dims[2] = {vec.size() / d2size, d2size};
    hid_t spid = H5Screate_simple(d2size > 1 ? 2 : 1, dims, nullptr);
    hid_t dsid = H5Dcreate(fid, name, HDF5Type<T>().Type(), spid,
                           H5P_DEFAULT, H5P_DEFAULT, H5P_DEFAULT);
    H5Dwrite(dsid, H5T_NATIVE_INT, H5S_ALL, H5S_ALL, H5P_DEFAULT, &vec[0]);
    H5Dclose(dsid);
    H5Sclose(spid);
}
````

He implemented a hierarchy using `HDF5Type<T>()`, specialized per type to return the corresponding native constant (`H5T_NATIVE_INT`, `H5T_NATIVE_FLOAT`, etc.).

But... inheritance, virtuals, and boilerplate made him wonder: *Is there a cleaner way?*


## The H5CPP Way: Keep It Simple and Modern (Steven Varga)

Steven Varga, author of **H5CPP**, weighed in with a minimalist approach inspired by modern C++ design—ditch virtual dispatch, avoid inheritance, and leverage function overloads and template inference.

### ✅ Solution

```cpp
inline hid_t h5type(const char&)                { return H5T_NATIVE_CHAR; }
inline hid_t h5type(const signed char&)         { return H5T_NATIVE_SCHAR; }
inline hid_t h5type(const unsigned char&)       { return H5T_NATIVE_UCHAR; }
inline hid_t h5type(const short&)               { return H5T_NATIVE_SHORT; }
inline hid_t h5type(const unsigned short&)      { return H5T_NATIVE_USHORT; }
inline hid_t h5type(const int&)                 { return H5T_NATIVE_INT; }
inline hid_t h5type(const unsigned int&)        { return H5T_NATIVE_UINT; }
inline hid_t h5type(const long&)                { return H5T_NATIVE_LONG; }
inline hid_t h5type(const unsigned long&)       { return H5T_NATIVE_ULONG; }
inline hid_t h5type(const long long&)           { return H5T_NATIVE_LLONG; }
inline hid_t h5type(const unsigned long long&)  { return H5T_NATIVE_ULLONG; }
inline hid_t h5type(const float&)               { return H5T_NATIVE_FLOAT; }
inline hid_t h5type(const double&)              { return H5T_NATIVE_DOUBLE; }
inline hid_t h5type(const long double&)         { return H5T_NATIVE_LDOUBLE; }

template<typename T>
inline hid_t h5type() { return h5type(T()); }

✅ Usage

hid_t type_id = h5type<T>();

Or, more directly:

hid_t dsid = H5Dcreate(fid, name, h5type<T>(), spid,
                       H5P_DEFAULT, H5P_DEFAULT, H5P_DEFAULT);

Benefits of This Pattern

Feature Virtual Class Approach Overload-Based Approach
Compile-time resolution ❌ ✅
Easy to extend for new types ⚠️ (needs specialization) ✅ (just another overload)
Runtime dispatch ✅ ❌
Readability ⚠️ verbose ✅ concise
Performance (no virtual calls) ❌ ✅

Conclusion

The overload-based mapping pattern presented by Steven Varga in response to the HDF forum thread reflects a clean, idiomatic modern C++ style:

  • Clear separation of logic
  • No need for inheritance boilerplate
  • Efficient and extensible

If you're designing a lightweight HDF5 backend—or building C++ infrastructure like H5CPP—this pattern is robust and production-ready.

  • in hdf5, c++
  • 7 min read

Optimizing HDF5 for Event-Based Time Series Access

Original Question (Tamas, Mar 30, 2017)

I am currently using HDF5 to store my data and I have a structure that is too slow for reading single events.
My first attempt was to have a table of all hits, including the event_id as a field, so a simple table of:

hit_id | event_id | dom_id | time | tot | triggered | pmt_id

This is extremely slow for reading single events, as I have to go through all hits to extract one event.
My second attempt was to create groups for each event, and place the hits in a dataset inside that group. This also turned out to be slow.
So far the only solution I have found is to go back to ROOT, which is a bit of a shame.
Has anyone worked with similar data structures in HDF5? What is the fastest way of reading a single event?

Response (Steven Varga, Mar 31, 2017)

Hello Tamas,

I use HDF5 to store irregular time series (IRTS) data in the financial domain, where performance and storage density both matter. Here’s the approach I’ve been using successfully for the past 5 years:

🧱 Dataset Structure

  • Data is partitioned per day, each stored as a variable-length dataset (VLEN) containing a stream of events.
  • The dataset holds a custom compound datatype, tailored for compactness and compatibility.

For example, the record type might look like:

[event_id, asset, price, timestamp, flags, ...]

This layout reduces overhead and improves both sequential read throughput and storage density.

🔧 Access Pattern

  • The system is designed for write-once, read-many, with sequential reads being the dominant access mode.
  • For processing, I use C++ iterators to walk through the event stream efficiently.
  • This low-level backend is then exposed to Julia, R, and Python bindings for analysis pipelines.

Because the datatype is stored directly in the file, it’s also viewable with HDFView for inspection or debugging.

🚀 Performance Context

  • Runs in an MPI cluster using C++, Julia, and Rcpp.
  • Delivers excellent performance under both full-stream and filtered-access scenarios.
  • Chunk size does affect access time significantly—tuning that is essential depending on your use case.

🧠 Optimization Trade-Off

To your question—"how to access all or only some events efficiently"—that’s inherently a bi-objective optimization:

  • You can gain speed by trading off space, e.g., pre-sorting, duplicating metadata, or chunking strategically.

So while HDF5 might need more careful tuning than ROOT out of the box, it gives you portability and schema clarity across toolchains.

Hope this helps, Steve

  • in hdf5, c++
  • 7 min read

HDF5 for In-Memory Circular Buffers: Reuse Schema in RAM

The Original Ask (David Schneider, Dec 10 2016)

David from SLAC/LCLS posed this neat challenge:

“Is it possible to implement an in‑memory circular buffer using HDF5? We'd like both offline (on‑disk append) and online (shared‑memory overwrite) access via the same HDF5 schema and API, possibly using SWMR for shared-memory consumption.” :contentReference[oaicite:1]{index=1} Basically, a single schema that works both for archival and real-time consumption—elegant.

1. Werner's Insight: The Virtual File Driver

Werner dropped the elegant solution:

“There is a virtual file driver that operates on a memory image of an HDF5 file. It should be no problem to have this one also operate on shared memory.” :contentReference[oaicite:2]{index=2}

That’s referencing HDF5’s core VFD—you can treat a pointer to memory (including shared memory via mmap or shm_open) as if it were an HDF5 file. The same dataset API (H5Dcreate, H5Dwrite, etc.) applies, so you can reuse your schema seamlessly.

2. Steve’s Real‑World Twist (HFT-inspired)

Steve Varga chimed with a production-grade twist:

“Boost’s circular/ring buffer handles one-writer-many-readers; tail flushing can be channeled to the writer or fault‑tolerant hosts. Combine with ZeroMQ + Protocol Buffers or Thrift.”
“For experiments—where failure isn't critical—you can just access HDF5 locally on cluster nodes using MPI + Grid Engine + serial HDF5.” :contentReference[oaicite:3]{index=3}

So if you're doing industrial-strength durability, go ring buffer + messaging middleware. For HPC experiments where speed and simplicity triump, stick with HDF5+MPI.

Quick API Sketch (Julia‑Flavored)

```julia using HDF5, SharedMemory # hypothetical module?

fid = h5open_sharedmem(shm_address, mode="r+")

Use HDF5 API as if working on a real file

dset = d_create(fid, "/buffer", datatype=Float64, dims=(N,), maxdims=(HDF5.UNLIMITED,)) write(dset, new_chunk) close(fid) ````

Summary Table

Scenario Approach Notes
On-disk appendable buffer HDF5 datasets (append mode) Standard functionality
In-memory circular buffer HDF5 via core VFD over a memory region Shared schema/API in RAM
High‑throughput, production-grade Boost ring buffer + messaging (ZeroMQ, ProtoBuf) More robust, fault-tolerant
Experimental/distributed HPC HDF5 per node + MPI/Scheduler (serial HDF5) Simple, performance-focused