H5CPP

compiler assisted static reflection
CRUD templates
Pythonic syntax

take a header file with POD struct


typedef unsigned long long int MyUInt;
namespace sn {
	namespace example {
		struct Record {
			MyUInt               field_01;
			char                 field_02;
			double            field_03[3];
			other::Record field_04[4];
		};
	}
}

typedefs are fine
nested namespace are OK
mapped to : H5T_NATIVE_CHAR
H5Tarray_create(H5T_NATIVE_DOUBLE,1, ... )
first `other::Record` is parsed: type_hid_t = ...
then the generated type is used: H5Tarray_create(type_hid_t, ...)

write your program

write your cpp program as if `generated.h` were already written 
#include "some_header_file.h"
#include <h5cpp/core>
	#include "generated.h"
#include <h5cpp/io>
int main(){
	std::vector<sn::example::Record> stream =
		...
	h5::fd_t fd = h5::create("example.h5",H5F_ACC_TRUNC);
	h5::pt_t pt = h5::create<sn::example::Record>(
		fd, "stream of struct",
		h5::max_dims{H5S_UNLIMITED,7}, h5::chunk{4,7} | h5::gzip{9} );
	...
}

sandwich the not-yet existing `generated.h`
write the TU translation unit as usual
using the POD type with one of the H5CPP CRUD like operators:
h5::create | h5::write | h5::read | h5::append | h5::acreate | h5::awrite | h5::aread
will trigger the `h5cpp` compiler to generate code

A header file with HDF5 Compound Type descriptors:

#ifndef H5CPP_GUARD_ErRrk
#define H5CPP_GUARD_ErRrk
namespace h5{
    template<> hid_t inline register_struct(){
        hsize_t at_00_[] ={7};            hid_t at_00 = H5Tarray_create(H5T_NATIVE_FLOAT,1,at_00_);
        hsize_t at_01_[] ={3};            hid_t at_01 = H5Tarray_create(H5T_NATIVE_DOUBLE,1,at_01_);
        hid_t ct_00 = H5Tcreate(H5T_COMPOUND, sizeof (sn::typecheck::Record));
        H5Tinsert(ct_00, "_char",	HOFFSET(sn::typecheck::Record,_char),H5T_NATIVE_CHAR);
		...
		H5Tclose(at_03); H5Tclose(at_04); H5Tclose(at_05); 
        return ct_02;
    };
}
H5CPP_REGISTER_STRUCT(sn::example::Record);
#endif

random include guards
within namespace
template specialization for h5::operators
compound types are recursively created
calls the template specialization when h5::operator needs it

Attributes:

do the right thing. Here are some examples, and come with an easy to use operator:

h5::ds_t ds = h5::write(fd,"some dataset with attributes", ... );
ds["att_01"] = 42 ;
ds["att_02"] = {1.,3.,4.,5.};
ds["att_03"] = {'1','3','4','5'};
ds["att_04"] = {"alpha", "beta","gamma","..."};
ds["att_05"] = "const char[N]";
ds["att_06"] = u8"const char[N]áééé";
ds["att_07"] = std::string( "std::string");
ds["att_08"] = record; // pod/compound datatype
ds["att_09"] = vector; // vector of pod/compound type
ds["att_10"] = matrix; // linear algebra object

obtain a handle by h5::create | h5::open | h5::write
rank N objects, even compound types when h5cpp compiler used
arrays of various element types
mapped to rank 0 variable length character types

MPI-IO made easy

H5CPP with parallel HDF5
#include <mpi.h>
#include <h5cpp/all>
...
int main(int argc, char** argv) {
    ...
    MPI_Init(NULL, NULL);
    MPI_Comm_size(MPI_COMM_WORLD, &size);
    MPI_Comm_rank(MPI_COMM_WORLD, &rank);
    ...
    { .. /* for H5CPP see next slide */.. }
    ...
    MPI_Barrier(MPI_COMM_WORLD);
    MPI_Finalize();
}

start your program as usual
write the MPI boilerplate

MPI-IO made easy

H5CPP with parallel HDF5
{
std::vector v(nrows);
auto fd = h5::create("hdf5_container_name.h5", H5F_ACC_TRUNC,
               h5::fcpl,
	           h5::mpiio({mpi_comm, mpi_info}) );
h5::write( fd, "dataset", v,
	h5::chunk{nrows,1}, h5::current_dims{nrows,size},h5::count{nrows,1}
	h5::offset{0,rank}, h5::block{1,1}, h5::stride{1,1},
	h5::collective );
}

use code-block to activate H5CPP RAII
arbitrary objects works, as long as you can get a pointer to memory
pass `h5::mpiio` file access property list
control IO with `rank`
set `h5::independent` | `h5::collective`

The pattern

#include <some_header_files>
    int main(int argc, char *argv[]) {
       sn::some_type_t object;
       h5::write( somewhere, object, ... );
       ...
       for( size_t i=0; i<huge_number; i+=batch_size)
          h5::read( somewhere, object, ...);
    }

take a program
with an object, having some memory layout
and intention to save its state to some device
or update the object state from an already saved state

A Closer Look at The Object

can be categorized by

How much space is used in total
Content: homogeneous vs heterogeneous
Number of dimensions or rank
Placement in memory: Contiguous vs Non-contiguous

Type trait utility?

std::is_compound != heterogenious
see: int vs int[N][...]
std::vector<int>().data() points to a contiguous memory, but when T = std::string it doesn't

C++ Type trait based approach may not be the best way to go...

heterogeneous types such as plain old struct, class, union...

require access to field names and may reside in non-contiguous memory. possible layouts:

Table approach, where each row is a heterogeneous datatype leads to fast indexing by rows.
OTOH accessing a single field will lead to I/O bandwidth loss.

Contiguous memory: single block, heterogeneous

Column layout provides efficient access by fields, with the added implementation complexity of each dataset per columns

non-contiguous memory: separate blocks, each homogeneous

 //a vector of pod struct
    struct coo_t {
       size_t row;
       size_t column;
       double value;
    };
    std::vector<coo_t> sparse_matrix;

 // each field of the struct is a vector
    struct csc_t {
       std::vector<size_t> rowind; // row indices
       std::vector<size_t> colptr; // start of new columns
       std::vector<double> values; // nonzero values
    };
    csc_t sparse_matrix;

NOTE: coordinate of points is efficient for incremental construction. Whereas Armadillo C++ arma::SpMat<T> uses Compressed Sparse Column representation

We need Introspection / Reflection method to retrieve field names of C++ class types

C++ mailing list and papers categorized as related to reflection

2019 8 entries

P1390R0 Suggested Reflection TS NB Resolutions Matúš Chochlík, Axel Naumann, and David Sankel
P1390R1 Reflection TS NB comment resolutions: summary and rationale Matúš Chochlík, Axel Naumann, and David Sankel
N4818 Working Draft, C++ Extensions for Reflection David Sankel
P1733R0 User-friendly and Evolution-friendly Reflection: A Compromise David Sankel, Daveed Vandevoorde
P1749R0 Access control for reflection Yehezkel Bernat
P1240R1 Scalable Reflection in C++ Daveed Vandevoorde, Wyatt Childers, Andrew Sutton, Faisal Vali, Daveed Vandevoorde
P1887R0 Typesafe Reflection on attributes Corentin Jabot

2018 13 entries

P0194R5 Static reflection by Matúš Chochlík, Axel Naumann, David Sankel
P0670R2 Static reflection of functions by Matúš Chochlík, Axel Naumann, David Sankel
P0954R0 What do we want to do with reflection? Bjarne Stroustrup
P0993R0 Value-based Reflection Andrew Sutton, Herb Sutter
P0572R2 Static reflection of bit fields Alex Christensen
P0670R3 Function reflection Matúš Chochlík, Axel Naumann, David Sankel
P1240R0 Scalable Reflection in C++ Andrew Sutton, Faisal Vali, Daveed Vandevoorde

2017 14 entries

P0194R3 Static reflection by Matúš Chochlík, Axel Naumann, David Sankel
P0385R2 Static reflection: Rationale, design and evolution by Matúš Chochlík, Axel Naumann, David Sankel
P0578R0 Static Reflection in a Nutshell by Matúš Chochlík, Axel Naumann, David Sankel
P0590R0 A design static reflection: Andrew Sutton, Herb Sutter
P0598R0 Reflect Through Values Instead of Types by Daveed Vandevoorde

2016 19 entries

P0194R0 Static reflection (revision 4) Matus Chochlik, Axel Naumann
P0255R0 C++ Static Reflection via template pack expansion Cleiton Santoia Silva, Daniel Auresco
P0256R0 C++ Reflection Light Cleiton Santoia Silva
P0327R0 Product types access Vicente J. Botet Escriba
P0341R0 parameter packs outside of templates Mike Spertus
Static reflection: Rationale, design and evolution Matúš Chochlík, Alex Naumann

introspection/reflection is non-trivial, not yet available as a language feature

LLVM/CLANG lib tooling based static reflection

...
    StatementMatcher h5templateMatcher = callExpr( allOf(
       hasDescendant( declRefExpr( to( varDecl().bind("variableDecl")  ) ) ),
       hasDescendant( declRefExpr( to(
          functionDecl( allOf(
            eachOf(
    		hasName("h5::write"), hasName("h5::create"), hasName("h5::read"),
                hasName("h5::append"),
    		hasName("h5::awrite"), hasName("h5::acreate"), hasName("h5::aread")
    	),
    ... ));

identify the relevant nodes
marked by I/O operators
visit the structure in reverse topological order
emit the templates describing the class with fields and types

P0993r0 Value-based Reflection, Andrew Sutton, Herb Sutter:

static reflection is a programming facility that exposes read-only data about entities in a translation unit compile-time values. Static reflection does not require support for runtime compilation since reflection values can be used with existing generative facilities (i.e., templates) or additional generative facilities (i.e., metaprogramming) to produce new code.

dynamic reflection:provides information for navigating source-code data structures at runtime. Language supporting dynamic reflection also tend to make additional facilities available for generating and JIT-compiling new code. Dynamic reflection and code generation are not in the scope of this work.

How about Containers? Let's take a look at N4436 C++ Detection Idiom

It is possible to identify if a container is STL like, provides direct access to its contiguous storage -- as std::vector<T> does, or alternatively iterators for scatter/gather operations

identify if there is direct access to contiguous memory
or iterator for non-contiguous layouts


    template <typename T> using value_type_f = typename T::value_type;
    template <typename T> using data_f = decltype(std::declval <T>().data());
    template <typename T> using size_f = decltype(std::declval <T>().size());
    template <typename T> using begin_f = decltype(std::declval <T>().begin());
    template <typename T> using end_f = decltype(std::declval <T>().end());
    template <typename T> using cbegin_f = decltype(std::declval <T>().cbegin());
    template <typename T> using cend_f = decltype(std::declval <T>().cend());

    template <typename T> using value = compat::detected_or <T, value_type_f, T>;
    template <typename T> using has_value_type = compat::is_detected <value_type_f, T>;
    template <typename T> using has_data = compat::is_detected <data_f, T>;
    template <typename T> using has_direct_access = compat::is_detected <data_f, T>;
    template <typename T> using has_size = compat::is_detected <size_f, T>;
    template <typename T> using has_begin = compat::is_detected <begin_f, T>;
    template <typename T> using has_end = compat::is_detected <end_f, T>;
    template <typename T> using has_cbegin = compat::is_detected <cbegin_f, T>;
    template <typename T> using has_cend = compat::is_detected <cend_f, T>;

    template <typename T> using has_iterator = std::integral_constant <bool, has_begin <T>::value && has_end <T>::value >;
    template <typename T> using has_const_iterator = std::integral_constant <bool, has_cbegin <T>::value && has_cend <T>::value >;

_{credit: WG21 N4436 C++ Detection Idiom by Walter Brown}

C++ Linear Algebra Systems calling BLAS/LAPACK specialized containers

are dedicated category, as they all must provide mechanism to pass/receive data to/from some BLAS system call, however the naming varies from system to system.

The differences can be mitigated with a combination of

type traits
feature detection idiom

library	direct access	vector size
arma	memptr()	n_elem
eigen	data()	size()
blaze	data()	n/a
blitz	data()	size()
itpp	_data()	length()
ublas	data().begin()	n/a
dlib	(0,0)	size()