H5CPP Documentation

Easy to use HDF5 C++17 compiler assisted templates for HDF5

Hierarchical Data Format prevalent in high performance scientific computing, sits directly on top of sequential or parallel file systems, providing block and stream operations on standardized or custom binary/text objects. Scientific computing platforms such as Python, R, Matlab, Fortran, Julia [and many more...] come with the necessary libraries to read write HDF5 dataset. This edition simplifies interactions with popular linear algebra libraries, provides compiler assisted seamless object persistence, Standard Template Library support.

Templates:

create dataset within an opened hdf5 file

file ::= const h5::fd_t& fd | const std::string& file_path;
dataspace ::= const h5::sp_t& dataspace | const h5::current_dims& current_dim [, const h5::max_dims& max_dims ] |
[,const h5::current_dims& current_dim] , const h5::max_dims& max_dims;
lcpl ::=
template <typename T> h5::ds_t create( file, const std::string& dataset_path, dataspace,
[, const h5::lcpl_t& lcpl] [, const h5::dcpl_t& dcpl] [, const h5::dapl_t& dapl] );

read a dataset and return a reference of the created object

dataset ::= (const h5::fd_t& fd | const std::string& file_path, const std::string& dataset_path ) | const h5::ds_t& ds;
template <typename T> T read( dataset
[, const h5::offset_t& offset] [, const h5::stride_t& stride] [, const h5::count_t& count]
[, const h5::dxpl_t& dxpl ] ) const;
template <typename T> h5::err_t read( dataset, T& ref
[, const h5::offset_t& offset] [, const h5::stride_t& stride] [, const h5::count_t& count]
[, const h5::dxpl_t& dxpl ] ) [noexcept] const;

write dataset into a specified location

dataset ::= (const h5::fd_t& fd | const std::string& file_path, const std::string& dataset_path ) | const h5::ds_t& ds;
template <typename T> h5::err_t write( dataset, const T* ptr
[,const hsize_t* offset] [,const hsize_t* stride] ,const hsize_t* count [, const h5::dxpl_t dxpl ] ) noexcept;
template <typename T> h5::err_t write( dataset, const T& ref
[,const h5::offset_t& offset] [,const h5::stride_t& stride] [,const& h5::dxcpl_t& dxpl] ) [noexept];

append to extendable C++/C struct dataset

#include <h5cpp/core>
#include "your_data_definition.h"
#include <h5cpp/io>
template <typename T> void h5::append(h5::pt_t& ds, const T& ref) [noexcept];

All file and dataset io descriptors implement raii idiom and close underlying resource when going out of scope, and may be seamlessly passed to HDF5 CAPI calls when implicit conversion enabled. Similarly templates can take CAPI hid_t identifiers as arguments where applicable provided conversion policy allows. See conversion policy for details.

how to use:

sudo make install will copy the header files and h5cpp.pc package config file into /usr/local/ or copy them and ship it with your project. There is no other dependency than hdf5 libraries and include files. However to activate the template specialization for any given library you must include that library first then h5cpp. In case the auto detection fails turn template specialization on by defining:

#define [ H5CPP_USE_BLAZE | H5CPP_USE_ARMADILLO | H5CPP_USE_EIGEN3 | H5CPP_USE_UBLAS_MATRIX
| H5CPP_USE_UBLAS_VECTOR | H5CPP_USE_ITPP_MATRIX | H5CPP_USE_ITPP_VECTOR | H5CPP_USE_BLITZ | H5CPP_USE_DLIB | H5CPP_USE_ETL ]

supported classes:

1 T := ([unsigned] ( int8_t | int16_t | int32_t | int64_t )) | ( float | double )
2 S := T | c/c++ struct | std::string
3 ref := std::vector<S>
4  | arma::Row<T> | arma::Col<T> | arma::Mat<T> | arma::Cube<T>
5  | Eigen::Matrix<T,Dynamic,Dynamic> | Eigen::Matrix<T,Dynamic,1> | Eigen::Matrix<T,1,Dynamic>
6  | Eigen::Array<T,Dynamic,Dynamic> | Eigen::Array<T,Dynamic,1> | Eigen::Array<T,1,Dynamic>
7  | blaze::DynamicVector<T,rowVector> | blaze::DynamicVector<T,colVector>
8  | blaze::DynamicVector<T,blaze::rowVector> | blaze::DynamicVector<T,blaze::colVector>
9  | blaze::DynamicMatrix<T,blaze::rowMajor> | blaze::DynamicMatrix<T,blaze::colMajor>
10  | itpp::Mat<T> | itpp::Vec<T>
11  | blitz::Array<T,1> | blitz::Array<T,2> | blitz::Array<T,3>
12  | dlib::Matrix<T> | dlib::Vector<T,1>
13  | ublas::matrix<T> | ublas::vector<T>
14 ptr := T*
15 accept := ref | ptr

In addition to the standard data types offered by BLAS/LAPACK systems and POD struct -s, std::vector also supports std::string data-types mapping N dimensional variable-length C like string HDF5 data-sets to std::vector<std::string> objects.

short examples:

to read/map a 10x5 matrix from a 3D array from location {3,4,1}

#include <armadillo>
#include <h5cpp/all>
...
auto fd = h5::open("some_file.h5", H5F_ACC_RDWR);
/* the RVO arma::Mat<double> object will have the size 10x5 filled*/
try {
/* will drop extents of unit dimension returns a 2D object */
auto M = h5::read<arma::mat>(fd,"path/to/object",
h5::offset{3,4,1}, h5::count{10,1,5}, h5::stride{3,1,1} ,h5::block{2,1,1} );
} catch (const std::runtime_error& ex ){
...
}
// fd closes underlying resource (raii idiom)

to write the entire matrix back to a different file

#include <Eigen/Dense>
#include <h5cpp/all>
h5::fd_t fd = h5::create("some_file.h5",H5F_ACC_TRUNC);
h5::write(fd,"/result",M);

to create an dataset recording a stream of struct into an extendable chunked dataset with GZIP level 9 compression:

#include <h5cpp/core>
#include "your_data_definition.h"
#include <h5cpp/io>
...
auto ds = h5::create<some_type>(fd,"bids", h5::max_dims{H5S_UNLIMITED}, h5::chunk{1000} | h5::gzip{9});

to append records to an HDF5 datastream

#include <h5cpp/core>
#include "your_data_definition.h"
#include <h5cpp/io>
auto fd = h5::create("NYSE high freq dataset.h5");
h5::pt_t pt = h5::packet_table::create<ns::nyse_stock_quote>( fd,
"price_quotes/2018-01-05.qte",h5::max_dims{H5S_UNLIMITED}, h5::chunk{1024} | h5::gzip{9} );
quote_update_t qu;
bool having_a_good_day{true};
while( having_a_good_day ){
try{
recieve_data_from_udp_stream( qu )
h5::append(pt, qu);
} catch ( ... ){
if( cant_fix_connection() )
having_a_good_day = false;
}
}

Requirements:

c++17 capable compiler or above until backporting to c++14 is completed. The following instructions will help you to install and set to default gcc 8.0 on ubuntu 16.04 LTS, whereas this document will help you with details

1 sudo add-apt-repository ppa:jonathonf/gcc-8.1
2 sudo apt-get update
3 sudo apt-get upgrade
4 sudo apt-get install gcc-8 g++-8
5 sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-8 100 --slave /usr/bin/g++ g++ /usr/bin/g++-8
Copyright (c) 2018 vargaconsulting, Toronto,ON Canada steve.nosp@m.n@va.nosp@m.rgaco.nosp@m.nsul.nosp@m.ting..nosp@m.ca