Relay I/O¶
Conduit Relay I/O provides optional Silo, HDF5, and ADIOS I/O interfaces.
These interfaces can be accessed through a generic path-based API, generic handle class, or through APIs specific to each underlying I/O interface. The specific APIs provide lower level control and allow reuse of handles, which is more efficient for most non-trivial use cases. The generic handle class strikes a balance between usability and efficiency.
Relay I/O Path-based Interface¶
The path-based Relay I/O interface allows you to read and write conduit::Nodes using any enabled I/O interface through a simple path-based (string) API. The underlying I/O interface is selected using the extension of the destination path or an explicit protocol argument.
The conduit_relay
library provides the following methods in the conduit::relay::io
namespace:
relay::io::save
- Saves the contents of the passed Node to a file. Works like a
Node::set
to the file: if the file exists, it is overwritten to reflect contents of the passed Node.relay::io::save_merged
- Merges the contents of the passed Node to a file. Works like a
Node::update
to the file: if the file exists, new data paths are appended, common paths are overwritten, and other existing paths are not changed.relay::io::load
- Loads the contents of a file into the passed Node. Works like a
Node::set
from the contents of the file: if the Node has existing data, it is overwritten to reflect contents of the file.relay::io::load_merged
- Merges the contents of a file into the passed Node. Works like a
Node::update
rom the contents of the file: if the Node has existing data, new data paths are appended, common paths are overwritten, and other existing paths are not changed.
The conduit_relay_mpi_io
library provides the conduit::relay::mpi::io
namespace which includes variants of these methods which take a MPI Communicator. These variants pass the communicator to the underlying I/O interface to enable collective I/O. Relay currently only supports collective I/O for ADIOS.
Relay I/O Path-based Interface Examples¶
Save and Load¶
- C++ Example:
// setup node to save
Node n;
n["a/my_data"] = 1.0;
n["a/b/my_string"] = "value";
std::cout << "\nNode to write:" << std::endl;
n.print();
//save to json using save
conduit::relay::io::save(n,"my_output.json");
//load back from json using load
Node n_load;
conduit::relay::io::load("my_output.json",n_load);
std::cout << "\nLoad result:" << std::endl;
n_load.print();
- Output:
Node to write:
a:
my_data: 1.0
b:
my_string: "value"
Load result:
a:
my_data: 1.0
b:
my_string: "value"
Save Merged¶
- C++ Example:
// setup node to save
Node n;
n["a/my_data"] = 1.0;
n["a/b/my_string"] = "value";
std::cout << "\nNode to write:" << std::endl;
n.print();
//save to hdf5 using save
conduit::relay::io::save(n,"my_output.hdf5");
// append a new path to the hdf5 file using save_merged
Node n2;
n2["a/b/new_data"] = 42.0;
std::cout << "\nNode to append:" << std::endl;
n2.print();
conduit::relay::io::save_merged(n2,"my_output.hdf5");
Node n_load;
// load back from hdf5 using load:
conduit::relay::io::load("my_output.hdf5",n_load);
std::cout << "\nLoad result:" << std::endl;
n_load.print();
- Output:
Node to write:
a:
my_data: 1.0
b:
my_string: "value"
Node to append:
a:
b:
new_data: 42.0
Load result:
a:
my_data: 1.0
b:
my_string: "value"
new_data: 42.0
Load Merged¶
- C++ Example:
// setup node to save
Node n;
n["a/my_data"] = 1.0;
n["a/b/my_string"] = "value";
std::cout << "\nNode to write:" << std::endl;
n.print();
//save to hdf5 using generic i/o save
conduit::relay::io::save(n,"my_output.hdf5");
// append to existing node with data from hdf5 file using load_merged
Node n_load;
n_load["a/b/new_data"] = 42.0;
std::cout << "\nNode to load into:" << std::endl;
n_load.print();
conduit::relay::io::load_merged("my_output.hdf5",n_load);
std::cout << "\nLoad result:" << std::endl;
n_load.print();
- Output:
Node to write:
a:
my_data: 1.0
b:
my_string: "value"
Node to load into:
a:
b:
new_data: 42.0
Load result:
a:
b:
new_data: 42.0
my_string: "value"
my_data: 1.0
Load from Subpath¶
- C++ Example:
// setup node to save
Node n;
n["path/to/my_data"] = 1.0;
std::cout << "\nNode to write:" << std::endl;
n.print();
//save to hdf5 using generic i/o save
conduit::relay::io::save(n,"my_output.hdf5");
// load only a subset of the tree
Node n_load;
conduit::relay::io::load("my_output.hdf5:path/to",n_load);
std::cout << "\nLoad result from 'path/to'" << std::endl;
n_load.print();
- Output:
Node to write:
path:
to:
my_data: 1.0
Load result from 'path/to'
my_data: 1.0
Save to Subpath¶
- C++ Example:
// setup node to save
Node n;
n["my_data"] = 1.0;
std::cout << "\nNode to write to 'path/to':" << std::endl;
n.print();
//save to hdf5 using generic i/o save
conduit::relay::io::save(n,"my_output.hdf5:path/to");
// load only a subset of the tree
Node n_load;
conduit::relay::io::load("my_output.hdf5",n_load);
std::cout << "\nLoad result:" << std::endl;
n_load.print();
- Output:
Node to write to 'path/to':
my_data: 1.0
Load result:
path:
to:
my_data: 1.0
Relay I/O Handle Interface¶
The relay::io::IOHandle
class provides a high level interface to query, read, and modify files.
It provides a generic interface that is more efficient than the path-based interface for protocols like HDF5 which support partial I/O and querying without reading the entire contents of a file. It also supports simpler built-in protocols (conduit_bin, json, etc) that do not support partial I/O for convenience. Its basic contract is that changes to backing (file on disk, etc) are not guaranteed to be reflected until the handle is closed. Relay I/O Handle supports reading AXOM Sidre DataStore Style files. Relay I/O Handle does not yet support Silo or ADIOS.
IOHandle has the following instance methods:
open
- Opens a handle. The underlying I/O interface is selected using the extension of the destination path or an explicit protocol argument. Supports reading and writing by default. Select a different mode by passing an options node that contains a
mode
child with one of the following strings:
rw
read + write (default mode)Supports both read and write operations. Creates file if it does not exist. r
read onlyOnly supports read operations. Throws an Error if you open a non-existing file or on any attempt to write. w
write onlyOnly supports write operations. Throws an Error on any attempt to read. Danger
Note: While you can read from and write to subpaths using a handle, IOHandle does not support opening a file with a subpath (e.g.
myhandle.open("file.hdf5:path/data")
).
read
- Merges the contents from the handle or contents from a subpath of the handle into the passed Node. Works like a
Node::update
from the handle: if the Node has existing data, new data paths are appended, common paths are overwritten, and other existing paths are not changed.write
- Writes the contents of the passed Node to the handle or to a subpath of the handle. Works like a
Node::update
to the handle: if the handle has existing data, new data paths are appended, common paths are overwritten, and other existing paths are not changed.has_path
- Checks if the handle contains a given path.
list_child_names
- Returns a list of the child names at a given path, or an empty list if the path does not exist.
remove
- Removes any data at and below a given path. With HDF5 the space may not be fully reclaimed.
close
- Closes a handle. This is when changes are realized to the backing (file on disc, etc).
Relay I/O Handle Examples¶
- C++ Example:
// setup node with example data to save
Node n;
n["a/data"] = 1.0;
n["a/more_data"] = 2.0;
n["a/b/my_string"] = "value";
std::cout << "\nNode to write:" << std::endl;
n.print();
// save to hdf5 file using the path-based api
conduit::relay::io::save(n,"my_output.hdf5");
// inspect and modify with an IOHandle
conduit::relay::io::IOHandle h;
h.open("my_output.hdf5");
// check for and read a path we are interested in
if( h.has_path("a/data") )
{
Node nread;
h.read("a/data",nread);
std::cout << "\nValue at \"a/data\" = "
<< nread.to_float64()
<< std::endl;
}
// check for and remove a path we don't want
if( h.has_path("a/more_data") )
{
h.remove("a/more_data");
std::cout << "\nRemoved \"a/more_data\""
<< std::endl;
}
// verify the data was removed
if( !h.has_path("a/more_data") )
{
std::cout << "\nPath \"a/more_data\" is no more"
<< std::endl;
}
std::cout << "\nWriting to \"a/c\""
<< std::endl;
// write some new data
n = 42.0;
h.write(n,"a/c");
// find the names of the children of "a"
std::vector<std::string> cld_names;
h.list_child_names("a",cld_names);
// print the names
std::cout << "\nChildren of \"a\": ";
std::vector<std::string>::const_iterator itr;
for (itr = cld_names.begin();
itr < cld_names.end();
++itr)
{
std::cout << "\"" << *itr << "\" ";
}
std::cout << std::endl;
Node nread;
// read the entire contents
h.read(nread);
std::cout << "\nRead Result:" << std::endl;
nread.print();
- Output:
Node to write:
a:
data: 1.0
more_data: 2.0
b:
my_string: "value"
Value at "a/data" = 1
Removed "a/more_data"
Path "a/more_data" is no more
Writing to "a/c"
Children of "a": "data" "b" "c"
Read Result:
a:
data: 1.0
b:
my_string: "value"
c: 42.0
- Python Example:
import conduit
import conduit.relay.io
n = conduit.Node()
n["a/data"] = 1.0
n["a/more_data"] = 2.0
n["a/b/my_string"] = "value"
print("\nNode to write:")
print(n)
# save to hdf5 file using the path-based api
conduit.relay.io.save(n,"my_output.hdf5")
# inspect and modify with an IOHandle
h = conduit.relay.io.IOHandle()
h.open("my_output.hdf5")
# check for and read a path we are interested in
if h.has_path("a/data"):
nread = conduit.Node()
h.read(nread,"a/data")
print('\nValue at "a/data" = {0}'.format(nread.value()))
# check for and remove a path we don't want
if h.has_path("a/more_data"):
h.remove("a/more_data")
print('\nRemoved "a/more_data"')
# verify the data was removed
if not h.has_path("a/more_data"):
print('\nPath "a/more_data" is no more')
# write some new data
print('\nWriting to "a/c"')
n.set(42.0)
h.write(n,"a/c")
# find the names of the children of "a"
cnames = h.list_child_names("a")
print('\nChildren of "a": {0}'.format(cnames))
nread = conduit.Node()
# read the entire contents
h.read(nread)
print("\nRead Result:")
print(nread)
- Output:
Node to write:
a:
data: 1.0
more_data: 2.0
b:
my_string: "value"
Value at "a/data" = 1.0
Removed "a/more_data"
Path "a/more_data" is no more
Writing to "a/c"
Children of "a": ['data', 'b', 'c']
Read Result:
a:
data: 1.0
b:
my_string: "value"
c: 42.0
- C++ Sidre Basic Example:
// this example reads a sample hdf5 sidre style file
std::string input_fname = relay_test_data_path(
"texample_sidre_basic_ds_demo.sidre_hdf5");
// open our sidre file for read with an IOHandle
conduit::relay::io::IOHandle h;
h.open(input_fname,"sidre_hdf5");
// find the names of the children at the root
std::vector<std::string> cld_names;
h.list_child_names(cld_names);
// print the names
std::cout << "\nChildren at root: ";
std::vector<std::string>::const_iterator itr;
for (itr = cld_names.begin();
itr < cld_names.end();
++itr)
{
std::cout << "\"" << *itr << "\" ";
}
Node nread;
// read the entire contents
h.read(nread);
std::cout << "\nRead Result:" << std::endl;
nread.print();
- Output:
Children at root: "my_scalars" "my_strings" "my_arrays"
Read Result:
my_scalars:
i64: 1
f64: 10.0
my_strings:
s0: "s0 string"
s1: "s1 string"
my_arrays:
a5_i64: [0, 1, 2, 3, 4]
a5_i64_ext: [0, 1, 2, 3, -5]
b_v1: [1.0, 1.0, 1.0]
b_v2: [2.0, 2.0, 2.0]
- Python Sidre Basic Example:
import conduit
import conduit.relay.io
# this example reads a sample hdf5 sidre style file
input_fname = relay_test_data_path("texample_sidre_basic_ds_demo.sidre_hdf5")
# open our sidre file for read with an IOHandle
h = conduit.relay.io.IOHandle()
h.open(input_fname,"sidre_hdf5")
# find the names of the children at the root
cnames = h.list_child_names()
print('\nChildren at root {0}'.format(cnames))
nread = conduit.Node()
# read the entire contents
h.read(nread);
print("Read Result:")
print(nread)
- Output:
Children at root ['my_scalars', 'my_strings', 'my_arrays']
Read Result:
my_scalars:
i64: 1
f64: 10.0
my_strings:
s0: "s0 string"
s1: "s1 string"
my_arrays:
a5_i64: [0, 1, 2, 3, 4]
a5_i64_ext: [0, 1, 2, 3, -5]
b_v1: [1.0, 1.0, 1.0]
b_v2: [2.0, 2.0, 2.0]
- C++ Sidre with Root File Example:
// this example reads a sample hdf5 sidre datastore, grouped by a root file
std::string input_fname = relay_test_data_path(
"out_spio_blueprint_example.root");
// read using the root file
conduit::relay::io::IOHandle h;
h.open(input_fname,"sidre_hdf5");
// find the names of the children at the root
std::vector<std::string> cld_names;
h.list_child_names(cld_names);
// the "root" (/) of the Sidre-based IOHandle to the datastore provides
// access to the root file itself, and all of the data groups
// print the names
std::cout << "\nChildren at root: ";
std::vector<std::string>::const_iterator itr;
for (itr = cld_names.begin();
itr < cld_names.end();
++itr)
{
std::cout << "\"" << *itr << "\" ";
}
Node nroot;
// read the entire root file contents
h.read("root",nroot);
std::cout << "\nRead \"root\" Result:" << std::endl;
nroot.print();
Node nread;
// read all of data group 0
h.read("0",nread);
std::cout << "\nRead \"0\" Result:" << std::endl;
nread.print();
// reset, or trees will blend in this case
nread.reset();
// read a subpath of data group 1
h.read("1/mesh",nread);
std::cout << "\nRead \"1/mesh\" Result:" << std::endl;
nread.print();
- Output:
Children at root: "root" "0" "1" "2" "3"
Read "root" Result:
blueprint_index:
mesh:
state:
number_of_domains: 4
coordsets:
coords:
type: "uniform"
coord_system:
axes:
x:
y:
type: "cartesian"
path: "mesh/coordsets/coords"
topologies:
mesh:
type: "uniform"
coordset: "coords"
path: "mesh/topologies/mesh"
fields:
field:
number_of_components: 1
topology: "mesh"
association: "element"
path: "mesh/fields/field"
rank:
number_of_components: 1
topology: "mesh"
association: "element"
path: "mesh/fields/rank"
file_pattern: "out_spio_blueprint_example/out_spio_blueprint_example_%07d.hdf5"
number_of_files: 4
number_of_trees: 4
protocol:
name: "sidre_hdf5"
version: "0.0"
tree_pattern: "datagroup_%07d"
Read "0" Result:
mesh:
coordsets:
coords:
dims:
i: 3
j: 3
origin:
x: 0.0
y: -10.0
spacing:
dx: 10.0
dy: 10.0
type: "uniform"
topologies:
mesh:
type: "uniform"
coordset: "coords"
fields:
field:
association: "element"
topology: "mesh"
volume_dependent: "false"
values: [0.0, 1.0, 2.0, 3.0]
rank:
association: "element"
topology: "mesh"
values: [0, 0, 0, 0]
Read "1/mesh" Result:
coordsets:
coords:
dims:
i: 3
j: 3
origin:
x: 20.0
y: -10.0
spacing:
dx: 10.0
dy: 10.0
type: "uniform"
topologies:
mesh:
type: "uniform"
coordset: "coords"
fields:
field:
association: "element"
topology: "mesh"
volume_dependent: "false"
values: [0.0, 1.0, 2.0, 3.0]
rank:
association: "element"
topology: "mesh"
values: [1, 1, 1, 1]
- Python Sidre with Root File Example:
import conduit
import conduit.relay.io
# this example reads a sample hdf5 sidre datastore,
# grouped by a root file
input_fname = relay_test_data_path("out_spio_blueprint_example.root")
# open our sidre datastore for read via root file with an IOHandle
h = conduit.relay.io.IOHandle()
h.open(input_fname,"sidre_hdf5")
# find the names of the children at the root
# the "root" (/) of the Sidre-based IOHandle to the datastore provides
# access to the root file itself, and all of the data groups
cnames = h.list_child_names()
print('\nChildren at root {0}'.format(cnames))
nroot = conduit.Node();
# read the entire root file contents
h.read(path="root",node=nroot);
print("Read 'root' Result:")
print(nroot)
nread = conduit.Node();
# read all of data group 0
h.read(path="0",node=nread);
print("Read '0' Result:")
print(nread)
#reset, or trees will blend in this case
nread.reset();
# read a subpath of data group 1
h.read(path="1/mesh",node=nread);
print("Read '1/mesh' Result:")
print(nread)
- Output:
Children at root ['root', '0', '1', '2', '3']
Read 'root' Result:
blueprint_index:
mesh:
state:
number_of_domains: 4
coordsets:
coords:
type: "uniform"
coord_system:
axes:
x:
y:
type: "cartesian"
path: "mesh/coordsets/coords"
topologies:
mesh:
type: "uniform"
coordset: "coords"
path: "mesh/topologies/mesh"
fields:
field:
number_of_components: 1
topology: "mesh"
association: "element"
path: "mesh/fields/field"
rank:
number_of_components: 1
topology: "mesh"
association: "element"
path: "mesh/fields/rank"
file_pattern: "out_spio_blueprint_example/out_spio_blueprint_example_%07d.hdf5"
number_of_files: 4
number_of_trees: 4
protocol:
name: "sidre_hdf5"
version: "0.0"
tree_pattern: "datagroup_%07d"
Read '0' Result:
mesh:
coordsets:
coords:
dims:
i: 3
j: 3
origin:
x: 0.0
y: -10.0
spacing:
dx: 10.0
dy: 10.0
type: "uniform"
topologies:
mesh:
type: "uniform"
coordset: "coords"
fields:
field:
association: "element"
topology: "mesh"
volume_dependent: "false"
values: [0.0, 1.0, 2.0, 3.0]
rank:
association: "element"
topology: "mesh"
values: [0, 0, 0, 0]
Read '1/mesh' Result:
coordsets:
coords:
dims:
i: 3
j: 3
origin:
x: 20.0
y: -10.0
spacing:
dx: 10.0
dy: 10.0
type: "uniform"
topologies:
mesh:
type: "uniform"
coordset: "coords"
fields:
field:
association: "element"
topology: "mesh"
volume_dependent: "false"
values: [0.0, 1.0, 2.0, 3.0]
rank:
association: "element"
topology: "mesh"
values: [1, 1, 1, 1]
Relay I/O HDF5 Interface¶
The Relay I/O HDF5 interface provides methods to read and write Nodes using HDF5 handles. It is also the interface used to implement the path-based and handle I/O interfaces for HDF5. This interface provides more control and allows more efficient reuse of I/O handles. It is only available in C++.
Relay I/O HDF5 Interface Examples¶
Here is a example exercising the basic parts of Relay I/O’s HDF5 interface, for
more detailed documentation see the conduit_relay_io_hdf5_api.hpp
header file.
HDF5 I/O Interface Basics¶
- C++ Example:
// setup node to save
Node n;
n["a/my_data"] = 1.0;
n["a/b/my_string"] = "value";
std::cout << "\nNode to write:" << std::endl;
n.print();
// open hdf5 file and obtain a handle
hid_t h5_id = conduit::relay::io::hdf5_create_file("myoutput.hdf5");
// write data
conduit::relay::io::hdf5_write(n,h5_id);
// close our file
conduit::relay::io::hdf5_close_file(h5_id);
// open our file to read
h5_id = conduit::relay::io::hdf5_open_file_for_read_write("myoutput.hdf5");
// check if a subpath exists
if(conduit::relay::io::hdf5_has_path(h5_id,"a/my_data"))
std::cout << "\nPath 'myoutput.hdf5:a/my_data' exists" << std::endl;
Node n_read;
// read a subpath (Note: read works like `load_merged`)
conduit::relay::io::hdf5_read(h5_id,"a/my_data",n_read);
std::cout << "\nData loaded:" << std::endl;
n_read.print();
// write more data to the file
n.reset();
// write data (appends data, works like `save_merged`)
// the Node tree needs to be compatible with the existing
// hdf5 state, adding new paths is always fine.
n["a/my_data"] = 3.1415;
n["a/b/c"] = 144;
// lists are also supported
n["a/my_list"].append() = 42.0;
n["a/my_list"].append() = 42;
conduit::relay::io::hdf5_write(n,h5_id);
// check if a subpath of a list exists
if(conduit::relay::io::hdf5_has_path(h5_id,"a/my_list/0"))
std::cout << "\nPath 'myoutput.hdf5:a/my_list/0' exists" << std::endl;
// Read the entire tree:
n_read.reset();
conduit::relay::io::hdf5_read(h5_id,n_read);
std::cout << "\nData loaded:" << std::endl;
n_read.print();
// other helpers:
// check if a path is a hdf5 file:
if(conduit::relay::io::is_hdf5_file("myoutput.hdf5"))
std::cout << "\nFile 'myoutput.hdf5' is a hdf5 file" << std::endl;
- Output:
Node to write:
a:
my_data: 1.0
b:
my_string: "value"
Path 'myoutput.hdf5:a/my_data' exists
Data loaded:
1.0
Path 'myoutput.hdf5:a/my_list/0' exists
Data loaded:
a:
my_data: 3.1415
b:
my_string: "value"
c: 144
my_list:
- 42.0
- 42
File 'myoutput.hdf5' is a hdf5 file
HDF5 I/O Options¶
- C++ Example:
Node io_about;
conduit::relay::io::about(io_about);
std::cout << "\nRelay I/O Info and Default Options:" << std::endl;
io_about.print();
Node &hdf5_opts = io_about["options/hdf5"];
// change the default chunking threshold to
// a smaller number to enable compression for
// a small array
hdf5_opts["chunking/threshold"] = 2000;
hdf5_opts["chunking/chunk_size"] = 2000;
std::cout << "\nNew HDF5 I/O Options:" << std::endl;
hdf5_opts.print();
// set options
conduit::relay::io::hdf5_set_options(hdf5_opts);
int num_vals = 5000;
Node n;
n["my_values"].set(DataType::float64(num_vals));
float64 *v_ptr = n["my_values"].value();
for(int i=0; i< num_vals; i++)
{
v_ptr[i] = float64(i);
}
// save using options
std::cout << "\nsaving data to 'myoutput_chunked.hdf5' " << std::endl;
conduit::relay::io::hdf5_save(n,"myoutput_chunked.hdf5");
- Output:
Relay I/O Info and Default Options:
protocols:
json: "enabled"
conduit_json: "enabled"
conduit_base64_json: "enabled"
yaml: "enabled"
conduit_bin: "enabled"
hdf5: "enabled"
sidre_hdf5: "enabled"
conduit_silo: "disabled"
conduit_silo_mesh: "disabled"
adios: "disabled"
options:
hdf5:
compact_storage:
enabled: "true"
threshold: 1024
chunking:
enabled: "true"
threshold: 2000000
chunk_size: 1000000
compression:
method: "gzip"
level: 5
New HDF5 I/O Options:
compact_storage:
enabled: "true"
threshold: 1024
chunking:
enabled: "true"
threshold: 2000
chunk_size: 2000
compression:
method: "gzip"
level: 5
saving data to 'myoutput_chunked.hdf5'
You can verify using h5stat
that the data set was written to the hdf5 file using chunking and
compression.