Basic Concepts¶
Node basics¶
The Node class is the primary object in conduit.
Think of it as a hierarchical variant object.
import conduit
n = conduit.Node()
n["my"] = "data"
print(n)
my: "data"
The Node class supports hierarchical construction.
n = conduit.Node()
n["my"] = "data";
n["a/b/c"] = "d";
n["a"]["b"]["e"] = 64.0;
print(n)
print("total bytes: {}\n".format(n.total_strided_bytes()))
my: "data"
a:
b:
c: "d"
e: 64.0
total bytes: 15
Borrowing from JSON (and other similar notations), collections of named nodes are called Objects and collections of unnamed nodes are called Lists, all other types are leaves that represent concrete data.
n = conduit.Node()
n["object_example/val1"] = "data"
n["object_example/val2"] = 10
n["object_example/val3"] = 3.1415
for i in range(5):
l_entry = n["list_example"].append()
l_entry.set(i)
print(n)
object_example:
val1: "data"
val2: 10
val3: 3.1415
list_example:
- 0
- 1
- 2
- 3
- 4
You can iterate through a Node’s children.
n = conduit.Node()
n["object_example/val1"] = "data"
n["object_example/val2"] = 10
n["object_example/val3"] = 3.1415
for i in range(5):
l_entry = n["list_example"].append()
l_entry.set(i)
print(n)
for v in n["object_example"].children():
print("{}: {}".format(v.name(),str(v.node())))
print()
for v in n["list_example"].children():
print(v.node())
object_example:
val1: "data"
val2: 10
val3: 3.1415
list_example:
- 0
- 1
- 2
- 3
- 4
val1: "data"
val2: 10
val3: 3.1415
0
1
2
3
4
Behind the scenes, Node instances manage a collection of memory spaces.
n = conduit.Node()
n["my"] = "data"
n["a/b/c"] = "d"
n["a"]["b"]["e"] = 64.0
print(n.info())
mem_spaces:
0x7fa45d800830:
path: "my"
type: "allocated"
bytes: 5
0x7fa45d82caa0:
path: "a/b/c"
type: "allocated"
bytes: 2
0x7fa45d816b10:
path: "a/b/e"
type: "allocated"
bytes: 8
total_bytes_allocated: 15
total_bytes_mmaped: 0
total_bytes_compact: 15
total_strided_bytes: 15
There is no absolute path construct, all paths are fetched relative to the current node (a leading
/ is ignored when fetching). Empty paths names are also ignored, fetching a///b is
equalvalent to fetching a/b.
Bitwidth Style Types¶
When sharing data in scientific codes, knowing the precision of the underlining types is very important.
Conduit uses well defined bitwidth style types (inspired by NumPy) for leaf values. In Python, leaves are provided as NumPy ndarrays.
n = conduit.Node()
n["test"] = numpy.uint32(100)
print(n)
test: 100
Standard Python numeric types will be mapped to bitwidth style types.
n = conduit.Node()
n["test"] = 10
print(n.schema())
{
"test": {"dtype":"int64","number_of_elements": 1,"offset": 0,"stride": 8,"element_bytes": 8,"endianness": "little"}
}
- Supported Bitwidth Style Types:
- signed integers: int8,int16,int32,int64
- unsigned integers: uint8,uint16,uint32,uint64
- floating point numbers: float32,float64
- Conduit provides these types by constructing a mapping for the current platform the from the following C++ types:
- char, short, int, long, long long, float, double, long double
When a set method is called on a leaf Node, if the data passed to the set is compatible with the Node’s Schema the data is simply copied.
Compatible Schemas¶
When passed a compatible Node, Node methods update and update_compatible
allow you to copy data into Node or extend a Node with new data without
changing existing allocations.
Schemas do not need to be identical to be compatible.
You can check if a Schema is compatible with another Schema using the Schema::compatible(Schema &test) method. Here is the criteria for checking if two Schemas are compatible:
- If the calling Schema describes an Object : The passed test Schema must describe an Object and the test Schema’s children must be compatible with the calling Schema’s children that have the same name.
- If the calling Schema describes a List: The passed test Schema must describe a List, the calling Schema must have at least as many children as the test Schema, and when compared in list order each of the test Schema’s children must be compatible with the calling Schema’s children.
- If the calling Schema describes a leaf data type: The calling Schema’s and test Schema’s dtype().id() and dtype().element_bytes() must match, and the calling Schema dtype().number_of_elements() must be greater than or equal than the test Schema’s.
Differences between C++ and Python APIs¶
In Python, Node objects are reference-counted containers that hold C++ Node pointers. This provides a Python API similar to using references in C++. However, you should be aware of some key differences.
This provides similar API in Python to using references in C++, however there are a few key differences to be aware of.
The [] operator is different in that it will return not only Nodes, but numpy arrays depending on the context:
# setup a node with a leaf array
n = conduit.Node()
data = numpy.zeros((5,),dtype=numpy.float64)
n["my/path/to/data"] = data
# this will be an ndarray
my_data = n["my/path/to/data"]
print("== this will be an ndarray == ")
print("data: ", my_data)
print("repr: ",repr(my_data))
print()
# this will be a node
n_my_path = n["my/path"]
print("== this will be a node == ")
print("{node}\n", n_my_path)
print("{schema}\n",n_my_path.schema().to_yaml())
== this will be an ndarray ==
data: [0. 0. 0. 0. 0.]
repr: array([0., 0., 0., 0., 0.])
== this will be a node ==
{node}
to:
data: [0.0, 0.0, 0.0, 0.0, 0.0]
{schema}
to:
data:
dtype: "float64"
number_of_elements: 5
offset: 0
stride: 8
element_bytes: 8
endianness: "little"
If you are expecting a Node, the best way to access a subpath is using
fetch() or fetch_existing():
# setup a node with a leaf array
n = conduit.Node()
data = numpy.zeros((5,),dtype=numpy.float64)
n["my/path/to/data"] = data
# this will be an ndarray
my_data = n["my/path/to/data"]
print("== this will be an ndarray == ")
print("data: ", my_data)
print("repr: ",repr(my_data))
print()
# equiv access via fetch (or fetch_existing)
# first fetch the node and then the array
my_data = n.fetch("my/path/to/data").value()
print("== this will be an ndarray == ")
print("data: ",my_data)
print("repr: ",repr(my_data))
print()
== this will be an ndarray ==
data: [0. 0. 0. 0. 0.]
repr: array([0., 0., 0., 0., 0.])
== this will be an ndarray ==
data: [0. 0. 0. 0. 0.]
repr: array([0., 0., 0., 0., 0.])
We use the const construct in C++ to provide additional seat belts for read-only
style access to Nodes. We don’t provide a similar overall construct in Python, but
the standard methods like fetch_existing() do support these types
of use cases:
# setup a node with a leaf array
n = conduit.Node()
data = numpy.zeros((5,),dtype=numpy.float64)
n["my/path/to/data"] = data
# access via fetch existing
# first fetch the node
n_data = n.fetch_existing("my/path/to/data")
# then the value
my_data = n_data.value()
print("== this will be an ndarray == ")
print("data: ",my_data)
print("repr: ",repr(my_data))
print()
# using fetch_existing,
# if the path doesn't exist - we will get an Exception
try:
n_data = n.fetch_existing("my/path/TYPO/data")
except Exception as e:
print("Here is what went wrong:")
print(e)
== this will be an ndarray ==
data: [0. 0. 0. 0. 0.]
repr: array([0., 0., 0., 0., 0.])
Here is what went wrong:
file: /Users/harrison37/Work/github/llnl/conduit/src/libs/conduit/conduit_node.cpp
line: 13050
message:
Cannot fetch non-existent child "TYPO" from Node(my/path)