Basic Concepts

Node basics

The Node class is the primary object in conduit.

Think of it as a hierarchical variant object.

Node n;
n["my"] = "data";
n.print(); 

my: "data"

The Node class supports hierarchical construction.

Node n;
n["my"] = "data";
n["a/b/c"] = "d";
n["a"]["b"]["e"] = 64.0;
n.print();

std::cout << "total bytes: " << n.total_strided_bytes() << std::endl;

my: "data"
a: 
  b: 
    c: "d"
    e: 64.0

total bytes: 15

Borrowing form JSON (and other similar notations), collections of named nodes are called Objects and collections of unnamed nodes are called Lists, all other types are leaves that represent concrete data.

Node n;
n["object_example/val1"] = "data";
n["object_example/val2"] = 10u;
n["object_example/val3"] = 3.1415;

for(int i = 0; i < 5 ; i++ )
{
    Node &list_entry = n["list_example"].append();
    list_entry.set(i);
}

n.print();

object_example: 
  val1: "data"
  val2: 10
  val3: 3.1415
list_example: 
  - 0
  - 1
  - 2
  - 3
  - 4

You can use a NodeIterator ( or a NodeConstIterator) to iterate through a Node’s children.

Node n;
n["object_example/val1"] = "data";
n["object_example/val2"] = 10u;
n["object_example/val3"] = 3.1415;

for(int i = 0; i < 5 ; i++ )
{
    Node &list_entry = n["list_example"].append();
    list_entry.set(i);
}

n.print();

NodeIterator itr = n["object_example"].children();
while(itr.has_next())
{
    Node &cld = itr.next();
    std::string cld_name = itr.name();
    std::cout << cld_name << ": " << cld.to_string() << std::endl;
}

std::cout << std::endl;

itr = n["list_example"].children();
while(itr.has_next())
{
    Node &cld = itr.next();
    std::cout << cld.to_string() << std::endl;
}

object_example: 
  val1: "data"
  val2: 10
  val3: 3.1415
list_example: 
  - 0
  - 1
  - 2
  - 3
  - 4

val1: "data"
val2: 10
val3: 3.1415

0
1
2
3
4

Behind the scenes, Node instances manage a collection of memory spaces.

Node n;
n["my"] = "data";
n["a/b/c"] = "d";
n["a"]["b"]["e"] = 64.0;

Node ninfo;
n.info(ninfo);
ninfo.print();

mem_spaces: 
  0x7ff3dc9049b0: 
    path: "my"
    type: "allocated"
    bytes: 5
    allocator_id: 0
  0x7ff3dc904d90: 
    path: "a/b/c"
    type: "allocated"
    bytes: 2
    allocator_id: 0
  0x7ff3dc904d80: 
    path: "a/b/e"
    type: "allocated"
    bytes: 8
    allocator_id: 0
total_bytes_allocated: 15
total_bytes_mmaped: 0
total_bytes_compact: 15
total_strided_bytes: 15

There is no absolute path construct, all paths are fetched relative to the current node (a leading / is ignored when fetching). Empty paths names are also ignored, fetching a///b is equalvalent to fetching a/b.

Bitwidth Style Types

When sharing data in scientific codes, knowing the precision of the underlining types is very important.

Conduit uses well defined bitwidth style types (inspired by NumPy) for leaf values.

Node n;
uint32 val = 100;
n["test"] = val;
n.print();
n.print_detailed();

test: 100


{
  "test": 
  {
    "dtype":"uint32",
    "number_of_elements": 1,
    "offset": 0,
    "stride": 4,
    "element_bytes": 4,
    "endianness": "little",
    "value": 100
  }
}

Standard C++ numeric types will be mapped by the compiler to bitwidth style types.

Node n;
int val = 100;
n["test"] = val;
n.print_detailed();

{
  "test": 
  {
    "dtype":"int32",
    "number_of_elements": 1,
    "offset": 0,
    "stride": 4,
    "element_bytes": 4,
    "endianness": "little",
    "value": 100
  }
}
Supported Bitwidth Style Types:
  • signed integers: int8,int16,int32,int64
  • unsigned integers: uint8,uint16,uint32,uint64
  • floating point numbers: float32,float64
Conduit provides these types by constructing a mapping for the current platform the from the following types:
  • char, short, int, long, long long, float, double, long double

When C++11 support is enabled, Conduit’s bitwidth style types will match the C++11 standard bitwidth types defined in <cstdint>.

When a set method is called on a leaf Node, if the data passed to the set is compatible with the Node’s Schema the data is simply copied.

Compatible Schemas

When passed a compatible Node, Node methods update and update_compatible allow you to copy data into Node or extend a Node with new data without changing existing allocations.

Schemas do not need to be identical to be compatible.

You can check if a Schema is compatible with another Schema using the Schema::compatible(Schema &test) method. Here is the criteria for checking if two Schemas are compatible:

  • If the calling Schema describes an Object : The passed test Schema must describe an Object and the test Schema’s children must be compatible with the calling Schema’s children that have the same name.
  • If the calling Schema describes a List: The passed test Schema must describe a List, the calling Schema must have at least as many children as the test Schema, and when compared in list order each of the test Schema’s children must be compatible with the calling Schema’s children.
  • If the calling Schema describes a leaf data type: The calling Schema’s and test Schema’s dtype().id() and dtype().element_bytes() must match, and the calling Schema dtype().number_of_elements() must be greater than or equal than the test Schema’s.

Node References

For most uses cases in C++, Node references are the best solution to build and manipulate trees. They allow you to avoid expensive copies and pass around sub-trees without worrying about valid pointers.

////////////////////////////////////////////////////////////
// In C++, use Node references!
////////////////////////////////////////////////////////////
// Using Node references is common (good) pattern!

// setup a node
Node root;
// set data in hierarchy 
root["my/nested/path"] = 0.0;
// display the contents
root.print();

// Get a ref to the node in the tree
Node &data = root["my/nested/path"];
// change the value
data = 42.0;
// display the contents
root.print();

my: 
  nested: 
    path: 0.0


my: 
  nested: 
    path: 42.0

In C++ the Node assignment operator that takes a Node input is really an alias to set. That is, if follows set (deep copy) semantics.

////////////////////////////////////////////////////////////
// C++ anti-pattern to avoid: copy instead of reference
////////////////////////////////////////////////////////////

// setup a node
Node root;
// set data in hierarchy 
root["my/nested/path"] = 0.0;

// display the contents
root.print();

// In this case, notice we aren't using a reference.
// This creates a copy, disconnected from the orignal tree!
// This is probably not what you are looking for ...
Node data = root["my/nested/path"];
// change the value
data = 42.0;

// display the contents
root.print();

my: 
  nested: 
    path: 0.0


my: 
  nested: 
    path: 0.0

const Nodes

If you aren’t careful, the ability to easily create dynamic trees can also undermine your process to consume them. For example, asking for an expected but non-existent path will return a reference to an empty Node. Surprise!

Methods like fetch_existing allow you to be more explicit when asking for expected data. In C++, const Node references are also common way to process trees in an read-only fashion. const methods will not modify the tree structure, so if you ask for a non-existent path, you will receive an error instead of reference to an empty Node.

// with non-const references, you can modify the node, 
// leading to surprises in cases were read-only 
// validation and processing is intended
void important_suprise(Node &data)
{
    // if this doesn't exist, we will get a new empty node
    // Note: we could also ask if the path exists via Node:has_path()
    int val = data["my/important/data"].to_int();
    std::cout << "\n==> important: " << val << std::endl;
}

// with const references,  the api provides checks
// that help
void important(const Node &data)
{
    // if this doesn't exist, const access will trigger exception here
    // Note: we could also ask if the path exists via Node:has_path()
    int val = data["my/important/data"].to_int();
    std::cout << "\n==> important: " << val << std::endl;
}
////////////////////////////////////////////////////////////
// In C++, leverage const refs for processing existing nodes
////////////////////////////////////////////////////////////

// setup a node
Node n1;
n1["my/important/but/mistyped/path/to/data"] = 42.0;

std::cout << "== n1 == " << std::endl;
n1.print();

// method with non-const arg drives on ...
try
{
    important_suprise(n1);
}
catch(conduit::Error &e)
{
    e.print();
}

// check n1, was it was modified ( yes ... )
std::cout << "n1 after calling `important_suprise`" << std::endl;
n1.print();

Node n2;
n2["my/important/but/mistyped/path/to/data"] = 42.0;

std::cout << "== n2 == " << std::endl;
n2.print();

// method with const arg lets us know, and also makes sure 
// the node structure isn't modified
try
{
    important(n2);
}
catch(conduit::Error &e)
{
    e.print();
}

// check n2, was it was modified ( no ... )
std::cout << "n2 after calling `important`" << std::endl;
n2.print();
== n1 == 

my: 
  important: 
    but: 
      mistyped: 
        path: 
          to: 
            data: 42.0


==> important: 0
n1 after calling `important_suprise`

my: 
  important: 
    but: 
      mistyped: 
        path: 
          to: 
            data: 42.0
    data: 

== n2 == 

my: 
  important: 
    but: 
      mistyped: 
        path: 
          to: 
            data: 42.0


file: /Users/harrison37/Work/github/llnl/conduit/src/libs/conduit/conduit_node.cpp
line: 14182
message: 
Cannot fetch non-existent child "data" from Node(my/important)

n2 after calling `important`

my: 
  important: 
    but: 
      mistyped: 
        path: 
          to: 
            data: 42.0