Conduit: Simplified Data Exchange for HPC Simulations
Conduit is an open source project from Lawrence Livermore National Laboratory that provides an intuitive model for describing hierarchical scientific data in C++, C, Fortran, and Python. It is used for data coupling between packages in-core, serialization, and I/O tasks.
Conduit’s Core API provides:
A flexible way to describe hierarchal data:
A JSON-inspired data model for describing hierarchical in-core scientific data.
A sane API to access hierarchal data:
A dynamic API for rapid construction and consumption of hierarchical objects.
Conduit is under active development and targets Linux, OSX, and Windows platforms. The C++ API underpins the other language APIs and currently has the most features. We are still filling out the C, Fortran, and Python APIs.
For more background, please see Presentations.
Conduit was built around the concept that an intuitive in-core data description capability simplifies many other common tasks in the HPC simulation eco-system. To this aim, Conduit’s Core API:
- Provides a runtime focused in-core data description API that does not require repacking or code generation.
- Supports a mix of externally owned and Conduit allocated memory semantics.
Conduit Project Resources¶
Github Source Repo
LLNL Collaboration Zone Bitbucket Server Source Repo (Mirror for LLNL Users)
The conduit library provides Conduit’s core data API. The relay and blueprint libraries provide higher-level services built on top of the core API.
Caveat Emptor: The APIs for relay and blueprint are still in flux.
- Provides Conduit’s Core API in C++ and subsets of Core API in Python, C, and Fortran.
- Optionally depends on Fortran and Python with NumPy
- I/O functionally beyond simple binary, memory mapped, and json-based text file I/O.
- A light-weight web server for REST and WebSocket clients.
- Interfaces for MPI communication using conduit::Node instances as payloads.
- Optionally depends on silo, hdf5, szip and mpi
- Cyrus Harrison (LLNL)
- Brian Ryujin (LLNL)
- Adam Kunen (LLNL)
- Kathleen Biagas (LLNL)
- Eric Brugger (LLNL)
- Aaron Black (LLNL)
- George Zagaris (LLNL)
- Kenny Weiss (LLNL)
- Matt Larsen (LLNL)
- Joe Ciurej (LLNL)
- George Aspesi (Harvey Mudd)
- Justin Bai (Harvey Mudd)
- Rupert Deese (Harvey Mudd)
- Linnea Shin (Harvey Mudd)
In 2014 and 2015 LLNL sponsored a Harvey Mudd Computer Science Clinic project focused on using Conduit in HPC Proxy apps. You can read about more details about the clinic project from this LLNL article: http://computation.llnl.gov/newsroom/hpc-partnership-harvey-mudd-college-and-livermore