Building¶
This page provides details on several ways to build Conduit.
If you are building features that depend on third party libraries we recommend using Spack, or uberenv, which leverages Spack. We also provide a Docker example that leverages Spack.
Getting Started¶
Clone the Conduit repo:
- From Github
git clone --recursive https://github.com/llnl/conduit.git
--recursive
is necessary because we are using a git submodule to pull in BLT (https://github.com/llnl/blt).
If you cloned without --recursive
, you can checkout this submodule using:
cd conduit
git submodule init
git submodule update
Configure a build:
config-build.sh
is a simple wrapper for the cmake call to configure conduit.
This creates a new out-of-source build directory build-debug
and a directory for the install install-debug
.
It optionally includes a host-config.cmake
file with detailed configuration options.
cd conduit
./config-build.sh
Build, test, and install Conduit:
cd build-debug
make -j 8
make test
make install
Build Options¶
The core Conduit library has no dependencies outside of the repo, however Conduit provides optional support for I/O and Communication (MPI) features that require externally built third party libraries.
Conduit’s build system supports the following CMake options:
- BUILD_SHARED_LIBS - Controls if shared (ON) or static (OFF) libraries are built. (default = ON)
- ENABLE_TESTS - Controls if unit tests are built. (default = ON)
- ENABLE_DOCS - Controls if the Conduit documentation is built (when sphinx and doxygen are found ). (default = ON)
- ENABLE_COVERAGE - Controls if code coverage compiler flags are used to build Conduit. (default = OFF)
- ENABLE_PYTHON - Controls if the Conduit Python module is built. (default = OFF)
The Conduit Python module will build for both Python 2 and Python 3. To select a specific Python, set the CMake variable PYTHON_EXECUTABLE to path of the desired python binary. The Conduit Python module requires Numpy. The selected Python instance must provide Numpy, or PYTHONPATH must be set to include a Numpy install compatible with the selected Python install.
- ENABLE_MPI - Controls if the conduit_relay_mpi library is built. (default = OFF)
We are using CMake’s standard FindMPI logic. To select a specific MPI set the CMake variables MPI_C_COMPILER and MPI_CXX_COMPILER, or the other FindMPI options for MPI include paths and MPI libraries.
To run the mpi unit tests on LLNL’s LC platforms, you may also need change the CMake variables MPIEXEC and MPIEXEC_NUMPROC_FLAG, so you can use srun and select a partition. (for an example see: src/host-configs/chaos_5_x86_64.cmake)
Warning
Starting in CMake 3.10, the FindMPI MPIEXEC variable was changed to MPIEXEC_EXECUTABLE. FindMPI will still set MPIEXEC, but any attempt to change it before calling FindMPI with your own cached value of MPIEXEC will not survive, so you need to set MPIEXEC_EXECUTABLE [reference].
- HDF5_DIR - Path to a HDF5 install (optional).
Controls if HDF5 I/O support is built into conduit_relay.
- SILO_DIR - Path to a Silo install (optional).
Controls if Silo I/O support is built into conduit_relay. When used, the following CMake variables must also be set:
- HDF5_DIR - Path to a HDF5 install. (Silo support depends on HDF5)
- BLT_SOURCE_DIR - Path to BLT. (default = “blt”)
Defaults to “blt”, where we expect the blt submodule. The most compelling reason to override is to share a single instance of BLT across multiple projects.
Installation Path Options¶
Conduit’s build system provides an install target that installs the Conduit libraires, headers, python modules, and documentation. These CMake options allow you to control install destination paths:
- CMAKE_INSTALL_PREFIX - Standard CMake install path option (optional).
- PYTHON_MODULE_INSTALL_PREFIX - Path to install Python modules into (optional).
When present and ENABLE_PYTHON is ON, Conduit’s Python modules will be installed to${PYTHON_MODULE_INSTALL_PREFIX}
directory instead of${CMAKE_INSTALL_PREFIX}/python-modules
.
Host Config Files¶
To handle build options, third party library paths, etc we rely on CMake’s initial-cache file mechanism.
cmake -C config_file.cmake
We call these initial-cache files host-config files, since we typically create a file for each platform or specific hosts if necessary.
The config-build.sh
script uses your machine’s hostname, the SYS_TYPE environment variable, and your platform name (via uname) to look for an existing host config file in the host-configs
directory at the root of the conduit repo. If found, it passes the host config file to CMake via the -C command line option.
cmake {other options} -C host-configs/{config_file}.cmake ../
You can find example files in the host-configs
directory.
These files use standard CMake commands. To properly seed the cache, CMake set commands need to specify CACHE
as follows:
set(CMAKE_VARIABLE_NAME {VALUE} CACHE PATH "")
Bootstrapping Third Party Dependencies¶
We use Spack (http://software.llnl.gov/spack) to automate builds of third party dependencies on OSX and Linux. Conduit builds on Windows as well, but there is no automated process to build dependencies necessary to support Conduit’s optional features.
Note
Conduit developers use bootstrap-env.sh
and scripts/uberenv/uberenv.py
to setup third party libraries for Conduit development.
This path uses the Conduit Spack package and extra settings, including Spack compiler and external third party package details for some platforms. For info on how to use the Conduit Spack package see Building Conduit and its Dependencies with Spack.
On OSX and Linux, you can use bootstrap-env.sh
(located at the root of the conduit repo) to help setup your development environment. This script uses scripts/uberenv/uberenv.py
, which leverages Spack to build all of the external third party libraries and tools used by Conduit. Fortran support is optional and all dependencies should build without a fortran compiler. After building these libraries and tools, it writes an initial host-config file and adds the Spack built CMake binary to your PATH so can immediately call the config-build.sh
helper script to configure a conduit build.
#build third party libs using spack
source bootstrap-env.sh
#copy the generated host-config file into the standard location
cp uberenv_libs/`hostname`*.cmake to host-configs/
# run the configure helper script
./config-build.sh
# or you can run the configure helper script and give it the
# path to a host-config file
./config-build.sh uberenv_libs/`hostname`*.cmake
When bootstrap-env.sh
runs uberenv.py
, all command line arguments are forwarded:
python scripts/uberenv/uberenv.py $@
So any options to bootstrap-env.sh
are effectively uberenv.py
options.
Uberenv Options for Building Third Party Dependencies¶
uberenv.py
has a few options that allow you to control how dependencies are built:
Option Description Default –prefix Destination directory uberenv_libs
–spec Spack spec linux: %gcc osx: %clang –compilers-yaml Spack compilers settings file scripts/uberenv/compilers.yaml
-k Ignore SSL Errors False
The -k
option exists for sites where SSL certificate interception undermines fetching
from github and https hosted source tarballs. When enabled, uberenv.py
clones spack using:
git -c http.sslVerify=false clone https://github.com/llnl/spack.git
And passes -k
to any spack commands that may fetch via https.
Default invocation on Linux:
python scripts/uberenv/uberenv.py --prefix uberenv_libs \
--spec %gcc \
--compilers-yaml scripts/uberenv/compilers.yaml
Default invocation on OSX:
python scripts/uberenv/uberenv.py --prefix uberenv_libs \
--spec %clang \
--compilers-yaml scripts/uberenv/compilers.yaml
For details on Spack’s spec syntax, see the Spack Specs & dependencies documentation.
You can edit scripts/uberenv/compilers.yaml
or use the –compilers-yaml option to change the compiler settings
used by Spack. See the Spack Compiler Configuration
documentation for details.
For OSX, the defaults in compilers.yaml
are X-Code’s clang and gfortran from https://gcc.gnu.org/wiki/GFortranBinaries#MacOS.
Note
The bootstrapping process ignores ~/.spack/compilers.yaml
to avoid conflicts
and surprises from a user’s specific Spack settings on HPC platforms.
When run, uberenv.py
checkouts a specific version of Spack from github as spack
in the
destination directory. It then uses Spack to build and install Conduit’s dependencies into
spack/opt/spack/
. Finally, it generates a host-config file {hostname}.cmake
in the
destination directory that specifies the compiler settings and paths to all of the dependencies.
Building Conduit and its Dependencies with Spack¶
As of 1/4/2017, Spack’s develop branch includes a recipe to build and install Conduit.
To install the latest released version of Conduit with all options (and also build all of its dependencies as necessary) run:
spack install conduit
To build and install Conduit’s github master branch run:
spack install conduit@master
The Conduit Spack package provides several variants that customize the options and dependencies used to build Conduit:
Variant Description Default shared Build Conduit as shared libraries ON (+shared) cmake Build CMake with Spack ON (+cmake) python Enable Conduit Python support ON (+python) mpi Enable Conduit MPI support ON (+mpi) hdf5 Enable Conduit HDF5 support ON (+hdf5) silo Enable Conduit Silo support ON (+silo) doc Build Conduit’s Documentation OFF (+docs)
Variants are enabled using +
and disabled using ~
. For example, to build Conduit with the minimum set of options (and dependencies) run:
spack install conduit~python~mpi~hdf5~silo~docs
You can specify specific versions of a dependency using ^
. For Example, to build Conduit with Python 3:
spack install conduit+python ^python@3
Supported CMake Versions¶
We recommend CMake 3.9. We test building Conduit with CMake 3.3.1, 3.8.1 and 3.9.4. Other versions of CMake may work, however CMake 3.4.x to 3.7.x have specific issues with finding and using HDF5 and Python.
Using Conduit in Another Project¶
Under src/examples
there are examples demonstrating how to use Conduit in a CMake-based build system (using-with-cmake
) and via a Makefile (using-with-make
).
Building Conduit in a Docker Container¶
Under src/examples/docker/ubuntu
there is an example Dockerfile
which can be used to create an ubuntu-based docker image with a build of the Conduit. There is also a script that demonstrates how to build a Docker image from the Dockerfile (example_build.sh
) and a script that runs this image in a Docker container (example_run.sh
). The Conduit repo is cloned into the image’s file system at /conduit
, the build directory is /conduit/build-debug
, and the install directory is /conduit/install-debug
.
Notes for Cray systems¶
HDF5 and gtest use runtime features such as dlopen
. Because of this, building static on Cray systems commonly yields the following flavor of compiler warning:
Using 'zzz' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
You can avoid related linking warnings by adding the -dynamic
compiler flag, or by setting the CRAYPE_LINK_TYPE environment variable:
export CRAYPE_LINK_TYPE=dynamic
Shared Memory Maps are read only
on Cray systems, so updates to data using Node::mmap
will not be seen between processes.