A singularity container for LAMMPS with NVHPC from scratch (unsecure root method)

A singularity container for LAMMPS with NVHPC from scratch (unsecure root method)

Building a custom LAMMPS singularity container using "singularity flow".

My local openHPC implementation runs on CentOS 5, and it is showing signs of bitrot recently. One issue I encountered was the outdated openSSL, and the openKIM. For whatever reason openKIM insists on secure connections, which are not supported by the openSSL in CentOS 5. I tried to compile a lot of things to circumvent this, but it was beginning to compromise maintainability of my node image, so I decided to give singularity a go.

The idea of singularity is that you ship the OS with the program. I can imagine how much headache it solves due to complex tangle of dependencies in modern Linux, still I think it is a lot of wasted disk space.

The general procedure I have followed for this can be found here.

I used the "standard development cycle" or "Singularity flow" to create this container. It consists of the following steps:

  • create a writable container (called a sandbox)
  • shell into the container with the --writable option and tinker with it interactively
  • record changes that we like in our definition file
  • rebuild the container from the definition file if we break it
  • rinse and repeat until we are happy with the result
  • rebuild the container from the final definition file as a read-only singularity image format (SIF) image for use in production

WARNING

I am building the container as root user, in my case that is a reasonably safe thing to do, because I am using the "official" Ubuntu container.

In general, it’s a bad idea to build a container as root. If your source is untrusted, you are downloading random code from the internet and running it as root on your machine.

Recent kernels allow --fakeroot option when building containers.

$ singularity build --fakeroot container.sif container.def

Then you can be the root user inside of your container without actually granting singularity elevated privileges on host system. This is a much safer way to build and interact with your container.

Also, I am keeping this container with all the build capabilities, so it is really huge ~12 Gb.

Base with NVHPC

Getting the Container and setting up the Ubuntu development environment

Let's download the official Ubuntu LTS 22.04 first.

singularity build ubuntu22.04.sif docker://ubuntu:latest
singularity inspect --deffile ubuntu22.04.sif

The image is rather small. I used the root user from now on. Please read the WARNING section carefully before proceeding, and use fakeroot if possible.

singularity build --sandbox lammps-forever-diamond ubuntu22.04.sif
singularity shell --nv --writable lammps-forever-diamond/

this will create a folder called "lammps-forever-diamond" which is basically your writable container. The next line gets a shell with nvidia driver passthrough. The --nv flag will:

  • Ensure that the /dev/nvidiaX device entries are available inside the container, so that the GPU cards in the host are accessible.

  • Locate and bind the basic CUDA libraries from the host into the container, so that they are available to the container, and match the kernel GPU driver on the host.

  • Set the LD_LIBRARY_PATH inside the container so that the bound-in version of the CUDA libraries are used by applications run inside the container.

To use the --nv flag to run a CUDA application inside a container you must ensure that:

  • The host has a working installation of the NVIDIA GPU driver, and a matching version of the basic NVIDIA/CUDA libraries. The host does not need to have an X server running, unless you want to run graphical apps from the container.

  • Either a working installation of the nvidia-container-cli tool is available on the PATH when you run singularity, or the NVIDIA libraries are in the system’s library search path.

  • The application inside your container was compiled for a CUDA version, and device capability level, that is supported by the host card and driver.

Since my image uses the nvidia driver, I did not have any issues with this step. Now lets install some basic necessities.

Apptainer> apt install wget curl gcc gfortran libxml2 gpg less vim cmake make cmake-curses-gui git clang unzip xz-utils

Installing NVHPC

This is from the NVIDA NVHPC web page.

curl https://developer.download.nvidia.com/hpc-sdk/ubuntu/DEB-GPG-KEY-NVIDIA-HPC-SDK | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-hpcsdk-archive-keyring.gpg
echo 'deb [signed-by=/usr/share/keyrings/nvidia-hpcsdk-archive-keyring.gpg] https://developer.download.nvidia.com/hpc-sdk/ubuntu/amd64 /' | sudo tee /etc/apt/sources.list.d/nvhpc.list
sudo apt-get update -y
sudo apt-get install -y nvhpc-22-11-cuda-multi

Modifying the environment variables in a singularity container

You are not invoking a singularity container as a user in Linux sense, hence normal initialization procedures (i.e. profile.d) do not work. You'll need to modify Singularity specific initialization scripts. I use the empty 90-environment.sh for my purposes, but probably there are better ways to do this.

Apptainer> vim /.singularity.d/env/90-environment.sh

For NVHPC, I have adapted the module file, and it finally looks like this:

#!/bin/sh
# Copyright (c) Contributors to the Apptainer project, established as
#   Apptainer a Series of LF Projects LLC.
#   For website terms of use, trademark policy, privacy policy and other
#   project policies see https://lfprojects.org/policies
# Copyright (c) 2018-2021, Sylabs Inc. All rights reserved.
# This software is licensed under a 3-clause BSD license. Please consult
# https://github.com/apptainer/apptainer/blob/main/LICENSE.md regarding your
# rights to use or distribute this software.

# Custom environment shell code should follow
nvhome=/opt/nvidia/hpc_sdk
target=Linux_x86_64
version=22.11

nvcudadir=$nvhome/$target/$version/cuda
nvcompdir=$nvhome/$target/$version/compilers
nvmathdir=$nvhome/$target/$version/math_libs
nvcommdir=$nvhome/$target/$version/comm_libs

export NVHPC=$nvhome
export NVHPC_ROOT=$nvhome/$target/$version
export CC=$nvcompdir/bin/nvc
export CXX=$nvcompdir/bin/nvc++
export FC=$nvcompdir/bin/nvfortran
export F90=$nvcompdir/bin/nvfortran
export F77=$nvcompdir/bin/nvfortran
export CPP=cpp

export PATH=$nvcudadir/bin:$PATH
export PATH=$nvcompdir/bin:$PATH
export PATH=$nvcommdir/mpi/bin:$PATH
export PATH=$nvcompdir/extras/qd/bin:$PATH

export LD_LIBRARY_PATH=$nvcudadir/lib64:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=$nvcudadir/extras/CUPTI/lib64:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=$nvcompdir/extras/qd/lib:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=$nvcompdir/lib:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=$nvmathdir/lib64:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=$nvcommdir/mpi/lib:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=$nvcommdir/nccl/lib:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=$nvcommdir/nvshmem/lib:$LD_LIBRARY_PATH


export CPATH=$nvmathdir/include:$CPATH
export CPATH=$nvmath$nvcommdir/mpi/include:$CPATH
export CPATH=$nvmath$nvcommdir/nccl/include:$CPATH
export CPATH=$nvmath$nvcommdir/nvshmem/include:$CPATH
export CPATH=$nvmath$nvcompdir/extras/qd/include/qd:$CPATH

export MANPATH=$nvcompdir/man:$MANPATH

export OPAL_PREFIX=$nvcommdir/mpi

#LAMMPS + openKIM
export PATH=/usr/local/lammps/:/usr/local/lammps/kim_build-prefix/bin/:$PATH
export KIM_API_CMAKE_PREFIX_DIR="/usr/local/lammps/kim_build-prefix"
export PKG_CONFIG_PATH="/usr/local/lammps/kim_build-prefix/lib/pkgconfig:$PKG_CONFIG_PATH"

exit the container, and re-enter using singularity shell --nv --writable lammps-forever-diamond/ and check if nvhpc is working correctly before proceeding to the next step.

LAMMPS

The "stable" image of LAMMPS does not work with NVHPC (due to CTZ builtin issue). Hence I downloaded the git version. You don't need to download it into the container, as you'll have access to the root directory in the host machine, but since I am using the git version, it is less of an hassle to keep it inside the container. 

Apptainer>
cd lammps-forever-diamond/root/
git clone https://github.com/lammps/lammps.git
mv lammps lammps-git
cd lammps-git
mkdir build
cd build
ccmake ../cmake

Or you can use cmake if you are not particulary interested with the curses interface. The following instructions are for the ccmake interface.  

First, press c to configure. It should detect the NVHPC in the system. Then I use the interface to enable/edit following:

//Build and install the LAMMPS shell
BUILD_LAMMPS_SHELL:BOOL=OFF

//Build MPI version
BUILD_MPI:BOOL=ON

//Build with OpenMP support
BUILD_OMP:BOOL=ON

//Build shared library
BUILD_SHARED_LIBS:BOOL=OFF

//Build the testing tree.
BUILD_TESTING:BOOL=ON

//Build and install LAMMPS tools (msi2lmp, binary2txt, chain)
BUILD_TOOLS:BOOL=ON

//Choose the type of build, options are: None Debug Release RelWithDebInfo
// MinSizeRel.
CMAKE_BUILD_TYPE:STRING=RelWithDebInfo

//Enable/Disable color output during build.
CMAKE_COLOR_MAKEFILE:BOOL=ON

//CXX compiler
CMAKE_CXX_COMPILER:FILEPATH=/opt/nvidia/hpc_sdk/Linux_x86_64/22.11/compilers/bin/nvc++

//C compiler
CMAKE_C_COMPILER:FILEPATH=/opt/nvidia/hpc_sdk/Linux_x86_64/22.11/compilers/bin/nvc

//Fortran compiler
CMAKE_Fortran_COMPILER:FILEPATH=/opt/nvidia/hpc_sdk/Linux_x86_64/22.11/compilers/bin/nvfortran

//Host side compiler used by NVCC
CUDA_HOST_COMPILER:FILEPATH=/opt/nvidia/hpc_sdk/Linux_x86_64/22.11/compilers/bin/nvc++


//Download KIM-API from OpenKIM instead of using an already installed
// one
DOWNLOAD_KIM:BOOL=ON

//Download the LATTE library instead of using an already installed
// one
DOWNLOAD_LATTE:BOOL=ON

//Download Plumed package instead of using an already installed
// one
DOWNLOAD_PLUMED:BOOL=ON

//Download the QUIP library instead of using an already installed
// one
DOWNLOAD_QUIP:BOOL=ON


//FFT library for KSPACE package
FFT:STRING=KISS


//API used by GPU package
GPU_API:STRING=cuda

//LAMMPS GPU CUDA SM primary architecture (e.g. sm_60)
GPU_ARCH:STRING=sm_50

//LAMMPS GPU precision
GPU_PREC:STRING=mixed

//Set extra unit tests verbose mode on/off. If on, extra tests
// are included.
KIM_EXTRA_UNITTESTS:STRING=OFF

//MD5 checksum of KIM tarball
KIM_MD5:STRING=ae1ddda2ef7017ea07934e519d023dca

//URL for KIM tarball
KIM_URL:STRING=https://s3.openkim.org/kim-api/kim-api-2.2.1.txz

//Base URL for LAMMPS downloads
LAMMPS_DOWNLOADS_URL:STRING=https://download.lammps.org


//MD5 checksum of PACE evaluator library tarball
PACELIB_MD5:STRING=f418d32b60e531063ac4285bf702b468

//URL for PACE evaluator library sources
PACELIB_URL:STRING=https://github.com/ICAMS/lammps-user-pace/archive/refs/tags/v.2023.01.3.tar.gz

//Build ADIOS Package
PKG_ADIOS:BOOL=OFF

//Build AMOEBA Package
PKG_AMOEBA:BOOL=OFF

//Build ASPHERE Package
PKG_ASPHERE:BOOL=OFF

//Build ATC Package
PKG_ATC:BOOL=OFF

//Build AWPMD Package
PKG_AWPMD:BOOL=OFF

//Build BOCS Package
PKG_BOCS:BOOL=OFF

//Build BODY Package
PKG_BODY:BOOL=ON

//Build BPM Package
PKG_BPM:BOOL=OFF

//Build BROWNIAN Package
PKG_BROWNIAN:BOOL=OFF

//Build CG-DNA Package
PKG_CG-DNA:BOOL=OFF

//Build CG-SPICA Package
PKG_CG-SPICA:BOOL=OFF

//Build CLASS2 Package
PKG_CLASS2:BOOL=OFF

//Build COLLOID Package
PKG_COLLOID:BOOL=OFF

//Build COLVARS Package
PKG_COLVARS:BOOL=OFF

//Build COMPRESS Package
PKG_COMPRESS:BOOL=OFF

//Arguments to supply to pkg-config
PKG_CONFIG_ARGN:STRING=

//pkg-config executable
PKG_CONFIG_EXECUTABLE:FILEPATH=PKG_CONFIG_EXECUTABLE-NOTFOUND

//Build CORESHELL Package
PKG_CORESHELL:BOOL=OFF

//Build DIELECTRIC Package
PKG_DIELECTRIC:BOOL=ON

//Build DIFFRACTION Package
PKG_DIFFRACTION:BOOL=ON

//Build DIPOLE Package
PKG_DIPOLE:BOOL=ON

//Build DPD-BASIC Package
PKG_DPD-BASIC:BOOL=OFF

//Build DPD-MESO Package
PKG_DPD-MESO:BOOL=OFF

//Build DPD-REACT Package
PKG_DPD-REACT:BOOL=OFF

//Build DPD-SMOOTH Package
PKG_DPD-SMOOTH:BOOL=OFF

//Build DRUDE Package
PKG_DRUDE:BOOL=ON

//Build EFF Package
PKG_EFF:BOOL=ON

//Build ELECTRODE Package
PKG_ELECTRODE:BOOL=ON

//Build EXTRA-COMPUTE Package
PKG_EXTRA-COMPUTE:BOOL=ON

//Build EXTRA-DUMP Package
PKG_EXTRA-DUMP:BOOL=ON

//Build EXTRA-FIX Package
PKG_EXTRA-FIX:BOOL=ON

//Build EXTRA-MOLECULE Package
PKG_EXTRA-MOLECULE:BOOL=ON

//Build EXTRA-PAIR Package
PKG_EXTRA-PAIR:BOOL=ON

//Build FEP Package
PKG_FEP:BOOL=OFF

//Build GPU Package
PKG_GPU:BOOL=ON

//Build GRANULAR Package
PKG_GRANULAR:BOOL=OFF

//Build H5MD Package
PKG_H5MD:BOOL=OFF

//Build INTEL Package
PKG_INTEL:BOOL=OFF

//Build INTERLAYER Package
PKG_INTERLAYER:BOOL=ON

//Build KIM Package
PKG_KIM:BOOL=ON

//Build KOKKOS Package
PKG_KOKKOS:BOOL=OFF

//Build KSPACE Package
PKG_KSPACE:BOOL=ON

//Build LATBOLTZ Package
PKG_LATBOLTZ:BOOL=OFF

//Build LATTE Package
PKG_LATTE:BOOL=OFF

//Build LEPTON Package
PKG_LEPTON:BOOL=OFF

//Build MACHDYN Package
PKG_MACHDYN:BOOL=OFF

//Build MANIFOLD Package
PKG_MANIFOLD:BOOL=OFF

//Build MANYBODY Package
PKG_MANYBODY:BOOL=ON

//Build MC Package
PKG_MC:BOOL=ON

//Build MDI Package
PKG_MDI:BOOL=OFF

//Build MEAM Package
PKG_MEAM:BOOL=ON

//Build MESONT Package
PKG_MESONT:BOOL=OFF

//Build MGPT Package
PKG_MGPT:BOOL=OFF

//Build MISC Package
PKG_MISC:BOOL=OFF

//Build ML-HDNNP Package
PKG_ML-HDNNP:BOOL=OFF

//Build ML-IAP Package
PKG_ML-IAP:BOOL=ON

//Build ML-PACE Package
PKG_ML-PACE:BOOL=OFF

//Build ML-POD Package
PKG_ML-POD:BOOL=OFF

//Build ML-QUIP Package
PKG_ML-QUIP:BOOL=OFF

//Build ML-RANN Package
PKG_ML-RANN:BOOL=OFF

//Build ML-SNAP Package
PKG_ML-SNAP:BOOL=ON

//Build MOFFF Package
PKG_MOFFF:BOOL=OFF

//Build MOLECULE Package
PKG_MOLECULE:BOOL=ON

//Build MOLFILE Package
PKG_MOLFILE:BOOL=ON

//Build MPIIO Package
PKG_MPIIO:BOOL=OFF

//Build MSCG Package
PKG_MSCG:BOOL=OFF

//Build NETCDF Package
PKG_NETCDF:BOOL=OFF

//Build OPENMP Package
PKG_OPENMP:BOOL=OFF

//Build OPT Package
PKG_OPT:BOOL=ON

//Build ORIENT Package
PKG_ORIENT:BOOL=ON

//Build PERI Package
PKG_PERI:BOOL=OFF

//Build PHONON Package
PKG_PHONON:BOOL=ON

//Build PLUGIN Package
PKG_PLUGIN:BOOL=OFF

//Build PLUMED Package
PKG_PLUMED:BOOL=ON

//Build POEMS Package
PKG_POEMS:BOOL=OFF

//Build PTM Package
PKG_PTM:BOOL=OFF

//Build PYTHON Package
PKG_PYTHON:BOOL=ON

//Build QEQ Package
PKG_QEQ:BOOL=OFF

//Build QMMM Package
PKG_QMMM:BOOL=OFF

//Build QTB Package
PKG_QTB:BOOL=OFF

//Build REACTION Package
PKG_REACTION:BOOL=OFF

//Build REAXFF Package
PKG_REAXFF:BOOL=ON

//Build REPLICA Package
PKG_REPLICA:BOOL=OFF

//Build RIGID Package
PKG_RIGID:BOOL=OFF

//Build SCAFACOS Package
PKG_SCAFACOS:BOOL=OFF

//Build SHOCK Package
PKG_SHOCK:BOOL=OFF

//Build SMTBQ Package
PKG_SMTBQ:BOOL=OFF

//Build SPH Package
PKG_SPH:BOOL=OFF

//Build SPIN Package
PKG_SPIN:BOOL=OFF

//Build SRD Package
PKG_SRD:BOOL=OFF

//Build TALLY Package
PKG_TALLY:BOOL=OFF

//Build UEF Package
PKG_UEF:BOOL=OFF

//Build VORONOI Package
PKG_VORONOI:BOOL=OFF

//Build VTK Package
PKG_VTK:BOOL=OFF

//Build YAFF Package
PKG_YAFF:BOOL=OFF

//MD5 checksum of PLUMED tarball
PLUMED_MD5:STRING=6bfe72ebdae63dc38a9ca27d9b0e08f8

//Linkage mode for Plumed2 library
PLUMED_MODE:STRING=static

//URL for PLUMED tarball
PLUMED_URL:STRING=https://github.com/plumed/plumed2/releases/download/v2.8.1/plumed-src-2.8.1.tgz

//Path to a library.
PNG_LIBRARY_DEBUG:FILEPATH=PNG_LIBRARY_DEBUG-NOTFOUND

//Path to a library.
PNG_LIBRARY_RELEASE:FILEPATH=PNG_LIBRARY_RELEASE-NOTFOUND

//Path to a file.
PNG_PNG_INCLUDE_DIR:PATH=PNG_PNG_INCLUDE_DIR-NOTFOUND

LAMMPS bugs with NVHPC

  • nvcc fatal   : Unsupported NVHPC compiler found. nvc++ is the only NVHPC compiler that is supported. Solution: "ccmake->advanced->change CUDA compiler from nvc to nvc++"
  • nvfortran-Error-Unknown switch: -std=gnu Solution: vim build/examples/simulators/simulator-model-example/CMakeFiles/simulator-model-example-fortran.dir/flags.make remove -std=gnu

Environment script

The environment variables go in .singularity.d/env/90-environment.sh 

nvhome=/opt/nvidia/hpc_sdk
target=Linux_x86_64
version=22.11

nvcudadir=$nvhome/$target/$version/cuda
nvcompdir=$nvhome/$target/$version/compilers
nvmathdir=$nvhome/$target/$version/math_libs
nvcommdir=$nvhome/$target/$version/comm_libs

export NVHPC=$nvhome
export NVHPC_ROOT=$nvhome/$target/$version
export CC=$nvcompdir/bin/nvc
export CXX=$nvcompdir/bin/nvc++
export FC=$nvcompdir/bin/nvfortran
export F90=$nvcompdir/bin/nvfortran
export F77=$nvcompdir/bin/nvfortran
export CPP=cpp

export PATH=$nvcudadir/bin:$PATH
export PATH=$nvcompdir/bin:$PATH
export PATH=$nvcommdir/mpi/bin:$PATH
export PATH=$nvcompdir/extras/qd/bin:$PATH

export LD_LIBRARY_PATH=$nvcudadir/lib64:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=$nvcudadir/extras/CUPTI/lib64:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=$nvcompdir/extras/qd/lib:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=$nvcompdir/lib:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=$nvmathdir/lib64:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=$nvcommdir/mpi/lib:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=$nvcommdir/nccl/lib:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=$nvcommdir/nvshmem/lib:$LD_LIBRARY_PATH


export CPATH=$nvmathdir/include:$CPATH
export CPATH=$nvmath$nvcommdir/mpi/include:$CPATH
export CPATH=$nvmath$nvcommdir/nccl/include:$CPATH
export CPATH=$nvmath$nvcommdir/nvshmem/include:$CPATH
export CPATH=$nvmath$nvcompdir/extras/qd/include/qd:$CPATH

export MANPATH=$nvcompdir/man:$MANPATH

export OPAL_PREFIX=$nvcommdir/mpi

export PATH=/usr/local/lammps/:/usr/local/lammps/kim_build-prefix/bin/:$PATH
export KIM_API_CMAKE_PREFIX_DIR="/usr/local/lammps/kim_build-prefix"
export PKG_CONFIG_PATH="/usr/local/lammps/kim_build-prefix/lib/pkgconfig:$PKG_CONFIG_PATH"

Creating the production sif from the sandbox

Exit from the sandbox and run

singularity build lammps-git-nvhpc-forever-diamond lammps-forever-diamond

 

Tags
Singularity
LAMMPS
NVHPC
HPC
singularity flow
apptainer
English