Introduction
OpenKIM is a curated repository of interatomic potentials and analytics for making classical molecular simulations of materials reliable, reproducible, and accessible. Basically it is an interatomic potential repository on steroids, and with a focus on scientific result reproducibility.
LAMMPS is a classical molecular dynamics code with a focus on materials modeling. It's an acronym for Large-scale Atomic/Molecular Massively Parallel Simulator.
Apptainer/Singularity is the most widely used container system for HPC
In this tutorial, I will summarize the basic usage of OpenKIM enabled LAMMPS Apptainer image in our Carbon cluster.
Reserve an interactive node
Follow the instructions on the Carbon documentation here
srun --ntasks=4 --time=1:00:00 --partition=forever-diamond --oversubscribe --pty bash -i
If available, this will give us 4 CPUs (and all the GPUs) in forever-diamond workstation for 1 hour.
Apptainer shell
apptainer shell command runs a shell within a container. We will use this on lammps-git-nvhpc-forever-diamond.sif to download OpenKIM potential to our user directory.
apptainer shell --nv /share/apps/singularity-containers/lammps-git-nvhpc-forever-diamond.sif
Apptainer>
The --nv command is for accessing GPUs, and it is mandatory for this particular image. You can check out other containers available in Carbon at the directory /share/apps/singularity-containers/
Apptainer> prompt indicates you have successfully obtained a shell. You'll notice that you are still at the directory you have invoked the command, but now it is as if you are using a different OS. Actually, this particular container is running Ubuntu 22.04, and it works a bit different from the Centos5 the Carbon runs.
Create a test input
Within the apptainer shell, you can still manipulate your home directory. Now create a directory called lammps-test, and create a LAMMPS test input there
Apptainer> mkdir lammps-test
Apptainer> cd lammps-test/
Apptainer> pwd
/home/obm/lammps-test
Apptainer> cat <<EOF >lammps-openkim-test.in
> kim init Sim_LAMMPS_MEAM_PascuetFernandez_2015_Al__SM_811588957187_000 metal
boundary p p p
lattice fcc 4.049
region simbox block 0 1 0 1 0 1 units lattice
create_box 1 simbox
create_atoms 1 box
mass 1 26.981539
kim interactions Al
run 0
variable Ec equal (pe/count(all))
print "Cohesive Energy = ${Ec} eV"
EOF
Apptainer> ls
lammps-openkim-test.in
KIM capabilities of LAMMPS are detailed here. In this simple tutorial we just replaced a fairly standard LAMMPS run with equivalent KIM commands
Download the OpenKIM potential
For this simple tutorial, what we need to do next is to add Sim_LAMMPS_MEAM_PascuetFernandez_2015_Al__SM_811588957187_000 potential to our user collection. The KIM command required for this is kim-api-collections-management. This is already set up and in path of our container.
Apptainer> kim-api-collections-management install user Sim_LAMMPS_MEAM_PascuetFernandez_2015_Al__SM_811588957187_000
Downloading.............. Sim_LAMMPS_MEAM_PascuetFernandez_2015_Al__SM_811588957187_000
Consolidate compiler generated dependencies of target Sim_LAMMPS_MEAM_PascuetFernandez_2015_Al__SM_811588957187_000
[100%] Built target Sim_LAMMPS_MEAM_PascuetFernandez_2015_Al__SM_811588957187_000
Install the project...
-- Install configuration: "RelWithDebInfo"
-- Installing: /home/obm/.kim-api/2.2.1+NVHPC.NVHPC.NVHPC.2023-01-14-21-40-28/simulator-models-dir/Sim_LAMMPS_MEAM_PascuetFernandez_2015_Al__SM_811588957187_000/libkim-api-simulator-model.so
-- Set runtime path of "/home/obm/.kim-api/2.2.1+NVHPC.NVHPC.NVHPC.2023-01-14-21-40-28/simulator-models-dir/Sim_LAMMPS_MEAM_PascuetFernandez_2015_Al__SM_811588957187_000/libkim-api-simulator-model.so" to ""
Success!
This will download the potential in your user directory ( /home/obm/.kim-api/ in my case). You don't need to do this again as long as you don't need a new potential!
In order to list available potentials, you can use kim-api-collections-management list. You'll see that the new potential is listed under the user collections.
Apptainer> kim-api-collections-management list
Knowledgebase of Interatomic Models (KIM) --- Model Collections Listing
===============================================================================
kim-api :
kim-api 2.2.1+NVHPC.NVHPC.NVHPC
kim-api-uid :
2.2.1+NVHPC.NVHPC.NVHPC.2023-01-14-21-40-28
kim-api configuration file:
/home/obm/.kim-api/2.2.1+NVHPC.NVHPC.NVHPC.2023-01-14-21-40-28/config
Environment Variables:
-------------------------------------------------------------------------------
KIM_API_CMAKE_PREFIX_DIR:
/usr/local/lammps/kim_build-prefix
KIM_API_CONFIGURATION_FILE:
--empty--
KIM_API_MODEL_DRIVERS_DIR:
--empty--
KIM_API_PORTABLE_MODELS_DIR:
--empty--
KIM_API_SIMULATOR_MODELS_DIR:
--empty--
===============================================================================
Current Working Directory Collection
-------------------------------------------------------------------------------
Model Drivers: '/home/obm/lammps-test'
--empty--
Portable Models: '/home/obm/lammps-test'
--empty--
Simulator Models: '/home/obm/lammps-test'
--empty--
Environment Variable Collection
-------------------------------------------------------------------------------
Model Drivers: --empty--
--empty--
Portable Models: --empty--
--empty--
Simulator Models: --empty--
--empty--
User Collection
-------------------------------------------------------------------------------
Model Drivers: '/home/obm/.kim-api/2.2.1+NVHPC.NVHPC.NVHPC.2023-01-14-21-40-28/model-drivers-dir'
--empty--
Portable Models: '/home/obm/.kim-api/2.2.1+NVHPC.NVHPC.NVHPC.2023-01-14-21-40-28/portable-models-dir'
--empty--
Simulator Models: '/home/obm/.kim-api/2.2.1+NVHPC.NVHPC.NVHPC.2023-01-14-21-40-28/simulator-models-dir'
Sim_LAMMPS_MEAM_PascuetFernandez_2015_Al__SM_811588957187_000
System Collection
-------------------------------------------------------------------------------
Model Drivers: '/usr/local/lammps/kim_build-prefix/libexec/kim-api/../../lib/kim-api/model-drivers'
LennardJones612__MD_414112407348_003
ex_model_driver_P_LJ
ex_model_driver_P_Morse
Portable Models: '/usr/local/lammps/kim_build-prefix/libexec/kim-api/../../lib/kim-api/portable-models'
LennardJones612_UniversalShifted__MO_959249795837_003
LennardJones_Ar
ex_model_Ar_P_LJ
ex_model_Ar_P_MLJ_Fortran
ex_model_Ar_P_Morse
ex_model_Ar_P_Morse_07C
ex_model_Ar_P_Morse_07C_w_Extensions
ex_model_Ar_P_Morse_MultiCutoff
ex_model_Ar_SLJ_MultiCutoff
Simulator Models: '/usr/local/lammps/kim_build-prefix/libexec/kim-api/../../lib/kim-api/simulator-models'
Sim_LAMMPS_LJcut_AkersonElliott_Alchemy_PbAu
You don't need to reserve a workstation to download OpenKIM potentials. You can get an apptainer shell also in Carbon Headnode. OpenKIM will place the potential in your home directory, so it will be accessible in the future from each and every workstation.
Run the test in the apptainer shell
Now we are ready to run the test.
Apptainer> mpirun lmp <lammps-openkim-test.in
LAMMPS (22 Dec 2022)
WARNING: Using I/O redirection is unreliable with parallel runs. Better to use the -in switch to read input files. (src/lammps.cpp:531)
OMP_NUM_THREADS environment is not set. Defaulting to 1 thread. (src/comm.cpp:98)
using 1 OpenMP thread(s) per MPI task
# Using KIM Simulator Model : Sim_LAMMPS_MEAM_PascuetFernandez_2015_Al__SM_811588957187_000
# For Simulator : LAMMPS 12 Dec 2018
# Running on : LAMMPS 22 Dec 2022
#
Lattice spacing in x,y,z = 4.049 4.049 4.049
Created orthogonal box = (0 0 0) to (4.049 4.049 4.049)
1 by 2 by 2 MPI processor grid
Created 4 atoms
using lattice units in orthogonal box = (0 0 0) to (4.049 4.049 4.049)
create_atoms CPU = 0.000 seconds
CITE-CITE-CITE-CITE-CITE-CITE-CITE-CITE-CITE-CITE-CITE-CITE-CITE
Your simulation uses code contributions which should be cited:
- OpenKIM Project: doi:10.1007/s11837-011-0102-6
- OpenKIM potential: https://openkim.org/cite/SM_811588957187_000#item-citation
The log file lists these citations in BibTeX format.
CITE-CITE-CITE-CITE-CITE-CITE-CITE-CITE-CITE-CITE-CITE-CITE-CITE
WARNING: No fixes with time integration, atoms won't move (src/verlet.cpp:60)
Neighbor list info ...
update: every = 1 steps, delay = 0 steps, check = yes
max neighbors/atom: 2000, page size: 100000
master list distance cutoff = 7
ghost atom cutoff = 7
binsize = 3.5, bins = 2 2 2
2 neighbor lists, perpetual/occasional/extra = 2 0 0
(1) pair meam/c, perpetual
attributes: full, newton on
pair build: full/bin/atomonly
stencil: full/bin/3d
bin: standard
(2) pair meam/c, perpetual, half/full from (1)
attributes: half, newton on
pair build: halffull/newton
stencil: none
bin: none
Setting up Verlet run ...
Unit style : metal
Current step : 0
Time step : 0.001
Per MPI rank memory allocation (min/avg/max) = 8.512 | 8.512 | 8.512 Mbytes
Step Temp E_pair E_mol TotEng Press
0 0 -13.439951 0 -13.439951 585.04952
Loop time of 2.995e-06 on 4 procs for 0 steps with 4 atoms
33.4% CPU use with 4 MPI tasks x 1 OpenMP threads
MPI task timing breakdown:
Section | min time | avg time | max time |%varavg| %total
---------------------------------------------------------------
Pair | 0 | 0 | 0 | 0.0 | 0.00
Neigh | 0 | 0 | 0 | 0.0 | 0.00
Comm | 0 | 0 | 0 | 0.0 | 0.00
Output | 0 | 0 | 0 | 0.0 | 0.00
Modify | 0 | 0 | 0 | 0.0 | 0.00
Other | | 2.995e-06 | | |100.00
Nlocal: 1 ave 1 max 1 min
Histogram: 4 0 0 0 0 0 0 0 0 0
Nghost: 287 ave 287 max 287 min
Histogram: 4 0 0 0 0 0 0 0 0 0
Neighs: 39 ave 39 max 39 min
Histogram: 4 0 0 0 0 0 0 0 0 0
FullNghs: 78 ave 78 max 78 min
Histogram: 4 0 0 0 0 0 0 0 0 0
Total # of neighbors = 312
Ave neighs/atom = 78
Neighbor list builds = 0
Dangerous builds = 0
Cohesive Energy = eV
Total wall time: 0:00:00
Now that the test is running, you can exit the apptainer shell by exit, and release forever-diamond by a second exit .Now you should be back to Carbon head node.
Create a job script for further runs
It is not a feasible strategy to run everything interactively, here is an example jobscript to run lammps apptainer from a job script.
#!/bin/bash -l
#SBATCH --partition=forever-diamond
#SBATCH --time=0-00:05:00
#SBATCH --ntasks=4
LAMMPS_RUN_DIRECTORY=/home/obm/lammps-test/
LAMMPS_INPUT=lammps-openkim-test.in
LAMMPS_OUTPUT=lammps-openkim-test.out
run_line="apptainer run --nv '-B$LAMMPS_RUN_DIRECTORY:/host_pwd' --pwd /host_pwd /share/apps/singularity-containers/lammps-git-nvhpc-forever-diamond.sif 'mpirun lmp <$LAMMPS_RUN_DIRECTORY/$LAMMPS_INPUT >$LAMMPS_RUN_DIRECTORY/$LAMMPS_OUTPUT'"
echo $run_line
eval $run_line