ITS is actively responding to the COVID-19 situation and making resources available for you. Learn more here.

Page tree

Overview

LAMMPS can be run using four main classes of resources:

  • small jobs using CPU cores in a single computer,
  • small jobs using CPU cores and 1-2 GPUs in a single computer,
  • large jobs using CPU cores in multiple computers connected over an Infiniband network, and
  • large jobs using CPU cores and 1-2 GPUs per computer in multiple computers connected over an Infiniband network.

Though LAMMPS can use multiple computers connected over an Ethernet network, it is not recommended on the HPC, since the HPC processor performance is bottlenecked by the Ethernet bandwidth and latency.

Input files for all the examples below are from the HECBioSim project's benchmark suite.

Refer to the Slurm Quick Start User Guide for more information on Slurm scripts.


Single-computer, non-GPU LAMMPS Job

Working from the HECBioSim 20k atom model, edit the file bench.in to ensure that the variable new is set to a value:

Section of LAMMPS input file
variable        new string "20k_multicore"

Create a Slurm job script named lammps_multicore.sh to run the job:

lammps_multicore.sh
#!/bin/bash
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=28

INPUT=bench.in
OUTPUT=bench.out
module load lammps
mpirun -np ${SLURM_NTASKS} lmp_mpi -in ${INPUT} -log ${OUTPUT}

Submit the job from the login node with the command sbatch lammps_multicore.sh, and when the job completes, you should have several new files named the same as your new variable setting (20k_multicore, in this case) and bench.out, containing the LAMMPS output.

Final directory listing for LAMMPS multicore job
[renfro@login NAMD]$ ls -lt
total 9720
-rw-r--r-- 1 renfro domain users   36460 Sep  1 20:23 bench.out
-rw-r--r-- 1 renfro domain users  470524 Sep  1 20:23 20k_multicore.vel
-rw-r--r-- 1 renfro domain users 2353676 Sep  1 20:23 20k_multicore.dcd
-rw-r--r-- 1 renfro domain users  470524 Sep  1 20:23 20k_multicore.coor
-rw-r--r-- 1 renfro domain users    1488 Sep  1 20:23 20k_multicore.xst
-rw-r--r-- 1 renfro domain users     261 Sep  1 20:23 20k_multicore.xsc
-rw-r--r-- 1 renfro domain users  470524 Sep  1 20:23 20k_multicore.vel.BAK
-rw-r--r-- 1 renfro domain users  470524 Sep  1 20:23 20k_multicore.coor.BAK
-rw-r--r-- 1 renfro domain users     262 Sep  1 20:23 20k_multicore.xsc.BAK
-rw-r--r-- 1 renfro domain users  470524 Sep  1 20:23 20k_multicore.vel.old
-rw-r--r-- 1 renfro domain users  470524 Sep  1 20:23 20k_multicore.coor.old
-rw-r--r-- 1 renfro domain users     262 Sep  1 20:23 20k_multicore.xsc.old
-rw-r--r-- 1 renfro domain users    1680 Sep  1 20:17 FFTW_NAMD_2.12_Linux-x86_64-multicore.txt
-rw-r--r-- 1 renfro domain users       0 Sep  1 20:17 slurm-1031.out
-rw-r----- 1 renfro domain users    1659 Sep  1 20:16 bench.in
-rw-r--r-- 1 renfro domain users     124 Sep  1 20:16 namd_multicore.sh
-rw-r--r-- 1 renfro domain users  470524 Dec  4  2013 relres.vel
-rw-r--r-- 1 renfro domain users     225 Dec  4  2013 relres.xsc
-rw-r--r-- 1 renfro domain users  470524 Dec  4  2013 relres.coor
-rw-r--r-- 1 renfro domain users  187048 Nov 14  2011 par_all27_prot_lipid.prm
-rw-r--r-- 1 renfro domain users 1548870 Nov 14  2011 solvated.pdb
-rw-r--r-- 1 renfro domain users 2013967 Nov 14  2011 solvated.psf

Single-computer, GPU-enabled LAMMPS Job

Working from the HECBioSim 20k atom model, edit the file bench.in to ensure that the variable new is set to a value:

Section of LAMMPS input file
variable        new string "20k_gpu"

Create a Slurm job script named lammps_gpu.sh to run the job:

lammps_gpu.sh
#!/bin/bash
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=28
#SBATCH --partition=gpu
#SBATCH --gres=gpu:2

INPUT=bench.in
OUTPUT=bench.out
module load cuda80/toolkit lammps
mpirun -np 28 lmp_gpu -sf gpu -pk gpu 2 -in ${INPUT} -log ${OUTPUT}

Submit the job from the login node with the command sbatch lammps_gpu.sh, and when the job completes, you should have several new files named the same as your new variable setting (20k_gpu, in this case) and bench.out, containing the LAMMPS output.

Final directory listing for LAMMPS GPU job
[renfro@login NAMD]$ ls -lt
total 2104540
-rw-r--r-- 1 renfro domain users   38291 Sep  1 21:26 bench.out
-rw-r--r-- 1 renfro domain users  470524 Sep  1 21:26 20k_gpu.vel
-rw-r--r-- 1 renfro domain users 2353676 Sep  1 21:26 20k_gpu.dcd
-rw-r--r-- 1 renfro domain users  470524 Sep  1 21:26 20k_gpu.coor
-rw-r--r-- 1 renfro domain users    1479 Sep  1 21:26 20k_gpu.xst
-rw-r--r-- 1 renfro domain users     261 Sep  1 21:26 20k_gpu.xsc
-rw-r--r-- 1 renfro domain users  470524 Sep  1 21:26 20k_gpu.vel.BAK
-rw-r--r-- 1 renfro domain users  470524 Sep  1 21:26 20k_gpu.coor.BAK
-rw-r--r-- 1 renfro domain users     262 Sep  1 21:26 20k_gpu.xsc.BAK
-rw-r--r-- 1 renfro domain users  470524 Sep  1 21:26 20k_gpu.vel.old
-rw-r--r-- 1 renfro domain users  470524 Sep  1 21:26 20k_gpu.coor.old
-rw-r--r-- 1 renfro domain users     265 Sep  1 21:26 20k_gpu.xsc.old
-rw-r--r-- 1 renfro domain users    1681 Sep  1 21:24 FFTW_NAMD_2.12_Linux-x86_64-multicore-CUDA.txt
-rw-r--r-- 1 renfro domain users       0 Sep  1 21:23 slurm-1033.out
-rw-r----- 1 renfro domain users    1653 Sep  1 21:20 bench.in
-rw-r--r-- 1 renfro domain users     269 Sep  1 21:20 namd_gpu.sh
-rw-r--r-- 1 renfro domain users  470524 Dec  4  2013 relres.vel
-rw-r--r-- 1 renfro domain users     225 Dec  4  2013 relres.xsc
-rw-r--r-- 1 renfro domain users  470524 Dec  4  2013 relres.coor
-rw-r--r-- 1 renfro domain users  187048 Nov 14  2011 par_all27_prot_lipid.prm
-rw-r--r-- 1 renfro domain users 1548870 Nov 14  2011 solvated.pdb
-rw-r--r-- 1 renfro domain users 2013967 Nov 14  2011 solvated.psf

Multi-computer, Infiniband, non-GPU LAMMPS Job

Working from the HECBioSim 20k atom model, edit the file bench.in to ensure that the variable new is set to a value:

Section of LAMMPS input file
variable        new string "20k_infiniband"

Create a Slurm job script named lammps_infiniband.sh to run the job:

lammps_infiniband.sh
#!/bin/bash
#SBATCH --nodes=5
#SBATCH --ntasks-per-node=28

INPUT=bench.in
OUTPUT=bench.out
module load lammps
mpirun --mca btl ^tcp -np ${SLURM_NTASKS} \
    lmp_mpi -in ${INPUT} -log ${OUTPUT}

Submit the job from the login node with the command sbatch lammps_infiniband.sh, and when the job completes, you should have several new files named the same as your new variable setting (20k_infiniband, in this case) and bench.out, containing the LAMMPS output.

Final directory listing for LAMMPS Infiniband job
[renfro@login NAMD]$ ls -lt
total 2104544
-rw-r--r-- 1 renfro domain users   38258 Sep  1 21:53 bench.out
-rw-r--r-- 1 renfro domain users  470524 Sep  1 21:53 20k_infiniband.vel
-rw-r--r-- 1 renfro domain users 2353676 Sep  1 21:53 20k_infiniband.dcd
-rw-r--r-- 1 renfro domain users  470524 Sep  1 21:53 20k_infiniband.coor
-rw-r--r-- 1 renfro domain users    1485 Sep  1 21:53 20k_infiniband.xst
-rw-r--r-- 1 renfro domain users     264 Sep  1 21:53 20k_infiniband.xsc
-rw-r--r-- 1 renfro domain users  470524 Sep  1 21:53 20k_infiniband.vel.BAK
-rw-r--r-- 1 renfro domain users  470524 Sep  1 21:53 20k_infiniband.coor.BAK
-rw-r--r-- 1 renfro domain users     265 Sep  1 21:53 20k_infiniband.xsc.BAK
-rw-r--r-- 1 renfro domain users     262 Sep  1 21:53 20k_infiniband.xsc.old
-rw-r--r-- 1 renfro domain users  470524 Sep  1 21:53 20k_infiniband.vel.old
-rw-r--r-- 1 renfro domain users  470524 Sep  1 21:53 20k_infiniband.coor.old
-rw-r--r-- 1 renfro domain users    1682 Sep  1 21:49 FFTW_NAMD_2.12_Linux-x86_64-ibverbs-smp.txt
-rw-r--r-- 1 renfro domain users     190 Sep  1 21:49 nodelist.1034
-rw-r--r-- 1 renfro domain users       0 Sep  1 21:49 slurm-1034.out
-rw-r--r-- 1 renfro domain users     429 Sep  1 21:49 namd_infiniband.sh
-rw-r----- 1 renfro domain users    1660 Sep  1 21:47 bench.in
-rw-r--r-- 1 renfro domain users  470524 Dec  4  2013 relres.vel
-rw-r--r-- 1 renfro domain users     225 Dec  4  2013 relres.xsc
-rw-r--r-- 1 renfro domain users  470524 Dec  4  2013 relres.coor
-rw-r--r-- 1 renfro domain users  187048 Nov 14  2011 par_all27_prot_lipid.prm
-rw-r--r-- 1 renfro domain users 1548870 Nov 14  2011 solvated.pdb
-rw-r--r-- 1 renfro domain users 2013967 Nov 14  2011 solvated.psf

Multi-computer, Infiniband, GPU-enabled LAMMPS Job

Working from the HECBioSim 20k atom model, edit the file bench.in to ensure that the variable new is set to a value:

Section of LAMMPS input file
variable        new string "20k_infiniband_gpu"

Create a Slurm job script named lammps_infiniband_gpu.sh to run the job:

lammps_infiniband_gpu.sh
#!/bin/bash
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=28
#SBATCH --partition=gpu
#SBATCH --gres=gpu:2
INPUT=bench.in
OUTPUT=bench.out
module load cuda80/toolkit lammps
mpirun --mca btl ^tcp -np ${SLURM_NTASKS} \
    lmp_gpu -sf gpu -pk gpu 2 -in ${INPUT} -log ${OUTPUT}

Submit the job from the login node with the command sbatch lammps_infiniband_gpu.sh, and when the job completes, you should have several new files named the same as your new variable setting (20k_infiniband_gpu, in this case) and bench.out, containing the LAMMPS output.

Final directory listing for LAMMPS Infiniband GPU job
[renfro@login NAMD]$ ls -lt
total 9740
-rw-r--r-- 1 renfro domain users   41540 Sep  2 13:02 bench.out
-rw-r--r-- 1 renfro domain users  470524 Sep  2 13:02 20k_infiniband_gpu.vel
-rw-r--r-- 1 renfro domain users 2353676 Sep  2 13:02 20k_infiniband_gpu.dcd
-rw-r--r-- 1 renfro domain users  470524 Sep  2 13:02 20k_infiniband_gpu.coor
-rw-r--r-- 1 renfro domain users    1467 Sep  2 13:02 20k_infiniband_gpu.xst
-rw-r--r-- 1 renfro domain users     264 Sep  2 13:02 20k_infiniband_gpu.xsc
-rw-r--r-- 1 renfro domain users     265 Sep  2 13:02 20k_infiniband_gpu.xsc.BAK
-rw-r--r-- 1 renfro domain users  470524 Sep  2 13:02 20k_infiniband_gpu.vel.BAK
-rw-r--r-- 1 renfro domain users  470524 Sep  2 13:02 20k_infiniband_gpu.coor.BAK
-rw-r--r-- 1 renfro domain users  470524 Sep  2 13:01 20k_infiniband_gpu.vel.old
-rw-r--r-- 1 renfro domain users  470524 Sep  2 13:01 20k_infiniband_gpu.coor.old
-rw-r--r-- 1 renfro domain users     265 Sep  2 13:01 20k_infiniband_gpu.xsc.old
-rw-r--r-- 1 renfro domain users    1681 Sep  2 12:58 FFTW_NAMD_2.12_Linux-x86_64-ibverbs-smp-CUDA.txt
-rw-r--r-- 1 renfro domain users      82 Sep  2 12:58 nodelist.1046
-rw-r--r-- 1 renfro domain users       0 Sep  2 12:58 slurm-1046.out
-rw-r----- 1 renfro domain users    1664 Sep  2 12:56 bench.in
-rw-r--r-- 1 renfro domain users     555 Sep  2 12:38 namd_infiniband_gpu.sh
-rw-r--r-- 1 renfro domain users  470524 Dec  4  2013 relres.vel
-rw-r--r-- 1 renfro domain users     225 Dec  4  2013 relres.xsc
-rw-r--r-- 1 renfro domain users  470524 Dec  4  2013 relres.coor
-rw-r--r-- 1 renfro domain users  187048 Nov 14  2011 par_all27_prot_lipid.prm
-rw-r--r-- 1 renfro domain users 1548870 Nov 14  2011 solvated.pdb
-rw-r--r-- 1 renfro domain users 2013967 Nov 14  2011 solvated.psf

Improving Performance for LAMMPS Jobs

Please read the LAMMPS documentation section Accelerating LAMMPS performance for more information about benchmarking LAMMPS and improving your job runtimes.

Though it's assumed that adding more processors to a given job will automatically increase performance, various model sizes will reach peak performance with different processor counts. Additionally, simply adding minimal GPU support to a input file might not improve performance without more extensive changes.


How helpful was this information?

Your Rating: Results: 1 Star2 Star3 Star4 Star5 Star 87 rates