ITS is actively responding to the COVID-19 situation and making resources available for you. Learn more here.

Page tree

Overview

This document shows two different Towhee inputs and Slurm job scripts: one for small jobs using a single CPU core, and a second for a set of jobs using multiple CPU cores on multiple compute nodes. Towhee does not support true parallel computation on a single job: its parallel support is limited to managing a set of otherwise independent single-core jobs.

Refer to the Slurm Quick Start User Guide for more information on Slurm scripts.

Single-CPU Towhee Job

Create two Towhee input files named towhee_input and towhee_initial. If you don't have your own, you can copy a sample set of files from the Towhee installation to your current directory on the login server:

Copying sample Towhee input files
[renfro@login new]$ cp /cm/shared/apps/towhee/7.2.0/Examples/Canonical_Ensemble/Amber_Villin/towhee_in* .
[renfro@login new]$ ls -al
total 2097156
drwxr-xr-x 2 renfro domain users    85 Sep  1 09:21 .
drwxr-xr-x 5 renfro domain users   111 Sep  1 09:17 ..
-rwxr-xr-x 1 renfro domain users 22113 Sep  1 09:24 towhee_initial
-rwxr-xr-x 1 renfro domain users  2488 Sep  1 09:24 towhee_input
[renfro@login new]$

Create a Slurm job script named towhee-serial.sh to run the job:

towhee-serial.sh
#!/bin/bash
#SBATCH --ntasks=1
#SBATCH --nodes=1

module load towhee
towhee -p none

Submit the job from the login node with the command sbatch towhee-serial.sh, and when the job completes, you should have several new files, including towhee_final, containing the Towhee output.

Final directory listing for Towhee serial job
[renfro@login new]$ ls -lt
total 3146128
-rw-r--r-- 1 renfro domain users   65501 Sep  1 09:28 slurm-984.out
-rw-r--r-- 1 renfro domain users     860 Sep  1 09:28 towhee_citations
-rw-r--r-- 1 renfro domain users   51438 Sep  1 09:28 box_01_step_00000000000010.pdb
-rw-r--r-- 1 renfro domain users       0 Sep  1 09:28 towhee_ostwald
-rw-r--r-- 1 renfro domain users      79 Sep  1 09:28 towhee_pvsrho
-rw-r--r-- 1 renfro domain users   48380 Sep  1 09:28 towhee_final
-rw-r--r-- 1 renfro domain users   51438 Sep  1 09:28 box_01_step_00000000000008.pdb
-rw-r--r-- 1 renfro domain users   51438 Sep  1 09:28 box_01_step_00000000000006.pdb
-rw-r--r-- 1 renfro domain users   51438 Sep  1 09:28 box_01_step_00000000000004.pdb
-rw-r--r-- 1 renfro domain users   51438 Sep  1 09:27 box_01_step_00000000000002.pdb
-rw-r--r-- 1 renfro domain users   51438 Sep  1 09:27 box_01_step_00000000000000.pdb
-rw-r--r-- 1 renfro domain users  506561 Sep  1 09:27 towhee_altinp
-rw-r--r-- 1 renfro domain users 3001334 Sep  1 09:27 towhee_cbmc
-rw-r--r-- 1 renfro domain users     174 Sep  1 09:27 towhee-serial.sh
-rwxr-xr-x 1 renfro domain users    2488 Sep  1 09:24 towhee_input
-rwxr-xr-x 1 renfro domain users   22113 Sep  1 09:24 towhee_initial
[renfro@login new]$

Multiple-CPU Towhee Job Set

Create a towhee_parallel file describing your set of jobs. In this example, we'll use a set of files from the Towhee installation:

Copying a set of Towhee input files
[renfro@login parallel]$ cp /cm/shared/apps/towhee/7.2.0/Examples/Parallel_Test/towhee_parallel .
[renfro@login parallel]$ cp -a /cm/shared/apps/towhee/7.2.0/Examples/* .

The file towhee_parallel contains a total count of the jobs, a filename to store standard output content, and a list of working directories for each of the jobs in the set. For the example shown, the first several lines will look like:

towhee_parallel file
#number of jobs
56
#stdout filename
par_answer_current
#working directories
/cm/shared/apps/towhee/7.2.0/Examples/Canonical_Ensemble/AVB1_Methane
/cm/shared/apps/towhee/7.2.0/Examples/Canonical_Ensemble/Amber_Villin

These paths will need to be edited to match the folder we've copied them into. Either search and replace the paths with a text editor, or with a utility like perl or sed.

Editing paths in towhee_parallel
[renfro@login parallel]$ perl -pi.bak -E "s#/cm/shared/apps/towhee/7.2.0/Examples#$PWD#g" towhee_parallel

Verify that your towhee_parallel file contains valid paths to your job directories. For the example shown, the first several lines will look like:

towhee_parallel file
#number of jobs
56
#stdout filename
par_answer_current
#working directories
/home/some/path/to/parallel/Canonical_Ensemble/AVB1_Methane
/home/some/path/to/parallel/Canonical_Ensemble/Amber_Villin

Create a Slurm job script named towhee-parallel.sh to run the set of jobs. Note that

  • the nodes and ntasks flags have been set to 2 and 56 instead of 1 and 1 compared to the single-CPU job,
  • the -p flag has been removed from the towhee command, and
  • towhee is being run through the mpirun command.
towhee-parallel.sh
#!/bin/bash
# Adapted from
# https://wiki.crc.nd.edu/w/index.php?title=Submitting_a_Towhee_Job_to_SGE
#SBATCH --nodes=2
#SBATCH --ntasks=56

module load towhee
mpirun -np ${SLURM_NTASKS} `which towhee`

Submit the job from the login node with the command sbatch towhee-parallel.sh, and when the job completes, you should have results in each of your job folders. One set of results is shown below:

Final directory listing for part of Towhee parallel job
[renfro@login parallel]$ ls -lt Canonical_Ensemble/Amber_Villin/
total 3992
-rwxr-xr-x 1 renfro domain users   65604 Sep  1 10:02 par_answer_current
-rw-r--r-- 1 renfro domain users     860 Sep  1 10:02 towhee_citations
-rw-r--r-- 1 renfro domain users   51438 Sep  1 10:02 box_01_step_00000000000010.pdb
-rw-r--r-- 1 renfro domain users      79 Sep  1 10:02 towhee_pvsrho
-rw-r--r-- 1 renfro domain users       0 Sep  1 10:02 towhee_ostwald
-rw-r--r-- 1 renfro domain users   48380 Sep  1 10:02 towhee_final
-rw-r--r-- 1 renfro domain users   51438 Sep  1 10:02 box_01_step_00000000000008.pdb
-rw-r--r-- 1 renfro domain users   51438 Sep  1 10:02 box_01_step_00000000000006.pdb
-rw-r--r-- 1 renfro domain users   51438 Sep  1 10:02 box_01_step_00000000000004.pdb
-rw-r--r-- 1 renfro domain users   51438 Sep  1 10:02 box_01_step_00000000000002.pdb
-rw-r--r-- 1 renfro domain users   51438 Sep  1 10:02 box_01_step_00000000000000.pdb
-rw-r--r-- 1 renfro domain users  506561 Sep  1 10:02 towhee_altinp
-rw-r--r-- 1 renfro domain users 3001334 Sep  1 10:02 towhee_cbmc
-rwxr-xr-x 1 renfro domain users    2488 Sep  1 09:10 towhee_input
-rwxr-xr-x 1 renfro domain users   65552 Sep  1 09:10 answer_current
-rwxr-xr-x 1 renfro domain users   11052 Sep  1 08:25 Makefile
-rwxr-xr-x 1 renfro domain users    9849 Aug 25  2016 Makefile.in
-rwxr-xr-x 1 renfro domain users      40 Jul 24  2014 header
-rwxr-xr-x 1 renfro domain users      84 Sep 20  2012 Makefile.am
-rwxr-xr-x 1 renfro domain users   22113 Sep 20  2012 towhee_initial
[renfro@login parallel]$

How helpful was this information?

Your Rating: Results: 1 Star2 Star3 Star4 Star5 Star 52 rates