HPC-recommendations

Table of Contents

• 7.1. Limited local disk space available for temporary files
• 7.2. Per-task or CPU memory reservation for LSF
• 7.3. LSF executor scratch space
• 7.4. Wall-time and minimum CPU settings
• 7.5. Setting specific Queue, CPU, RAM and DISK per pipeline or workflow module
• 7.6. Setting specific Memory settings for SGE/OGE/UGE HPC clusters

(back to main documentation)

7. LSF and SLURM Recommendations

You should ensure that your local cluster queues and relevant cluster requirements are reflected in the nextflow.config file in the biomodal script folder. However, some settings may be less obvious so we added a few recommendations below.

(back to main documentation) | (back to top)

7.1. Limited local disk space available for temporary files

If your cluster has limited local disk space available for temporary files, please consider binding a larger local tmp directory location using the following settings in the nextflow.config file in the biomodal script folder. This will allow the duet containers to use this directory for temporary files (/tmp).

singularity {
  enabled    = true
  autoMounts = true
  runOptions = "--bind /<your local tmp path>:/tmp:rw"
}

Additionally, add the following to the process section:

process {
  executor = "lsf"
  containerOptions = "--bind /<your local tmp path>:/tmp"
}

You should also check, or consider setting, the TMPDIR and/or SINGULARITY_TMPDIR environment variables on the head/controller node before any jobs are submitted. This should ensure that the temporary files are written to the correct location.

export TMPDIR=<your local tmp path>
export SINGULARITY_TMPDIR=<your local tmp path>

Similarly, you can check that the optional SINGULARITY_CACHEDIR environment variable is pointing to the same location as the libraryDir = "<some-location>/singularity-images" in the nextflow.config file in the biomodal script folder to ensure that the singularity containers are cached in the correct location.

export SINGULARITY_CACHEDIR=<same-path-as-libraryDir>

Please note that dependent on the software on your cluster, you need to set Singularity or Apptainer environment variables respctively.

Singularity variable	Apptainer variable
SINGULARITYENV_TMPDIR	APPTAINERENV_TMPDIR
SINGULARITYENV_NXF_TASK_WORKDIR	APPTAINERENV_NXF_TASK_WORKDIR
SINGULARITYENV_NXF_DEBUG	APPTAINERENV_NXF_DEBUG

(back to main documentation) | (back to top)

7.2. Per-task or CPU memory reservation for LSF

Default LSF cluster settings works with a per-core memory limits mode, so it divides the requested memory by the number of requested cpus. If your LFS cluster is configered differently, it is recommended that you try to add the following settings to the LSF executor section in the nextflow.config file in the biomodal script folder.

executor {
  name = "lsf"
  perTaskReserve = false
  perJobMemLimit = true
}

(back to main documentation) | (back to top)

7.3. LSF executor scratch space

You can dynamically use LSF scratch space per job using the following settings in the nextflow.config file in the biomodal script folder.

process {
  executor = "lsf"
  scratch = "$SCRATCHDIR/$LSB_JOBID"
}

(back to main documentation) | (back to top)

7.4. Wall-time and minimum CPU settings

In some clusters, it is recommended to set a “wall-time”, i.e. max time a job can run before it is terminated. There may also be a “minimum CPU” requirement to start jobs in your cluster. You can adjust the Nextflow time and cpuNum parameters in the nextflow.config file in the biomodal script folder.

process {
  executor = "slurm"
  time = "24h"
  params.cpuNum = 1
}

(back to main documentation) | (back to top)

7.5. Setting specific Queue, CPU, RAM and DISK per pipeline or workflow module

In the nextflow.config file in the biomodal script folder, you can set specific Queues, CPU, RAM or DISK requirements per module by using the withName parameter. For example, you can set the CPU, RAM or DISK requirements for the BWA_MEM2 and PRELUDE modules as follows:

process { 
  //Example of default queue settings 
  cpus     = 16  
  memory   = "16GB"
  queue    = "<add the name name of your default queue>"  
  time     = "24h"
  disk     = "800GB"

  //Example of setting per module specific queuesettings 
  withName: 'BWA_MEM2' { memory = "64GB"
                         queue = "<add the name name of a larger queue>" 
                         time = "48h" }
  withName: 'PRELUDE'  { cpus = 32
                         memory = "32GB"
                         queue = "<add the name name of a larger queue>" }
}

7.6. Setting specific Memory settings for SGE/OGE/UGE HPC clusters

If you are on a SGE/OGE/UGE HPC cluster that is not supporting h_rt, h_rss or mem_free settings, you can try to add one of the following settings. Either globaly:

process { 
  //Example of default queue settings 
  cpus     = 16  
  memory   = "16GB"
  queue    = "<name-of-small-queue>"  
  time     = "24h"
  disk     = "800GB"
  clusterOptions = { "-l h_vmem=${task.memory.toString().replaceAll(/[\sB]/,'')}" }

or diretly in the relevant withName sections if you like to remove other memory specific cluster settings:

withName: 'BWA_MEM2' { cpus = 32
                       clusterOptions = "-l h_vmem=64GB"
                       memory = null
                       queue = "<name-of-preferred-queue>" 
                       time = "48h" }  

If your cluster expects a per core rather than per job memory limit, you should adjuse the h_vmem setting to reflect the per core memory limit.

(back to main documentation) | (back to top) | (Next)