 |
 |
 |
|---|
Table of Contents
Overview
Below is a table detailing minimum resource requirements for each module in the duet pipeline and D(h)MR workflow, realtive to each profile. Your cloud tenancy or HPC cluster should be able to allocate a set of VMs/nodes with these resources.
The module specific settings and profiles can be found in the biomodal-duet-x.x.x/modules and biomodal-duet-x.x.x/conf directories. We do not recommend editing these files directly as any local change will not be reflected when you upgrade the pipeline.
You can edit resource requirements by referencing specific modules the nextflow.config file in the biomodal script folder to accomodate available hardware resources. Please see examples in section 7.5. Setting specific Queue, CPU, RAM and DISK per pipeline or workflow module for more information on how to increase or decrease the resource requirements. However, the duet pipeline is less likely to run successfully if the minimum resources are too restrictive.
Please note that the module requirement is per sample per node on a HPC cluster node or cloud VM.
You need to multiply the number of samples by the resource requirement if you intend to only use a single node or VM.
If you are running the pipeline or D(h)MR workflow on a larger local node you can set a maximum celing for CPU and RAM per module. Please see section 3.5.1. Limit local resource requirements in the documentation for information of how to use parameters to limit the resource requirements for the local and local_deep_seq profiles.
Module requirements
| Module / Requirement type | CPU (cores) | RAM (GB) | Disk (GB) | CPU (cores) | RAM (GB) | Disk (GB) | CPU (cores) | RAM (GB) | Disk (GB) |
|---|---|---|---|---|---|---|---|---|---|
| Profile name: | default | default | default | deep_seq | deep_seq | deep_seq | super_seq | super_seq | super_seq |
| BAMLET | 2 | 2 | – | – | – | – | – | – | – |
| ASM | 16 | 32 | – | – | – | – | 32 | 128 | 1600 |
| BAMLET | 2 | 2 | – | – | – | – | – | – | – |
| BAM_TO_CRAM | 6 | 6 | – | – | – | – | – | – | – |
| BAMLET_GENOME | 2 | 2 | – | 8 | 8 | 800 | 8 | 8 | 3200 |
| BAMLET_GENOME_PRIMARY_ASSEMBLY | 2 | 2 | – | 8 | 8 | 800 | 8 | 8 | 3200 |
| BAMLET_LONG_CTRL | 2 | 2 | – | 4 | 4 | 800 | 4 | 4 | 1600 |
| BAMLET_SHORT_CTRL | 2 | 2 | – | 4 | 4 | 800 | 4 | 4 | 1600 |
| BEDGRAPHS | 2 | 8 | – | – | – | – | – | – | – |
| BIGWIGS | 2 | 8 | – | – | – | – | – | – | – |
| BWAKIT | 16 | 16 | – | – | – | – | – | – | – |
| BWA_MEM2 (Human genome) | 16 | 64 | – | 32 | 64 | 800 | 96 | 128 | 3200 |
| BWA_MEM2 (Mouse genome) | 32 | 128 | – | 32 | 128 | 800 | 96 | 128 | 3200 |
| COMBINE_GENOME_AND_CTRLS | 1 | 2 | – | – | – | – | – | – | – |
| CONCATENATE_SAMPLE_STORES (chg_chh_contexts) | 16 | 32 | – | – | – | – | – | – | – |
| CONCATENATE_SAMPLE_STORES | 8 | 16 | – | – | – | – | – | – | – |
| COUPLET | 32 | 32 | – | – | – | – | 96 | 320 | 1600 |
| CURATE_QUANT | 4 | 8 | – | – | – | – | – | – | – |
| CURATE_QUANT (chg_chh_contexts) | 8 | 16 | – | – | – | – | – | – | – |
| CUTADAPT | 16 | 16 | – | – | – | – | 32 | 32 | 1600 |
| DEDUP_GENOME | 1 | 4 | – | 4 | 16 | 800 | 4 | 16 | 1600 |
| DEDUP_MERGED_BAM | 1 | 4 | – | 1 | 16 | 800 | 4 | 16 | 1600 |
| DEDUP_GENOME_INTERVAL | 1 | 4 | – | – | – | – | 4 | 16 | 1600 |
| DEDUP_LONG_CTRL | 1 | 4 | – | 2 | 8 | 800 | 4 | 16 | 800 |
| DEEPTOOLS_PLOTFINGERPRINT | 4 | 16 | – | – | – | – | 8 | 32 | 1600 |
| DOWNSAMPLE_BAM | 2 | 2 | – | – | – | – | – | – | – |
| DOWNSAMPLE_LONG_CTRL | 2 | 2 | – | – | – | – | – | – | – |
| DQSPY | 8 | 32 | – | – | – | – | – | – | – |
| DQSREPORT | 4 | 16 | – | – | – | – | – | – | – |
| EPIQUANT_BAM2ZARR | 8 | 8 | – | 16 | 16 | – | 32 | 32 | – |
| EPIQUANT_MBIAS or GENOME_MBIAS | 4 | 4 | – | 16 | 16 | – | – | – | – |
| EPIQUANT_QUANT | 8 | 8 | – | 16 | 32 | 800 | 32 | 32 | 1600 |
| EPIQUANT_SUMMARY | 1 | 1 | – | – | – | – | – | – | – |
| FASTQC | 2 | 4 | – | – | – | – | – | – | – |
| FASTQC_RAW | 2 | 4 | – | – | – | – | – | – | – |
| FASTQC_RESOLVED | 2 | 4 | – | – | – | – | – | – | – |
| FASTQC_TRIMMED | 2 | 4 | – | – | – | – | – | – | – |
| FILTER_BAM | 8 | 16 | – | – | – | – | – | – | – |
| FILTER_GENOME | 8 | 16 | – | – | – | – | 8 | 16 | 1600 |
| FLEXBAR_PRE_RESOLUTION | 2 | 2 | – | – | – | – | – | – | – |
| FLEXBAR_POST_RESOLUTION | 2 | – | – | – | – | – | – | – | – |
| FRAGMENT_LENGTH | 2 | 8 | – | – | – | – | 8 | 32 | 1600 |
| GENETIC_ACCURACY | 8 | 16 | – | – | – | – | – | – | – |
| HAPLOTYPE_CALLER | 16 | 64 | – | 32 | 64 | – | 32 | 128 | 1600 |
| INLINE_UMIS | 16 | 32 | 500 | – | – | – | – | – | 1600 |
| JOIN_ZARR_STORES | 16 | 32 | – | – | – | – | – | – | – |
| JOIN_ZARR_STORES (GC) | 8 | 16 | – | – | – | – | – | – | – |
| LONG_CTRL_GENETIC_ACCURACY | 1 | 1 | – | – | – | – | – | – | – |
| MAKE_BED_FILES | 1 | 1 | – | – | – | – | – | – | – |
| MAKE_FASTA_INDEX | 1 | 1 | – | – | – | – | – | – | – |
| MAKE_GENOME_PRIMARY_ASSEMBLY | 1 | 1 | – | – | – | – | – | – | – |
| MAKE_GENOME_SIZES_NAMES | 1 | 1 | – | – | – | – | – | – | – |
| MAKE_INTERVAL_FILE | 1 | 1 | – | – | – | – | – | – | – |
| MAKE_MOD_REF_FILES | 1 | 1 | – | – | – | – | – | – | – |
| MAKE_NON_POSITIONAL_QTABLE | 1 | 1 | – | – | – | – | – | – | – |
| MAKE_STORE (chg_chh_contexts) | 16 | 32 | – | – | – | – | – | – | – |
| MAKE_STORE | 8 | 16 | – | – | – | – | – | – | – |
| MAKE_TARGET_INTERVALS | 1 | 1 | – | – | – | – | – | – | – |
| MERGE_DEDUP_STATS | 2 | 2 | – | – | – | – | – | – | – |
| MERGE_UMI_STATS | 2 | 2 | – | – | – | – | – | – | – |
| MODALITY_DMR | (dynamic allocation) 8 | (dynamic allocation) 16 | – | – | – | – | – | – | – |
| MODALITY_EXPORT | (dynamic allocation) 4 | (dynamic allocation) 16 | (dynamic allocation) 6 | (dynamic allocation) 128 | – | – | – | – | – |
| MULTIQC | 2 | 8 | – | – | – | – | – | – | – |
| MUTECT2 | 16 | 64 | – | 32 | 64 | – | 64 | 256 | 1600 |
| ONE_STEP_BAMLET | 8 | 8 | – | 16 | 16 | 800 | 16 | 16 | 1600 |
| PICARD_GCBIAS | 2 | 8 | – | – | – | – | 8 | 32 | 1600 |
| PICARD_MARKDUPLICATES | 1 | 4 | – | – | – | – | – | – | – |
| PICARD_TARGETED_METRICS | 2 | 8 | – | 8 | 32 | – | 8 | 32 | – |
| PICARD_TARGETED_METRICS (twist methylome) | 8 | 32 | – | 8 | 32 | – | 8 | 32 | – |
| PIPELINE_REPORT | 1 | 1 | – | – | – | – | – | – | – |
| PRELUDE | 32 | 32 | – | 64 | 64 | – | 96 | 96 | 1600 |
| PRESEQ | 4 | 8 | – | – | – | – | – | – | – |
| QUALIMAP_BAMQC | 4 | 16 | – | – | – | – | 8 | 32 | 1600 |
| REDUCE_FASTA | 1 | 1 | – | – | – | – | – | – | – |
| REHEADER_VCF | 1 | 1 | – | – | – | – | – | – | – |
| SAMTOOLS_MERGE | 8 | 2 | – | – | – | – | – | – | – |
| SAMTOOLS_MERGE_INTERVALS | 16 | 16 | 200 | – | – | – | 8 | 8 | 3200 |
| SAMTOOLS_MERGE_LANES (Human genome) | 8 | 8 | – | 8 | 8 | 800 | 8 | 8 | 3200 |
| SAMTOOLS_MERGE_LANES (Mouse genome) | 8 | 8 | – | 16 | 64 | 800 | 16 | 64 | 3200 |
| SAMTOOLS_STATS | 2 | 2 | – | – | – | 800 | – | – | 1600 |
| SEQTK_SAMPLE | 8 | 8 | – | 32 | 64 | – | 32 | 64 | 1600 |
| SPLIT_FASTQS | 1 | 1 | – | – | – | 800 | – | – | 1600 |
| SPLIT_INTERVALS | 1 | 4 | – | – | – | – | – | – | – |
| SUMMARY_REPORT | 2 | 8 | – | – | – | – | – | – | – |
| TSS_BIAS | 2 | 8 | – | – | – | – | – | – | 1600 |
| UMI_DEDUP_GENOME | 1 | 4 | – | 4 | 16 | 800 | 4 | 16 | 1600 |
| UMI_DEDUP_LONG_CTRL | 1 | 4 | – | – | – | – | 4 | 16 | 800 |
Reference pipeline module requirements
| Module / Requirement type | CPU (cores) | RAM (GB) |
|---|---|---|
| COMBINE_GENOME_AND_CTRLS | 1 | 2 |
| MAKE_BWA_MEM2_INDEX | 2 | 128 |
| MAKE_BWA_MEM2_INDEX (>=4.6Gbp) | 2 | 320 |
| MAKE_FASTA_INDEX_GENOME | 1 | 1 |
| MAKE_FASTA_INDEX_SHORT_CTRLS | 1 | 1 |
| MAKE_FASTA_INDEX_LONG_CTRLS | 1 | 1 |
| MAKE_BED_FILES_GENOME | 1 | 1 |
| MAKE_GENOME_PRIMARY_ASSEMBLY | 1 | 1 |
| MAKE_BED_FILES_SHORT_CTRLS | 1 | 1 |
| MAKE_BED_FILES_LONG_CTRLS | 1 | 1 |
| MAKE_SHORT_CTRLS_SIZES_NAMES | 1 | 1 |
| MAKE_LONG_CTRLS_SIZES_NAMES | 1 | 1 |
| MAKE_GENOME_SIZES_NAMES | 1 | 1 |
| MAKE_MOD_REF_FILES | 1 | 1 |
| MAKE_DICT_GENOME | 2 | 4 |
| MAKE_DICT_COMBINED | 2 | 4 |
