Cosigt documentation
⌘ K

# Pipeline Setup

This section provides detailed instructions for setting up the cosigt pipeline on your system.

# Prerequisites

Cosigt is a Snakemake (opens new window) workflow. Detailed instructions for installing Snakemake are available in the official documentation (opens new window). Below is a minimal installation using micromamba (opens new window).

WARNING

Cosigt now requires Snakemake 8 or newer and has been tested with 9.23.1. Earlier releases of the pipeline targeted Snakemake 7; that version is no longer supported, because the workflow relies on the executor-plugin architecture introduced in Snakemake 8.

# Install micromamba
"${SHELL}" <(curl -L micro.mamba.pm/install.sh)
# Create the environment
micromamba create \
    -n cosigt \
    -c conda-forge -c bioconda \
    snakemake=9.23.1 \
    apptainer
# Activate it
micromamba activate cosigt
# Confirm
snakemake --version
# 9.23.1
apptainer --version
# apptainer version 1.3.x

Cosigt combines numerous tools, which can present setup challenges. To simplify deployment and enhance reproducibility we provide Docker (opens new window) containers with pre-compiled binaries for all required software. These are managed automatically when a working Apptainer (opens new window) (formerly Singularity) installation is available, and this is the default. Alternatively the pipeline can run with Conda (opens new window) environments, provided per rule. If you prefer neither, all required tools must be installed and on your $PATH.

# Cluster execution

Running on a cluster needs the matching Snakemake executor plugin. Install only the one your site uses:

pip install snakemake-executor-plugin-slurm
# or
pip install snakemake-executor-plugin-lsf
# or
pip install snakemake-executor-plugin-cluster-generic

make check verifies that the plugin required by your chosen profile is present and prints the exact install command if it is missing, so you do not have to work this out in advance.

TIP

Profiles for local, slurm, lsf and cluster-generic ship with the pipeline in cosigt_smk/profiles/.

# Getting started

Clone the repository and run the three commands that drive the whole pipeline:

git clone https://github.com/davidebolo1993/cosigt
cd cosigt

make init     # create config/ from the examples
make check    # validate the environment and your configuration
make run      # run the pipeline

# make init

Copies the example files into cosigt_smk/config/ and writes a .cosigt.mk holding your default settings. Existing files are never overwritten, so it is safe to re-run.

cosigt_smk/config/
├── config.yaml         # the main configuration
├── samples.tsv         # sample -> alignment
├── regions.bed         # regions to genotype
├── assemblies.tsv      # chromosome -> assembly FASTA
├── alleles.tsv         # region -> allele FASTA   (allele_source: custom)
├── graphs.tsv          # region -> .og or .gfa    (allele_source: graph)
└── truth_graphs.tsv    # region -> truth graph    (benchmark_mode: leave_all_out)

Edit these before running make check. See the → Configuration section for every available key.

# make check

check validates everything needed for the settings you give it and fails with a specific message rather than part-way through a run:

  • Snakemake is on $PATH.
  • The executor plugin required by PROFILE is installed; for local it reports the cores it will use.
  • The runtime required by SOFTWARE is available — apptainer (accepting either apptainer or singularity), conda (including the 24.7.1 minimum Snakemake needs), or, for none, that every tool the current configuration will invoke is on $PATH. That list is derived from your config, so it accounts for read_mode, vcf, gtf and the visualisation switches.
  • The config file, sample table, region BED, assembly or allele tables, and every referenced file and index.
  • Finally it composes the Apptainer flags this run needs, and writes the settings it just validated to .cosigt.mk.
cosigt check  (profile=local software=apptainer target=cosigt cores=32)

environment:
  ok    snakemake (9.23.1)
  ok    local execution, 32 of 32 detected cores
  ok    apptainer (apptainer version 1.3.6)

configuration and inputs:
  ok    config, sample table, regions, indexes and input files
  ok    region metadata and flagger blacklist written

apptainer flags (used automatically by 'make run'):
  -B /group/soranzo,/project/ham,/tmp -e

Wrote .cosigt.mk: profile=local software=apptainer target=cosigt cores=32
All checks passed. Run the pipeline with: make run

Those flags are the bind mounts covering every configured input and output location, collapsed to the shortest set of parent directories, plus -e (--cleanenv), which pggb requires. make run picks them up automatically. Because check persists what it validated, make check PROFILE=slurm followed by a bare make run executes the configuration that was actually checked. Command-line variables still override the file.

# make run

Runs the selected target. It re-reads the flags written by check, so run check first, and again after changing any input path.

The targets are:

target what it builds
cosigt the default: genotypes every sample over every region
graph the per-region graphs, their clustering and the optional plots — and nothing that reads a sample's alignment
benchmark the QV tables and plot, see → Benchmarking
refine refined region boundaries, written to refine/regions_refined.bed

graph is a strict subset of cosigt: it stops once the graph side is complete, skipping read extraction, k-mer filtering, realignment, graph injection, coverage and genotyping. Use it to get the expensive pggb construction done once, or before the reads are available. A later TARGET=cosigt reuses everything it produced and only runs the per-sample half.

Validation follows the target. graph and refine never read a sample's alignment, so neither requires samples.tsv or the BAM/CRAM files to exist, and under SOFTWARE=none each asks only for the tools it actually invokes — graph wants odgi, pggb, impg and friends, not bwa-mem2, kfilt or gafpack.

Re-run check after changing TARGET

The Apptainer bind mounts are derived from what the target reads, so those composed for graph omit the directories holding your alignments. make run compares the two and refuses to start on a mismatch rather than failing inside the container.

# Settings

All three commands take the same variables, either on the command line or persisted in .cosigt.mk by make init and make check:

Variable Default Meaning
PROFILE local local, slurm, lsf, cluster-generic, a path, or none
SOFTWARE apptainer apptainer, conda, or none
TARGET cosigt cosigt, graph, refine, or benchmark
CORES all detected passed to --cores. On cluster profiles set this to the largest core count a compute node offers — it is detected from wherever make runs, and it also caps the resources Snakemake aggregates into a group job
SMK_ARGS empty extra Snakemake arguments, e.g. -n for a dry run
make run PROFILE=slurm                     # cluster run
make run SMK_ARGS=-n                       # dry run
make run TARGET=graph                      # build the graphs only, no genotyping
make run TARGET=benchmark                  # benchmarking instead of genotyping
make run SOFTWARE=conda                    # conda environments instead of containers
make run PROFILE=cluster-generic SMK_ARGS='--cluster-generic-submit-cmd "qsub"'

# Tools

Below are the tools used across the workflow and the versions pinned by the conda environments. For manual installation, refer to the Dockerfiles in this repository (opens new window).

Tool Version
bcftools (opens new window) 1.23.1
bedtools (opens new window) 2.31.1
bwa (opens new window) 0.7.18
bwa-mem2 (opens new window) 2.2.1
cosigt (opens new window) 0.2
gafpack (opens new window) 0.1.3
gfainject (opens new window) 0.2.0
impg (opens new window) 0.3.3
kfilt (opens new window) 0.1.1
meryl (opens new window) 1.4.1
minimap2 (opens new window) 2.31
miniprot (opens new window) 0.18
odgi (opens new window) 0.9.3
pangene (opens new window) r231 (v1.1)
panplexity (opens new window) 0.1.1
pggb (opens new window) 0.7.4
samtools (opens new window) 1.23.1
svim-asm (opens new window) 1.0.3
wally (opens new window) 0.7.1

Which of these are actually needed depends on your configuration: bwa only for read_mode: ancient, kfilt/meryl only for read_mode: short, bcftools only with vcf: true, pangene/miniprot only with gtf/proteins, and so on. make check SOFTWARE=none reports exactly the set your configuration requires.

Calculations and plots are performed in R. The package list is in the renv Dockerfile (opens new window).

WARNING

r-dbscan is pinned to 1.2.2 deliberately. Releases from 1.2.3 onward crash with a segmentation fault inside dbcv(), which the clustering step calls, and because a segfault terminates the R process it cannot be caught. Do not relax this pin.

TIP

Two conda pins differ slightly from the containers, because the versions the containers build are not on bioconda: gfainject (0.2.0 vs 0.2.1) and bedtools (2.31.1 vs 2.31.0, since 2.31.0 cannot coexist with samtools 1.23.1). The benchmark target is container-only, as its compute_qv helper has no conda package.

ON THIS PAGE
Prerequisites
Cluster execution
Getting started
make init
make check
make run
Settings
Tools