Running Pipelines¶

Execute Nextflow and WDL workflows using the Flow CLI.

Overview¶

The Flow platform supports running bioinformatics pipelines on Google Cloud Batch. Pipelines can be:

Nextflow workflows - Standard nf-core and custom Nextflow pipelines
WDL workflows - WDL/Cromwell-based workflows

Prerequisites¶

Installed CLI - See Installation
Authenticated - Run flow login
Project access - You need access to a project with GCP credentials configured

Available Pipelines¶

List available pipelines:

flow pipelines list

Get details about a specific pipeline:

flow pipelines info nextflow-minimal
# or for WDL:
flow pipelines info wdl-minimal

Pipeline Types¶

Type	Description	Engine
`direct_nextflow`	Runs standard Nextflow pipeline (e.g., nf-core workflows)	Nextflow
`direct_wdl`	Runs WDL workflow via Cromwell	Cromwell
`container_entrypoint`	Runs container with custom entrypoint (e.g., nextflow-minimal)	Custom

Submitting a Run¶

Basic Submission¶

flow runs submit \
  --project PROJECT_ID \
  --pipeline nextflow-minimal \
  -P analysis_name=my_analysis

With Parameters¶

Pass parameters using -P KEY=VALUE:

flow runs submit \
  --project PROJECT_ID \
  --pipeline nextflow-minimal \
  -P nf_profile_name=client_config_test \
  -P analysis_name=my_analysis \
  -P sample_id=sample1 \
  -P fastq_tumor=gs://bucket/R1.fastq.gz \
  -P fastq_tumor=gs://bucket/R2.fastq.gz

Array Parameters

For parameters that accept multiple values (like paired FASTQ files), use -P key=value multiple times with the same key.

With Watch Mode¶

Automatically monitor the run after submission:

flow runs submit \
  --project PROJECT_ID \
  --pipeline nextflow-minimal \
  -P analysis_name=test \
  --watch

Monitoring Runs¶

Check Status¶

flow runs status RUN_ID

Output includes: - Current status (queued, running, succeeded, failed, cancelled) - Timestamps (created, started, finished) - Parameters used - Output path (on success) - Error message (on failure)

Watch Until Completion¶

flow runs watch RUN_ID --interval 15

The watch command polls the status and displays updates:

[12:00:15] Status: queued
[12:00:30] Status: running
           Started: 2026-01-08T12:00:25Z
[12:05:30] Status: succeeded

Run completed successfully!
Output: gs://bucket/outputs/run-abc123/

Press Ctrl+C to stop watching (the run continues in the background).

List Runs¶

View recent runs for a project:

flow runs list --project PROJECT_ID --limit 10

Cancelling Runs¶

Cancel a running or queued run:

flow runs cancel RUN_ID

You'll be prompted to confirm the cancellation.

Example: nextflow-minimal Pipeline¶

The nextflow-minimal pipeline processes FASTQ files through a minimal Nextflow workflow.

Required Parameters¶

Parameter	Description
`nf_profile_name`	Nextflow profile (e.g., `client_config_test`)
`analysis_name`	Name for the analysis
`sample_id`	Sample identifier
`fastq_tumor`	Path(s) to FASTQ files (use twice for paired)

Full Example¶

# Login
flow login -e user@example.com -p yourpassword

# Verify pipeline exists
flow pipelines info nextflow-minimal

# Submit run
flow runs submit \
  --project 01234567-89ab-cdef-0123-456789abcdef \
  --pipeline nextflow-minimal \
  -P nf_profile_name=client_config_test \
  -P analysis_name=test_$(date +%Y%m%d_%H%M) \
  -P sample_id=sample1 \
  -P fastq_tumor=gs://bucket-intelliseq-demo/test-data/sample_small_R1.fastq.gz \
  -P fastq_tumor=gs://bucket-intelliseq-demo/test-data/sample_small_R2.fastq.gz \
  --watch

Expected Output¶

On successful completion:

Run submitted successfully!
  ID: abc123-def456-...
  Name: nextflow-minimal-1736339200
  Status: queued

Watching run status (Ctrl+C to stop)...
[12:00:15] Status: queued
[12:00:30] Status: running
           Started: 2026-01-08T12:00:25Z
[12:05:30] Status: succeeded

Run completed successfully!
Output: gs://bucket/outputs/nextflow-minimal-1736339200/

Example: WDL Minimal Pipeline¶

The wdl-minimal pipeline demonstrates WDL/Cromwell execution on Flow.

WDL Pipeline Features¶

Cromwell execution - WDL workflows run via Cromwell
No inputs required - The minimal pipeline runs without parameters
Metadata output - Results include Cromwell metadata JSON

Submitting WDL Run¶

# Check pipeline info
flow pipelines info wdl-minimal

# Submit WDL run (no parameters needed for minimal pipeline)
flow runs submit \
  --project PROJECT_ID \
  --pipeline wdl-minimal \
  --watch

Example: Hereditary Mock Pipeline¶

The hereditary-mock pipeline demonstrates WDL execution with actual inputs (FASTQ files).

Pipeline Overview¶

Type: WDL/Cromwell (direct_wdl)
Purpose: Mock hereditary analysis for testing input handling
Required inputs: case_id, child_fastq files

Required Parameters¶

Parameter	Type	Description
`case_id`	String	Case identifier for the analysis
`child_fastq`	Array[File]	Child/proband FASTQ files (R1, R2)

Optional inputs include mother_fastq, father_fastq, patient metadata, HPO terms.

Submitting Hereditary Mock Run¶

# Check pipeline info
flow pipelines info hereditary-mock

# Submit with FASTQ inputs
flow runs submit \
  --project PROJECT_ID \
  --pipeline hereditary-mock \
  -P case_id=case-001 \
  -P child_fastq=gs://bucket/sample_R1.fastq.gz \
  -P child_fastq=gs://bucket/sample_R2.fastq.gz \
  --watch

Complete Walkthrough¶

1. Login to Flow

flow login -e your@email.com -p yourpassword

2. Check pipeline details

flow pipelines info hereditary-mock

Output:

Pipeline: Hereditary Mock Pipeline
  Slug: hereditary-mock
  Type: direct_wdl
  Container: broadinstitute/cromwell:86
  Tags: wdl, hereditary, mock, cromwell, clinical

  Description: Mock hereditary analysis pipeline for WDL/Cromwell testing.

  Required inputs: case_id, child_fastq

3. Submit the run with FASTQ inputs

flow runs submit \
  --project 01c965b9-3366-4a09-81f0-dcc88375e76a \
  --pipeline hereditary-mock \
  -P case_id=test-case-001 \
  -P child_fastq=gs://bucket-intelliseq-demo/test-data/regions_R1.fastq.gz \
  -P child_fastq=gs://bucket-intelliseq-demo/test-data/regions_R2.fastq.gz \
  --watch

Output:

Run submitted successfully!
  ID: hereditary-mock-20260112-134500
  Status: queued

Watching run status (Ctrl+C to stop)...
[13:45:15] Status: queued
[13:46:00] Status: running
           Started: 2026-01-12T13:45:50Z
[13:48:00] Status: succeeded
           Finished: 2026-01-12T13:47:45Z

Run completed successfully!
Duration: 2m 30s

4. Check run status

flow runs status hereditary-mock-20260112-134500

WDL Pipeline Output¶

Run submitted successfully!
  ID: wdl-minimal-pipeline-20260110-...
  Status: queued

Watching run status (Ctrl+C to stop)...
[12:00:15] Status: queued
[12:01:30] Status: running
[12:03:30] Status: succeeded

Run completed successfully!
Output: gs://bucket/outputs/wdl-minimal-pipeline-.../

WDL with Parameters¶

For WDL pipelines that require inputs:

flow runs submit \
  --project PROJECT_ID \
  --pipeline germline-pipeline \
  -P sample_id=sample1 \
  -P input_bam=gs://bucket/sample.bam

Complete WDL Walkthrough (Minimal Pipeline)¶

Step-by-step guide to run the WDL minimal pipeline:

1. Login to Flow

flow login -e your@email.com -p yourpassword

2. List available pipelines

flow pipelines list

Output:

Available Pipelines:
  hereditary-mock    Hereditary Mock Pipeline (direct_wdl)
  wdl-minimal        WDL Minimal Pipeline (direct_wdl)
  nextflow-minimal   Nextflow Minimal Pipeline (container_entrypoint)
  nfcore-demo        nf-core Demo Pipeline (direct_nextflow)

3. Get WDL pipeline details

flow pipelines info wdl-minimal

Output:

Pipeline: WDL Minimal Pipeline
  Slug: wdl-minimal
  Type: direct_wdl
  Container: broadinstitute/cromwell:86
  Tags: wdl, minimal, cromwell, test

  Description: Minimal WDL pipeline for testing Cromwell execution.

  This pipeline has no required inputs.

4. Submit the WDL run

flow runs submit \
  --project 01c965b9-3366-4a09-81f0-dcc88375e76a \
  --pipeline wdl-minimal \
  --watch

Output:

Run submitted successfully!
  ID: wdl-minimal-pipeline-20260110-210636
  Status: queued

Watching run status (Ctrl+C to stop)...
[21:06:51] Status: queued
[21:07:06] Status: queued
[21:08:06] Status: running
           Started: 2026-01-10T21:08:00Z
[21:09:06] Status: running
[21:09:36] Status: succeeded
           Finished: 2026-01-10T21:09:30Z

Run completed successfully!
Duration: 1m 30s

5. Check run details

flow runs status wdl-minimal-pipeline-20260110-210636

Output:

Run: wdl-minimal-pipeline-20260110-210636
  Pipeline: WDL Minimal Pipeline
  Status: succeeded
  Created: 2026-01-10T21:06:36Z
  Started: 2026-01-10T21:08:00Z
  Finished: 2026-01-10T21:09:30Z
  Duration: 1m 30s

What happens during execution:

GCP Batch creates a VM with the Cromwell container
The WDL file is downloaded from workflows.intelliseq.com
Cromwell parses and validates the workflow
Tasks are executed locally in the container
Results are captured in Cromwell metadata
Job completes and status is updated

Order-Run Association¶

Runs can be associated with clinical orders from the Miner service. This enables tracking which pipeline runs were executed for which clinical cases.

Submitting a Run with Order ID¶

Use --order-id (or -o) to associate a run with an order:

flow runs submit \
  --project PROJECT_ID \
  --pipeline hereditary-mock \
  --order-id ORDER_ID \
  -P case_id=case-001 \
  -P child_fastq=gs://bucket/R1.fastq.gz \
  -P child_fastq=gs://bucket/R2.fastq.gz \
  --watch

Listing Runs by Order¶

View all runs associated with a specific order:

flow runs list -p PROJECT_ID --order-id ORDER_ID

Output:

ID                                       NAME                                STATUS
------------------------------------------------------------------------------------------
8e9beaa2-3be1-482a-a313-75ea64876ec3     hereditary-mock-20260113-000500     succeeded

Checking Order ID in Run Status¶

When viewing run details, the associated order ID is displayed:

flow runs status RUN_ID

Output:

Run: hereditary-mock-20260113-000500
  ID: 8e9beaa2-3be1-482a-a313-75ea64876ec3
  Status: succeeded
  Pipeline ID: 302ca72c-df9c-4733-8b3d-174ab82e5fc4
  Order ID: ORDER_ID
  Profile: local
  Created: 2026-01-13T00:05:00Z
  Started: 2026-01-13T00:06:00Z
  Finished: 2026-01-13T00:08:00Z
  Output: gs://bucket/nf-results/hereditary-mock-20260113-000500

Complete Order-Run Workflow¶

This example shows the full workflow from login to run completion with order association:

1. Login

flow login --token YOUR_PAT_TOKEN
# Or: flow login -e email@example.com -p password

2. Submit run with order association

flow runs submit \
  --project 01c965b9-3366-4a09-81f0-dcc88375e76a \
  --pipeline hereditary-mock \
  --order-id my-clinical-order-001 \
  -P case_id=patient-12345 \
  -P child_fastq=gs://bucket-intelliseq-demo/test-data/regions_R1.fastq.gz \
  -P child_fastq=gs://bucket-intelliseq-demo/test-data/regions_R2.fastq.gz \
  -t clinical \
  -t hereditary \
  --watch

3. Check run status

flow runs status hereditary-mock-20260113-...

4. List all runs for the order

flow runs list -p PROJECT_ID --order-id my-clinical-order-001

API Access¶

For programmatic access, the order-run association is also available via the API:

Submit run with order_id:

curl -X POST https://compute.flow.labpgx.com/api/v1/runs \
  -H "Authorization: Bearer TOKEN" \
  -H "X-Project-ID: PROJECT_ID" \
  -H "Content-Type: application/json" \
  -d '{
    "pipeline_id": "PIPELINE_ID",
    "order_id": "ORDER_ID",
    "params": {"case_id": "test"},
    "tags": ["clinical"]
  }'

Get runs by order:

curl https://compute.flow.labpgx.com/api/v1/runs/by-order/ORDER_ID

Troubleshooting¶

"Not authenticated"¶

Run flow login to authenticate:

flow login -e your@email.com -p yourpassword

"Resource not found" for Pipeline¶

The pipeline may not be configured. Check available pipelines:

flow pipelines list

Run Stuck in "queued"¶

Possible causes: - GCP Batch quota limits - Service account permission issues - Network connectivity problems

Check with your administrator.

"credentials" Error¶

The project needs GCP credentials configured: 1. Open the Admin Console 2. Navigate to your project 3. Add GCP credentials under Settings

Run Failed¶

Check the error message:

flow runs status RUN_ID

Common issues: - Invalid input file paths - Insufficient compute resources - Pipeline-specific parameter errors

Next Steps¶

Commands Reference - Full command documentation
Compute Service - Web interface for pipelines