Use dsub in the All of Us Researcher Workbench

This document describes dsub support in All Of Us Researcher Workbench, and is intended for users with Controlled Tier access who are processing large volumes of data.  This document assumes basic familiarity with the Google cloud, including buckets and docker.  

This document focuses on using dsub for genomic data, but it is applicable to processing any data that is accessible in Researcher Workbench.

Note: we suggest using us-central region when launching Google Batch API for batch jobs because our CDR bucket and your buckets live in us-central1. If you launch API jobs in other regions, you will incur network egress charges.

Introduction 

dsub is a command-line tool that makes it easy to submit and run batch scripts in the cloud. With dsub, you can write a shell script and then submit it to a job scheduler from Jupyter. Unlike Cromwell WDL and Nextflow, dsub does not use a DSL. dsub supports Google Cloud as the backend batch job runner. Refer to the dsub documentation for additional guidance on writing dsub commands and more example dsub scripts. 

Refer to the dsub tutorial notebooks in the Controlled Tier Featured Workspace for more guidance on getting started with dsub. 

Within the Researcher Workbench, Google Batch API is the executor of dsub tasks. 

If you have any feedback or questions on using dsub, reach out to support@researchallofus.org 

Suggestion while running dsub tasks

With dsub, you can check the status of a job at any time by running the dstat command. This will work both in a Notebook and in a Terminal session. 

See https://github.com/DataBiosphere/dsub/blob/main/docs/troubleshooting.md for more examples.

Potential Limitations

Public docker images from Google Artifact Registry (GAR) are the only types of images officially supported for tasks, but dockerhub images (built on linux/amd64) may also work in Researcher Workbench workflows. 

For a full list of limitations applicable to all batch processing, refer to Overview of Batch Processing.

Docker container images in GAR

Each of these docker images are a lightweight, standalone, executable package of software that includes everything needed to run an application: code, runtime, system tools, system libraries and settings.  Below is a list of common tools and the associated public docker images in GAR that will work with dsub:  

• GATK:  us.gcr.io/broad-gatk/gatk:{version} (eg, us.gcr.io/broad-gatk/gatk-4.2.5.0)
  • The canonical image for the GATK/
• Tools available in RW jupyter notebooks:  us.gcr.io/broad-dsp-gcr-public/terra-jupyter-aou:{version}
  • You can find vcftools, PLINK, and other tools here.  A complete list can be found in here: https://github.com/DataBiosphere/terra-docker/blob/master/terra-jupyter-aou/Dockerfile
  • This image needs a lot of boot-disk-size, --boot-disk-size (default is 10) needs to be set at least 50-100 to start with.