Pipeline’s Repository Structure

To be picked up correctly by some of the commands, a repository needs to be set up as follow:

• A descriptions folder to store workflow description files (CWL and WDL).

• A dockerfiles folder to store Docker images. Each image should have its own subfolder with all the required components and the Dockerfile. Subfolder names will be used to tag the corresponding images together with the version from the VERSION file.

• A portal_objects folder to store the objects representing metadata for the pipeline. This folder should include several subfolders:

  • A workflows folder to store metadata for Workflow objects as YAML files.

  • A metaworkflows folder to store metadata for Pipeline objects as YAML files.

  • A file_format.yaml file to store metadata for File Format objects.

  • A file_reference.yaml file to store metadata for File Reference objects.

  • A software.yaml file to store metadata for Software objects.

• A PIPELINE one line file with the pipeline name.

• A VERSION one line file with the pipeline version.

Example foo_bar pipeline:

pipeline-foo_bar
│
├── descriptions
│   ├── foo.cwl
│   └── bar.wdl
│
├── dockerfiles
│   │
│   ├── image_foo
│   │   ├── foo.sh
│   │   └── Dockerfile
│   │
│   └── image_bar
│       ├── bar.py
│       └── Dockerfile
│
├── portal_objects
│   │
│   ├── workflows
│   │   ├── foo.yaml
│   │   └── bar.yaml
│   │
│   ├── metaworkflows
│   │   └── foo_bar.yaml
│   │
│   ├── file_format.yaml
│   ├── file_reference.yaml
│   └── software.yaml
│
├── PIPELINE
└── VERSION

Real examples can be found linked as submodules in our pipelines repository for CGAP project here: https://github.com/dbmi-bgm/cgap-pipeline-main.