From d6bfc449dfc6979511e746a52f6fddf0e30e7853 Mon Sep 17 00:00:00 2001
From: Ruben Vorderman <r.h.p.vorderman@lumc.nl>
Date: Fri, 11 Feb 2022 12:47:06 +0100
Subject: [PATCH] Speed up CI by using conda caching and only checking changed
files
Squashed commit of the following:
commit 7fa743cc028b8e2c86bde49244834ee13c13e95b
Author: Ruben Vorderman <r.h.p.vorderman@lumc.nl>
Date: Fri Feb 11 12:34:33 2022 +0100
Add comment about activate environment
commit 2de7802e03f90cd6e26b3d8287fcb0c6b8b81d11
Author: Ruben Vorderman <r.h.p.vorderman@lumc.nl>
Date: Fri Feb 11 12:26:38 2022 +0100
Invalidate cache
commit 8ca394d41361acf2511249e3e29688baf0705004
Author: Ruben Vorderman <r.h.p.vorderman@lumc.nl>
Date: Fri Feb 11 12:26:27 2022 +0100
Consolidate steps
commit 31d09c6f0e86d4625bfa3a6e94a7ced910c7410c
Author: Ruben Vorderman <r.h.p.vorderman@lumc.nl>
Date: Fri Feb 11 11:27:53 2022 +0100
Use correct path for caching
commit 7e1374ed323bb38d674da09d7270def4a2192d00
Author: Ruben Vorderman <r.h.p.vorderman@lumc.nl>
Date: Fri Feb 11 11:22:10 2022 +0100
Do not cache conda packages
commit deffd8a0776e15a4df58a1398fcbcb8b0f1430f0
Author: Ruben Vorderman <r.h.p.vorderman@lumc.nl>
Date: Fri Feb 11 11:20:59 2022 +0100
Remove unnecessary whitespace
commit 8e97bcd4dfd8ee459a23f1931465875c0a41fd49
Author: Ruben Vorderman <r.h.p.vorderman@lumc.nl>
Date: Fri Feb 11 11:20:03 2022 +0100
Remove debugging task
commit 8338cd4b843245d781d7028f1f1acad45c8c7d0d
Author: Ruben Vorderman <r.h.p.vorderman@lumc.nl>
Date: Fri Feb 11 11:17:18 2022 +0100
Try to change path
commit 6a75baa36eee340d7a6d766c89163e960a6203b0
Author: Ruben Vorderman <r.h.p.vorderman@lumc.nl>
Date: Fri Feb 11 11:12:18 2022 +0100
Delete path line in current github env
commit cbbb9fe67cb796a010c01760ca2e05986f979ced
Author: Ruben Vorderman <r.h.p.vorderman@lumc.nl>
Date: Fri Feb 11 11:05:50 2022 +0100
Properly activate
commit 671568b7c8d79a5141429068a32b72814110b361
Author: Ruben Vorderman <r.h.p.vorderman@lumc.nl>
Date: Fri Feb 11 10:59:46 2022 +0100
Also printenv
commit 4c8945e8d5305753482538389ddc8af892f493f9
Author: Ruben Vorderman <r.h.p.vorderman@lumc.nl>
Date: Fri Feb 11 10:56:45 2022 +0100
Manual activate
commit a925c53a99836e81eb0e2b21075356370906c641
Author: Ruben Vorderman <r.h.p.vorderman@lumc.nl>
Date: Fri Feb 11 10:53:10 2022 +0100
Reset cache number
commit 645ed2b4504d067ea1b26a0922943ef3d5c34622
Author: Ruben Vorderman <r.h.p.vorderman@lumc.nl>
Date: Fri Feb 11 10:51:09 2022 +0100
Activate environment path
commit 5852d29fb538b80f06a738677e7ae271c6c57fa3
Author: Ruben Vorderman <r.h.p.vorderman@lumc.nl>
Date: Fri Feb 11 10:31:07 2022 +0100
Proper setting for cache
commit 83f14a939d662d628ca47dc7b82bbc114f164541
Author: Ruben Vorderman <r.h.p.vorderman@lumc.nl>
Date: Fri Feb 11 10:03:45 2022 +0100
List environments
commit 59267fbba267c0b1726733e390ff471d7012cefa
Author: Ruben Vorderman <r.h.p.vorderman@lumc.nl>
Date: Fri Feb 11 10:01:58 2022 +0100
Activate environment manually
commit 0a4d2cd5644407308fcc78356a8aef55de86c0c6
Author: Ruben Vorderman <r.h.p.vorderman@lumc.nl>
Date: Fri Feb 11 09:57:32 2022 +0100
List environments
commit 0bc8fa939eb35a6eb352bb58b1235efecd34056f
Author: Ruben Vorderman <r.h.p.vorderman@lumc.nl>
Date: Fri Feb 11 09:52:02 2022 +0100
Add mambaforge comment
commit 719d92a0b5245be891d1b5c0eb38d8048abdc5a1
Author: Ruben Vorderman <r.h.p.vorderman@lumc.nl>
Date: Fri Feb 11 09:44:18 2022 +0100
Use normal conda, since environment is cached
commit e5efbb75109f40cfa8b7b33280ec9707a31970d1
Author: Ruben Vorderman <r.h.p.vorderman@lumc.nl>
Date: Fri Feb 11 09:38:21 2022 +0100
Also cache environments.txt
commit 4fa66afb6606ceeb7be577df9f20704d96fc3af0
Author: Ruben Vorderman <r.h.p.vorderman@lumc.nl>
Date: Fri Feb 11 09:34:46 2022 +0100
Check home
commit 2ac42e42829141650585780d27f39d06ebaf8f75
Author: Ruben Vorderman <r.h.p.vorderman@lumc.nl>
Date: Wed Feb 9 17:00:27 2022 +0100
Add an annoying but effective manual check
commit 78d88eae8cb3d1ca44709ce90bcffeb7c5786c1b
Author: Ruben Vorderman <r.h.p.vorderman@lumc.nl>
Date: Wed Feb 9 16:54:29 2022 +0100
Cache correct path
commit c05c94561785b1d5e198588dc210313014f3913d
Author: Ruben Vorderman <r.h.p.vorderman@lumc.nl>
Date: Wed Feb 9 16:45:51 2022 +0100
Rename workflow
commit 1c67f010c589c1c1fb407ac32e8ed74afdb3ddfd
Author: Ruben Vorderman <r.h.p.vorderman@lumc.nl>
Date: Wed Feb 9 16:45:05 2022 +0100
Use correct quotes
commit 7f9d2e559697e9d9d1f6df3514c8269612e7bcee
Author: Ruben Vorderman <r.h.p.vorderman@lumc.nl>
Date: Wed Feb 9 16:42:25 2022 +0100
Only check changed wdl files
commit 0e2a15b38e206fdb96d2d8b225999d6e5c9e6e73
Author: Ruben Vorderman <r.h.p.vorderman@lumc.nl>
Date: Wed Feb 9 16:34:35 2022 +0100
remove v parameter
commit 89348dde8a84cd1d935999255c64428c99db7042
Author: Ruben Vorderman <r.h.p.vorderman@lumc.nl>
Date: Wed Feb 9 16:19:02 2022 +0100
Remove newline
commit 752b8cb4a8407908348d8424fdc4b89d3219fdad
Author: Ruben Vorderman <r.h.p.vorderman@lumc.nl>
Date: Wed Feb 9 16:17:33 2022 +0100
Git fetch develop
commit 9216a3f846268ba00d0fe922055536b06dc975b3
Author: Ruben Vorderman <r.h.p.vorderman@lumc.nl>
Date: Wed Feb 9 15:53:13 2022 +0100
Specifically check origin
commit b54c140de4fc0bf31d7c95384831aedb253f35a3
Author: Ruben Vorderman <r.h.p.vorderman@lumc.nl>
Date: Wed Feb 9 15:44:50 2022 +0100
Only chek files that are different from the base with womtool validate
commit d963818753272aa18311d3d29276c3db6241e85d
Author: Ruben Vorderman <r.h.p.vorderman@lumc.nl>
Date: Wed Feb 9 15:33:50 2022 +0100
Correctly use data
commit 8113bfdd2e1feda6047e13da79885a3131c000e6
Author: Ruben Vorderman <r.h.p.vorderman@lumc.nl>
Date: Wed Feb 9 15:32:48 2022 +0100
Set correct env cache param
commit 4f7af2ed0365887be9147954290c4b807673afdd
Author: Ruben Vorderman <r.h.p.vorderman@lumc.nl>
Date: Wed Feb 9 15:30:23 2022 +0100
Add lint-evnironment
commit b026b5a8a77ea131b229a50cb28e0d301915cfb8
Author: Ruben Vorderman <r.h.p.vorderman@lumc.nl>
Date: Wed Feb 9 15:28:50 2022 +0100
Use mamba env update
commit 41fda1a9f52d56578a76f8bf185db86da2128a0e
Author: Ruben Vorderman <r.h.p.vorderman@lumc.nl>
Date: Wed Feb 9 15:25:21 2022 +0100
Use cache
commit fd1a64261bea956b6b31a26f5eaa38ce4a63121c
Author: Ruben Vorderman <r.h.p.vorderman@lumc.nl>
Date: Wed Feb 9 14:31:17 2022 +0100
Add missing done statement
commit 4a64eb43535f48e0558ba6c5dc408178784ef207
Merge: a36a227 f234b0e
Author: Ruben Vorderman <r.h.p.vorderman@lumc.nl>
Date: Wed Feb 9 14:23:48 2022 +0100
Merge branch 'develop' into BIOWDL-583
commit a36a2274116732bc8e3229a267fe35ee4d61e7da
Author: Ruben Vorderman <r.h.p.vorderman@lumc.nl>
Date: Wed Feb 9 14:23:26 2022 +0100
Implement all checks in lint.yml directly
commit 391bb0de9619e75293599a1be1d24322fd466f4c
Author: Ruben Vorderman <r.h.p.vorderman@lumc.nl>
Date: Wed Feb 9 14:11:33 2022 +0100
Use a separate lint file
commit 832a131cee403ec0ac7d983d6e82fd567ce1b246
Author: Ruben Vorderman <r.h.p.vorderman@lumc.nl>
Date: Tue Dec 14 16:32:30 2021 +0100
Use mamba-forge and mamba to install dependencies
---
.github/PULL_REQUEST_TEMPLATE.md | 1 +
.github/lint-environment.yml | 9 ++++
.github/workflows/ci.yml | 30 -----------
.github/workflows/lint.yml | 93 ++++++++++++++++++++++++++++++++
4 files changed, 103 insertions(+), 30 deletions(-)
create mode 100644 .github/lint-environment.yml
delete mode 100644 .github/workflows/ci.yml
create mode 100644 .github/workflows/lint.yml
diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md
index 3b4ec9a..372071e 100644
--- a/.github/PULL_REQUEST_TEMPLATE.md
+++ b/.github/PULL_REQUEST_TEMPLATE.md
@@ -2,3 +2,4 @@
- [ ] Pull request details were added to CHANGELOG.md.
- [ ] Documentation was updated (if required).
- [ ] `parameter_meta` was added/updated (if required).
+- [ ] Submodule branches are on develop or a tagged commit.
diff --git a/.github/lint-environment.yml b/.github/lint-environment.yml
new file mode 100644
index 0000000..63b538f
--- /dev/null
+++ b/.github/lint-environment.yml
@@ -0,0 +1,9 @@
+name: biowdl-lint
+channels:
+ - conda-forge
+ - bioconda
+ - defaults
+dependencies:
+ - cromwell
+ - wdl-aid
+ - miniwdl
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
deleted file mode 100644
index 7856611..0000000
--- a/.github/workflows/ci.yml
+++ /dev/null
@@ -1,30 +0,0 @@
-name: Continuous integration
-
-on:
- pull_request:
- paths_ignore:
- - "docs/**"
-
-defaults:
- run:
- # This is needed for miniconda, see:
- # https://github.com/marketplace/actions/setup-miniconda#important
- shell: bash -l {0}
-
-jobs:
- lint:
- runs-on: ubuntu-latest
- name: Womtool validate and submodule up to date.
- steps:
- - uses: actions/checkout@v2.3.4
- with:
- submodules: recursive
- - name: install miniconda
- uses: conda-incubator/setup-miniconda@v2.0.1
- with:
- channels: conda-forge,bioconda,defaults
- # Conda-incubator uses 'test' environment by default.
- - name: install requirements
- run: conda install -n test cromwell miniwdl wdl-aid
- - name: run linting
- run: bash scripts/biowdl_lint.sh
diff --git a/.github/workflows/lint.yml b/.github/workflows/lint.yml
new file mode 100644
index 0000000..e6edbba
--- /dev/null
+++ b/.github/workflows/lint.yml
@@ -0,0 +1,93 @@
+name: Linting
+
+on:
+ pull_request:
+ paths_ignore:
+ - "docs/**"
+
+defaults:
+ run:
+ # This is needed for miniconda, see:
+ # https://github.com/marketplace/actions/setup-miniconda#important
+ shell: bash -l {0}
+
+jobs:
+ lint:
+ runs-on: ubuntu-latest
+ name: Linting checks
+ steps:
+ - uses: actions/checkout@v2.3.4
+ with:
+ submodules: recursive
+
+ - name: Set cache date
+ run: echo "DATE=$(date +'%Y%m%d')" >> $GITHUB_ENV
+
+ - name: Cache conda environment
+ uses: actions/cache@v2.1.7
+ env:
+ # Increase this value to manually invalidate the cache
+ CACHE_NUMBER: 0
+ with:
+ path: /usr/share/miniconda/envs/biowdl-lint
+ key:
+ ${{runner.os}}-biowdl-lint-${{ env.CACHE_NUMBER }}-${{env.DATE}}-${{ hashFiles('.github/lint-environment.yml') }}
+ id: env_cache
+
+ # Use the builtin conda. This is the fastest installation. It may not be
+ # the fastest for resolving, but the package cache mitigates that problem.
+ # Since this installs fastest, it is fastest for all runs where a cache
+ # hit occurs.
+ - name: install miniconda
+ uses: conda-incubator/setup-miniconda@v2.1.1
+ with:
+ channels: conda-forge,bioconda,defaults
+ channel-priority: strict
+ auto-activate-base: false
+ use-only-tar-bz2: true # Needed for proper caching according to the documentation.
+ # activate-environment is broken! This always seems to create a new environment.
+ # Activation is therefore done separately.
+
+ - name: Create test environment if no cache is present
+ run: conda env create -n biowdl-lint -f .github/lint-environment.yml
+ if: steps.env_cache.outputs.cache-hit != 'true'
+
+ - name: Activate test environment
+ # The new PATH should be passed to the environment, otherwise it won't register.
+ run: |
+ conda activate biowdl-lint
+ echo "PATH=$PATH" >> $GITHUB_ENV
+
+ - name: Fetch develop branch for comparisons
+ run: git fetch --depth=1 origin develop
+
+ - name: run womtool validate
+ # Only check files that have changed from the base reference.
+ # Womtool validate checks very slowly, so this saves a lot of time.
+ run: |
+ bash -c "
+ for WDL_FILE in $(git diff --name-only origin/${{github.base_ref}} | grep -E '*.wdl$'); do
+ womtool validate $WDL_FILE
+ done
+ "
+ - name: run miniwdl check
+ run: bash -c 'miniwdl check $(git ls-files *.wdl)'
+
+ - name: Check copyright headers
+ run: |
+ bash -c '
+ for WDL_FILE in $(git diff --name-only origin/${{github.base_ref}} | grep -E '*.wdl$'); do
+ grep Copyright $WDL_FILE || bash -c "echo No copyright header in $WDL_FILE && exit 1"
+ done
+ '
+ - name: Check parameter_meta for inputs
+ run: |
+ bash -c "
+ for WDL_FILE in $(git diff --name-only origin/${{github.base_ref}} | grep -E '*.wdl$'); do
+ wdl-aid --strict $WDL_FILE > /dev/null 2> wdl-aid_stderr ||
+ if grep -z 'ValueError: Missing parameter_meta for inputs:' wdl-aid_stderr
+ then
+ exit 1
+ fi
+ done
+ "
--
GitLab