Speed up CI by using conda caching and only checking changed files

Squashed commit of the following: commit 7fa743cc028b8e2c86bde49244834ee13c13e95b Author: Ruben Vorderman <r.h.p.vorderman@lumc.nl> Date: Fri Feb 11 12:34:33 2022 +0100 Add comment about activate environment commit 2de7802e03f90cd6e26b3d8287fcb0c6b8b81d11 Author: Ruben Vorderman <r.h.p.vorderman@lumc.nl> Date: Fri Feb 11 12:26:38 2022 +0100 Invalidate cache commit 8ca394d41361acf2511249e3e29688baf0705004 Author: Ruben Vorderman <r.h.p.vorderman@lumc.nl> Date: Fri Feb 11 12:26:27 2022 +0100 Consolidate steps commit 31d09c6f0e86d4625bfa3a6e94a7ced910c7410c Author: Ruben Vorderman <r.h.p.vorderman@lumc.nl> Date: Fri Feb 11 11:27:53 2022 +0100 Use correct path for caching commit 7e1374ed323bb38d674da09d7270def4a2192d00 Author: Ruben Vorderman <r.h.p.vorderman@lumc.nl> Date: Fri Feb 11 11:22:10 2022 +0100 Do not cache conda packages commit deffd8a0776e15a4df58a1398fcbcb8b0f1430f0 Author: Ruben Vorderman <r.h.p.vorderman@lumc.nl> Date: Fri Feb 11 11:20:59 2022 +0100 Remove unnecessary whitespace commit 8e97bcd4dfd8ee459a23f1931465875c0a41fd49 Author: Ruben Vorderman <r.h.p.vorderman@lumc.nl> Date: Fri Feb 11 11:20:03 2022 +0100 Remove debugging task commit 8338cd4b843245d781d7028f1f1acad45c8c7d0d Author: Ruben Vorderman <r.h.p.vorderman@lumc.nl> Date: Fri Feb 11 11:17:18 2022 +0100 Try to change path commit 6a75baa36eee340d7a6d766c89163e960a6203b0 Author: Ruben Vorderman <r.h.p.vorderman@lumc.nl> Date: Fri Feb 11 11:12:18 2022 +0100 Delete path line in current github env commit cbbb9fe67cb796a010c01760ca2e05986f979ced Author: Ruben Vorderman <r.h.p.vorderman@lumc.nl> Date: Fri Feb 11 11:05:50 2022 +0100 Properly activate commit 671568b7c8d79a5141429068a32b72814110b361 Author: Ruben Vorderman <r.h.p.vorderman@lumc.nl> Date: Fri Feb 11 10:59:46 2022 +0100 Also printenv commit 4c8945e8d5305753482538389ddc8af892f493f9 Author: Ruben Vorderman <r.h.p.vorderman@lumc.nl> Date: Fri Feb 11 10:56:45 2022 +0100 Manual activate commit a925c53a99836e81eb0e2b21075356370906c641 Author: Ruben Vorderman <r.h.p.vorderman@lumc.nl> Date: Fri Feb 11 10:53:10 2022 +0100 Reset cache number commit 645ed2b4504d067ea1b26a0922943ef3d5c34622 Author: Ruben Vorderman <r.h.p.vorderman@lumc.nl> Date: Fri Feb 11 10:51:09 2022 +0100 Activate environment path commit 5852d29fb538b80f06a738677e7ae271c6c57fa3 Author: Ruben Vorderman <r.h.p.vorderman@lumc.nl> Date: Fri Feb 11 10:31:07 2022 +0100 Proper setting for cache commit 83f14a939d662d628ca47dc7b82bbc114f164541 Author: Ruben Vorderman <r.h.p.vorderman@lumc.nl> Date: Fri Feb 11 10:03:45 2022 +0100 List environments commit 59267fbba267c0b1726733e390ff471d7012cefa Author: Ruben Vorderman <r.h.p.vorderman@lumc.nl> Date: Fri Feb 11 10:01:58 2022 +0100 Activate environment manually commit 0a4d2cd5644407308fcc78356a8aef55de86c0c6 Author: Ruben Vorderman <r.h.p.vorderman@lumc.nl> Date: Fri Feb 11 09:57:32 2022 +0100 List environments commit 0bc8fa939eb35a6eb352bb58b1235efecd34056f Author: Ruben Vorderman <r.h.p.vorderman@lumc.nl> Date: Fri Feb 11 09:52:02 2022 +0100 Add mambaforge comment commit 719d92a0b5245be891d1b5c0eb38d8048abdc5a1 Author: Ruben Vorderman <r.h.p.vorderman@lumc.nl> Date: Fri Feb 11 09:44:18 2022 +0100 Use normal conda, since environment is cached commit e5efbb75109f40cfa8b7b33280ec9707a31970d1 Author: Ruben Vorderman <r.h.p.vorderman@lumc.nl> Date: Fri Feb 11 09:38:21 2022 +0100 Also cache environments.txt commit 4fa66afb6606ceeb7be577df9f20704d96fc3af0 Author: Ruben Vorderman <r.h.p.vorderman@lumc.nl> Date: Fri Feb 11 09:34:46 2022 +0100 Check home commit 2ac42e42 Author: Ruben Vorderman <r.h.p.vorderman@lumc.nl> Date: Wed Feb 9 17:00:27 2022 +0100 Add an annoying but effective manual check commit 78d88eae Author: Ruben Vorderman <r.h.p.vorderman@lumc.nl> Date: Wed Feb 9 16:54:29 2022 +0100 Cache correct path commit c05c9456 Author: Ruben Vorderman <r.h.p.vorderman@lumc.nl> Date: Wed Feb 9 16:45:51 2022 +0100 Rename workflow commit 1c67f010 Author: Ruben Vorderman <r.h.p.vorderman@lumc.nl> Date: Wed Feb 9 16:45:05 2022 +0100 Use correct quotes commit 7f9d2e55 Author: Ruben Vorderman <r.h.p.vorderman@lumc.nl> Date: Wed Feb 9 16:42:25 2022 +0100 Only check changed wdl files commit 0e2a15b3 Author: Ruben Vorderman <r.h.p.vorderman@lumc.nl> Date: Wed Feb 9 16:34:35 2022 +0100 remove v parameter commit 89348dde Author: Ruben Vorderman <r.h.p.vorderman@lumc.nl> Date: Wed Feb 9 16:19:02 2022 +0100 Remove newline commit 752b8cb4 Author: Ruben Vorderman <r.h.p.vorderman@lumc.nl> Date: Wed Feb 9 16:17:33 2022 +0100 Git fetch develop commit 9216a3f8 Author: Ruben Vorderman <r.h.p.vorderman@lumc.nl> Date: Wed Feb 9 15:53:13 2022 +0100 Specifically check origin commit b54c140d Author: Ruben Vorderman <r.h.p.vorderman@lumc.nl> Date: Wed Feb 9 15:44:50 2022 +0100 Only chek files that are different from the base with womtool validate commit d9638187 Author: Ruben Vorderman <r.h.p.vorderman@lumc.nl> Date: Wed Feb 9 15:33:50 2022 +0100 Correctly use data commit 8113bfdd Author: Ruben Vorderman <r.h.p.vorderman@lumc.nl> Date: Wed Feb 9 15:32:48 2022 +0100 Set correct env cache param commit 4f7af2ed Author: Ruben Vorderman <r.h.p.vorderman@lumc.nl> Date: Wed Feb 9 15:30:23 2022 +0100 Add lint-evnironment commit b026b5a8 Author: Ruben Vorderman <r.h.p.vorderman@lumc.nl> Date: Wed Feb 9 15:28:50 2022 +0100 Use mamba env update commit 41fda1a9 Author: Ruben Vorderman <r.h.p.vorderman@lumc.nl> Date: Wed Feb 9 15:25:21 2022 +0100 Use cache commit fd1a6426 Author: Ruben Vorderman <r.h.p.vorderman@lumc.nl> Date: Wed Feb 9 14:31:17 2022 +0100 Add missing done statement commit 4a64eb43 Merge: a36a2274 f234b0e8 Author: Ruben Vorderman <r.h.p.vorderman@lumc.nl> Date: Wed Feb 9 14:23:48 2022 +0100 Merge branch 'develop' into BIOWDL-583 commit a36a2274 Author: Ruben Vorderman <r.h.p.vorderman@lumc.nl> Date: Wed Feb 9 14:23:26 2022 +0100 Implement all checks in lint.yml directly commit 391bb0de Author: Ruben Vorderman <r.h.p.vorderman@lumc.nl> Date: Wed Feb 9 14:11:33 2022 +0100 Use a separate lint file commit 832a131c Author: Ruben Vorderman <r.h.p.vorderman@lumc.nl> Date: Tue Dec 14 16:32:30 2021 +0100 Use mamba-forge and mamba to install dependencies

Speed up CI by using conda caching and only checking changed files
d6bfc449 · Ruben Vorderman · f234b0e8 · d6bfc449 · d6bfc449 · f234b0e8
Commit d6bfc449 authored 3 years ago by Ruben Vorderman
--- a/.github/PULL_REQUEST_TEMPLATE.md
+++ b/.github/PULL_REQUEST_TEMPLATE.md
@@ -2,3 +2,4 @@
 - [ ] Pull request details were added to CHANGELOG.md.
 - [ ] Documentation was updated (if required).
 - [ ] `parameter_meta` was added/updated (if required).
+- [ ] Submodule branches are on develop or a tagged commit.
--- a/.github/lint-environment.yml
+++ b/.github/lint-environment.yml
+name: biowdl-lint
+channels:
+  - conda-forge
+  - bioconda
+  - defaults
+dependencies:
+  - cromwell
+  - wdl-aid
+  - miniwdl
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
-name: Continuous integration
-
-on: 
-  pull_request:
-    paths_ignore:
-      - "docs/**"
-
-defaults:
-  run:
-    # This is needed for miniconda, see:
-    # https://github.com/marketplace/actions/setup-miniconda#important
-    shell: bash -l {0}  
-
-jobs:
-  lint:
-    runs-on: ubuntu-latest
-    name: Womtool validate and submodule up to date.
-    steps:
-      - uses: actions/checkout@v2.3.4
-        with:
-          submodules: recursive
-      - name: install miniconda
-        uses: conda-incubator/setup-miniconda@v2.0.1
-        with:
-          channels: conda-forge,bioconda,defaults
-        # Conda-incubator uses 'test' environment by default.
-      - name: install requirements
-        run: conda install -n test cromwell miniwdl wdl-aid
-      - name: run linting
-        run: bash scripts/biowdl_lint.sh
--- a/.github/workflows/lint.yml
+++ b/.github/workflows/lint.yml
+name: Linting
+
+on: 
+  pull_request:
+    paths_ignore:
+      - "docs/**"
+
+defaults:
+  run:
+    # This is needed for miniconda, see:
+    # https://github.com/marketplace/actions/setup-miniconda#important
+    shell: bash -l {0}  
+
+jobs:
+  lint:
+    runs-on: ubuntu-latest
+    name: Linting checks
+    steps:
+      - uses: actions/checkout@v2.3.4
+        with:
+          submodules: recursive
+      
+      - name: Set cache date
+        run: echo "DATE=$(date +'%Y%m%d')" >> $GITHUB_ENV
+
+      - name: Cache conda environment
+        uses: actions/cache@v2.1.7
+        env:
+          # Increase this value to manually invalidate the cache
+          CACHE_NUMBER: 0
+        with:
+          path: /usr/share/miniconda/envs/biowdl-lint
+          key:
+            ${{runner.os}}-biowdl-lint-${{ env.CACHE_NUMBER }}-${{env.DATE}}-${{ hashFiles('.github/lint-environment.yml') }}
+        id: env_cache
+      
+      # Use the builtin conda. This is the fastest installation. It may not be
+      # the fastest for resolving, but the package cache mitigates that problem.
+      # Since this installs fastest, it is fastest for all runs where a cache
+      # hit occurs.
+      - name: install miniconda
+        uses: conda-incubator/setup-miniconda@v2.1.1
+        with:
+          channels: conda-forge,bioconda,defaults
+          channel-priority: strict
+          auto-activate-base: false
+          use-only-tar-bz2: true  # Needed for proper caching according to the documentation.
+        # activate-environment is broken! This always seems to create a new environment.
+        # Activation is therefore done separately.
+      
+      - name: Create test environment if no cache is present
+        run: conda env create -n biowdl-lint -f .github/lint-environment.yml
+        if: steps.env_cache.outputs.cache-hit != 'true'
+      
+      - name: Activate test environment
+        # The new PATH should be passed to the environment, otherwise it won't register.
+        run: |
+          conda activate biowdl-lint
+          echo "PATH=$PATH" >> $GITHUB_ENV
+      
+      - name: Fetch develop branch for comparisons
+        run: git fetch --depth=1 origin develop
+
+      - name: run womtool validate
+        # Only check files that have changed from the base reference.
+        # Womtool validate checks very slowly, so this saves a lot of time.
+        run: |
+          bash -c "
+            for WDL_FILE in $(git diff --name-only origin/${{github.base_ref}} | grep -E '*.wdl$'); do
+              womtool validate $WDL_FILE
+            done
+          "
+      - name: run miniwdl check
+        run: bash -c 'miniwdl check $(git ls-files *.wdl)'
+
+      - name: Check copyright headers
+        run: | 
+          bash -c '
+            for WDL_FILE in $(git diff --name-only origin/${{github.base_ref}} | grep -E '*.wdl$'); do
+              grep Copyright $WDL_FILE || bash -c "echo No copyright header in $WDL_FILE && exit 1"
+            done
+          '
+      - name: Check parameter_meta for inputs
+        run: |
+          bash -c "
+            for WDL_FILE in $(git diff --name-only origin/${{github.base_ref}} | grep -E '*.wdl$'); do
+              wdl-aid --strict $WDL_FILE > /dev/null 2> wdl-aid_stderr || 
+              if grep -z  'ValueError: Missing parameter_meta for inputs:' wdl-aid_stderr
+              then
+                 exit 1
+              fi
+            done
+          "