You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/04/16 07:59:40 UTC
[GitHub] [airflow] potiuk opened a new pull request #8393: Bring back CI
optimisations
potiuk opened a new pull request #8393: Bring back CI optimisations
URL: https://github.com/apache/airflow/pull/8393
---
Make sure to mark the boxes below before creating PR: [x]
- [x] Description above provides context of the change
- [x] Unit tests coverage for changes (not needed for documentation changes)
- [x] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)"
- [x] Relevant documentation is updated including usage instructions.
- [x] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
---
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] ashb commented on a change in pull request #8393: Bring
back CI optimisations
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #8393: Bring back CI optimisations
URL: https://github.com/apache/airflow/pull/8393#discussion_r409466577
##########
File path: scripts/ci/ci_flake8.sh
##########
@@ -50,6 +50,8 @@ function run_flake8() {
fi
}
+get_ci_environment
Review comment:
Can this be called from prepare_ci_build? It seems everywhere we call one we call the other.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] turbaszek commented on a change in pull request #8393:
Bring back CI optimisations
Posted by GitBox <gi...@apache.org>.
turbaszek commented on a change in pull request #8393: Bring back CI optimisations
URL: https://github.com/apache/airflow/pull/8393#discussion_r410037855
##########
File path: .github/workflows/ci.yml
##########
@@ -80,134 +80,152 @@ jobs:
name: Build docs
runs-on: ubuntu-latest
env:
- TRAVIS_JOB_NAME: "Build documentation"
PYTHON_VERSION: 3.6
+ CI_JOB_TYPE: "Documentation"
steps:
- uses: actions/checkout@master
- - name: "Build documentation"
+ - name: "Build CI image"
+ run: ./scripts/ci/ci_prepare_image_on_ci.sh
+ - name: "Build docs"
run: ./scripts/ci/ci_docs.sh
tests-p36-postgres-integrations:
- name: "Tests [Postgres9.6][Py3.6][integrations]"
+ name: "[Pg9.6][Py3.6][integrations]"
runs-on: ubuntu-latest
needs: [statics, statics-tests]
env:
- TRAVIS_JOB_NAME: "Tests [Postgres9.6][Py3.6][integrations]"
BACKEND: postgres
PYTHON_VERSION: 3.6
POSTGRES_VERSION: 9.6
ENABLED_INTEGRATIONS: "cassandra kerberos mongo openldap rabbitmq redis"
RUN_INTEGRATION_TESTS: all
+ CI_JOB_TYPE: "Tests"
steps:
- uses: actions/checkout@master
- - name: "Tests [Postgres9.6][Py3.6][integrations]"
+ - name: "Build CI image"
+ run: ./scripts/ci/ci_prepare_image_on_ci.sh
+ - name: "Tests"
run: ./scripts/ci/ci_run_airflow_testing.sh
tests-p36-postgres-providers:
Review comment:
This will limit number of concurrent jobs, so quite a good idea
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #8393: Bring
back CI optimisations
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #8393: Bring back CI optimisations
URL: https://github.com/apache/airflow/pull/8393#discussion_r409433130
##########
File path: .github/workflows/ci.yml
##########
@@ -19,58 +19,58 @@
---
name: Airflow
Review comment:
Airflow CI Build :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on issue #8393: Bring back CI
optimisations
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #8393: Bring back CI optimisations
URL: https://github.com/apache/airflow/pull/8393#issuecomment-614546418
Should be fixed. I also fixed variables retrieved from github ..
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] turbaszek commented on issue #8393: Bring back CI
optimisations
Posted by GitBox <gi...@apache.org>.
turbaszek commented on issue #8393: Bring back CI optimisations
URL: https://github.com/apache/airflow/pull/8393#issuecomment-615084010
So here's some input from Github support:
> > But the job was still running (?) and it got cancelled. Have you faced such issues?
> There are a few reasons a job would be cancelled– timeouts , fail-fast setting or the user has cancelled the job.
In this case it appears that the job was cancelled due to the fail-fast setting in which the default is true, the one failure must have triggered this.
https://help.github.com/en/actions/reference/workflow-syntax-for-github-actions#jobsjob_idstrategyfail-fast
"When set to true, GitHub cancels all in-progress jobs if any matrix job fails. Default: true"
And this one is I think worth considering @potiuk
> > Also are there any plans to allow reruns of single build? Reruning 70min of workflows doesn't sounds as best option :)
> I'm not aware of any plans for this, another approach would be to have the jobs spread out into separate workflows.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #8393: Bring
back CI optimisations
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #8393: Bring back CI optimisations
URL: https://github.com/apache/airflow/pull/8393#discussion_r409466152
##########
File path: .github/workflows/ci.yml
##########
@@ -19,58 +19,58 @@
---
name: Airflow
Review comment:
CI Build os fine for me :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #8393: Bring
back CI optimisations
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #8393: Bring back CI optimisations
URL: https://github.com/apache/airflow/pull/8393#discussion_r409473686
##########
File path: scripts/ci/ci_flake8.sh
##########
@@ -50,6 +50,8 @@ function run_flake8() {
fi
}
+get_ci_environment
Review comment:
```
get_ci_environment
# In case of CRON jobs we run builds without cache and upgrade to latest requirements
if [[ "${CI_EVENT_TYPE:=}" == "schedule" ]]; then
echo
echo "Disabling cache for scheduled jobs"
echo
export DOCKER_CACHE="no-cache"
export PULL_BASE_IMAGES="true"
export UPGRADE_TO_LATEST_REQUIREMENTS="true"
fi
prepare_ci_build
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] turbaszek commented on a change in pull request #8393:
Bring back CI optimisations
Posted by GitBox <gi...@apache.org>.
turbaszek commented on a change in pull request #8393: Bring back CI optimisations
URL: https://github.com/apache/airflow/pull/8393#discussion_r409458571
##########
File path: .github/workflows/ci.yml
##########
@@ -19,58 +19,58 @@
---
name: Airflow
Review comment:
Long one :D The name is displayed in the README badge (so in case of change we should update it too)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] turbaszek commented on a change in pull request #8393:
Bring back CI optimisations
Posted by GitBox <gi...@apache.org>.
turbaszek commented on a change in pull request #8393: Bring back CI optimisations
URL: https://github.com/apache/airflow/pull/8393#discussion_r410708095
##########
File path: .github/workflows/ci.yml
##########
@@ -17,272 +17,275 @@
#
---
-name: Airflow
+name: CI Build
on:
+ schedule:
+ - cron: '10 2 * * *'
push:
- branches: ['*']
+ branches: ['master', 'v1-10-test', 'v1-10-stable']
pull_request:
- branches: [master]
+ branches: ['master', 'v1-10-test', 'v1-10-stable']
env:
- BUILD_ID: ${{github.sha }}
MOUNT_LOCAL_SOURCES: "false"
- MOUNT_HOST_AIRFLOW_VOLUME: "true"
FORCE_ANSWER_TO_QUESTIONS: "yes"
SKIP_CHECK_REMOTE_IMAGE: "true"
+ SKIP_CI_IMAGE_CHECK: "true"
DB_RESET: "true"
VERBOSE: "true"
- CI: "true"
- # Should be a target branch
- TRAVIS_BRANCH: "master"
- TRAVIS: "true"
+ UPGRADE_TO_LATEST_REQUIREMENTS: "false"
+ ENABLED_INTEGRATIONS: "cassandra kerberos mongo openldap presto rabbitmq redis"
+ PYTHON_MAJOR_MINOR_VERSION: 3.6
+ # TODO: Enable cache registry once a master build succeeds uploading the images to Github registry cache
+ DISABLE_CACHE_REGISTRY: "true"
+ CACHE_REGISTRY: "docker.pkg.github.com"
+ CACHE_IMAGE_PREFIX: ${{ github.repository }}
+ CACHE_REGISTRY_USERNAME: ${{ github.actor }}
+ CACHE_REGISTRY_PASSWORD: ${{ secrets.GITHUB_TOKEN }}
+
jobs:
- statics:
- name: Static checks
+
+ static-checks-1:
+ name: "Checks: pylint, mypy"
runs-on: ubuntu-latest
env:
- PYTHON_VERSION: 3.6
- AIRFLOW_MOUNT_SOURCE_DIR_FOR_STATIC_CHECKS: "true"
- TRAVIS_JOB_NAME: "Static"
+ MOUNT_SOURCE_DIR_FOR_STATIC_CHECKS: "true"
+ CI_JOB_TYPE: "Pylint checks for tests"
steps:
- uses: actions/checkout@master
- - name: free disk space
- run: |
- sudo swapoff -a
- sudo rm -f /swapfile
- sudo apt clean
- docker rmi $(docker image ls -aq)
- df -h
- uses: actions/setup-python@v1
with:
- python-version: ${{ env.PYTHON_VERSION }}
- - name: "Before install"
- run: ./scripts/ci/ci_before_install.sh
- - name: "Static checks"
- if: success()
+ python-version: '3.x'
+ - name: Cache pre-commit env
+ uses: actions/cache@v1
env:
- PYTHON_VERSION: 3.6
- run: ./scripts/ci/ci_run_all_static_checks.sh
+ cache-name: cache-pre-commit
+ with:
+ path: ~/.cache/pre-commit
Review comment:
Can you explain the purpose of this cache?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #8393: Bring
back CI optimisations
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #8393: Bring back CI optimisations
URL: https://github.com/apache/airflow/pull/8393#discussion_r410101124
##########
File path: .github/workflows/ci.yml
##########
@@ -80,134 +80,152 @@ jobs:
name: Build docs
runs-on: ubuntu-latest
env:
- TRAVIS_JOB_NAME: "Build documentation"
PYTHON_VERSION: 3.6
+ CI_JOB_TYPE: "Documentation"
steps:
- uses: actions/checkout@master
- - name: "Build documentation"
+ - name: "Build CI image"
+ run: ./scripts/ci/ci_prepare_image_on_ci.sh
+ - name: "Build docs"
run: ./scripts/ci/ci_docs.sh
tests-p36-postgres-integrations:
- name: "Tests [Postgres9.6][Py3.6][integrations]"
+ name: "[Pg9.6][Py3.6][integrations]"
runs-on: ubuntu-latest
needs: [statics, statics-tests]
env:
- TRAVIS_JOB_NAME: "Tests [Postgres9.6][Py3.6][integrations]"
BACKEND: postgres
PYTHON_VERSION: 3.6
POSTGRES_VERSION: 9.6
ENABLED_INTEGRATIONS: "cassandra kerberos mongo openldap rabbitmq redis"
RUN_INTEGRATION_TESTS: all
+ CI_JOB_TYPE: "Tests"
steps:
- uses: actions/checkout@master
- - name: "Tests [Postgres9.6][Py3.6][integrations]"
+ - name: "Build CI image"
+ run: ./scripts/ci/ci_prepare_image_on_ci.sh
+ - name: "Tests"
run: ./scripts/ci/ci_run_airflow_testing.sh
tests-p36-postgres-providers:
Review comment:
Working on it. I will also (I think) incorporate a great idea from @turbaszek -> It turns out that (similarly as in GitLab POC I tried before) each project has a private GitHub Docker registry that we can use to store the images, so for every build we will just build one CI image with latest sources and use it for all the jobs. I discussed it with Tomek this morning and I thought a bit about it and I can do it very easily now within this CI "optimisation" build.
Bear with me. I will finish it today :).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #8393: Bring
back CI optimisations
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #8393: Bring back CI optimisations
URL: https://github.com/apache/airflow/pull/8393#discussion_r410725421
##########
File path: .github/workflows/ci.yml
##########
@@ -17,272 +17,275 @@
#
---
-name: Airflow
+name: CI Build
on:
+ schedule:
+ - cron: '10 2 * * *'
push:
- branches: ['*']
+ branches: ['master', 'v1-10-test', 'v1-10-stable']
pull_request:
- branches: [master]
+ branches: ['master', 'v1-10-test', 'v1-10-stable']
env:
- BUILD_ID: ${{github.sha }}
MOUNT_LOCAL_SOURCES: "false"
- MOUNT_HOST_AIRFLOW_VOLUME: "true"
FORCE_ANSWER_TO_QUESTIONS: "yes"
SKIP_CHECK_REMOTE_IMAGE: "true"
+ SKIP_CI_IMAGE_CHECK: "true"
DB_RESET: "true"
VERBOSE: "true"
- CI: "true"
- # Should be a target branch
- TRAVIS_BRANCH: "master"
- TRAVIS: "true"
+ UPGRADE_TO_LATEST_REQUIREMENTS: "false"
+ ENABLED_INTEGRATIONS: "cassandra kerberos mongo openldap presto rabbitmq redis"
+ PYTHON_MAJOR_MINOR_VERSION: 3.6
+ # TODO: Enable cache registry once a master build succeeds uploading the images to Github registry cache
+ DISABLE_CACHE_REGISTRY: "true"
+ CACHE_REGISTRY: "docker.pkg.github.com"
+ CACHE_IMAGE_PREFIX: ${{ github.repository }}
+ CACHE_REGISTRY_USERNAME: ${{ github.actor }}
+ CACHE_REGISTRY_PASSWORD: ${{ secrets.GITHUB_TOKEN }}
+
jobs:
- statics:
- name: Static checks
+
+ static-checks-1:
+ name: "Checks: pylint, mypy"
runs-on: ubuntu-latest
env:
- PYTHON_VERSION: 3.6
- AIRFLOW_MOUNT_SOURCE_DIR_FOR_STATIC_CHECKS: "true"
- TRAVIS_JOB_NAME: "Static"
+ MOUNT_SOURCE_DIR_FOR_STATIC_CHECKS: "true"
+ CI_JOB_TYPE: "Pylint checks for tests"
steps:
- uses: actions/checkout@master
- - name: free disk space
- run: |
- sudo swapoff -a
- sudo rm -f /swapfile
- sudo apt clean
- docker rmi $(docker image ls -aq)
- df -h
- uses: actions/setup-python@v1
with:
- python-version: ${{ env.PYTHON_VERSION }}
- - name: "Before install"
- run: ./scripts/ci/ci_before_install.sh
- - name: "Static checks"
- if: success()
+ python-version: '3.x'
+ - name: Cache pre-commit env
+ uses: actions/cache@v1
env:
- PYTHON_VERSION: 3.6
- run: ./scripts/ci/ci_run_all_static_checks.sh
+ cache-name: cache-pre-commit
+ with:
+ path: ~/.cache/pre-commit
+ key: ${{ env.cache-name }}-${{ github.job }}-${{ hashFiles('.pre-commit-config.yaml') }}
+ - name: "Free space"
+ run: ./scripts/ci/ci_free_space_on_ci.sh
+ - name: "Build CI image"
+ run: ./scripts/ci/ci_prepare_image_on_ci.sh
+ - name: "Pylint checks for tests"
+ run: |
+ python -m pip install pre-commit \
+ --constraint requirements/requirements-python${PYTHON_MAJOR_MINOR_VERSION}.txt
+ ./scripts/ci/ci_run_static_checks.sh pylint-tests mypy
- statics-tests:
- name: Pylint for tests
+ static-checks-2:
+ name: "Checks: All other"
runs-on: ubuntu-latest
env:
- PYTHON_VERSION: 3.6
- AIRFLOW_MOUNT_SOURCE_DIR_FOR_STATIC_CHECKS: "true"
- TRAVIS_JOB_NAME: "Static"
+ SKIP: pylint-tests,mypy
+ MOUNT_SOURCE_DIR_FOR_STATIC_CHECKS: "true"
+ CI_JOB_TYPE: "Static checks"
steps:
- uses: actions/checkout@master
- uses: actions/setup-python@v1
with:
- python-version: ${{ env.PYTHON_VERSION }}
- - name: "Before install"
- run: ./scripts/ci/ci_before_install.sh
- - name: "Static checks tests"
- if: success()
+ python-version: '3.x'
+ - name: Cache pre-commit env
+ uses: actions/cache@v1
env:
- PYTHON_VERSION: 3.6
- run: ./scripts/ci/ci_run_static_checks_pylint_tests.sh
+ cache-name: cache-pre-commit
+ with:
+ path: ~/.cache/pre-commit
+ key: ${{ env.cache-name }}-${{ github.job }}-${{ hashFiles('.pre-commit-config.yaml') }}
+ - name: "Free space"
+ run: ./scripts/ci/ci_free_space_on_ci.sh
+ - name: "Build CI image"
+ run: ./scripts/ci/ci_prepare_image_on_ci.sh
+ - name: "Static checks"
+ run: |
+ python -m pip install pre-commit \
+ --constraint requirements/requirements-python${PYTHON_MAJOR_MINOR_VERSION}.txt
+ ./scripts/ci/ci_run_static_checks.sh
docs:
name: Build docs
runs-on: ubuntu-latest
env:
- TRAVIS_JOB_NAME: "Build documentation"
- PYTHON_VERSION: 3.6
+ CI_JOB_TYPE: "Documentation"
steps:
- uses: actions/checkout@master
- - name: "Build documentation"
+ - name: "Build CI image ${{ matrix.python-version }}"
+ run: ./scripts/ci/ci_prepare_image_on_ci.sh
+ - name: "Build docs"
run: ./scripts/ci/ci_docs.sh
- tests-p36-postgres-integrations:
- name: "Tests [Postgres9.6][Py3.6][integrations]"
+ build-prod-image:
+ name: "Build prod image"
runs-on: ubuntu-latest
- needs: [statics, statics-tests]
+ strategy:
+ matrix:
+ python-version: [3.6, 3.7]
env:
- TRAVIS_JOB_NAME: "Tests [Postgres9.6][Py3.6][integrations]"
- BACKEND: postgres
- PYTHON_VERSION: 3.6
- POSTGRES_VERSION: 9.6
- ENABLED_INTEGRATIONS: "cassandra kerberos mongo openldap rabbitmq redis"
- RUN_INTEGRATION_TESTS: all
+ PYTHON_MAJOR_MINOR_VERSION: ${{ matrix.python-version }}
+ CI_JOB_TYPE: "Prod image"
steps:
- uses: actions/checkout@master
- - name: free disk space
- run: |
- sudo swapoff -a
- sudo rm -f /swapfile
- sudo apt clean
- docker rmi $(docker image ls -aq)
- df -h
- - name: "Tests [Postgres9.6][Py3.6][integrations]"
- run: ./scripts/ci/ci_run_airflow_testing.sh
+ - name: "Build PROD image ${{ matrix.python-version }}"
+ run: ./scripts/ci/ci_build_production_images.sh
- tests-p36-postgres-providers:
- name: "Tests [Postgres10][Py3.6][providers]"
+ prepare-backport-packages:
+ name: "Backport packages"
runs-on: ubuntu-latest
- needs: [statics, statics-tests]
env:
- TRAVIS_JOB_NAME: "Tests [Postgres10][Py3.6][providers]"
- BACKEND: postgres
- POSTGRES_VERSION: 10
- PYTHON_VERSION: 3.6
+ INSTALL_AIRFLOW_VERSION: "1.10.10"
+ CI_JOB_TYPE: "Prepare & test packages"
steps:
- uses: actions/checkout@master
- - name: "Tests [Postgres10][Py3.6][providers]"
- run: ./scripts/ci/ci_run_airflow_testing.sh tests/providers
+ - uses: actions/setup-python@v1
+ with:
+ python-version: '3.x'
+ - name: "Prepare & test backport packages"
+ run: |
+ python -m pip install bowler sdist wheel
+ ./scripts/ci/ci_prepare_and_test_backport_packages.sh
- tests-p36-postgres-core:
- name: "Tests [Postgres9.6][Py3.6][core]"
+ pyfiles:
+ name: "Count changed python files"
runs-on: ubuntu-latest
- needs: [statics, statics-tests]
- env:
- TRAVIS_JOB_NAME: "Tests [Postgres9.6][Py3.6][core]"
- BACKEND: postgres
- POSTGRES_VERSION: 9.6
- PYTHON_VERSION: 3.6
+ outputs:
+ count: ${{ steps.pyfiles.outputs.count }}
steps:
- uses: actions/checkout@master
- - name: "Tests [Postgres9.6][Py3.6][core]"
- run: ./scripts/ci/ci_run_airflow_testing.sh --ignore=tests/providers
-
+ - name: "Get count of changed python files"
+ run: |
+ set +e
+ ./scripts/ci/ci_count_changed_files.sh ${GITHUB_SHA} '\.py$'
+ echo "::set-output name=count::$?"
+ id: pyfiles
- tests-p37-sqlite-integrations:
- name: "Tests [Sqlite][3.7][integrations]"
+ tests-postgres:
+ name: "Tests: Pg${{ matrix.postgres-version }}, Py{{ matrix.python-version}}"
runs-on: ubuntu-latest
- needs: [statics, statics-tests]
+ needs: [static-checks-1, static-checks-2, pyfiles]
+ strategy:
+ matrix:
+ python-version: [3.6, 3.7]
+ postgres-version: [9.6, 10]
+ fail-fast: false
env:
- TRAVIS_JOB_NAME: "Tests [Sqlite][3.7][integrations]"
- BACKEND: sqlite
- PYTHON_VERSION: 3.7
- ENABLED_INTEGRATIONS: "cassandra kerberos mongo openldap rabbitmq redis"
- RUN_INTEGRATION_TESTS: all
+ BACKEND: postgres
+ PYTHON_MAJOR_MINOR_VERSION: ${{ matrix.python-version }}
+ POSTGRES_VERSION: ${{ matrix.postgres-version }}
+ RUN_TESTS: "true"
+ CI_JOB_TYPE: "Tests"
+ # For pull requests only run tests when python files changed
+ if: needs.pyfiles.outputs.count != '0' || github.event_name != 'pull_request'
steps:
- uses: actions/checkout@master
- - name: free disk space
- run: |
- sudo swapoff -a
- sudo rm -f /swapfile
- sudo apt clean
- docker rmi $(docker image ls -aq)
- df -h
- - name: "Tests [Sqlite][3.7][integrations]"
+ - uses: actions/setup-python@v1
+ with:
+ python-version: '3.x'
+ - name: "Free space"
+ run: ./scripts/ci/ci_free_space_on_ci.sh
+ - name: "Build CI image ${{ matrix.python-version }}"
+ run: ./scripts/ci/ci_prepare_image_on_ci.sh
+ - name: "Tests"
run: ./scripts/ci/ci_run_airflow_testing.sh
- tests-p36-sqlite:
- name: "Tests [Sqlite][Py3.6]"
+ tests-mysql:
+ name: "Tests: MySQL${{ matrix.mysql-version }}, Py{{ matrix.python-version}}"
runs-on: ubuntu-latest
- needs: [statics, statics-tests]
+ needs: [static-checks-1, static-checks-2, pyfiles]
+ strategy:
+ matrix:
+ python-version: [3.6, 3.7]
+ mysql-version: [5.7]
+ fail-fast: false
env:
- TRAVIS_JOB_NAME: "Tests [Sqlite][Py3.6]"
- BACKEND: sqlite
- PYTHON_VERSION: 3.6
+ BACKEND: mysql
+ PYTHON_MAJOR_MINOR_VERSION: ${{ matrix.python-version }}
+ MYSQL_VERSION: ${{ matrix.mysql-version }}
+ RUN_TESTS: "true"
+ CI_JOB_TYPE: "Tests"
+ # For pull requests only run tests when python files changed
+ if: needs.pyfiles.outputs.count != '0' || github.event_name != 'pull_request'
steps:
- uses: actions/checkout@master
- - name: "Tests [Sqlite][Py3.6]"
+ - uses: actions/setup-python@v1
+ with:
+ python-version: '3.x'
+ - name: "Free space"
+ run: ./scripts/ci/ci_free_space_on_ci.sh
+ - name: "Build CI image ${{ matrix.python-version }}"
+ run: ./scripts/ci/ci_prepare_image_on_ci.sh
+ - name: "Tests"
run: ./scripts/ci/ci_run_airflow_testing.sh
- tests-p36-mysql-integrations:
- name: "Tests [MySQL][Py3.6][integrations]"
+ tests-sqlite:
+ name: "Tests: Sqlite Py{{ matrix.python-version}}"
runs-on: ubuntu-latest
- needs: [statics, statics-tests]
+ needs: [static-checks-1, static-checks-2, pyfiles]
+ strategy:
+ matrix:
+ python-version: [3.6, 3.7]
+ fail-fast: false
env:
- TRAVIS_JOB_NAME: "Tests [MySQL][Py3.6][integrations]"
BACKEND: sqlite
- PYTHON_VERSION: 3.6
- MYSQL_VERSION: 5.7
- ENABLED_INTEGRATIONS: "cassandra kerberos mongo openldap rabbitmq redis"
- RUN_INTEGRATION_TESTS: all
+ PYTHON_MAJOR_MINOR_VERSION: ${{ matrix.python-version }}
+ RUN_TESTS: "true"
+ CI_JOB_TYPE: "Tests"
+ # For pull requests only run tests when python files changed
+ if: needs.pyfiles.outputs.count != '0' || github.event_name != 'pull_request'
steps:
- uses: actions/checkout@master
- - name: free disk space
- run: |
- sudo swapoff -a
- sudo rm -f /swapfile
- sudo apt clean
- docker rmi $(docker image ls -aq)
- df -h
- - name: "Tests [MySQL][Py3.6][integrations]"
+ - uses: actions/setup-python@v1
+ with:
+ python-version: '3.x'
+ - name: "Free space"
+ run: ./scripts/ci/ci_free_space_on_ci.sh
+ - name: "Build CI image ${{ matrix.python-version }}"
+ run: ./scripts/ci/ci_prepare_image_on_ci.sh
+ - name: "Tests"
run: ./scripts/ci/ci_run_airflow_testing.sh
- tests-p36-mysql-providers:
- name: "Tests [MySQL5.7][Py3.7][providers][kerberos]"
- runs-on: ubuntu-latest
- needs: [statics, statics-tests]
- env:
- TRAVIS_JOB_NAME: "Tests [MySQL5.7][Py3.7][providers][kerberos]"
- BACKEND: mysql
- PYTHON_VERSION: 3.7
- MYSQL_VERSION: 5.7
- ENABLED_INTEGRATIONS: "kerberos"
- steps:
- - uses: actions/checkout@master
- - name: "Tests [MySQL5.7][Py3.7][providers][kerberos]"
- run: ./scripts/ci/ci_run_airflow_testing.sh tests/providers
-
- tests-p37-mysql-kerberos:
- name: "Tests [MySQL5.7][Py3.7][core][kerberos]"
- runs-on: ubuntu-latest
- needs: [statics, statics-tests]
- env:
- TRAVIS_JOB_NAME: "Tests [MySQL5.7][Py3.7][core][kerberos]"
- BACKEND: mysql
- PYTHON_VERSION: 3.7
- MYSQL_VERSION: 5.7
- ENABLED_INTEGRATIONS: "kerberos"
- steps:
- - uses: actions/checkout@master
- - name: free disk space
- run: |
- sudo swapoff -a
- sudo rm -f /swapfile
- sudo apt clean
- docker rmi $(docker image ls -aq)
- df -h
- - name: "Tests [MySQL5.7][Py3.7][core][kerberos]"
- run: ./scripts/ci/ci_run_airflow_testing.sh --ignore=tests/providers
-
requirements:
- name: "Generate requirements"
+ name: "Requirements: Py ${{ matrix.python-version}} "
runs-on: ubuntu-latest
- needs: [statics, statics-tests]
strategy:
matrix:
python-version: [3.6, 3.7]
+ env:
+ PYTHON_MAJOR_MINOR_VERSION: ${{ matrix.python-version }}
+ SHOW_GENERATE_REQUIREMENTS_INSTRUCTIONS: true
+ CI_JOB_TYPE: "Requirements"
steps:
- uses: actions/checkout@master
- uses: actions/setup-python@v1
+ - name: "Build CI image ${{ matrix.python-version }}"
+ run: ./scripts/ci/ci_prepare_image_on_ci.sh
- name: "Generate requirements"
- env:
- PYTHON_MAJOR_MINOR_VERSION: ${{ matrix.python-version }}
- SHOW_GENERATE_REQUIREMENTS_INSTRUCTIONS: true
run: ./scripts/ci/ci_generate_requirements.sh
- build-image:
- name: "Build production image"
+ push-images-to-github-cache:
+ name: "Build and push images to github registry"
runs-on: ubuntu-latest
- needs: [statics, statics-tests]
+ needs:
+ - tests-sqlite
+ - tests-postgres
+ - tests-mysql
+ - requirements
+ - prepare-backport-packages
+ - build-prod-image
+ - docs
Review comment:
Yes. unfortunately we cannot do one CI image build and store it in the registry. When you run pull requests from the forks, the GITHUB_TOKEN has only read permissions and you are not able to push image to Github Image Registry :(. And when you think about it, it makes perfect sense (Imagine someone changing the build and overriding all temporary images). However I believe (still need to verify it) that we can pull the images from the Github Registy rather than from the DockerHub and it should significantly speed up Pullng the images for rebuilds (it takes 3-4 minutes now). I yet has to confirm it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #8393: Bring
back CI optimisations
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #8393: Bring back CI optimisations
URL: https://github.com/apache/airflow/pull/8393#discussion_r410036296
##########
File path: .github/workflows/ci.yml
##########
@@ -80,134 +80,152 @@ jobs:
name: Build docs
runs-on: ubuntu-latest
env:
- TRAVIS_JOB_NAME: "Build documentation"
PYTHON_VERSION: 3.6
+ CI_JOB_TYPE: "Documentation"
steps:
- uses: actions/checkout@master
- - name: "Build documentation"
+ - name: "Build CI image"
+ run: ./scripts/ci/ci_prepare_image_on_ci.sh
+ - name: "Build docs"
run: ./scripts/ci/ci_docs.sh
tests-p36-postgres-integrations:
- name: "Tests [Postgres9.6][Py3.6][integrations]"
+ name: "[Pg9.6][Py3.6][integrations]"
runs-on: ubuntu-latest
needs: [statics, statics-tests]
env:
- TRAVIS_JOB_NAME: "Tests [Postgres9.6][Py3.6][integrations]"
BACKEND: postgres
PYTHON_VERSION: 3.6
POSTGRES_VERSION: 9.6
ENABLED_INTEGRATIONS: "cassandra kerberos mongo openldap rabbitmq redis"
RUN_INTEGRATION_TESTS: all
+ CI_JOB_TYPE: "Tests"
steps:
- uses: actions/checkout@master
- - name: "Tests [Postgres9.6][Py3.6][integrations]"
+ - name: "Build CI image"
+ run: ./scripts/ci/ci_prepare_image_on_ci.sh
+ - name: "Tests"
run: ./scripts/ci/ci_run_airflow_testing.sh
tests-p36-postgres-providers:
Review comment:
Actually I guess we have far less limits now for GitHub actions so rather than splitting the tests we can combine the tests back.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #8393: Bring
back CI optimisations
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #8393: Bring back CI optimisations
URL: https://github.com/apache/airflow/pull/8393#discussion_r409475660
##########
File path: tests/cli/commands/test_webserver_command.py
##########
@@ -144,7 +144,7 @@ def test_cli_webserver_foreground_with_pid(self):
proc.terminate()
proc.wait()
- @unittest.skipIf("TRAVIS" in os.environ and bool(os.environ["TRAVIS"]),
+ @unittest.skipIf("CI" in os.environ and bool(os.environ["CI"]),
Review comment:
But yeah.. I will take the risk now :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] turbaszek commented on a change in pull request #8393:
Bring back CI optimisations
Posted by GitBox <gi...@apache.org>.
turbaszek commented on a change in pull request #8393: Bring back CI optimisations
URL: https://github.com/apache/airflow/pull/8393#discussion_r409383392
##########
File path: scripts/ci/_utils.sh
##########
@@ -917,34 +925,113 @@ function match_files_regexp() {
export FILE_MATCHES
}
+function get_ci_environment() {
+ if [[ ${CI:=} != "true" ]]; then
+ echo
+ echo "This is not a CI environment!. Set target branch to master and evert type to manual"
+ echo
+ export CI_EVENT_TYPE="manual"
+ export CI_TARGET_BRANCH="master"
+ else
+ if [[ ${TRAVIS:=} == "true" ]]; then
+ if [[ "${TRAVIS_PULL_REQUEST:=}" == "true" ]]; then
+ export CI_EVENT_TYPE="pull_request"
+ export CI_PR_REPO="${TRAVIS_PULL_REQUEST_SLUG}"
+ export CI_PR_BRANCH="${TRAVIS_PULL_REQUEST_BRANCH}"
+ elif [[ "${TRAVIS_EVENT_TYPE:=}" == "cron" ]]; then
+ export CI_EVENT_TYPE="schedule"
+ export CI_PR_REPO=""
+ export CI_PR_BRANCH=""
Review comment:
Can we move those variables before elifs? It seems that we are repeating it few times
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #8393: Bring
back CI optimisations
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #8393: Bring back CI optimisations
URL: https://github.com/apache/airflow/pull/8393#discussion_r409472112
##########
File path: scripts/ci/ci_flake8.sh
##########
@@ -50,6 +50,8 @@ function run_flake8() {
fi
}
+get_ci_environment
Review comment:
Not really. There is one place that between get_ci_environment and prepare_ci_build we check if this is a "cron" job (We find it out in get_ci_environment) and if it is, we override few variables to make the build from scratch.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #8393: Bring
back CI optimisations
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #8393: Bring back CI optimisations
URL: https://github.com/apache/airflow/pull/8393#discussion_r409473686
##########
File path: scripts/ci/ci_flake8.sh
##########
@@ -50,6 +50,8 @@ function run_flake8() {
fi
}
+get_ci_environment
Review comment:
```
get_ci_environment
# In case of CRON jobs we run builds without cache and upgrade to latest requirements
if [[ "${CI_EVENT_TYPE:=}" == "schedule" ]]; then
echo
echo "Disabling cache for scheduled jobs"
echo
export DOCKER_CACHE="no-cache"
export PULL_BASE_IMAGES="true"
export UPGRADE_TO_LATEST_REQUIREMENTS="true"
fi
```
prepare_ci_build
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #8393: Bring
back CI optimisations
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #8393: Bring back CI optimisations
URL: https://github.com/apache/airflow/pull/8393#discussion_r409430675
##########
File path: scripts/ci/_utils.sh
##########
@@ -917,34 +925,113 @@ function match_files_regexp() {
export FILE_MATCHES
}
+function get_ci_environment() {
+ if [[ ${CI:=} != "true" ]]; then
+ echo
+ echo "This is not a CI environment!. Set target branch to master and evert type to manual"
+ echo
+ export CI_EVENT_TYPE="manual"
+ export CI_TARGET_BRANCH="master"
+ else
+ if [[ ${TRAVIS:=} == "true" ]]; then
+ if [[ "${TRAVIS_PULL_REQUEST:=}" == "true" ]]; then
+ export CI_EVENT_TYPE="pull_request"
+ export CI_PR_REPO="${TRAVIS_PULL_REQUEST_SLUG}"
+ export CI_PR_BRANCH="${TRAVIS_PULL_REQUEST_BRANCH}"
+ elif [[ "${TRAVIS_EVENT_TYPE:=}" == "cron" ]]; then
+ export CI_EVENT_TYPE="schedule"
+ export CI_PR_REPO=""
+ export CI_PR_BRANCH=""
Review comment:
Yep.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on issue #8393: Bring back CI
optimisations
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #8393: Bring back CI optimisations
URL: https://github.com/apache/airflow/pull/8393#issuecomment-614483335
I brought back most of the niceties and optimisations lost yesterday during emergency move to Travis (that includes faster doc-only builds, cron jobs building from the scratch, colors in the output of tests and static check, better look of the static checks (We do not need verbose as we have timestamp in Github Actions) ...
Pls. take a look. It's really nice output now for most of the tests. I am also working on making the Kubernetes tests work on GA and hopefully we will part our ways with Travis :).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on issue #8393: Bring back CI
optimisations
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #8393: Bring back CI optimisations
URL: https://github.com/apache/airflow/pull/8393#issuecomment-614588240
Comments addressed :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #8393: Bring
back CI optimisations
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #8393: Bring back CI optimisations
URL: https://github.com/apache/airflow/pull/8393#discussion_r409467548
##########
File path: .github/workflows/ci.yml
##########
@@ -80,134 +80,152 @@ jobs:
name: Build docs
runs-on: ubuntu-latest
env:
- TRAVIS_JOB_NAME: "Build documentation"
PYTHON_VERSION: 3.6
+ CI_JOB_TYPE: "Documentation"
steps:
- uses: actions/checkout@master
- - name: "Build documentation"
+ - name: "Build CI image"
+ run: ./scripts/ci/ci_prepare_image_on_ci.sh
+ - name: "Build docs"
run: ./scripts/ci/ci_docs.sh
tests-p36-postgres-integrations:
- name: "Tests [Postgres9.6][Py3.6][integrations]"
+ name: "[Pg9.6][Py3.6][integrations]"
runs-on: ubuntu-latest
needs: [statics, statics-tests]
env:
- TRAVIS_JOB_NAME: "Tests [Postgres9.6][Py3.6][integrations]"
BACKEND: postgres
PYTHON_VERSION: 3.6
POSTGRES_VERSION: 9.6
ENABLED_INTEGRATIONS: "cassandra kerberos mongo openldap rabbitmq redis"
RUN_INTEGRATION_TESTS: all
+ CI_JOB_TYPE: "Tests"
steps:
- uses: actions/checkout@master
- - name: "Tests [Postgres9.6][Py3.6][integrations]"
+ - name: "Build CI image"
+ run: ./scripts/ci/ci_prepare_image_on_ci.sh
+ - name: "Tests"
run: ./scripts/ci/ci_run_airflow_testing.sh
tests-p36-postgres-providers:
- name: "Tests [Postgres10][Py3.6][providers]"
+ name: "[Pg10][Py3.6][prov]"
runs-on: ubuntu-latest
needs: [statics, statics-tests]
env:
- TRAVIS_JOB_NAME: "Tests [Postgres10][Py3.6][providers]"
BACKEND: postgres
POSTGRES_VERSION: 10
PYTHON_VERSION: 3.6
+ CI_JOB_TYPE: "Tests"
steps:
- uses: actions/checkout@master
- - name: "Tests [Postgres10][Py3.6][providers]"
+ - name: "Build CI image"
+ run: ./scripts/ci/ci_prepare_image_on_ci.sh
+ - name: "Tests"
run: ./scripts/ci/ci_run_airflow_testing.sh tests/providers
tests-p36-postgres-core:
- name: "Tests [Postgres9.6][Py3.6][core]"
+ name: "[Pg9.6][Py3.6][core]"
runs-on: ubuntu-latest
needs: [statics, statics-tests]
env:
- TRAVIS_JOB_NAME: "Tests [Postgres9.6][Py3.6][core]"
BACKEND: postgres
POSTGRES_VERSION: 9.6
PYTHON_VERSION: 3.6
+ CI_JOB_TYPE: "Tests"
steps:
- uses: actions/checkout@master
- - name: "Tests [Postgres9.6][Py3.6][core]"
+ - name: "Build CI image"
+ run: ./scripts/ci/ci_prepare_image_on_ci.sh
+ - name: "Tests"
run: ./scripts/ci/ci_run_airflow_testing.sh --ignore=tests/providers
tests-p37-sqlite-integrations:
- name: "Tests [Sqlite][3.7][integrations]"
+ name: "[Sqlt][3.7][int]"
Review comment:
OK OK
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] dimberman commented on a change in pull request #8393:
Bring back CI optimisations
Posted by GitBox <gi...@apache.org>.
dimberman commented on a change in pull request #8393: Bring back CI optimisations
URL: https://github.com/apache/airflow/pull/8393#discussion_r409694983
##########
File path: .github/workflows/ci.yml
##########
@@ -80,134 +80,152 @@ jobs:
name: Build docs
runs-on: ubuntu-latest
env:
- TRAVIS_JOB_NAME: "Build documentation"
PYTHON_VERSION: 3.6
+ CI_JOB_TYPE: "Documentation"
steps:
- uses: actions/checkout@master
- - name: "Build documentation"
+ - name: "Build CI image"
+ run: ./scripts/ci/ci_prepare_image_on_ci.sh
+ - name: "Build docs"
run: ./scripts/ci/ci_docs.sh
tests-p36-postgres-integrations:
- name: "Tests [Postgres9.6][Py3.6][integrations]"
+ name: "[Pg9.6][Py3.6][integrations]"
runs-on: ubuntu-latest
needs: [statics, statics-tests]
env:
- TRAVIS_JOB_NAME: "Tests [Postgres9.6][Py3.6][integrations]"
BACKEND: postgres
PYTHON_VERSION: 3.6
POSTGRES_VERSION: 9.6
ENABLED_INTEGRATIONS: "cassandra kerberos mongo openldap rabbitmq redis"
RUN_INTEGRATION_TESTS: all
+ CI_JOB_TYPE: "Tests"
steps:
- uses: actions/checkout@master
- - name: "Tests [Postgres9.6][Py3.6][integrations]"
+ - name: "Build CI image"
+ run: ./scripts/ci/ci_prepare_image_on_ci.sh
+ - name: "Tests"
run: ./scripts/ci/ci_run_airflow_testing.sh
tests-p36-postgres-providers:
Review comment:
We can do all of this using a matrix
```
strategy:
matrix:
postgres: [Pg9, Pg10]
python: [3.6, 3.7]
strategy: [providers, core]
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #8393: Bring
back CI optimisations
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #8393: Bring back CI optimisations
URL: https://github.com/apache/airflow/pull/8393#discussion_r409474658
##########
File path: tests/cli/commands/test_webserver_command.py
##########
@@ -144,7 +144,7 @@ def test_cli_webserver_foreground_with_pid(self):
proc.terminate()
proc.wait()
- @unittest.skipIf("TRAVIS" in os.environ and bool(os.environ["TRAVIS"]),
+ @unittest.skipIf("CI" in os.environ and bool(os.environ["CI"]),
Review comment:
I want to check it afterwards. I will make a separate PR for that.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] turbaszek commented on a change in pull request #8393:
Bring back CI optimisations
Posted by GitBox <gi...@apache.org>.
turbaszek commented on a change in pull request #8393: Bring back CI optimisations
URL: https://github.com/apache/airflow/pull/8393#discussion_r409384030
##########
File path: .github/workflows/ci.yml
##########
@@ -19,58 +19,58 @@
---
name: Airflow
on:
+ schedule:
+ # Schedule cron jobs every day at 2.10 am
+ - cron: '10 2 * * *'
push:
- branches: ['*']
+ branches: ['master', 'v1-10-test', 'v1-10-stable']
Review comment:
Should we do same for pull request just in case?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on issue #8393: Bring back CI
optimisations
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #8393: Bring back CI optimisations
URL: https://github.com/apache/airflow/pull/8393#issuecomment-614632798
> It seems that our tests are still timeouting :<
See: https://github.com/apache/airflow/pull/8403 - this might help.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #8393: Bring
back CI optimisations
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #8393: Bring back CI optimisations
URL: https://github.com/apache/airflow/pull/8393#discussion_r409467400
##########
File path: .github/workflows/ci.yml
##########
@@ -17,60 +17,60 @@
#
---
-name: Airflow
+name: Airflow CI Build
on:
+ schedule:
+ # Schedule cron jobs every day at 2.10 am
+ - cron: '10 2 * * *'
push:
- branches: ['*']
+ branches: ['master', 'v1-10-test', 'v1-10-stable']
pull_request:
- branches: [master]
+ branches: ['master', 'v1-10-test', 'v1-10-stable']
env:
- BUILD_ID: ${{github.sha }}
MOUNT_LOCAL_SOURCES: "false"
MOUNT_HOST_AIRFLOW_VOLUME: "true"
FORCE_ANSWER_TO_QUESTIONS: "yes"
SKIP_CHECK_REMOTE_IMAGE: "true"
DB_RESET: "true"
VERBOSE: "true"
- CI: "true"
Review comment:
CI is set to "true" automatically by both Travis and Github Actions.
- https://help.github.com/en/actions/configuring-and-managing-workflows/using-environment-variables
- https://docs.travis-ci.com/user/environment-variables/#default-environment-variables
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] ashb commented on a change in pull request #8393: Bring
back CI optimisations
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #8393: Bring back CI optimisations
URL: https://github.com/apache/airflow/pull/8393#discussion_r409463247
##########
File path: .github/workflows/ci.yml
##########
@@ -17,60 +17,60 @@
#
---
-name: Airflow
+name: Airflow CI Build
on:
+ schedule:
+ # Schedule cron jobs every day at 2.10 am
+ - cron: '10 2 * * *'
push:
- branches: ['*']
+ branches: ['master', 'v1-10-test', 'v1-10-stable']
pull_request:
- branches: [master]
+ branches: ['master', 'v1-10-test', 'v1-10-stable']
env:
- BUILD_ID: ${{github.sha }}
MOUNT_LOCAL_SOURCES: "false"
MOUNT_HOST_AIRFLOW_VOLUME: "true"
FORCE_ANSWER_TO_QUESTIONS: "yes"
SKIP_CHECK_REMOTE_IMAGE: "true"
DB_RESET: "true"
VERBOSE: "true"
- CI: "true"
Review comment:
Why did re remove this? It seems like a useful thing to know/have.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] turbaszek commented on issue #8393: Bring back CI
optimisations
Posted by GitBox <gi...@apache.org>.
turbaszek commented on issue #8393: Bring back CI optimisations
URL: https://github.com/apache/airflow/pull/8393#issuecomment-614614478
It seems that our tests are still timeouting :<
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] ashb commented on a change in pull request #8393: Bring
back CI optimisations
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #8393: Bring back CI optimisations
URL: https://github.com/apache/airflow/pull/8393#discussion_r409467485
##########
File path: tests/cli/commands/test_webserver_command.py
##########
@@ -144,7 +144,7 @@ def test_cli_webserver_foreground_with_pid(self):
proc.terminate()
proc.wait()
- @unittest.skipIf("TRAVIS" in os.environ and bool(os.environ["TRAVIS"]),
+ @unittest.skipIf("CI" in os.environ and bool(os.environ["CI"]),
Review comment:
Is this still needed? It seems odd we don't have permission to do something in our own docker container.
(I guess this is a separate issue)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] turbaszek commented on a change in pull request #8393:
Bring back CI optimisations
Posted by GitBox <gi...@apache.org>.
turbaszek commented on a change in pull request #8393: Bring back CI optimisations
URL: https://github.com/apache/airflow/pull/8393#discussion_r409385842
##########
File path: .github/workflows/ci.yml
##########
@@ -19,58 +19,58 @@
---
name: Airflow
Review comment:
It's probably a nit but maybe we should consider another name like "CI" or "Build"?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] ashb commented on a change in pull request #8393: Bring
back CI optimisations
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #8393: Bring back CI optimisations
URL: https://github.com/apache/airflow/pull/8393#discussion_r409465658
##########
File path: scripts/ci/_utils.sh
##########
@@ -134,6 +134,14 @@ function initialize_common_environment {
"--env" "PYTHONDONTWRITEBYTECODE" \
)
fi
+
+ if [[ ${CI:=} == "true" ]]; then
+ CI_CODECOV_ENV="$(bash <(curl -s https://codecov.io/env))"
Review comment:
What crazy magic is this?! :)
Should this be limited to just running from Github/Travis and not CI in general?
(I can tell what it's doing. Nothing to do here)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] kaxil commented on a change in pull request #8393: Bring
back CI optimisations
Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #8393: Bring back CI optimisations
URL: https://github.com/apache/airflow/pull/8393#discussion_r410119916
##########
File path: .github/workflows/ci.yml
##########
@@ -80,134 +80,152 @@ jobs:
name: Build docs
runs-on: ubuntu-latest
env:
- TRAVIS_JOB_NAME: "Build documentation"
PYTHON_VERSION: 3.6
+ CI_JOB_TYPE: "Documentation"
steps:
- uses: actions/checkout@master
- - name: "Build documentation"
+ - name: "Build CI image"
+ run: ./scripts/ci/ci_prepare_image_on_ci.sh
+ - name: "Build docs"
run: ./scripts/ci/ci_docs.sh
tests-p36-postgres-integrations:
- name: "Tests [Postgres9.6][Py3.6][integrations]"
+ name: "[Pg9.6][Py3.6][integrations]"
runs-on: ubuntu-latest
needs: [statics, statics-tests]
env:
- TRAVIS_JOB_NAME: "Tests [Postgres9.6][Py3.6][integrations]"
BACKEND: postgres
PYTHON_VERSION: 3.6
POSTGRES_VERSION: 9.6
ENABLED_INTEGRATIONS: "cassandra kerberos mongo openldap rabbitmq redis"
RUN_INTEGRATION_TESTS: all
+ CI_JOB_TYPE: "Tests"
steps:
- uses: actions/checkout@master
- - name: "Tests [Postgres9.6][Py3.6][integrations]"
+ - name: "Build CI image"
+ run: ./scripts/ci/ci_prepare_image_on_ci.sh
+ - name: "Tests"
run: ./scripts/ci/ci_run_airflow_testing.sh
tests-p36-postgres-providers:
- name: "Tests [Postgres10][Py3.6][providers]"
+ name: "[Pg10][Py3.6][prov]"
runs-on: ubuntu-latest
needs: [statics, statics-tests]
env:
- TRAVIS_JOB_NAME: "Tests [Postgres10][Py3.6][providers]"
BACKEND: postgres
POSTGRES_VERSION: 10
PYTHON_VERSION: 3.6
+ CI_JOB_TYPE: "Tests"
steps:
- uses: actions/checkout@master
- - name: "Tests [Postgres10][Py3.6][providers]"
+ - name: "Build CI image"
+ run: ./scripts/ci/ci_prepare_image_on_ci.sh
+ - name: "Tests"
run: ./scripts/ci/ci_run_airflow_testing.sh tests/providers
tests-p36-postgres-core:
- name: "Tests [Postgres9.6][Py3.6][core]"
+ name: "[Pg9.6][Py3.6][core]"
runs-on: ubuntu-latest
needs: [statics, statics-tests]
env:
- TRAVIS_JOB_NAME: "Tests [Postgres9.6][Py3.6][core]"
BACKEND: postgres
POSTGRES_VERSION: 9.6
PYTHON_VERSION: 3.6
+ CI_JOB_TYPE: "Tests"
steps:
- uses: actions/checkout@master
- - name: "Tests [Postgres9.6][Py3.6][core]"
+ - name: "Build CI image"
+ run: ./scripts/ci/ci_prepare_image_on_ci.sh
+ - name: "Tests"
run: ./scripts/ci/ci_run_airflow_testing.sh --ignore=tests/providers
tests-p37-sqlite-integrations:
- name: "Tests [Sqlite][3.7][integrations]"
+ name: "[Sqlite][3.7][int]"
runs-on: ubuntu-latest
needs: [statics, statics-tests]
env:
- TRAVIS_JOB_NAME: "Tests [Sqlite][3.7][integrations]"
BACKEND: sqlite
PYTHON_VERSION: 3.7
ENABLED_INTEGRATIONS: "cassandra kerberos mongo openldap rabbitmq redis"
RUN_INTEGRATION_TESTS: all
+ CI_JOB_TYPE: "Tests"
steps:
- uses: actions/checkout@master
- - name: "Tests [Sqlite][3.7][integrations]"
+ - name: "Build CI image"
+ run: ./scripts/ci/ci_prepare_image_on_ci.sh
+ - name: "Tests"
run: ./scripts/ci/ci_run_airflow_testing.sh
tests-p36-sqlite:
- name: "Tests [Sqlite][Py3.6]"
+ name: "[Sqlite][Py3.6]"
runs-on: ubuntu-latest
needs: [statics, statics-tests]
env:
- TRAVIS_JOB_NAME: "Tests [Sqlite][Py3.6]"
BACKEND: sqlite
PYTHON_VERSION: 3.6
+ CI_JOB_TYPE: "Tests"
steps:
- uses: actions/checkout@master
- - name: "Tests [Sqlite][Py3.6]"
+ - name: "Build CI image"
+ run: ./scripts/ci/ci_prepare_image_on_ci.sh
+ - name: "Tests"
run: ./scripts/ci/ci_run_airflow_testing.sh
tests-p36-mysql-integrations:
- name: "Tests [MySQL][Py3.6][integrations]"
+ name: "[MySQL5.7][Py3.6][int]"
runs-on: ubuntu-latest
needs: [statics, statics-tests]
env:
- TRAVIS_JOB_NAME: "Tests [MySQL][Py3.6][integrations]"
BACKEND: sqlite
PYTHON_VERSION: 3.6
MYSQL_VERSION: 5.7
ENABLED_INTEGRATIONS: "cassandra kerberos mongo openldap rabbitmq redis"
RUN_INTEGRATION_TESTS: all
+ CI_JOB_TYPE: "Tests"
steps:
- uses: actions/checkout@master
- - name: "Tests [MySQL][Py3.6][integrations]"
+ - name: "Build CI image"
+ run: ./scripts/ci/ci_prepare_image_on_ci.sh
+ - name: "Tests"
run: ./scripts/ci/ci_run_airflow_testing.sh
tests-p36-mysql-providers:
- name: "Tests [MySQL5.7][Py3.7][providers][kerberos]"
+ name: "[MySQL5.7][Py3.7][prov][kerb]"
runs-on: ubuntu-latest
needs: [statics, statics-tests]
env:
- TRAVIS_JOB_NAME: "Tests [MySQL5.7][Py3.7][providers][kerberos]"
BACKEND: mysql
PYTHON_VERSION: 3.7
MYSQL_VERSION: 5.7
ENABLED_INTEGRATIONS: "kerberos"
+ CI_JOB_TYPE: "Tests"
steps:
- uses: actions/checkout@master
- - name: "Tests [MySQL5.7][Py3.7][providers][kerberos]"
+ - name: "Build CI image"
+ run: ./scripts/ci/ci_prepare_image_on_ci.sh
+ - name: "Tests"
run: ./scripts/ci/ci_run_airflow_testing.sh tests/providers
tests-p37-mysql-kerberos:
- name: "Tests [MySQL5.7][Py3.7][core][kerberos]"
+ name: "[MySQL5.7][Py3.7][core][kerb]"
runs-on: ubuntu-latest
needs: [statics, statics-tests]
env:
- TRAVIS_JOB_NAME: "Tests [MySQL5.7][Py3.7][core][kerberos]"
BACKEND: mysql
PYTHON_VERSION: 3.7
MYSQL_VERSION: 5.7
ENABLED_INTEGRATIONS: "kerberos"
+ CI_JOB_TYPE: "Tests"
steps:
- uses: actions/checkout@master
- - name: "Tests [MySQL5.7][Py3.7][core][kerberos]"
+ - name: "Build CI image"
+ run: ./scripts/ci/ci_prepare_image_on_ci.sh
+ - name: "Tests"
run: ./scripts/ci/ci_run_airflow_testing.sh --ignore=tests/providers
requirements:
- name: "Generate requirements"
+ name: "Generate reqs"
runs-on: ubuntu-latest
needs: [statics, statics-tests]
strategy:
Review comment:
No, there might be other errors that we might miss and will have to wait for CI to run again to know about it. So fail-fast should be false
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] kaxil commented on a change in pull request #8393: Bring
back CI optimisations
Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #8393: Bring back CI optimisations
URL: https://github.com/apache/airflow/pull/8393#discussion_r410119916
##########
File path: .github/workflows/ci.yml
##########
@@ -80,134 +80,152 @@ jobs:
name: Build docs
runs-on: ubuntu-latest
env:
- TRAVIS_JOB_NAME: "Build documentation"
PYTHON_VERSION: 3.6
+ CI_JOB_TYPE: "Documentation"
steps:
- uses: actions/checkout@master
- - name: "Build documentation"
+ - name: "Build CI image"
+ run: ./scripts/ci/ci_prepare_image_on_ci.sh
+ - name: "Build docs"
run: ./scripts/ci/ci_docs.sh
tests-p36-postgres-integrations:
- name: "Tests [Postgres9.6][Py3.6][integrations]"
+ name: "[Pg9.6][Py3.6][integrations]"
runs-on: ubuntu-latest
needs: [statics, statics-tests]
env:
- TRAVIS_JOB_NAME: "Tests [Postgres9.6][Py3.6][integrations]"
BACKEND: postgres
PYTHON_VERSION: 3.6
POSTGRES_VERSION: 9.6
ENABLED_INTEGRATIONS: "cassandra kerberos mongo openldap rabbitmq redis"
RUN_INTEGRATION_TESTS: all
+ CI_JOB_TYPE: "Tests"
steps:
- uses: actions/checkout@master
- - name: "Tests [Postgres9.6][Py3.6][integrations]"
+ - name: "Build CI image"
+ run: ./scripts/ci/ci_prepare_image_on_ci.sh
+ - name: "Tests"
run: ./scripts/ci/ci_run_airflow_testing.sh
tests-p36-postgres-providers:
- name: "Tests [Postgres10][Py3.6][providers]"
+ name: "[Pg10][Py3.6][prov]"
runs-on: ubuntu-latest
needs: [statics, statics-tests]
env:
- TRAVIS_JOB_NAME: "Tests [Postgres10][Py3.6][providers]"
BACKEND: postgres
POSTGRES_VERSION: 10
PYTHON_VERSION: 3.6
+ CI_JOB_TYPE: "Tests"
steps:
- uses: actions/checkout@master
- - name: "Tests [Postgres10][Py3.6][providers]"
+ - name: "Build CI image"
+ run: ./scripts/ci/ci_prepare_image_on_ci.sh
+ - name: "Tests"
run: ./scripts/ci/ci_run_airflow_testing.sh tests/providers
tests-p36-postgres-core:
- name: "Tests [Postgres9.6][Py3.6][core]"
+ name: "[Pg9.6][Py3.6][core]"
runs-on: ubuntu-latest
needs: [statics, statics-tests]
env:
- TRAVIS_JOB_NAME: "Tests [Postgres9.6][Py3.6][core]"
BACKEND: postgres
POSTGRES_VERSION: 9.6
PYTHON_VERSION: 3.6
+ CI_JOB_TYPE: "Tests"
steps:
- uses: actions/checkout@master
- - name: "Tests [Postgres9.6][Py3.6][core]"
+ - name: "Build CI image"
+ run: ./scripts/ci/ci_prepare_image_on_ci.sh
+ - name: "Tests"
run: ./scripts/ci/ci_run_airflow_testing.sh --ignore=tests/providers
tests-p37-sqlite-integrations:
- name: "Tests [Sqlite][3.7][integrations]"
+ name: "[Sqlite][3.7][int]"
runs-on: ubuntu-latest
needs: [statics, statics-tests]
env:
- TRAVIS_JOB_NAME: "Tests [Sqlite][3.7][integrations]"
BACKEND: sqlite
PYTHON_VERSION: 3.7
ENABLED_INTEGRATIONS: "cassandra kerberos mongo openldap rabbitmq redis"
RUN_INTEGRATION_TESTS: all
+ CI_JOB_TYPE: "Tests"
steps:
- uses: actions/checkout@master
- - name: "Tests [Sqlite][3.7][integrations]"
+ - name: "Build CI image"
+ run: ./scripts/ci/ci_prepare_image_on_ci.sh
+ - name: "Tests"
run: ./scripts/ci/ci_run_airflow_testing.sh
tests-p36-sqlite:
- name: "Tests [Sqlite][Py3.6]"
+ name: "[Sqlite][Py3.6]"
runs-on: ubuntu-latest
needs: [statics, statics-tests]
env:
- TRAVIS_JOB_NAME: "Tests [Sqlite][Py3.6]"
BACKEND: sqlite
PYTHON_VERSION: 3.6
+ CI_JOB_TYPE: "Tests"
steps:
- uses: actions/checkout@master
- - name: "Tests [Sqlite][Py3.6]"
+ - name: "Build CI image"
+ run: ./scripts/ci/ci_prepare_image_on_ci.sh
+ - name: "Tests"
run: ./scripts/ci/ci_run_airflow_testing.sh
tests-p36-mysql-integrations:
- name: "Tests [MySQL][Py3.6][integrations]"
+ name: "[MySQL5.7][Py3.6][int]"
runs-on: ubuntu-latest
needs: [statics, statics-tests]
env:
- TRAVIS_JOB_NAME: "Tests [MySQL][Py3.6][integrations]"
BACKEND: sqlite
PYTHON_VERSION: 3.6
MYSQL_VERSION: 5.7
ENABLED_INTEGRATIONS: "cassandra kerberos mongo openldap rabbitmq redis"
RUN_INTEGRATION_TESTS: all
+ CI_JOB_TYPE: "Tests"
steps:
- uses: actions/checkout@master
- - name: "Tests [MySQL][Py3.6][integrations]"
+ - name: "Build CI image"
+ run: ./scripts/ci/ci_prepare_image_on_ci.sh
+ - name: "Tests"
run: ./scripts/ci/ci_run_airflow_testing.sh
tests-p36-mysql-providers:
- name: "Tests [MySQL5.7][Py3.7][providers][kerberos]"
+ name: "[MySQL5.7][Py3.7][prov][kerb]"
runs-on: ubuntu-latest
needs: [statics, statics-tests]
env:
- TRAVIS_JOB_NAME: "Tests [MySQL5.7][Py3.7][providers][kerberos]"
BACKEND: mysql
PYTHON_VERSION: 3.7
MYSQL_VERSION: 5.7
ENABLED_INTEGRATIONS: "kerberos"
+ CI_JOB_TYPE: "Tests"
steps:
- uses: actions/checkout@master
- - name: "Tests [MySQL5.7][Py3.7][providers][kerberos]"
+ - name: "Build CI image"
+ run: ./scripts/ci/ci_prepare_image_on_ci.sh
+ - name: "Tests"
run: ./scripts/ci/ci_run_airflow_testing.sh tests/providers
tests-p37-mysql-kerberos:
- name: "Tests [MySQL5.7][Py3.7][core][kerberos]"
+ name: "[MySQL5.7][Py3.7][core][kerb]"
runs-on: ubuntu-latest
needs: [statics, statics-tests]
env:
- TRAVIS_JOB_NAME: "Tests [MySQL5.7][Py3.7][core][kerberos]"
BACKEND: mysql
PYTHON_VERSION: 3.7
MYSQL_VERSION: 5.7
ENABLED_INTEGRATIONS: "kerberos"
+ CI_JOB_TYPE: "Tests"
steps:
- uses: actions/checkout@master
- - name: "Tests [MySQL5.7][Py3.7][core][kerberos]"
+ - name: "Build CI image"
+ run: ./scripts/ci/ci_prepare_image_on_ci.sh
+ - name: "Tests"
run: ./scripts/ci/ci_run_airflow_testing.sh --ignore=tests/providers
requirements:
- name: "Generate requirements"
+ name: "Generate reqs"
runs-on: ubuntu-latest
needs: [statics, statics-tests]
strategy:
Review comment:
No, there might be other errors that we might miss and will have to wait for CI to run again to know about it
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] turbaszek commented on a change in pull request #8393:
Bring back CI optimisations
Posted by GitBox <gi...@apache.org>.
turbaszek commented on a change in pull request #8393: Bring back CI optimisations
URL: https://github.com/apache/airflow/pull/8393#discussion_r410048080
##########
File path: .github/workflows/ci.yml
##########
@@ -80,134 +80,152 @@ jobs:
name: Build docs
runs-on: ubuntu-latest
env:
- TRAVIS_JOB_NAME: "Build documentation"
PYTHON_VERSION: 3.6
+ CI_JOB_TYPE: "Documentation"
steps:
- uses: actions/checkout@master
- - name: "Build documentation"
+ - name: "Build CI image"
+ run: ./scripts/ci/ci_prepare_image_on_ci.sh
+ - name: "Build docs"
run: ./scripts/ci/ci_docs.sh
tests-p36-postgres-integrations:
- name: "Tests [Postgres9.6][Py3.6][integrations]"
+ name: "[Pg9.6][Py3.6][integrations]"
runs-on: ubuntu-latest
needs: [statics, statics-tests]
env:
- TRAVIS_JOB_NAME: "Tests [Postgres9.6][Py3.6][integrations]"
BACKEND: postgres
PYTHON_VERSION: 3.6
POSTGRES_VERSION: 9.6
ENABLED_INTEGRATIONS: "cassandra kerberos mongo openldap rabbitmq redis"
RUN_INTEGRATION_TESTS: all
+ CI_JOB_TYPE: "Tests"
steps:
- uses: actions/checkout@master
- - name: "Tests [Postgres9.6][Py3.6][integrations]"
+ - name: "Build CI image"
+ run: ./scripts/ci/ci_prepare_image_on_ci.sh
+ - name: "Tests"
run: ./scripts/ci/ci_run_airflow_testing.sh
tests-p36-postgres-providers:
- name: "Tests [Postgres10][Py3.6][providers]"
+ name: "[Pg10][Py3.6][prov]"
runs-on: ubuntu-latest
needs: [statics, statics-tests]
env:
- TRAVIS_JOB_NAME: "Tests [Postgres10][Py3.6][providers]"
BACKEND: postgres
POSTGRES_VERSION: 10
PYTHON_VERSION: 3.6
+ CI_JOB_TYPE: "Tests"
steps:
- uses: actions/checkout@master
- - name: "Tests [Postgres10][Py3.6][providers]"
+ - name: "Build CI image"
+ run: ./scripts/ci/ci_prepare_image_on_ci.sh
+ - name: "Tests"
run: ./scripts/ci/ci_run_airflow_testing.sh tests/providers
tests-p36-postgres-core:
- name: "Tests [Postgres9.6][Py3.6][core]"
+ name: "[Pg9.6][Py3.6][core]"
runs-on: ubuntu-latest
needs: [statics, statics-tests]
env:
- TRAVIS_JOB_NAME: "Tests [Postgres9.6][Py3.6][core]"
BACKEND: postgres
POSTGRES_VERSION: 9.6
PYTHON_VERSION: 3.6
+ CI_JOB_TYPE: "Tests"
steps:
- uses: actions/checkout@master
- - name: "Tests [Postgres9.6][Py3.6][core]"
+ - name: "Build CI image"
+ run: ./scripts/ci/ci_prepare_image_on_ci.sh
+ - name: "Tests"
run: ./scripts/ci/ci_run_airflow_testing.sh --ignore=tests/providers
tests-p37-sqlite-integrations:
- name: "Tests [Sqlite][3.7][integrations]"
+ name: "[Sqlite][3.7][int]"
runs-on: ubuntu-latest
needs: [statics, statics-tests]
env:
- TRAVIS_JOB_NAME: "Tests [Sqlite][3.7][integrations]"
BACKEND: sqlite
PYTHON_VERSION: 3.7
ENABLED_INTEGRATIONS: "cassandra kerberos mongo openldap rabbitmq redis"
RUN_INTEGRATION_TESTS: all
+ CI_JOB_TYPE: "Tests"
steps:
- uses: actions/checkout@master
- - name: "Tests [Sqlite][3.7][integrations]"
+ - name: "Build CI image"
+ run: ./scripts/ci/ci_prepare_image_on_ci.sh
+ - name: "Tests"
run: ./scripts/ci/ci_run_airflow_testing.sh
tests-p36-sqlite:
- name: "Tests [Sqlite][Py3.6]"
+ name: "[Sqlite][Py3.6]"
runs-on: ubuntu-latest
needs: [statics, statics-tests]
env:
- TRAVIS_JOB_NAME: "Tests [Sqlite][Py3.6]"
BACKEND: sqlite
PYTHON_VERSION: 3.6
+ CI_JOB_TYPE: "Tests"
steps:
- uses: actions/checkout@master
- - name: "Tests [Sqlite][Py3.6]"
+ - name: "Build CI image"
+ run: ./scripts/ci/ci_prepare_image_on_ci.sh
+ - name: "Tests"
run: ./scripts/ci/ci_run_airflow_testing.sh
tests-p36-mysql-integrations:
- name: "Tests [MySQL][Py3.6][integrations]"
+ name: "[MySQL5.7][Py3.6][int]"
runs-on: ubuntu-latest
needs: [statics, statics-tests]
env:
- TRAVIS_JOB_NAME: "Tests [MySQL][Py3.6][integrations]"
BACKEND: sqlite
PYTHON_VERSION: 3.6
MYSQL_VERSION: 5.7
ENABLED_INTEGRATIONS: "cassandra kerberos mongo openldap rabbitmq redis"
RUN_INTEGRATION_TESTS: all
+ CI_JOB_TYPE: "Tests"
steps:
- uses: actions/checkout@master
- - name: "Tests [MySQL][Py3.6][integrations]"
+ - name: "Build CI image"
+ run: ./scripts/ci/ci_prepare_image_on_ci.sh
+ - name: "Tests"
run: ./scripts/ci/ci_run_airflow_testing.sh
tests-p36-mysql-providers:
- name: "Tests [MySQL5.7][Py3.7][providers][kerberos]"
+ name: "[MySQL5.7][Py3.7][prov][kerb]"
runs-on: ubuntu-latest
needs: [statics, statics-tests]
env:
- TRAVIS_JOB_NAME: "Tests [MySQL5.7][Py3.7][providers][kerberos]"
BACKEND: mysql
PYTHON_VERSION: 3.7
MYSQL_VERSION: 5.7
ENABLED_INTEGRATIONS: "kerberos"
+ CI_JOB_TYPE: "Tests"
steps:
- uses: actions/checkout@master
- - name: "Tests [MySQL5.7][Py3.7][providers][kerberos]"
+ - name: "Build CI image"
+ run: ./scripts/ci/ci_prepare_image_on_ci.sh
+ - name: "Tests"
run: ./scripts/ci/ci_run_airflow_testing.sh tests/providers
tests-p37-mysql-kerberos:
- name: "Tests [MySQL5.7][Py3.7][core][kerberos]"
+ name: "[MySQL5.7][Py3.7][core][kerb]"
runs-on: ubuntu-latest
needs: [statics, statics-tests]
env:
- TRAVIS_JOB_NAME: "Tests [MySQL5.7][Py3.7][core][kerberos]"
BACKEND: mysql
PYTHON_VERSION: 3.7
MYSQL_VERSION: 5.7
ENABLED_INTEGRATIONS: "kerberos"
+ CI_JOB_TYPE: "Tests"
steps:
- uses: actions/checkout@master
- - name: "Tests [MySQL5.7][Py3.7][core][kerberos]"
+ - name: "Build CI image"
+ run: ./scripts/ci/ci_prepare_image_on_ci.sh
+ - name: "Tests"
run: ./scripts/ci/ci_run_airflow_testing.sh --ignore=tests/providers
requirements:
- name: "Generate requirements"
+ name: "Generate reqs"
runs-on: ubuntu-latest
needs: [statics, statics-tests]
strategy:
Review comment:
```suggestion
strategy:
fail-fast: false
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #8393: Bring
back CI optimisations
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #8393: Bring back CI optimisations
URL: https://github.com/apache/airflow/pull/8393#discussion_r409471211
##########
File path: scripts/ci/_utils.sh
##########
@@ -134,6 +134,14 @@ function initialize_common_environment {
"--env" "PYTHONDONTWRITEBYTECODE" \
)
fi
+
+ if [[ ${CI:=} == "true" ]]; then
+ CI_CODECOV_ENV="$(bash <(curl -s https://codecov.io/env))"
Review comment:
That's standard way of using Codecov when running codecov inside docker:
https://docs.codecov.io/docs/testing-with-docker
It detects automatically which CI codecov it is running on (it supports pretty much all of the CIs ) and it return appropriate "-e ENV" sequence to pass the right variables to inside docker.
This is needed because codecov puts together code coverage from multiple jobs running in different jobs/ machines so it needs to have a way to know that they all belong to the same build. And codecove does it via CI-specific environment variables.
Previously we passed the necessary TRAVIS_ * variables in base.yaml but the way it is done now it will work well for any CI this script will run on.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #8393: Bring
back CI optimisations
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #8393: Bring back CI optimisations
URL: https://github.com/apache/airflow/pull/8393#discussion_r410288174
##########
File path: .github/workflows/ci.yml
##########
@@ -80,134 +80,152 @@ jobs:
name: Build docs
runs-on: ubuntu-latest
env:
- TRAVIS_JOB_NAME: "Build documentation"
PYTHON_VERSION: 3.6
+ CI_JOB_TYPE: "Documentation"
steps:
- uses: actions/checkout@master
- - name: "Build documentation"
+ - name: "Build CI image"
+ run: ./scripts/ci/ci_prepare_image_on_ci.sh
+ - name: "Build docs"
run: ./scripts/ci/ci_docs.sh
tests-p36-postgres-integrations:
- name: "Tests [Postgres9.6][Py3.6][integrations]"
+ name: "[Pg9.6][Py3.6][integrations]"
runs-on: ubuntu-latest
needs: [statics, statics-tests]
env:
- TRAVIS_JOB_NAME: "Tests [Postgres9.6][Py3.6][integrations]"
BACKEND: postgres
PYTHON_VERSION: 3.6
POSTGRES_VERSION: 9.6
ENABLED_INTEGRATIONS: "cassandra kerberos mongo openldap rabbitmq redis"
RUN_INTEGRATION_TESTS: all
+ CI_JOB_TYPE: "Tests"
steps:
- uses: actions/checkout@master
- - name: "Tests [Postgres9.6][Py3.6][integrations]"
+ - name: "Build CI image"
+ run: ./scripts/ci/ci_prepare_image_on_ci.sh
+ - name: "Tests"
run: ./scripts/ci/ci_run_airflow_testing.sh
tests-p36-postgres-providers:
- name: "Tests [Postgres10][Py3.6][providers]"
+ name: "[Pg10][Py3.6][prov]"
runs-on: ubuntu-latest
needs: [statics, statics-tests]
env:
- TRAVIS_JOB_NAME: "Tests [Postgres10][Py3.6][providers]"
BACKEND: postgres
POSTGRES_VERSION: 10
PYTHON_VERSION: 3.6
+ CI_JOB_TYPE: "Tests"
steps:
- uses: actions/checkout@master
- - name: "Tests [Postgres10][Py3.6][providers]"
+ - name: "Build CI image"
+ run: ./scripts/ci/ci_prepare_image_on_ci.sh
+ - name: "Tests"
run: ./scripts/ci/ci_run_airflow_testing.sh tests/providers
tests-p36-postgres-core:
- name: "Tests [Postgres9.6][Py3.6][core]"
+ name: "[Pg9.6][Py3.6][core]"
runs-on: ubuntu-latest
needs: [statics, statics-tests]
env:
- TRAVIS_JOB_NAME: "Tests [Postgres9.6][Py3.6][core]"
BACKEND: postgres
POSTGRES_VERSION: 9.6
PYTHON_VERSION: 3.6
+ CI_JOB_TYPE: "Tests"
steps:
- uses: actions/checkout@master
- - name: "Tests [Postgres9.6][Py3.6][core]"
+ - name: "Build CI image"
+ run: ./scripts/ci/ci_prepare_image_on_ci.sh
+ - name: "Tests"
run: ./scripts/ci/ci_run_airflow_testing.sh --ignore=tests/providers
tests-p37-sqlite-integrations:
- name: "Tests [Sqlite][3.7][integrations]"
+ name: "[Sqlite][3.7][int]"
runs-on: ubuntu-latest
needs: [statics, statics-tests]
env:
- TRAVIS_JOB_NAME: "Tests [Sqlite][3.7][integrations]"
BACKEND: sqlite
PYTHON_VERSION: 3.7
ENABLED_INTEGRATIONS: "cassandra kerberos mongo openldap rabbitmq redis"
RUN_INTEGRATION_TESTS: all
+ CI_JOB_TYPE: "Tests"
steps:
- uses: actions/checkout@master
- - name: "Tests [Sqlite][3.7][integrations]"
+ - name: "Build CI image"
+ run: ./scripts/ci/ci_prepare_image_on_ci.sh
+ - name: "Tests"
run: ./scripts/ci/ci_run_airflow_testing.sh
tests-p36-sqlite:
- name: "Tests [Sqlite][Py3.6]"
+ name: "[Sqlite][Py3.6]"
runs-on: ubuntu-latest
needs: [statics, statics-tests]
env:
- TRAVIS_JOB_NAME: "Tests [Sqlite][Py3.6]"
BACKEND: sqlite
PYTHON_VERSION: 3.6
+ CI_JOB_TYPE: "Tests"
steps:
- uses: actions/checkout@master
- - name: "Tests [Sqlite][Py3.6]"
+ - name: "Build CI image"
+ run: ./scripts/ci/ci_prepare_image_on_ci.sh
+ - name: "Tests"
run: ./scripts/ci/ci_run_airflow_testing.sh
tests-p36-mysql-integrations:
- name: "Tests [MySQL][Py3.6][integrations]"
+ name: "[MySQL5.7][Py3.6][int]"
runs-on: ubuntu-latest
needs: [statics, statics-tests]
env:
- TRAVIS_JOB_NAME: "Tests [MySQL][Py3.6][integrations]"
BACKEND: sqlite
PYTHON_VERSION: 3.6
MYSQL_VERSION: 5.7
ENABLED_INTEGRATIONS: "cassandra kerberos mongo openldap rabbitmq redis"
RUN_INTEGRATION_TESTS: all
+ CI_JOB_TYPE: "Tests"
steps:
- uses: actions/checkout@master
- - name: "Tests [MySQL][Py3.6][integrations]"
+ - name: "Build CI image"
+ run: ./scripts/ci/ci_prepare_image_on_ci.sh
+ - name: "Tests"
run: ./scripts/ci/ci_run_airflow_testing.sh
tests-p36-mysql-providers:
- name: "Tests [MySQL5.7][Py3.7][providers][kerberos]"
+ name: "[MySQL5.7][Py3.7][prov][kerb]"
runs-on: ubuntu-latest
needs: [statics, statics-tests]
env:
- TRAVIS_JOB_NAME: "Tests [MySQL5.7][Py3.7][providers][kerberos]"
BACKEND: mysql
PYTHON_VERSION: 3.7
MYSQL_VERSION: 5.7
ENABLED_INTEGRATIONS: "kerberos"
+ CI_JOB_TYPE: "Tests"
steps:
- uses: actions/checkout@master
- - name: "Tests [MySQL5.7][Py3.7][providers][kerberos]"
+ - name: "Build CI image"
+ run: ./scripts/ci/ci_prepare_image_on_ci.sh
+ - name: "Tests"
run: ./scripts/ci/ci_run_airflow_testing.sh tests/providers
tests-p37-mysql-kerberos:
- name: "Tests [MySQL5.7][Py3.7][core][kerberos]"
+ name: "[MySQL5.7][Py3.7][core][kerb]"
runs-on: ubuntu-latest
needs: [statics, statics-tests]
env:
- TRAVIS_JOB_NAME: "Tests [MySQL5.7][Py3.7][core][kerberos]"
BACKEND: mysql
PYTHON_VERSION: 3.7
MYSQL_VERSION: 5.7
ENABLED_INTEGRATIONS: "kerberos"
+ CI_JOB_TYPE: "Tests"
steps:
- uses: actions/checkout@master
- - name: "Tests [MySQL5.7][Py3.7][core][kerberos]"
+ - name: "Build CI image"
+ run: ./scripts/ci/ci_prepare_image_on_ci.sh
+ - name: "Tests"
run: ./scripts/ci/ci_run_airflow_testing.sh --ignore=tests/providers
requirements:
- name: "Generate requirements"
+ name: "Generate reqs"
runs-on: ubuntu-latest
needs: [statics, statics-tests]
strategy:
Review comment:
With the curent rate of failures I am afraid we will have those failures more often on the perfectly ok builds :). But sure. Let's try!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #8393: Bring
back CI optimisations
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #8393: Bring back CI optimisations
URL: https://github.com/apache/airflow/pull/8393#discussion_r410102971
##########
File path: .github/workflows/ci.yml
##########
@@ -80,134 +80,152 @@ jobs:
name: Build docs
runs-on: ubuntu-latest
env:
- TRAVIS_JOB_NAME: "Build documentation"
PYTHON_VERSION: 3.6
+ CI_JOB_TYPE: "Documentation"
steps:
- uses: actions/checkout@master
- - name: "Build documentation"
+ - name: "Build CI image"
+ run: ./scripts/ci/ci_prepare_image_on_ci.sh
+ - name: "Build docs"
run: ./scripts/ci/ci_docs.sh
tests-p36-postgres-integrations:
- name: "Tests [Postgres9.6][Py3.6][integrations]"
+ name: "[Pg9.6][Py3.6][integrations]"
runs-on: ubuntu-latest
needs: [statics, statics-tests]
env:
- TRAVIS_JOB_NAME: "Tests [Postgres9.6][Py3.6][integrations]"
BACKEND: postgres
PYTHON_VERSION: 3.6
POSTGRES_VERSION: 9.6
ENABLED_INTEGRATIONS: "cassandra kerberos mongo openldap rabbitmq redis"
RUN_INTEGRATION_TESTS: all
+ CI_JOB_TYPE: "Tests"
steps:
- uses: actions/checkout@master
- - name: "Tests [Postgres9.6][Py3.6][integrations]"
+ - name: "Build CI image"
+ run: ./scripts/ci/ci_prepare_image_on_ci.sh
+ - name: "Tests"
run: ./scripts/ci/ci_run_airflow_testing.sh
tests-p36-postgres-providers:
- name: "Tests [Postgres10][Py3.6][providers]"
+ name: "[Pg10][Py3.6][prov]"
runs-on: ubuntu-latest
needs: [statics, statics-tests]
env:
- TRAVIS_JOB_NAME: "Tests [Postgres10][Py3.6][providers]"
BACKEND: postgres
POSTGRES_VERSION: 10
PYTHON_VERSION: 3.6
+ CI_JOB_TYPE: "Tests"
steps:
- uses: actions/checkout@master
- - name: "Tests [Postgres10][Py3.6][providers]"
+ - name: "Build CI image"
+ run: ./scripts/ci/ci_prepare_image_on_ci.sh
+ - name: "Tests"
run: ./scripts/ci/ci_run_airflow_testing.sh tests/providers
tests-p36-postgres-core:
- name: "Tests [Postgres9.6][Py3.6][core]"
+ name: "[Pg9.6][Py3.6][core]"
runs-on: ubuntu-latest
needs: [statics, statics-tests]
env:
- TRAVIS_JOB_NAME: "Tests [Postgres9.6][Py3.6][core]"
BACKEND: postgres
POSTGRES_VERSION: 9.6
PYTHON_VERSION: 3.6
+ CI_JOB_TYPE: "Tests"
steps:
- uses: actions/checkout@master
- - name: "Tests [Postgres9.6][Py3.6][core]"
+ - name: "Build CI image"
+ run: ./scripts/ci/ci_prepare_image_on_ci.sh
+ - name: "Tests"
run: ./scripts/ci/ci_run_airflow_testing.sh --ignore=tests/providers
tests-p37-sqlite-integrations:
- name: "Tests [Sqlite][3.7][integrations]"
+ name: "[Sqlite][3.7][int]"
runs-on: ubuntu-latest
needs: [statics, statics-tests]
env:
- TRAVIS_JOB_NAME: "Tests [Sqlite][3.7][integrations]"
BACKEND: sqlite
PYTHON_VERSION: 3.7
ENABLED_INTEGRATIONS: "cassandra kerberos mongo openldap rabbitmq redis"
RUN_INTEGRATION_TESTS: all
+ CI_JOB_TYPE: "Tests"
steps:
- uses: actions/checkout@master
- - name: "Tests [Sqlite][3.7][integrations]"
+ - name: "Build CI image"
+ run: ./scripts/ci/ci_prepare_image_on_ci.sh
+ - name: "Tests"
run: ./scripts/ci/ci_run_airflow_testing.sh
tests-p36-sqlite:
- name: "Tests [Sqlite][Py3.6]"
+ name: "[Sqlite][Py3.6]"
runs-on: ubuntu-latest
needs: [statics, statics-tests]
env:
- TRAVIS_JOB_NAME: "Tests [Sqlite][Py3.6]"
BACKEND: sqlite
PYTHON_VERSION: 3.6
+ CI_JOB_TYPE: "Tests"
steps:
- uses: actions/checkout@master
- - name: "Tests [Sqlite][Py3.6]"
+ - name: "Build CI image"
+ run: ./scripts/ci/ci_prepare_image_on_ci.sh
+ - name: "Tests"
run: ./scripts/ci/ci_run_airflow_testing.sh
tests-p36-mysql-integrations:
- name: "Tests [MySQL][Py3.6][integrations]"
+ name: "[MySQL5.7][Py3.6][int]"
runs-on: ubuntu-latest
needs: [statics, statics-tests]
env:
- TRAVIS_JOB_NAME: "Tests [MySQL][Py3.6][integrations]"
BACKEND: sqlite
PYTHON_VERSION: 3.6
MYSQL_VERSION: 5.7
ENABLED_INTEGRATIONS: "cassandra kerberos mongo openldap rabbitmq redis"
RUN_INTEGRATION_TESTS: all
+ CI_JOB_TYPE: "Tests"
steps:
- uses: actions/checkout@master
- - name: "Tests [MySQL][Py3.6][integrations]"
+ - name: "Build CI image"
+ run: ./scripts/ci/ci_prepare_image_on_ci.sh
+ - name: "Tests"
run: ./scripts/ci/ci_run_airflow_testing.sh
tests-p36-mysql-providers:
- name: "Tests [MySQL5.7][Py3.7][providers][kerberos]"
+ name: "[MySQL5.7][Py3.7][prov][kerb]"
runs-on: ubuntu-latest
needs: [statics, statics-tests]
env:
- TRAVIS_JOB_NAME: "Tests [MySQL5.7][Py3.7][providers][kerberos]"
BACKEND: mysql
PYTHON_VERSION: 3.7
MYSQL_VERSION: 5.7
ENABLED_INTEGRATIONS: "kerberos"
+ CI_JOB_TYPE: "Tests"
steps:
- uses: actions/checkout@master
- - name: "Tests [MySQL5.7][Py3.7][providers][kerberos]"
+ - name: "Build CI image"
+ run: ./scripts/ci/ci_prepare_image_on_ci.sh
+ - name: "Tests"
run: ./scripts/ci/ci_run_airflow_testing.sh tests/providers
tests-p37-mysql-kerberos:
- name: "Tests [MySQL5.7][Py3.7][core][kerberos]"
+ name: "[MySQL5.7][Py3.7][core][kerb]"
runs-on: ubuntu-latest
needs: [statics, statics-tests]
env:
- TRAVIS_JOB_NAME: "Tests [MySQL5.7][Py3.7][core][kerberos]"
BACKEND: mysql
PYTHON_VERSION: 3.7
MYSQL_VERSION: 5.7
ENABLED_INTEGRATIONS: "kerberos"
+ CI_JOB_TYPE: "Tests"
steps:
- uses: actions/checkout@master
- - name: "Tests [MySQL5.7][Py3.7][core][kerberos]"
+ - name: "Build CI image"
+ run: ./scripts/ci/ci_prepare_image_on_ci.sh
+ - name: "Tests"
run: ./scripts/ci/ci_run_airflow_testing.sh --ignore=tests/providers
requirements:
- name: "Generate requirements"
+ name: "Generate reqs"
runs-on: ubuntu-latest
needs: [statics, statics-tests]
strategy:
Review comment:
I think actually until we can re-run individual jobs, fail-fast is better idea. We will have to re-run all of them anyway, so it's better if they all fail fast if one of them fails.
WDYT @turbaszek ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] ashb commented on a change in pull request #8393: Bring
back CI optimisations
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #8393: Bring back CI optimisations
URL: https://github.com/apache/airflow/pull/8393#discussion_r409464808
##########
File path: .github/workflows/ci.yml
##########
@@ -80,134 +80,152 @@ jobs:
name: Build docs
runs-on: ubuntu-latest
env:
- TRAVIS_JOB_NAME: "Build documentation"
PYTHON_VERSION: 3.6
+ CI_JOB_TYPE: "Documentation"
steps:
- uses: actions/checkout@master
- - name: "Build documentation"
+ - name: "Build CI image"
+ run: ./scripts/ci/ci_prepare_image_on_ci.sh
+ - name: "Build docs"
run: ./scripts/ci/ci_docs.sh
tests-p36-postgres-integrations:
- name: "Tests [Postgres9.6][Py3.6][integrations]"
+ name: "[Pg9.6][Py3.6][integrations]"
runs-on: ubuntu-latest
needs: [statics, statics-tests]
env:
- TRAVIS_JOB_NAME: "Tests [Postgres9.6][Py3.6][integrations]"
BACKEND: postgres
PYTHON_VERSION: 3.6
POSTGRES_VERSION: 9.6
ENABLED_INTEGRATIONS: "cassandra kerberos mongo openldap rabbitmq redis"
RUN_INTEGRATION_TESTS: all
+ CI_JOB_TYPE: "Tests"
steps:
- uses: actions/checkout@master
- - name: "Tests [Postgres9.6][Py3.6][integrations]"
+ - name: "Build CI image"
+ run: ./scripts/ci/ci_prepare_image_on_ci.sh
+ - name: "Tests"
run: ./scripts/ci/ci_run_airflow_testing.sh
tests-p36-postgres-providers:
- name: "Tests [Postgres10][Py3.6][providers]"
+ name: "[Pg10][Py3.6][prov]"
runs-on: ubuntu-latest
needs: [statics, statics-tests]
env:
- TRAVIS_JOB_NAME: "Tests [Postgres10][Py3.6][providers]"
BACKEND: postgres
POSTGRES_VERSION: 10
PYTHON_VERSION: 3.6
+ CI_JOB_TYPE: "Tests"
steps:
- uses: actions/checkout@master
- - name: "Tests [Postgres10][Py3.6][providers]"
+ - name: "Build CI image"
+ run: ./scripts/ci/ci_prepare_image_on_ci.sh
+ - name: "Tests"
run: ./scripts/ci/ci_run_airflow_testing.sh tests/providers
tests-p36-postgres-core:
- name: "Tests [Postgres9.6][Py3.6][core]"
+ name: "[Pg9.6][Py3.6][core]"
runs-on: ubuntu-latest
needs: [statics, statics-tests]
env:
- TRAVIS_JOB_NAME: "Tests [Postgres9.6][Py3.6][core]"
BACKEND: postgres
POSTGRES_VERSION: 9.6
PYTHON_VERSION: 3.6
+ CI_JOB_TYPE: "Tests"
steps:
- uses: actions/checkout@master
- - name: "Tests [Postgres9.6][Py3.6][core]"
+ - name: "Build CI image"
+ run: ./scripts/ci/ci_prepare_image_on_ci.sh
+ - name: "Tests"
run: ./scripts/ci/ci_run_airflow_testing.sh --ignore=tests/providers
tests-p37-sqlite-integrations:
- name: "Tests [Sqlite][3.7][integrations]"
+ name: "[Sqlt][3.7][int]"
Review comment:
```suggestion
name: "[Sqlite][3.7][int]"
```
please.
It's not that long and i've never seen Sqlite abbreviated before.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] turbaszek commented on a change in pull request #8393:
Bring back CI optimisations
Posted by GitBox <gi...@apache.org>.
turbaszek commented on a change in pull request #8393: Bring back CI optimisations
URL: https://github.com/apache/airflow/pull/8393#discussion_r410708187
##########
File path: .github/workflows/ci.yml
##########
@@ -17,272 +17,275 @@
#
---
-name: Airflow
+name: CI Build
on:
+ schedule:
+ - cron: '10 2 * * *'
push:
- branches: ['*']
+ branches: ['master', 'v1-10-test', 'v1-10-stable']
pull_request:
- branches: [master]
+ branches: ['master', 'v1-10-test', 'v1-10-stable']
env:
- BUILD_ID: ${{github.sha }}
MOUNT_LOCAL_SOURCES: "false"
- MOUNT_HOST_AIRFLOW_VOLUME: "true"
FORCE_ANSWER_TO_QUESTIONS: "yes"
SKIP_CHECK_REMOTE_IMAGE: "true"
+ SKIP_CI_IMAGE_CHECK: "true"
DB_RESET: "true"
VERBOSE: "true"
- CI: "true"
- # Should be a target branch
- TRAVIS_BRANCH: "master"
- TRAVIS: "true"
+ UPGRADE_TO_LATEST_REQUIREMENTS: "false"
+ ENABLED_INTEGRATIONS: "cassandra kerberos mongo openldap presto rabbitmq redis"
+ PYTHON_MAJOR_MINOR_VERSION: 3.6
+ # TODO: Enable cache registry once a master build succeeds uploading the images to Github registry cache
+ DISABLE_CACHE_REGISTRY: "true"
+ CACHE_REGISTRY: "docker.pkg.github.com"
+ CACHE_IMAGE_PREFIX: ${{ github.repository }}
+ CACHE_REGISTRY_USERNAME: ${{ github.actor }}
+ CACHE_REGISTRY_PASSWORD: ${{ secrets.GITHUB_TOKEN }}
+
jobs:
- statics:
- name: Static checks
+
+ static-checks-1:
+ name: "Checks: pylint, mypy"
runs-on: ubuntu-latest
env:
- PYTHON_VERSION: 3.6
- AIRFLOW_MOUNT_SOURCE_DIR_FOR_STATIC_CHECKS: "true"
- TRAVIS_JOB_NAME: "Static"
+ MOUNT_SOURCE_DIR_FOR_STATIC_CHECKS: "true"
+ CI_JOB_TYPE: "Pylint checks for tests"
steps:
- uses: actions/checkout@master
- - name: free disk space
- run: |
- sudo swapoff -a
- sudo rm -f /swapfile
- sudo apt clean
- docker rmi $(docker image ls -aq)
- df -h
- uses: actions/setup-python@v1
with:
- python-version: ${{ env.PYTHON_VERSION }}
- - name: "Before install"
- run: ./scripts/ci/ci_before_install.sh
- - name: "Static checks"
- if: success()
+ python-version: '3.x'
+ - name: Cache pre-commit env
+ uses: actions/cache@v1
env:
- PYTHON_VERSION: 3.6
- run: ./scripts/ci/ci_run_all_static_checks.sh
+ cache-name: cache-pre-commit
+ with:
+ path: ~/.cache/pre-commit
+ key: ${{ env.cache-name }}-${{ github.job }}-${{ hashFiles('.pre-commit-config.yaml') }}
+ - name: "Free space"
+ run: ./scripts/ci/ci_free_space_on_ci.sh
+ - name: "Build CI image"
+ run: ./scripts/ci/ci_prepare_image_on_ci.sh
+ - name: "Pylint checks for tests"
+ run: |
+ python -m pip install pre-commit \
+ --constraint requirements/requirements-python${PYTHON_MAJOR_MINOR_VERSION}.txt
+ ./scripts/ci/ci_run_static_checks.sh pylint-tests mypy
- statics-tests:
- name: Pylint for tests
+ static-checks-2:
+ name: "Checks: All other"
runs-on: ubuntu-latest
env:
- PYTHON_VERSION: 3.6
- AIRFLOW_MOUNT_SOURCE_DIR_FOR_STATIC_CHECKS: "true"
- TRAVIS_JOB_NAME: "Static"
+ SKIP: pylint-tests,mypy
+ MOUNT_SOURCE_DIR_FOR_STATIC_CHECKS: "true"
+ CI_JOB_TYPE: "Static checks"
steps:
- uses: actions/checkout@master
- uses: actions/setup-python@v1
with:
- python-version: ${{ env.PYTHON_VERSION }}
- - name: "Before install"
- run: ./scripts/ci/ci_before_install.sh
- - name: "Static checks tests"
- if: success()
+ python-version: '3.x'
+ - name: Cache pre-commit env
+ uses: actions/cache@v1
env:
- PYTHON_VERSION: 3.6
- run: ./scripts/ci/ci_run_static_checks_pylint_tests.sh
+ cache-name: cache-pre-commit
+ with:
+ path: ~/.cache/pre-commit
+ key: ${{ env.cache-name }}-${{ github.job }}-${{ hashFiles('.pre-commit-config.yaml') }}
+ - name: "Free space"
+ run: ./scripts/ci/ci_free_space_on_ci.sh
+ - name: "Build CI image"
+ run: ./scripts/ci/ci_prepare_image_on_ci.sh
+ - name: "Static checks"
+ run: |
+ python -m pip install pre-commit \
+ --constraint requirements/requirements-python${PYTHON_MAJOR_MINOR_VERSION}.txt
+ ./scripts/ci/ci_run_static_checks.sh
docs:
name: Build docs
runs-on: ubuntu-latest
env:
- TRAVIS_JOB_NAME: "Build documentation"
- PYTHON_VERSION: 3.6
+ CI_JOB_TYPE: "Documentation"
steps:
- uses: actions/checkout@master
- - name: "Build documentation"
+ - name: "Build CI image ${{ matrix.python-version }}"
+ run: ./scripts/ci/ci_prepare_image_on_ci.sh
+ - name: "Build docs"
run: ./scripts/ci/ci_docs.sh
- tests-p36-postgres-integrations:
- name: "Tests [Postgres9.6][Py3.6][integrations]"
+ build-prod-image:
+ name: "Build prod image"
runs-on: ubuntu-latest
- needs: [statics, statics-tests]
Review comment:
Is there any particular reason for skipping this? (Or is it just for faster feedback during development?)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #8393: Bring
back CI optimisations
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #8393: Bring back CI optimisations
URL: https://github.com/apache/airflow/pull/8393#discussion_r410724908
##########
File path: .github/workflows/ci.yml
##########
@@ -17,272 +17,275 @@
#
---
-name: Airflow
+name: CI Build
on:
+ schedule:
+ - cron: '10 2 * * *'
push:
- branches: ['*']
+ branches: ['master', 'v1-10-test', 'v1-10-stable']
pull_request:
- branches: [master]
+ branches: ['master', 'v1-10-test', 'v1-10-stable']
env:
- BUILD_ID: ${{github.sha }}
MOUNT_LOCAL_SOURCES: "false"
- MOUNT_HOST_AIRFLOW_VOLUME: "true"
FORCE_ANSWER_TO_QUESTIONS: "yes"
SKIP_CHECK_REMOTE_IMAGE: "true"
+ SKIP_CI_IMAGE_CHECK: "true"
DB_RESET: "true"
VERBOSE: "true"
- CI: "true"
- # Should be a target branch
- TRAVIS_BRANCH: "master"
- TRAVIS: "true"
+ UPGRADE_TO_LATEST_REQUIREMENTS: "false"
+ ENABLED_INTEGRATIONS: "cassandra kerberos mongo openldap presto rabbitmq redis"
+ PYTHON_MAJOR_MINOR_VERSION: 3.6
+ # TODO: Enable cache registry once a master build succeeds uploading the images to Github registry cache
+ DISABLE_CACHE_REGISTRY: "true"
+ CACHE_REGISTRY: "docker.pkg.github.com"
+ CACHE_IMAGE_PREFIX: ${{ github.repository }}
+ CACHE_REGISTRY_USERNAME: ${{ github.actor }}
+ CACHE_REGISTRY_PASSWORD: ${{ secrets.GITHUB_TOKEN }}
+
jobs:
- statics:
- name: Static checks
+
+ static-checks-1:
+ name: "Checks: pylint, mypy"
runs-on: ubuntu-latest
env:
- PYTHON_VERSION: 3.6
- AIRFLOW_MOUNT_SOURCE_DIR_FOR_STATIC_CHECKS: "true"
- TRAVIS_JOB_NAME: "Static"
+ MOUNT_SOURCE_DIR_FOR_STATIC_CHECKS: "true"
+ CI_JOB_TYPE: "Pylint checks for tests"
steps:
- uses: actions/checkout@master
- - name: free disk space
- run: |
- sudo swapoff -a
- sudo rm -f /swapfile
- sudo apt clean
- docker rmi $(docker image ls -aq)
- df -h
- uses: actions/setup-python@v1
with:
- python-version: ${{ env.PYTHON_VERSION }}
- - name: "Before install"
- run: ./scripts/ci/ci_before_install.sh
- - name: "Static checks"
- if: success()
+ python-version: '3.x'
+ - name: Cache pre-commit env
+ uses: actions/cache@v1
env:
- PYTHON_VERSION: 3.6
- run: ./scripts/ci/ci_run_all_static_checks.sh
+ cache-name: cache-pre-commit
+ with:
+ path: ~/.cache/pre-commit
Review comment:
Yep. it speeds up (by about a minute) pre-commit initialization. When you run pre-commit first time it builds local virtualenvs for all the pre-commits requiring it. It takes about a minute and the .venvs are stored in ~/.cache/pre-commit . This way we save about a minute for static checks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #8393: Bring
back CI optimisations
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #8393: Bring back CI optimisations
URL: https://github.com/apache/airflow/pull/8393#discussion_r409432803
##########
File path: .github/workflows/ci.yml
##########
@@ -19,58 +19,58 @@
---
name: Airflow
on:
+ schedule:
+ # Schedule cron jobs every day at 2.10 am
+ - cron: '10 2 * * *'
push:
- branches: ['*']
+ branches: ['master', 'v1-10-test', 'v1-10-stable']
Review comment:
Yep
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] ashb commented on a change in pull request #8393: Bring
back CI optimisations
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #8393: Bring back CI optimisations
URL: https://github.com/apache/airflow/pull/8393#discussion_r409463677
##########
File path: .github/workflows/ci.yml
##########
@@ -17,60 +17,60 @@
#
---
-name: Airflow
+name: Airflow CI Build
on:
+ schedule:
+ # Schedule cron jobs every day at 2.10 am
+ - cron: '10 2 * * *'
push:
- branches: ['*']
+ branches: ['master', 'v1-10-test', 'v1-10-stable']
pull_request:
- branches: [master]
+ branches: ['master', 'v1-10-test', 'v1-10-stable']
env:
- BUILD_ID: ${{github.sha }}
MOUNT_LOCAL_SOURCES: "false"
MOUNT_HOST_AIRFLOW_VOLUME: "true"
FORCE_ANSWER_TO_QUESTIONS: "yes"
SKIP_CHECK_REMOTE_IMAGE: "true"
DB_RESET: "true"
VERBOSE: "true"
- CI: "true"
Review comment:
And is still used in scripts.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] turbaszek commented on a change in pull request #8393:
Bring back CI optimisations
Posted by GitBox <gi...@apache.org>.
turbaszek commented on a change in pull request #8393: Bring back CI optimisations
URL: https://github.com/apache/airflow/pull/8393#discussion_r410708271
##########
File path: .github/workflows/ci.yml
##########
@@ -17,272 +17,275 @@
#
---
-name: Airflow
+name: CI Build
on:
+ schedule:
+ - cron: '10 2 * * *'
push:
- branches: ['*']
+ branches: ['master', 'v1-10-test', 'v1-10-stable']
pull_request:
- branches: [master]
+ branches: ['master', 'v1-10-test', 'v1-10-stable']
env:
- BUILD_ID: ${{github.sha }}
MOUNT_LOCAL_SOURCES: "false"
- MOUNT_HOST_AIRFLOW_VOLUME: "true"
FORCE_ANSWER_TO_QUESTIONS: "yes"
SKIP_CHECK_REMOTE_IMAGE: "true"
+ SKIP_CI_IMAGE_CHECK: "true"
DB_RESET: "true"
VERBOSE: "true"
- CI: "true"
- # Should be a target branch
- TRAVIS_BRANCH: "master"
- TRAVIS: "true"
+ UPGRADE_TO_LATEST_REQUIREMENTS: "false"
+ ENABLED_INTEGRATIONS: "cassandra kerberos mongo openldap presto rabbitmq redis"
+ PYTHON_MAJOR_MINOR_VERSION: 3.6
+ # TODO: Enable cache registry once a master build succeeds uploading the images to Github registry cache
+ DISABLE_CACHE_REGISTRY: "true"
+ CACHE_REGISTRY: "docker.pkg.github.com"
+ CACHE_IMAGE_PREFIX: ${{ github.repository }}
+ CACHE_REGISTRY_USERNAME: ${{ github.actor }}
+ CACHE_REGISTRY_PASSWORD: ${{ secrets.GITHUB_TOKEN }}
+
jobs:
- statics:
- name: Static checks
+
+ static-checks-1:
+ name: "Checks: pylint, mypy"
runs-on: ubuntu-latest
env:
- PYTHON_VERSION: 3.6
- AIRFLOW_MOUNT_SOURCE_DIR_FOR_STATIC_CHECKS: "true"
- TRAVIS_JOB_NAME: "Static"
+ MOUNT_SOURCE_DIR_FOR_STATIC_CHECKS: "true"
+ CI_JOB_TYPE: "Pylint checks for tests"
steps:
- uses: actions/checkout@master
- - name: free disk space
- run: |
- sudo swapoff -a
- sudo rm -f /swapfile
- sudo apt clean
- docker rmi $(docker image ls -aq)
- df -h
- uses: actions/setup-python@v1
with:
- python-version: ${{ env.PYTHON_VERSION }}
- - name: "Before install"
- run: ./scripts/ci/ci_before_install.sh
- - name: "Static checks"
- if: success()
+ python-version: '3.x'
+ - name: Cache pre-commit env
+ uses: actions/cache@v1
env:
- PYTHON_VERSION: 3.6
- run: ./scripts/ci/ci_run_all_static_checks.sh
+ cache-name: cache-pre-commit
+ with:
+ path: ~/.cache/pre-commit
+ key: ${{ env.cache-name }}-${{ github.job }}-${{ hashFiles('.pre-commit-config.yaml') }}
+ - name: "Free space"
+ run: ./scripts/ci/ci_free_space_on_ci.sh
+ - name: "Build CI image"
+ run: ./scripts/ci/ci_prepare_image_on_ci.sh
+ - name: "Pylint checks for tests"
+ run: |
+ python -m pip install pre-commit \
+ --constraint requirements/requirements-python${PYTHON_MAJOR_MINOR_VERSION}.txt
+ ./scripts/ci/ci_run_static_checks.sh pylint-tests mypy
- statics-tests:
- name: Pylint for tests
+ static-checks-2:
+ name: "Checks: All other"
runs-on: ubuntu-latest
env:
- PYTHON_VERSION: 3.6
- AIRFLOW_MOUNT_SOURCE_DIR_FOR_STATIC_CHECKS: "true"
- TRAVIS_JOB_NAME: "Static"
+ SKIP: pylint-tests,mypy
+ MOUNT_SOURCE_DIR_FOR_STATIC_CHECKS: "true"
+ CI_JOB_TYPE: "Static checks"
steps:
- uses: actions/checkout@master
- uses: actions/setup-python@v1
with:
- python-version: ${{ env.PYTHON_VERSION }}
- - name: "Before install"
- run: ./scripts/ci/ci_before_install.sh
- - name: "Static checks tests"
- if: success()
+ python-version: '3.x'
+ - name: Cache pre-commit env
+ uses: actions/cache@v1
env:
- PYTHON_VERSION: 3.6
- run: ./scripts/ci/ci_run_static_checks_pylint_tests.sh
+ cache-name: cache-pre-commit
+ with:
+ path: ~/.cache/pre-commit
+ key: ${{ env.cache-name }}-${{ github.job }}-${{ hashFiles('.pre-commit-config.yaml') }}
+ - name: "Free space"
+ run: ./scripts/ci/ci_free_space_on_ci.sh
+ - name: "Build CI image"
+ run: ./scripts/ci/ci_prepare_image_on_ci.sh
+ - name: "Static checks"
+ run: |
+ python -m pip install pre-commit \
+ --constraint requirements/requirements-python${PYTHON_MAJOR_MINOR_VERSION}.txt
+ ./scripts/ci/ci_run_static_checks.sh
docs:
name: Build docs
runs-on: ubuntu-latest
env:
- TRAVIS_JOB_NAME: "Build documentation"
- PYTHON_VERSION: 3.6
+ CI_JOB_TYPE: "Documentation"
steps:
- uses: actions/checkout@master
- - name: "Build documentation"
+ - name: "Build CI image ${{ matrix.python-version }}"
+ run: ./scripts/ci/ci_prepare_image_on_ci.sh
+ - name: "Build docs"
run: ./scripts/ci/ci_docs.sh
- tests-p36-postgres-integrations:
- name: "Tests [Postgres9.6][Py3.6][integrations]"
+ build-prod-image:
+ name: "Build prod image"
runs-on: ubuntu-latest
- needs: [statics, statics-tests]
+ strategy:
+ matrix:
+ python-version: [3.6, 3.7]
env:
- TRAVIS_JOB_NAME: "Tests [Postgres9.6][Py3.6][integrations]"
- BACKEND: postgres
- PYTHON_VERSION: 3.6
- POSTGRES_VERSION: 9.6
- ENABLED_INTEGRATIONS: "cassandra kerberos mongo openldap rabbitmq redis"
- RUN_INTEGRATION_TESTS: all
+ PYTHON_MAJOR_MINOR_VERSION: ${{ matrix.python-version }}
+ CI_JOB_TYPE: "Prod image"
steps:
- uses: actions/checkout@master
- - name: free disk space
- run: |
- sudo swapoff -a
- sudo rm -f /swapfile
- sudo apt clean
- docker rmi $(docker image ls -aq)
- df -h
- - name: "Tests [Postgres9.6][Py3.6][integrations]"
- run: ./scripts/ci/ci_run_airflow_testing.sh
+ - name: "Build PROD image ${{ matrix.python-version }}"
+ run: ./scripts/ci/ci_build_production_images.sh
- tests-p36-postgres-providers:
- name: "Tests [Postgres10][Py3.6][providers]"
+ prepare-backport-packages:
+ name: "Backport packages"
runs-on: ubuntu-latest
- needs: [statics, statics-tests]
env:
- TRAVIS_JOB_NAME: "Tests [Postgres10][Py3.6][providers]"
- BACKEND: postgres
- POSTGRES_VERSION: 10
- PYTHON_VERSION: 3.6
+ INSTALL_AIRFLOW_VERSION: "1.10.10"
+ CI_JOB_TYPE: "Prepare & test packages"
steps:
- uses: actions/checkout@master
- - name: "Tests [Postgres10][Py3.6][providers]"
- run: ./scripts/ci/ci_run_airflow_testing.sh tests/providers
+ - uses: actions/setup-python@v1
+ with:
+ python-version: '3.x'
+ - name: "Prepare & test backport packages"
+ run: |
+ python -m pip install bowler sdist wheel
+ ./scripts/ci/ci_prepare_and_test_backport_packages.sh
- tests-p36-postgres-core:
- name: "Tests [Postgres9.6][Py3.6][core]"
+ pyfiles:
+ name: "Count changed python files"
runs-on: ubuntu-latest
- needs: [statics, statics-tests]
- env:
- TRAVIS_JOB_NAME: "Tests [Postgres9.6][Py3.6][core]"
- BACKEND: postgres
- POSTGRES_VERSION: 9.6
- PYTHON_VERSION: 3.6
+ outputs:
+ count: ${{ steps.pyfiles.outputs.count }}
steps:
- uses: actions/checkout@master
- - name: "Tests [Postgres9.6][Py3.6][core]"
- run: ./scripts/ci/ci_run_airflow_testing.sh --ignore=tests/providers
-
+ - name: "Get count of changed python files"
+ run: |
+ set +e
+ ./scripts/ci/ci_count_changed_files.sh ${GITHUB_SHA} '\.py$'
+ echo "::set-output name=count::$?"
+ id: pyfiles
- tests-p37-sqlite-integrations:
- name: "Tests [Sqlite][3.7][integrations]"
+ tests-postgres:
+ name: "Tests: Pg${{ matrix.postgres-version }}, Py{{ matrix.python-version}}"
runs-on: ubuntu-latest
- needs: [statics, statics-tests]
+ needs: [static-checks-1, static-checks-2, pyfiles]
+ strategy:
+ matrix:
+ python-version: [3.6, 3.7]
+ postgres-version: [9.6, 10]
+ fail-fast: false
env:
- TRAVIS_JOB_NAME: "Tests [Sqlite][3.7][integrations]"
- BACKEND: sqlite
- PYTHON_VERSION: 3.7
- ENABLED_INTEGRATIONS: "cassandra kerberos mongo openldap rabbitmq redis"
- RUN_INTEGRATION_TESTS: all
+ BACKEND: postgres
+ PYTHON_MAJOR_MINOR_VERSION: ${{ matrix.python-version }}
+ POSTGRES_VERSION: ${{ matrix.postgres-version }}
+ RUN_TESTS: "true"
+ CI_JOB_TYPE: "Tests"
+ # For pull requests only run tests when python files changed
+ if: needs.pyfiles.outputs.count != '0' || github.event_name != 'pull_request'
steps:
- uses: actions/checkout@master
- - name: free disk space
- run: |
- sudo swapoff -a
- sudo rm -f /swapfile
- sudo apt clean
- docker rmi $(docker image ls -aq)
- df -h
- - name: "Tests [Sqlite][3.7][integrations]"
+ - uses: actions/setup-python@v1
+ with:
+ python-version: '3.x'
+ - name: "Free space"
+ run: ./scripts/ci/ci_free_space_on_ci.sh
+ - name: "Build CI image ${{ matrix.python-version }}"
+ run: ./scripts/ci/ci_prepare_image_on_ci.sh
+ - name: "Tests"
run: ./scripts/ci/ci_run_airflow_testing.sh
- tests-p36-sqlite:
- name: "Tests [Sqlite][Py3.6]"
+ tests-mysql:
+ name: "Tests: MySQL${{ matrix.mysql-version }}, Py{{ matrix.python-version}}"
runs-on: ubuntu-latest
- needs: [statics, statics-tests]
+ needs: [static-checks-1, static-checks-2, pyfiles]
+ strategy:
+ matrix:
+ python-version: [3.6, 3.7]
+ mysql-version: [5.7]
+ fail-fast: false
env:
- TRAVIS_JOB_NAME: "Tests [Sqlite][Py3.6]"
- BACKEND: sqlite
- PYTHON_VERSION: 3.6
+ BACKEND: mysql
+ PYTHON_MAJOR_MINOR_VERSION: ${{ matrix.python-version }}
+ MYSQL_VERSION: ${{ matrix.mysql-version }}
+ RUN_TESTS: "true"
+ CI_JOB_TYPE: "Tests"
+ # For pull requests only run tests when python files changed
+ if: needs.pyfiles.outputs.count != '0' || github.event_name != 'pull_request'
steps:
- uses: actions/checkout@master
- - name: "Tests [Sqlite][Py3.6]"
+ - uses: actions/setup-python@v1
+ with:
+ python-version: '3.x'
+ - name: "Free space"
+ run: ./scripts/ci/ci_free_space_on_ci.sh
+ - name: "Build CI image ${{ matrix.python-version }}"
+ run: ./scripts/ci/ci_prepare_image_on_ci.sh
+ - name: "Tests"
run: ./scripts/ci/ci_run_airflow_testing.sh
- tests-p36-mysql-integrations:
- name: "Tests [MySQL][Py3.6][integrations]"
+ tests-sqlite:
+ name: "Tests: Sqlite Py{{ matrix.python-version}}"
runs-on: ubuntu-latest
- needs: [statics, statics-tests]
+ needs: [static-checks-1, static-checks-2, pyfiles]
+ strategy:
+ matrix:
+ python-version: [3.6, 3.7]
+ fail-fast: false
env:
- TRAVIS_JOB_NAME: "Tests [MySQL][Py3.6][integrations]"
BACKEND: sqlite
- PYTHON_VERSION: 3.6
- MYSQL_VERSION: 5.7
- ENABLED_INTEGRATIONS: "cassandra kerberos mongo openldap rabbitmq redis"
- RUN_INTEGRATION_TESTS: all
+ PYTHON_MAJOR_MINOR_VERSION: ${{ matrix.python-version }}
+ RUN_TESTS: "true"
+ CI_JOB_TYPE: "Tests"
+ # For pull requests only run tests when python files changed
+ if: needs.pyfiles.outputs.count != '0' || github.event_name != 'pull_request'
steps:
- uses: actions/checkout@master
- - name: free disk space
- run: |
- sudo swapoff -a
- sudo rm -f /swapfile
- sudo apt clean
- docker rmi $(docker image ls -aq)
- df -h
- - name: "Tests [MySQL][Py3.6][integrations]"
+ - uses: actions/setup-python@v1
+ with:
+ python-version: '3.x'
+ - name: "Free space"
+ run: ./scripts/ci/ci_free_space_on_ci.sh
+ - name: "Build CI image ${{ matrix.python-version }}"
+ run: ./scripts/ci/ci_prepare_image_on_ci.sh
+ - name: "Tests"
run: ./scripts/ci/ci_run_airflow_testing.sh
- tests-p36-mysql-providers:
- name: "Tests [MySQL5.7][Py3.7][providers][kerberos]"
- runs-on: ubuntu-latest
- needs: [statics, statics-tests]
- env:
- TRAVIS_JOB_NAME: "Tests [MySQL5.7][Py3.7][providers][kerberos]"
- BACKEND: mysql
- PYTHON_VERSION: 3.7
- MYSQL_VERSION: 5.7
- ENABLED_INTEGRATIONS: "kerberos"
- steps:
- - uses: actions/checkout@master
- - name: "Tests [MySQL5.7][Py3.7][providers][kerberos]"
- run: ./scripts/ci/ci_run_airflow_testing.sh tests/providers
-
- tests-p37-mysql-kerberos:
- name: "Tests [MySQL5.7][Py3.7][core][kerberos]"
- runs-on: ubuntu-latest
- needs: [statics, statics-tests]
- env:
- TRAVIS_JOB_NAME: "Tests [MySQL5.7][Py3.7][core][kerberos]"
- BACKEND: mysql
- PYTHON_VERSION: 3.7
- MYSQL_VERSION: 5.7
- ENABLED_INTEGRATIONS: "kerberos"
- steps:
- - uses: actions/checkout@master
- - name: free disk space
- run: |
- sudo swapoff -a
- sudo rm -f /swapfile
- sudo apt clean
- docker rmi $(docker image ls -aq)
- df -h
- - name: "Tests [MySQL5.7][Py3.7][core][kerberos]"
- run: ./scripts/ci/ci_run_airflow_testing.sh --ignore=tests/providers
-
requirements:
- name: "Generate requirements"
+ name: "Requirements: Py ${{ matrix.python-version}} "
runs-on: ubuntu-latest
- needs: [statics, statics-tests]
strategy:
matrix:
python-version: [3.6, 3.7]
+ env:
+ PYTHON_MAJOR_MINOR_VERSION: ${{ matrix.python-version }}
+ SHOW_GENERATE_REQUIREMENTS_INSTRUCTIONS: true
+ CI_JOB_TYPE: "Requirements"
steps:
- uses: actions/checkout@master
- uses: actions/setup-python@v1
+ - name: "Build CI image ${{ matrix.python-version }}"
+ run: ./scripts/ci/ci_prepare_image_on_ci.sh
- name: "Generate requirements"
- env:
- PYTHON_MAJOR_MINOR_VERSION: ${{ matrix.python-version }}
- SHOW_GENERATE_REQUIREMENTS_INSTRUCTIONS: true
run: ./scripts/ci/ci_generate_requirements.sh
- build-image:
- name: "Build production image"
+ push-images-to-github-cache:
+ name: "Build and push images to github registry"
runs-on: ubuntu-latest
- needs: [statics, statics-tests]
+ needs:
+ - tests-sqlite
+ - tests-postgres
+ - tests-mysql
+ - requirements
+ - prepare-backport-packages
+ - build-prod-image
+ - docs
Review comment:
Nice one!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #8393: Bring
back CI optimisations
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #8393: Bring back CI optimisations
URL: https://github.com/apache/airflow/pull/8393#discussion_r410724983
##########
File path: .github/workflows/ci.yml
##########
@@ -17,272 +17,275 @@
#
---
-name: Airflow
+name: CI Build
on:
+ schedule:
+ - cron: '10 2 * * *'
push:
- branches: ['*']
+ branches: ['master', 'v1-10-test', 'v1-10-stable']
pull_request:
- branches: [master]
+ branches: ['master', 'v1-10-test', 'v1-10-stable']
env:
- BUILD_ID: ${{github.sha }}
MOUNT_LOCAL_SOURCES: "false"
- MOUNT_HOST_AIRFLOW_VOLUME: "true"
FORCE_ANSWER_TO_QUESTIONS: "yes"
SKIP_CHECK_REMOTE_IMAGE: "true"
+ SKIP_CI_IMAGE_CHECK: "true"
DB_RESET: "true"
VERBOSE: "true"
- CI: "true"
- # Should be a target branch
- TRAVIS_BRANCH: "master"
- TRAVIS: "true"
+ UPGRADE_TO_LATEST_REQUIREMENTS: "false"
+ ENABLED_INTEGRATIONS: "cassandra kerberos mongo openldap presto rabbitmq redis"
+ PYTHON_MAJOR_MINOR_VERSION: 3.6
+ # TODO: Enable cache registry once a master build succeeds uploading the images to Github registry cache
+ DISABLE_CACHE_REGISTRY: "true"
+ CACHE_REGISTRY: "docker.pkg.github.com"
+ CACHE_IMAGE_PREFIX: ${{ github.repository }}
+ CACHE_REGISTRY_USERNAME: ${{ github.actor }}
+ CACHE_REGISTRY_PASSWORD: ${{ secrets.GITHUB_TOKEN }}
+
jobs:
- statics:
- name: Static checks
+
+ static-checks-1:
+ name: "Checks: pylint, mypy"
runs-on: ubuntu-latest
env:
- PYTHON_VERSION: 3.6
- AIRFLOW_MOUNT_SOURCE_DIR_FOR_STATIC_CHECKS: "true"
- TRAVIS_JOB_NAME: "Static"
+ MOUNT_SOURCE_DIR_FOR_STATIC_CHECKS: "true"
+ CI_JOB_TYPE: "Pylint checks for tests"
steps:
- uses: actions/checkout@master
- - name: free disk space
- run: |
- sudo swapoff -a
- sudo rm -f /swapfile
- sudo apt clean
- docker rmi $(docker image ls -aq)
- df -h
- uses: actions/setup-python@v1
with:
- python-version: ${{ env.PYTHON_VERSION }}
- - name: "Before install"
- run: ./scripts/ci/ci_before_install.sh
- - name: "Static checks"
- if: success()
+ python-version: '3.x'
+ - name: Cache pre-commit env
+ uses: actions/cache@v1
env:
- PYTHON_VERSION: 3.6
- run: ./scripts/ci/ci_run_all_static_checks.sh
+ cache-name: cache-pre-commit
+ with:
+ path: ~/.cache/pre-commit
+ key: ${{ env.cache-name }}-${{ github.job }}-${{ hashFiles('.pre-commit-config.yaml') }}
+ - name: "Free space"
+ run: ./scripts/ci/ci_free_space_on_ci.sh
+ - name: "Build CI image"
+ run: ./scripts/ci/ci_prepare_image_on_ci.sh
+ - name: "Pylint checks for tests"
+ run: |
+ python -m pip install pre-commit \
+ --constraint requirements/requirements-python${PYTHON_MAJOR_MINOR_VERSION}.txt
+ ./scripts/ci/ci_run_static_checks.sh pylint-tests mypy
- statics-tests:
- name: Pylint for tests
+ static-checks-2:
+ name: "Checks: All other"
runs-on: ubuntu-latest
env:
- PYTHON_VERSION: 3.6
- AIRFLOW_MOUNT_SOURCE_DIR_FOR_STATIC_CHECKS: "true"
- TRAVIS_JOB_NAME: "Static"
+ SKIP: pylint-tests,mypy
+ MOUNT_SOURCE_DIR_FOR_STATIC_CHECKS: "true"
+ CI_JOB_TYPE: "Static checks"
steps:
- uses: actions/checkout@master
- uses: actions/setup-python@v1
with:
- python-version: ${{ env.PYTHON_VERSION }}
- - name: "Before install"
- run: ./scripts/ci/ci_before_install.sh
- - name: "Static checks tests"
- if: success()
+ python-version: '3.x'
+ - name: Cache pre-commit env
+ uses: actions/cache@v1
env:
- PYTHON_VERSION: 3.6
- run: ./scripts/ci/ci_run_static_checks_pylint_tests.sh
+ cache-name: cache-pre-commit
+ with:
+ path: ~/.cache/pre-commit
+ key: ${{ env.cache-name }}-${{ github.job }}-${{ hashFiles('.pre-commit-config.yaml') }}
+ - name: "Free space"
+ run: ./scripts/ci/ci_free_space_on_ci.sh
+ - name: "Build CI image"
+ run: ./scripts/ci/ci_prepare_image_on_ci.sh
+ - name: "Static checks"
+ run: |
+ python -m pip install pre-commit \
+ --constraint requirements/requirements-python${PYTHON_MAJOR_MINOR_VERSION}.txt
+ ./scripts/ci/ci_run_static_checks.sh
docs:
name: Build docs
runs-on: ubuntu-latest
env:
- TRAVIS_JOB_NAME: "Build documentation"
- PYTHON_VERSION: 3.6
+ CI_JOB_TYPE: "Documentation"
steps:
- uses: actions/checkout@master
- - name: "Build documentation"
+ - name: "Build CI image ${{ matrix.python-version }}"
+ run: ./scripts/ci/ci_prepare_image_on_ci.sh
+ - name: "Build docs"
run: ./scripts/ci/ci_docs.sh
- tests-p36-postgres-integrations:
- name: "Tests [Postgres9.6][Py3.6][integrations]"
+ build-prod-image:
+ name: "Build prod image"
runs-on: ubuntu-latest
- needs: [statics, statics-tests]
Review comment:
It's still WIP. Unfortunately the tests on GA are super-unstable and I am trying to find a way to stabilise them.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] ashb commented on a change in pull request #8393: Bring
back CI optimisations
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #8393: Bring back CI optimisations
URL: https://github.com/apache/airflow/pull/8393#discussion_r409462854
##########
File path: .github/workflows/ci.yml
##########
@@ -19,58 +19,58 @@
---
name: Airflow
Review comment:
I having both "Airflow" and "Build" is redundant here, and one or the other would be better
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services