You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/01/08 10:34:56 UTC
[GitHub] [airflow] potiuk opened a new pull request #13562: Optimize wait for prod images to be run after ci images
potiuk opened a new pull request #13562:
URL: https://github.com/apache/airflow/pull/13562
This change slightly optimizes waiting for prod images. Waiting
for images necessarily takes one slot from jobs queue, even if
those jobs do nothing at all. This is unfortunate, but seems
there is no easy way to do it differently because GitHub
Actions does not support yet cross-workflow dependencies.
This PR optimizes waiting for images in the way that PROD images
and CI images waiting happen sequentially rather than in parallell.
PROD images are, as of #13557 prepared sequentially which
means that there is no point to wait for PROD images in parallel
to CI images.
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk merged pull request #13562: Optimize wait for prod images to be run after ci images
Posted by GitBox <gi...@apache.org>.
potiuk merged pull request #13562:
URL: https://github.com/apache/airflow/pull/13562
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on a change in pull request #13562: Optimize wait for prod images to be run after ci images
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #13562:
URL: https://github.com/apache/airflow/pull/13562#discussion_r553878463
##########
File path: .github/workflows/ci.yml
##########
@@ -834,7 +834,7 @@ jobs:
timeout-minutes: 120
name: "Wait for PROD images"
runs-on: ubuntu-20.04
- needs: [build-info]
+ needs: [build-info, ci-images]
Review comment:
Add a comment saying why we are doing this please, becuase we _might_ be able to reverse this once we have self-hosted runners set up
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on a change in pull request #13562: Optimize wait for prod images to be run after ci images
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #13562:
URL: https://github.com/apache/airflow/pull/13562#discussion_r553878463
##########
File path: .github/workflows/ci.yml
##########
@@ -834,7 +834,7 @@ jobs:
timeout-minutes: 120
name: "Wait for PROD images"
runs-on: ubuntu-20.04
- needs: [build-info]
+ needs: [build-info, ci-images]
Review comment:
Add a comment saying why we are doing this please, becuase we _might_ be able to reverse this once we have self-hosted runners set up
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #13562: Optimize wait for prod images to be run after ci images
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #13562:
URL: https://github.com/apache/airflow/pull/13562#issuecomment-756737291
I tihnk this one is actually even MUST now. With the delays in queue we have, the prod wait can timeout before the prod image is built
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #13562: Optimize wait for prod images to be run after ci images
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #13562:
URL: https://github.com/apache/airflow/pull/13562#issuecomment-756737291
I tihnk this one is actually even MUST now. With the delays in queue we have, the prod wait can timeout before the prod image is built
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #13562: Optimize wait for prod images to be run after ci images
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #13562:
URL: https://github.com/apache/airflow/pull/13562#discussion_r553895149
##########
File path: .github/workflows/ci.yml
##########
@@ -834,7 +834,7 @@ jobs:
timeout-minutes: 120
name: "Wait for PROD images"
runs-on: ubuntu-20.04
- needs: [build-info]
+ needs: [build-info, ci-images]
Review comment:
I could add it, but I think this is a 'target' optimization. Even if we switch to self-hosted runners, we want to keep it in.
Currently our dependency model looks like:
1) Workflow run builds CI image -> once done CI Iimage waiting finishes and CI tests run
2) in Worklflow Run PROD images are build using the CI images from step 1 (this is needed to build provider packages because prod images are build from packages now).
Step 2) depends on 1), so there will never be the case that PROD images will be available before CI images.. So waiting for PROD images in parallel to CI images simply makes no sense.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk merged pull request #13562: Optimize wait for prod images to be run after ci images
Posted by GitBox <gi...@apache.org>.
potiuk merged pull request #13562:
URL: https://github.com/apache/airflow/pull/13562
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #13562: Optimize wait for prod images to be run after ci images
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #13562:
URL: https://github.com/apache/airflow/pull/13562#discussion_r553895149
##########
File path: .github/workflows/ci.yml
##########
@@ -834,7 +834,7 @@ jobs:
timeout-minutes: 120
name: "Wait for PROD images"
runs-on: ubuntu-20.04
- needs: [build-info]
+ needs: [build-info, ci-images]
Review comment:
I could add it, but I think this is a 'target' optimization. Even if we switch to self-hosted runners, we want to keep it in.
Currently our dependency model looks like:
1) Workflow run builds CI image -> once done CI Iimage waiting finishes and CI tests run
2) in Worklflow Run PROD images are build using the CI images from step 1 (this is needed to build provider packages because prod images are build from packages now).
Step 2) depends on 1), so there will never be the case that PROD images will be available before CI images.. So waiting for PROD images in parallel to CI images simply makes no sense.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org