You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/08/28 17:46:56 UTC
[GitHub] [airflow] potiuk opened a new pull request #17883: Avoids race condition on parallel builds
potiuk opened a new pull request #17883:
URL: https://github.com/apache/airflow/pull/17883
We need to set the "experimental" flag in CI in order to use
`docker manifest` command to check for presence of the images
in ghcr.io. In order to use them we need to enable experimental
features via ~/.docker/config.json.
Sometimes, very rarely we had the case that the config file
got broken and the problem turned out to be that we tried
to do this experimental replacement in parallel by several
running "wait image" commands (:facepalm: here for myself)
that were apparenlty overriding the same config.json
at the same time in non-atomic way, which (very rarely)
led to corrupted file.
This change moves the experimental replacement to top-level
scripts which are never run in parallel.
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/main/UPDATING.md).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #17883: Avoid race condition when setting experimental docker flag
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #17883:
URL: https://github.com/apache/airflow/pull/17883#issuecomment-909169603
It were bad - downgraded constraints (the downgrade should be fixed by #17939 )
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr closed pull request #17883: Avoid race condition when setting experimental docker flag
Posted by GitBox <gi...@apache.org>.
uranusjr closed pull request #17883:
URL: https://github.com/apache/airflow/pull/17883
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mrbaguvix commented on a change in pull request #17883: Avoid race condition when setting experimental docker flag
Posted by GitBox <gi...@apache.org>.
mrbaguvix commented on a change in pull request #17883:
URL: https://github.com/apache/airflow/pull/17883#discussion_r702332249
##########
File path: scripts/ci/images/ci_prepare_prod_image_on_ci.sh
##########
@@ -29,10 +29,13 @@ export VERBOSE="true"
# Builds or waits for the PROD image in the CI environment
function build_prod_images_on_ci() {
+ build_images::configure_docker_registry
build_images::prepare_prod_build
if [[ ${GITHUB_REGISTRY_WAIT_FOR_IMAGE} == "true" ]]; then
- build_images::wait_for_image_tag "${AIRFLOW_PROD_IMAGE}" ":${GITHUB_REGISTRY_PULL_IMAGE_TAG}"
+ local image_name_with_tag="${AIRFLOW_PROD_IMAGE}:${GITHUB_REGISTRY_PULL_IMAGE_TAG}"
+ push_pull_remove_images::wait_for_image "${image_name_with_tag}"
+ push_pull_remove_images::pull_image "${image_name_with_tag}"
Review comment:
Good stuff!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr closed pull request #17883: Avoid race condition when setting experimental docker flag
Posted by GitBox <gi...@apache.org>.
uranusjr closed pull request #17883:
URL: https://github.com/apache/airflow/pull/17883
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed pull request #17883: Avoid race condition when setting experimental docker flag
Posted by GitBox <gi...@apache.org>.
potiuk closed pull request #17883:
URL: https://github.com/apache/airflow/pull/17883
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #17883: Avoid race condition when setting experimental docker flag
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #17883:
URL: https://github.com/apache/airflow/pull/17883#issuecomment-909169603
It were bad - downgraded constraints (the downgrade should be fixed by #17939 )
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #17883: Stop using docker manifest to check for image presence
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #17883:
URL: https://github.com/apache/airflow/pull/17883#issuecomment-923007468
All Green. This will remove another source of random CI failures.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed pull request #17883: Avoid race condition when setting experimental docker flag
Posted by GitBox <gi...@apache.org>.
potiuk closed pull request #17883:
URL: https://github.com/apache/airflow/pull/17883
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr closed pull request #17883: Avoid race condition when setting experimental docker flag
Posted by GitBox <gi...@apache.org>.
uranusjr closed pull request #17883:
URL: https://github.com/apache/airflow/pull/17883
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #17883: Avoids race condition on parallel builds
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #17883:
URL: https://github.com/apache/airflow/pull/17883#issuecomment-907662897
One more fix to accidentally failing tests . Such a :facepalm: on me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #17883: Avoid race condition when setting experimental docker flag
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #17883:
URL: https://github.com/apache/airflow/pull/17883#issuecomment-909169603
It were bad - downgraded constraints (the downgrade should be fixed by #17939 )
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk merged pull request #17883: Stop using docker manifest to check for image presence
Posted by GitBox <gi...@apache.org>.
potiuk merged pull request #17883:
URL: https://github.com/apache/airflow/pull/17883
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed pull request #17883: Stop using docker manifest to check for image presence
Posted by GitBox <gi...@apache.org>.
potiuk closed pull request #17883:
URL: https://github.com/apache/airflow/pull/17883
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #17883: Stop using docker manifest to check for image presence
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #17883:
URL: https://github.com/apache/airflow/pull/17883#issuecomment-922345567
Rebuilding to see if the failures are transient.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed pull request #17883: Stop using docker manifest to check for image presence
Posted by GitBox <gi...@apache.org>.
potiuk closed pull request #17883:
URL: https://github.com/apache/airflow/pull/17883
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #17883: Stop using docker manifest to check for image presence
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #17883:
URL: https://github.com/apache/airflow/pull/17883#issuecomment-922262917
I removed completely the "experimental" flag @uranusjr - this only caused troubles and I actually need to pull the image anyway (for verification) once it is ready - so rather than checking if image is there via `manifest` I am pulling it straight away now - this will be cleaner, less experimental and simpler (and it will work better I hope - sometimes the config.json got irreparably broken.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #17883: Stop using docker manifest to check for image presence
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #17883:
URL: https://github.com/apache/airflow/pull/17883#issuecomment-922382916
Just MSSQL tests wich has to be fixed anywyay. But it's good
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed pull request #17883: Avoid race condition when setting experimental docker flag
Posted by GitBox <gi...@apache.org>.
potiuk closed pull request #17883:
URL: https://github.com/apache/airflow/pull/17883
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org