You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/05/21 06:35:48 UTC

[GitHub] [airflow] potiuk opened a new pull request #8943: Python base images are stored in cache

potiuk opened a new pull request #8943:
URL: https://github.com/apache/airflow/pull/8943


   All PRs will used cached "latest good" version of the python
   base images from our GitHub registry. The python versions in
   the Github Registry will only get updated after a master
   build (which pulls latest Python image from DockerHub) builds
   and passes test correctly.
   
   This is to avoid problems that we had recently with Python
   patchlevel releases breaking our Docker builds.
   
   ---
   Make sure to mark the boxes below before creating PR: [x]
   
   - [x] Description above provides context of the change
   - [x] Unit tests coverage for changes (not needed for documentation changes)
   - [x] Target Github ISSUE in description if exists
   - [x] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)"
   - [x] Relevant documentation is updated including usage instructions.
   - [x] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   ---
   In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information.
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on a change in pull request #8943: Python base images are stored in cache

Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #8943:
URL: https://github.com/apache/airflow/pull/8943#discussion_r428558788



##########
File path: scripts/ci/_utils.sh
##########
@@ -227,6 +227,9 @@ function initialize_common_environment {
 
     # Artifact name suffix for SVN packaging
     export VERSION_SUFFIX_FOR_SVN=""
+
+    # Whether base python images should be pulled from cache
+    export PULL_PYTHON_BASE_IMAGES_FROM_CACHE="true"

Review comment:
       Indeed good catch. It was wrong place as well I moved it higher next to push images where it belongs.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk merged pull request #8943: Python base images are stored in cache

Posted by GitBox <gi...@apache.org>.
potiuk merged pull request #8943:
URL: https://github.com/apache/airflow/pull/8943


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] ashb commented on a change in pull request #8943: Python base images are stored in cache

Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #8943:
URL: https://github.com/apache/airflow/pull/8943#discussion_r428521369



##########
File path: scripts/ci/_utils.sh
##########
@@ -227,6 +227,9 @@ function initialize_common_environment {
 
     # Artifact name suffix for SVN packaging
     export VERSION_SUFFIX_FOR_SVN=""
+
+    # Whether base python images should be pulled from cache
+    export PULL_PYTHON_BASE_IMAGES_FROM_CACHE="true"

Review comment:
       Won't this overwrite the value set by the ci.yml file?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] ashb commented on a change in pull request #8943: Python base images are stored in cache

Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #8943:
URL: https://github.com/apache/airflow/pull/8943#discussion_r428522890



##########
File path: scripts/ci/_utils.sh
##########
@@ -227,6 +227,9 @@ function initialize_common_environment {
 
     # Artifact name suffix for SVN packaging
     export VERSION_SUFFIX_FOR_SVN=""
+
+    # Whether base python images should be pulled from cache
+    export PULL_PYTHON_BASE_IMAGES_FROM_CACHE="true"

Review comment:
       ```suggestion
       export PULL_PYTHON_BASE_IMAGES_FROM_CACHE="${PULL_PYTHON_BASE_IMAGES_FROM_CACHE:="true"}"
   ```
   
   no?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on pull request #8943: Python base images are stored in cache

Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #8943:
URL: https://github.com/apache/airflow/pull/8943#issuecomment-632011170


   Seems we are back to the usual speed :)
   
   And caching the python image works as well:
   
   ```
   2020-05-21T10:02:14.4034205Z Pulling the image docker.pkg.github.com/apache/airflow/python:3.6-slim-buster
   2020-05-21T10:02:14.4034585Z 
   2020-05-21T10:02:14.4046176Z docker pull docker.pkg.github.com/apache/airflow/python:3.6-slim-buster
   2020-05-21T10:02:14.6005022Z 3.6-slim-buster: Pulling from apache/airflow/python
   2020-05-21T10:02:14.6022126Z afb6ec6fdc1c: Pulling fs layer
   2020-05-21T10:02:14.6022350Z 2cf47e503ea9: Pulling fs layer
   2020-05-21T10:02:14.6022519Z 9c34fc54941b: Pulling fs layer
   2020-05-21T10:02:14.6022698Z 10f0f49e307b: Pulling fs layer
   2020-05-21T10:02:14.6022873Z 37894e158256: Pulling fs layer
   2020-05-21T10:02:14.6023054Z 10f0f49e307b: Waiting
   2020-05-21T10:02:14.6023208Z 37894e158256: Waiting
   2020-05-21T10:02:15.0856863Z 2cf47e503ea9: Verifying Checksum
   2020-05-21T10:02:15.0857004Z 2cf47e503ea9: Download complete
   2020-05-21T10:02:15.3039416Z 10f0f49e307b: Verifying Checksum
   2020-05-21T10:02:15.3046121Z 10f0f49e307b: Download complete
   2020-05-21T10:02:15.7686670Z 37894e158256: Verifying Checksum
   2020-05-21T10:02:15.7686890Z 37894e158256: Download complete
   2020-05-21T10:02:15.8947685Z 9c34fc54941b: Verifying Checksum
   2020-05-21T10:02:15.8947871Z 9c34fc54941b: Download complete
   2020-05-21T10:02:16.1443974Z afb6ec6fdc1c: Verifying Checksum
   2020-05-21T10:02:16.1445262Z afb6ec6fdc1c: Download complete
   2020-05-21T10:02:17.1856984Z afb6ec6fdc1c: Pull complete
   2020-05-21T10:02:17.3173382Z 2cf47e503ea9: Pull complete
   2020-05-21T10:02:18.0080849Z 9c34fc54941b: Pull complete
   2020-05-21T10:02:18.0520775Z 10f0f49e307b: Pull complete
   2020-05-21T10:02:18.2215212Z 37894e158256: Pull complete
   2020-05-21T10:02:18.2241378Z Digest: sha256:cdb9505dab15e22df319d862de7b5315150f60f77cd855f4c57e9e422f49a384
   2020-05-21T10:02:18.2260884Z Status: Downloaded newer image for docker.pkg.github.com/apache/airflow/python:3.6-slim-buster
   2020-05-21T10:02:18.2265431Z docker.pkg.github.com/apache/airflow/python:3.6-slim-buster
   2020-05-21T10:02:18.2284012Z 
   2020-05-21T10:02:18.2286890Z docker tag docker.pkg.github.com/apache/airflow/python:3.6-slim-buster python:3.6-slim-buster
   ```
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org