You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/04/16 11:34:04 UTC

[GitHub] [airflow] potiuk opened a new pull request #8400: Less frequent dockerhub builds

potiuk opened a new pull request #8400: Less frequent dockerhub builds
URL: https://github.com/apache/airflow/pull/8400
 
 
   ---
   Make sure to mark the boxes below before creating PR: [x]
   
   - [x] Description above provides context of the change
   - [x] Unit tests coverage for changes (not needed for documentation changes)
   - [x] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)"
   - [x] Relevant documentation is updated including usage instructions.
   - [x] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   ---
   In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information.
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on a change in pull request #8400: Less frequent dockerhub builds

Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #8400: Less frequent dockerhub builds
URL: https://github.com/apache/airflow/pull/8400#discussion_r409659454
 
 

 ##########
 File path: scripts/ci/ci_build_dockerhub.sh
 ##########
 @@ -67,6 +67,25 @@ echo
 # shellcheck source=scripts/ci/_script_init.sh
 . "$( dirname "${BASH_SOURCE[0]}" )/_script_init.sh"
 
+DOCKER_IMAGE_SPEC="${DOCKERHUB_USER}/${DOCKERHUB_REPO}:${DOCKER_TAG}"
+
+IMAGE_CREATED_SINCE=$(docker image ls "${DOCKER_IMAGE_SPEC}" --format '{{ .CreatedSince }}')
+
+if [[ ${IMAGE_CREATED_SINCE} =~ .*days.* || ${IMAGE_CREATED_SINCE} =~ .*weeks.* ||
+      ${IMAGE_CREATED_SINCE} =~ .*months.* || ${IMAGE_CREATED_SINCE} =~ .*years.* ||
+      ${IMAGE_CREATED_SINCE} =~ [0-9][0-9]\ hours ]]; then
 
 Review comment:
   Then 48 hours and above that there are days.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on issue #8400: Less frequent dockerhub builds

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #8400: Less frequent dockerhub builds
URL: https://github.com/apache/airflow/pull/8400#issuecomment-614794174
 
 
   See yourself @dimberman -> maybe you can find how to do it. I believe you have access to https://hub.docker.com/repository/docker/apache/airflow :)

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #8400: Less frequent dockerhub builds

Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #8400: Less frequent dockerhub builds
URL: https://github.com/apache/airflow/pull/8400#issuecomment-614628834
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/8400?src=pr&el=h1) Report
   > Merging [#8400](https://codecov.io/gh/apache/airflow/pull/8400?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/6d5710785204120b57dc61e351e12ae0b9f142b0&el=desc) will **not change** coverage.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/airflow/pull/8400/graphs/tree.svg?width=650&height=150&src=pr&token=WdLKlKHOAU)](https://codecov.io/gh/apache/airflow/pull/8400?src=pr&el=tree)
   
   ```diff
   @@          Coverage Diff           @@
   ##           master   #8400   +/-   ##
   ======================================
     Coverage    6.24%   6.24%           
   ======================================
     Files         941     941           
     Lines       45573   45573           
   ======================================
     Hits         2846    2846           
     Misses      42727   42727           
   ```
   
   
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/8400?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/8400?src=pr&el=footer). Last update [6d57107...5a9ed06](https://codecov.io/gh/apache/airflow/pull/8400?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on a change in pull request #8400: Less frequent dockerhub builds

Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #8400: Less frequent dockerhub builds
URL: https://github.com/apache/airflow/pull/8400#discussion_r409507252
 
 

 ##########
 File path: scripts/ci/ci_build_dockerhub.sh
 ##########
 @@ -51,12 +51,12 @@ else
    echo "DOCKER_TAG=${DOCKER_TAG}"
 fi
 
-[[ ${DOCKER_TAG:=} =~ ${DEFAULT_BRANCH}-python([0-9.]*)(.*) ]] && export PYTHON_MAJOR_MINOR_VERSION=${BASH_REMATCH[1]}
+[[ ${DOCKER_TAG:=} =~ .*-python([0-9.]*)(.*) ]] && export PYTHON_MAJOR_MINOR_VERSION=${BASH_REMATCH[1]}
 
 Review comment:
   This way in the future we can have other branches as well and we will also be able to use the scripts for automated "tag" release building as well.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io commented on issue #8400: Less frequent dockerhub builds

Posted by GitBox <gi...@apache.org>.
codecov-io commented on issue #8400: Less frequent dockerhub builds
URL: https://github.com/apache/airflow/pull/8400#issuecomment-614628834
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/8400?src=pr&el=h1) Report
   > Merging [#8400](https://codecov.io/gh/apache/airflow/pull/8400?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/6d5710785204120b57dc61e351e12ae0b9f142b0&el=desc) will **not change** coverage.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/airflow/pull/8400/graphs/tree.svg?width=650&height=150&src=pr&token=WdLKlKHOAU)](https://codecov.io/gh/apache/airflow/pull/8400?src=pr&el=tree)
   
   ```diff
   @@          Coverage Diff           @@
   ##           master   #8400   +/-   ##
   ======================================
     Coverage    6.24%   6.24%           
   ======================================
     Files         941     941           
     Lines       45573   45573           
   ======================================
     Hits         2846    2846           
     Misses      42727   42727           
   ```
   
   
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/8400?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/8400?src=pr&el=footer). Last update [6d57107...5a9ed06](https://codecov.io/gh/apache/airflow/pull/8400?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #8400: Less frequent dockerhub builds

Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #8400: Less frequent dockerhub builds
URL: https://github.com/apache/airflow/pull/8400#issuecomment-614628834
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/8400?src=pr&el=h1) Report
   > Merging [#8400](https://codecov.io/gh/apache/airflow/pull/8400?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/6d5710785204120b57dc61e351e12ae0b9f142b0&el=desc) will **decrease** coverage by `0.00%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/airflow/pull/8400/graphs/tree.svg?width=650&height=150&src=pr&token=WdLKlKHOAU)](https://codecov.io/gh/apache/airflow/pull/8400?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff            @@
   ##           master   #8400      +/-   ##
   =========================================
   - Coverage    6.24%   6.24%   -0.01%     
   =========================================
     Files         941     941              
     Lines       45573   45606      +33     
   =========================================
     Hits         2846    2846              
   - Misses      42727   42760      +33     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/8400?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [airflow/utils/cli.py](https://codecov.io/gh/apache/airflow/pull/8400/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9jbGkucHk=) | `0.00% <0.00%> (ø)` | |
   | [airflow/cli/cli\_parser.py](https://codecov.io/gh/apache/airflow/pull/8400/diff?src=pr&el=tree#diff-YWlyZmxvdy9jbGkvY2xpX3BhcnNlci5weQ==) | `0.00% <0.00%> (ø)` | |
   | [airflow/cli/commands/config\_command.py](https://codecov.io/gh/apache/airflow/pull/8400/diff?src=pr&el=tree#diff-YWlyZmxvdy9jbGkvY29tbWFuZHMvY29uZmlnX2NvbW1hbmQucHk=) | `0.00% <0.00%> (ø)` | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/8400?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/8400?src=pr&el=footer). Last update [6d57107...2215ff8](https://codecov.io/gh/apache/airflow/pull/8400?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #8400: Less frequent dockerhub builds

Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #8400: Less frequent dockerhub builds
URL: https://github.com/apache/airflow/pull/8400#issuecomment-614628834
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/8400?src=pr&el=h1) Report
   > Merging [#8400](https://codecov.io/gh/apache/airflow/pull/8400?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/6d5710785204120b57dc61e351e12ae0b9f142b0&el=desc) will **not change** coverage.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/airflow/pull/8400/graphs/tree.svg?width=650&height=150&src=pr&token=WdLKlKHOAU)](https://codecov.io/gh/apache/airflow/pull/8400?src=pr&el=tree)
   
   ```diff
   @@          Coverage Diff           @@
   ##           master   #8400   +/-   ##
   ======================================
     Coverage    6.24%   6.24%           
   ======================================
     Files         941     941           
     Lines       45573   45573           
   ======================================
     Hits         2846    2846           
     Misses      42727   42727           
   ```
   
   
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/8400?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/8400?src=pr&el=footer). Last update [6d57107...5a9ed06](https://codecov.io/gh/apache/airflow/pull/8400?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on issue #8400: Less frequent dockerhub builds

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #8400: Less frequent dockerhub builds
URL: https://github.com/apache/airflow/pull/8400#issuecomment-614783967
 
 
   @dimberman It always lags quite a bit as DockerHub is very slow just "check if the image needs to reb uilt" takes anything between 10 and 20 minutes.  Currently we build so many images (including the PROD one)  that when we have a few merges, the build queue grows faster than it is cleared and pretty much every day we end up in a situation that the build waits a long time (700mins = 12 hours is pretty normal if we have a series of master merges) + if we have a v1-10-stable push, that's even longer. 
   
   From CI point of view it's OK. The whole system is built in the way that if the image is old, it - at most - run CI a bit slower. Assuming that master/v1-10-stable are constantly moving, I do not think this is a big issue to add such "10 hours" skip. 
   
   I also opened INFRA ticket to see if we can push the image from Github Actions (see my comment below. But I doubt it will be OK from the security POV:
   > Currently we have quite a lot of builds happening in DockerHub (including production image). It takes a long time to build the images and they are queuing now. I have created an infra ticket to ask if we can push images from Github Actions (https://issues.apache.org/jira/projects/INFRA/issues/INFRA-20124) but I have no high hopes for it - it requires storing secrets in GA. That's why I propose that build scripts will check if the image is <10 hours old and only actually build it if it is.
   
   Alternatively we might want to rebuild the images regularly - every night for example, but this is a bit problematic (I explained it to @turbaszek above) - we could generate a trigger URL and use it in an external service (there is no cron build support in DockerHub. But I am not sure it's safe.
   
   I realise it is not perfect, do you have any ideas how we can do it differently ?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on issue #8400: Less frequent dockerhub builds

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #8400: Less frequent dockerhub builds
URL: https://github.com/apache/airflow/pull/8400#issuecomment-614797872
 
 
   And if you ask - no - we cannot define our own secrets in Github Actions to store such URL. We have no permissions to do it. :( ... This is what is the INFRA ticket about as well...

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] turbaszek commented on a change in pull request #8400: Less frequent dockerhub builds

Posted by GitBox <gi...@apache.org>.
turbaszek commented on a change in pull request #8400: Less frequent dockerhub builds
URL: https://github.com/apache/airflow/pull/8400#discussion_r409575775
 
 

 ##########
 File path: scripts/ci/ci_build_dockerhub.sh
 ##########
 @@ -67,6 +67,25 @@ echo
 # shellcheck source=scripts/ci/_script_init.sh
 . "$( dirname "${BASH_SOURCE[0]}" )/_script_init.sh"
 
+DOCKER_IMAGE_SPEC="${DOCKERHUB_USER}/${DOCKERHUB_REPO}:${DOCKER_TAG}"
+
+IMAGE_CREATED_SINCE=$(docker image ls "${DOCKER_IMAGE_SPEC}" --format '{{ .CreatedSince }}')
+
+if [[ ${IMAGE_CREATED_SINCE} =~ .*days.* || ${IMAGE_CREATED_SINCE} =~ .*weeks.* ||
+      ${IMAGE_CREATED_SINCE} =~ .*months.* || ${IMAGE_CREATED_SINCE} =~ .*years.* ||
+      ${IMAGE_CREATED_SINCE} =~ [0-9][0-9]\ hours ]]; then
 
 Review comment:
   If we decide to not build image on every change then maybe we can run it every 24h? 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] kaxil commented on a change in pull request #8400: Less frequent dockerhub builds

Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #8400: Less frequent dockerhub builds
URL: https://github.com/apache/airflow/pull/8400#discussion_r409667116
 
 

 ##########
 File path: scripts/ci/ci_build_dockerhub.sh
 ##########
 @@ -67,6 +67,25 @@ echo
 # shellcheck source=scripts/ci/_script_init.sh
 . "$( dirname "${BASH_SOURCE[0]}" )/_script_init.sh"
 
+DOCKER_IMAGE_SPEC="${DOCKERHUB_USER}/${DOCKERHUB_REPO}:${DOCKER_TAG}"
+
+IMAGE_CREATED_SINCE=$(docker image ls "${DOCKER_IMAGE_SPEC}" --format '{{ .CreatedSince }}')
+
+if [[ ${IMAGE_CREATED_SINCE} =~ .*days.* || ${IMAGE_CREATED_SINCE} =~ .*weeks.* ||
+      ${IMAGE_CREATED_SINCE} =~ .*months.* || ${IMAGE_CREATED_SINCE} =~ .*years.* ||
+      ${IMAGE_CREATED_SINCE} =~ [0-9][0-9]\ hours ]]; then
 
 Review comment:
   ```bash
   ❯ export tta="9 hours"
   ❯ if [[ "$tta" =~ [0-9][0-9]\ hours ]]; then
   then> echo "Rebuilding the image."
   then> else
   else> echo "Skipping rebuilding the image."
   else> fi
   Skipping rebuilding the image.
   ```

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on a change in pull request #8400: Less frequent dockerhub builds

Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #8400: Less frequent dockerhub builds
URL: https://github.com/apache/airflow/pull/8400#discussion_r409506332
 
 

 ##########
 File path: scripts/ci/ci_build_dockerhub.sh
 ##########
 @@ -51,12 +51,12 @@ else
    echo "DOCKER_TAG=${DOCKER_TAG}"
 fi
 
-[[ ${DOCKER_TAG:=} =~ ${DEFAULT_BRANCH}-python([0-9.]*)(.*) ]] && export PYTHON_MAJOR_MINOR_VERSION=${BASH_REMATCH[1]}
+[[ ${DOCKER_TAG:=} =~ .*-python([0-9.]*)(.*) ]] && export PYTHON_MAJOR_MINOR_VERSION=${BASH_REMATCH[1]}
 
 Review comment:
   It was not needed. It's remnant from the old script and ${DEFAULT_BRANCH} was actually empty :)

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] dimberman commented on issue #8400: Less frequent dockerhub builds

Posted by GitBox <gi...@apache.org>.
dimberman commented on issue #8400: Less frequent dockerhub builds
URL: https://github.com/apache/airflow/pull/8400#issuecomment-614791092
 
 
   @potiuk perhaps we can do hourly/bi-hourly jobs? So instead of build-on-merge we just build at the top of the hour? (we could manually trigger if it's an important fix)

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on a change in pull request #8400: Less frequent dockerhub builds

Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #8400: Less frequent dockerhub builds
URL: https://github.com/apache/airflow/pull/8400#discussion_r409508684
 
 

 ##########
 File path: scripts/ci/ci_build_dockerhub.sh
 ##########
 @@ -67,6 +67,26 @@ echo
 # shellcheck source=scripts/ci/_script_init.sh
 . "$( dirname "${BASH_SOURCE[0]}" )/_script_init.sh"
 
+DOCKER_IMAGE_SPEC="${DOCKERHUB_USER}/${DOCKERHUB_REPO}:${DOCKER_TAG}"
+
+#docker pull "${DOCKER_IMAGE_SPEC}"
 
 Review comment:
   Actually it is not needed. I will enable caching in DockerHub and I will let DockerHub handle the pull for me so that I won't have to do it in the script. I will remove it.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on a change in pull request #8400: Less frequent dockerhub builds

Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #8400: Less frequent dockerhub builds
URL: https://github.com/apache/airflow/pull/8400#discussion_r409676501
 
 

 ##########
 File path: scripts/ci/ci_build_dockerhub.sh
 ##########
 @@ -67,6 +67,25 @@ echo
 # shellcheck source=scripts/ci/_script_init.sh
 . "$( dirname "${BASH_SOURCE[0]}" )/_script_init.sh"
 
+DOCKER_IMAGE_SPEC="${DOCKERHUB_USER}/${DOCKERHUB_REPO}:${DOCKER_TAG}"
+
+IMAGE_CREATED_SINCE=$(docker image ls "${DOCKER_IMAGE_SPEC}" --format '{{ .CreatedSince }}')
+
+if [[ ${IMAGE_CREATED_SINCE} =~ .*days.* || ${IMAGE_CREATED_SINCE} =~ .*weeks.* ||
+      ${IMAGE_CREATED_SINCE} =~ .*months.* || ${IMAGE_CREATED_SINCE} =~ .*years.* ||
+      ${IMAGE_CREATED_SINCE} =~ [0-9][0-9]\ hours ]]; then
 
 Review comment:
   So basically I want to rebuild for everything = or above 10 hours.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] kaxil commented on a change in pull request #8400: Less frequent dockerhub builds

Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #8400: Less frequent dockerhub builds
URL: https://github.com/apache/airflow/pull/8400#discussion_r409626652
 
 

 ##########
 File path: scripts/ci/ci_build_dockerhub.sh
 ##########
 @@ -67,6 +67,25 @@ echo
 # shellcheck source=scripts/ci/_script_init.sh
 . "$( dirname "${BASH_SOURCE[0]}" )/_script_init.sh"
 
+DOCKER_IMAGE_SPEC="${DOCKERHUB_USER}/${DOCKERHUB_REPO}:${DOCKER_TAG}"
+
+IMAGE_CREATED_SINCE=$(docker image ls "${DOCKER_IMAGE_SPEC}" --format '{{ .CreatedSince }}')
+
+if [[ ${IMAGE_CREATED_SINCE} =~ .*days.* || ${IMAGE_CREATED_SINCE} =~ .*weeks.* ||
+      ${IMAGE_CREATED_SINCE} =~ .*months.* || ${IMAGE_CREATED_SINCE} =~ .*years.* ||
+      ${IMAGE_CREATED_SINCE} =~ [0-9][0-9]\ hours ]]; then
 
 Review comment:
   Does it show `09 hours` in the `docker image ls ${DOCKER_IMAGE_SPEC}" --format '{{ .CreatedSince }}'`?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #8400: Less frequent dockerhub builds

Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #8400: Less frequent dockerhub builds
URL: https://github.com/apache/airflow/pull/8400#issuecomment-614628834
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/8400?src=pr&el=h1) Report
   > Merging [#8400](https://codecov.io/gh/apache/airflow/pull/8400?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/6d5710785204120b57dc61e351e12ae0b9f142b0&el=desc) will **decrease** coverage by `0.00%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/airflow/pull/8400/graphs/tree.svg?width=650&height=150&src=pr&token=WdLKlKHOAU)](https://codecov.io/gh/apache/airflow/pull/8400?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff            @@
   ##           master   #8400      +/-   ##
   =========================================
   - Coverage    6.24%   6.24%   -0.01%     
   =========================================
     Files         941     941              
     Lines       45573   45606      +33     
   =========================================
     Hits         2846    2846              
   - Misses      42727   42760      +33     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/8400?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [airflow/utils/cli.py](https://codecov.io/gh/apache/airflow/pull/8400/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9jbGkucHk=) | `0.00% <0.00%> (ø)` | |
   | [airflow/cli/cli\_parser.py](https://codecov.io/gh/apache/airflow/pull/8400/diff?src=pr&el=tree#diff-YWlyZmxvdy9jbGkvY2xpX3BhcnNlci5weQ==) | `0.00% <0.00%> (ø)` | |
   | [airflow/cli/commands/config\_command.py](https://codecov.io/gh/apache/airflow/pull/8400/diff?src=pr&el=tree#diff-YWlyZmxvdy9jbGkvY29tbWFuZHMvY29uZmlnX2NvbW1hbmQucHk=) | `0.00% <0.00%> (ø)` | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/8400?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/8400?src=pr&el=footer). Last update [6d57107...2215ff8](https://codecov.io/gh/apache/airflow/pull/8400?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] dimberman commented on issue #8400: Less frequent dockerhub builds

Posted by GitBox <gi...@apache.org>.
dimberman commented on issue #8400: Less frequent dockerhub builds
URL: https://github.com/apache/airflow/pull/8400#issuecomment-614771005
 
 
   @potiuk Should we only be building master every 10 hours? I feel like master should be up-to-date with whatever the master branch is doing.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on issue #8400: Less frequent dockerhub builds

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #8400: Less frequent dockerhub builds
URL: https://github.com/apache/airflow/pull/8400#issuecomment-614796124
 
 
   I mean - > we CAN have a corn Job setup (wherever) calling that trigger URL every 2-3 hours.... But is it safe to store such URL somewhere? Is it something that is sustainable long term (who will take care that it is actually still working  :)) .... Who will watch the watcher. ? 
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #8400: Less frequent dockerhub builds

Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #8400: Less frequent dockerhub builds
URL: https://github.com/apache/airflow/pull/8400#issuecomment-614628834
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/8400?src=pr&el=h1) Report
   > Merging [#8400](https://codecov.io/gh/apache/airflow/pull/8400?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/6d5710785204120b57dc61e351e12ae0b9f142b0&el=desc) will **not change** coverage.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/airflow/pull/8400/graphs/tree.svg?width=650&height=150&src=pr&token=WdLKlKHOAU)](https://codecov.io/gh/apache/airflow/pull/8400?src=pr&el=tree)
   
   ```diff
   @@          Coverage Diff           @@
   ##           master   #8400   +/-   ##
   ======================================
     Coverage    6.24%   6.24%           
   ======================================
     Files         941     941           
     Lines       45573   45573           
   ======================================
     Hits         2846    2846           
     Misses      42727   42727           
   ```
   
   
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/8400?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/8400?src=pr&el=footer). Last update [6d57107...5a9ed06](https://codecov.io/gh/apache/airflow/pull/8400?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on a change in pull request #8400: Less frequent dockerhub builds

Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #8400: Less frequent dockerhub builds
URL: https://github.com/apache/airflow/pull/8400#discussion_r409676031
 
 

 ##########
 File path: scripts/ci/ci_build_dockerhub.sh
 ##########
 @@ -67,6 +67,25 @@ echo
 # shellcheck source=scripts/ci/_script_init.sh
 . "$( dirname "${BASH_SOURCE[0]}" )/_script_init.sh"
 
+DOCKER_IMAGE_SPEC="${DOCKERHUB_USER}/${DOCKERHUB_REPO}:${DOCKER_TAG}"
+
+IMAGE_CREATED_SINCE=$(docker image ls "${DOCKER_IMAGE_SPEC}" --format '{{ .CreatedSince }}')
+
+if [[ ${IMAGE_CREATED_SINCE} =~ .*days.* || ${IMAGE_CREATED_SINCE} =~ .*weeks.* ||
+      ${IMAGE_CREATED_SINCE} =~ .*months.* || ${IMAGE_CREATED_SINCE} =~ .*years.* ||
+      ${IMAGE_CREATED_SINCE} =~ [0-9][0-9]\ hours ]]; then
 
 Review comment:
   9 hours is correct -> skipping....

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on a change in pull request #8400: Less frequent dockerhub builds

Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #8400: Less frequent dockerhub builds
URL: https://github.com/apache/airflow/pull/8400#discussion_r409589220
 
 

 ##########
 File path: scripts/ci/ci_build_dockerhub.sh
 ##########
 @@ -67,6 +67,25 @@ echo
 # shellcheck source=scripts/ci/_script_init.sh
 . "$( dirname "${BASH_SOURCE[0]}" )/_script_init.sh"
 
+DOCKER_IMAGE_SPEC="${DOCKERHUB_USER}/${DOCKERHUB_REPO}:${DOCKER_TAG}"
+
+IMAGE_CREATED_SINCE=$(docker image ls "${DOCKER_IMAGE_SPEC}" --format '{{ .CreatedSince }}')
+
+if [[ ${IMAGE_CREATED_SINCE} =~ .*days.* || ${IMAGE_CREATED_SINCE} =~ .*weeks.* ||
+      ${IMAGE_CREATED_SINCE} =~ .*months.* || ${IMAGE_CREATED_SINCE} =~ .*years.* ||
+      ${IMAGE_CREATED_SINCE} =~ [0-9][0-9]\ hours ]]; then
 
 Review comment:
   That's not possible on DockerHub. The build needs to be triggered by push to repo, there are no time triggers :(. We could use an external service for that and generate a special URL to trigger the builds but if it leaks we can be DDos-ed and it adds unnecessary third party. So I figured best thing is to check how fresh the image is.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on issue #8400: Less frequent dockerhub builds

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #8400: Less frequent dockerhub builds
URL: https://github.com/apache/airflow/pull/8400#issuecomment-614667765
 
 
   Green :). Review needed 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk edited a comment on issue #8400: Less frequent dockerhub builds

Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #8400: Less frequent dockerhub builds
URL: https://github.com/apache/airflow/pull/8400#issuecomment-614783967
 
 
   @dimberman It always lags quite a bit as DockerHub is very slow just "check if the image needs to be rebuilt aka Docker build from cache" takes anything between 10 and 20 minutes.  Currently we build so many images (including the PROD one)  that when we have a few merges, the build queue grows faster than it is cleared and pretty much every day we end up in a situation that the build waits a long time (700mins = 12 hours is pretty normal if we have a series of master merges) + if we have a v1-10-stable push, that's even longer. 
   
   From CI point of view it's OK. The whole system is built in the way that if the image is old, it - at most - run CI a bit slower. Assuming that master/v1-10-stable are constantly moving, I do not think this is a big issue to add such "10 hours" skip. 
   
   I also opened INFRA ticket to see if we can push the image from Github Actions (see my comment below. But I doubt it will be OK from the security POV:
   > Currently we have quite a lot of builds happening in DockerHub (including production image). It takes a long time to build the images and they are queuing now. I have created an infra ticket to ask if we can push images from Github Actions (https://issues.apache.org/jira/projects/INFRA/issues/INFRA-20124) but I have no high hopes for it - it requires storing secrets in GA. That's why I propose that build scripts will check if the image is <10 hours old and only actually build it if it is.
   
   Alternatively we might want to rebuild the images regularly - every night for example, but this is a bit problematic (I explained it to @turbaszek above) - we could generate a trigger URL and use it in an external service (there is no cron build support in DockerHub. But I am not sure it's safe.
   
   I realise it is not perfect, do you have any ideas how we can do it differently ?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on issue #8400: Less frequent dockerhub builds

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #8400: Less frequent dockerhub builds
URL: https://github.com/apache/airflow/pull/8400#issuecomment-614786735
 
 
   ![Screenshot from 2020-04-16 19-23-12](https://user-images.githubusercontent.com/595491/79486944-c838f300-8017-11ea-8fbb-132bd45ef54e.png)
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] kaxil commented on a change in pull request #8400: Less frequent dockerhub builds

Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #8400: Less frequent dockerhub builds
URL: https://github.com/apache/airflow/pull/8400#discussion_r409668928
 
 

 ##########
 File path: scripts/ci/ci_build_dockerhub.sh
 ##########
 @@ -67,6 +67,25 @@ echo
 # shellcheck source=scripts/ci/_script_init.sh
 . "$( dirname "${BASH_SOURCE[0]}" )/_script_init.sh"
 
+DOCKER_IMAGE_SPEC="${DOCKERHUB_USER}/${DOCKERHUB_REPO}:${DOCKER_TAG}"
+
+IMAGE_CREATED_SINCE=$(docker image ls "${DOCKER_IMAGE_SPEC}" --format '{{ .CreatedSince }}')
+
+if [[ ${IMAGE_CREATED_SINCE} =~ .*days.* || ${IMAGE_CREATED_SINCE} =~ .*weeks.* ||
+      ${IMAGE_CREATED_SINCE} =~ .*months.* || ${IMAGE_CREATED_SINCE} =~ .*years.* ||
+      ${IMAGE_CREATED_SINCE} =~ [0-9][0-9]\ hours ]]; then
 
 Review comment:
   Maybe we need the following:
   
   ```suggestion
         ${IMAGE_CREATED_SINCE} =~ [0-9]?\ hours ]]; then
   ```

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] kaxil commented on a change in pull request #8400: Less frequent dockerhub builds

Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #8400: Less frequent dockerhub builds
URL: https://github.com/apache/airflow/pull/8400#discussion_r409669217
 
 

 ##########
 File path: scripts/ci/ci_build_dockerhub.sh
 ##########
 @@ -67,6 +67,25 @@ echo
 # shellcheck source=scripts/ci/_script_init.sh
 . "$( dirname "${BASH_SOURCE[0]}" )/_script_init.sh"
 
+DOCKER_IMAGE_SPEC="${DOCKERHUB_USER}/${DOCKERHUB_REPO}:${DOCKER_TAG}"
+
+IMAGE_CREATED_SINCE=$(docker image ls "${DOCKER_IMAGE_SPEC}" --format '{{ .CreatedSince }}')
+
+if [[ ${IMAGE_CREATED_SINCE} =~ .*days.* || ${IMAGE_CREATED_SINCE} =~ .*weeks.* ||
+      ${IMAGE_CREATED_SINCE} =~ .*months.* || ${IMAGE_CREATED_SINCE} =~ .*years.* ||
+      ${IMAGE_CREATED_SINCE} =~ [0-9][0-9]\ hours ]]; then
 
 Review comment:
   ```bash
   ❯ export tta="9 hours"
   ❯ if [[ "$tta" =~ [0-9]?\ hours ]]; then
   echo "Rebuilding the image."
   else
   echo "Skipping rebuilding the image."
   fi
   Rebuilding the image.
   ```

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on issue #8400: Less frequent dockerhub builds

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #8400: Less frequent dockerhub builds
URL: https://github.com/apache/airflow/pull/8400#issuecomment-614787569
 
 
   This is our Dockerhub build confifguration ^^
   
   * 4 images to build from every master build
   * 8 images for v1-10-test
   
   Unfortuantely we have 1 worker queue from DockerHub that builds them :(
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on a change in pull request #8400: Less frequent dockerhub builds

Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #8400: Less frequent dockerhub builds
URL: https://github.com/apache/airflow/pull/8400#discussion_r409675720
 
 

 ##########
 File path: scripts/ci/ci_build_dockerhub.sh
 ##########
 @@ -67,6 +67,25 @@ echo
 # shellcheck source=scripts/ci/_script_init.sh
 . "$( dirname "${BASH_SOURCE[0]}" )/_script_init.sh"
 
+DOCKER_IMAGE_SPEC="${DOCKERHUB_USER}/${DOCKERHUB_REPO}:${DOCKER_TAG}"
+
+IMAGE_CREATED_SINCE=$(docker image ls "${DOCKER_IMAGE_SPEC}" --format '{{ .CreatedSince }}')
+
+if [[ ${IMAGE_CREATED_SINCE} =~ .*days.* || ${IMAGE_CREATED_SINCE} =~ .*weeks.* ||
+      ${IMAGE_CREATED_SINCE} =~ .*months.* || ${IMAGE_CREATED_SINCE} =~ .*years.* ||
+      ${IMAGE_CREATED_SINCE} =~ [0-9][0-9]\ hours ]]; then
 
 Review comment:
   But that's OK. I wanted to rebuild it when we hit 10 ..

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on a change in pull request #8400: Less frequent dockerhub builds

Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #8400: Less frequent dockerhub builds
URL: https://github.com/apache/airflow/pull/8400#discussion_r409677558
 
 

 ##########
 File path: scripts/ci/ci_build_dockerhub.sh
 ##########
 @@ -67,6 +67,25 @@ echo
 # shellcheck source=scripts/ci/_script_init.sh
 . "$( dirname "${BASH_SOURCE[0]}" )/_script_init.sh"
 
+DOCKER_IMAGE_SPEC="${DOCKERHUB_USER}/${DOCKERHUB_REPO}:${DOCKER_TAG}"
+
+IMAGE_CREATED_SINCE=$(docker image ls "${DOCKER_IMAGE_SPEC}" --format '{{ .CreatedSince }}')
+
+if [[ ${IMAGE_CREATED_SINCE} =~ .*days.* || ${IMAGE_CREATED_SINCE} =~ .*weeks.* ||
+      ${IMAGE_CREATED_SINCE} =~ .*months.* || ${IMAGE_CREATED_SINCE} =~ .*years.* ||
+      ${IMAGE_CREATED_SINCE} =~ [0-9][0-9]\ hours ]]; then
 
 Review comment:
   I wanted to make aproper calculation from UTC time but doing it in bash is kinda complex

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] kaxil commented on a change in pull request #8400: Less frequent dockerhub builds

Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #8400: Less frequent dockerhub builds
URL: https://github.com/apache/airflow/pull/8400#discussion_r409669217
 
 

 ##########
 File path: scripts/ci/ci_build_dockerhub.sh
 ##########
 @@ -67,6 +67,25 @@ echo
 # shellcheck source=scripts/ci/_script_init.sh
 . "$( dirname "${BASH_SOURCE[0]}" )/_script_init.sh"
 
+DOCKER_IMAGE_SPEC="${DOCKERHUB_USER}/${DOCKERHUB_REPO}:${DOCKER_TAG}"
+
+IMAGE_CREATED_SINCE=$(docker image ls "${DOCKER_IMAGE_SPEC}" --format '{{ .CreatedSince }}')
+
+if [[ ${IMAGE_CREATED_SINCE} =~ .*days.* || ${IMAGE_CREATED_SINCE} =~ .*weeks.* ||
+      ${IMAGE_CREATED_SINCE} =~ .*months.* || ${IMAGE_CREATED_SINCE} =~ .*years.* ||
+      ${IMAGE_CREATED_SINCE} =~ [0-9][0-9]\ hours ]]; then
 
 Review comment:
   ```bash
   ❯ export tta="9 hours"
   ❯ if [[ "$tta" =~ [0-9]?\ hours ]]; then
   echo "Rebuilding the image."
   else
   echo "Skipping rebuilding the image."
   fi
   Rebuilding the image.
   ```

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] kaxil commented on a change in pull request #8400: Less frequent dockerhub builds

Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #8400: Less frequent dockerhub builds
URL: https://github.com/apache/airflow/pull/8400#discussion_r409503082
 
 

 ##########
 File path: scripts/ci/ci_build_dockerhub.sh
 ##########
 @@ -67,6 +67,26 @@ echo
 # shellcheck source=scripts/ci/_script_init.sh
 . "$( dirname "${BASH_SOURCE[0]}" )/_script_init.sh"
 
+DOCKER_IMAGE_SPEC="${DOCKERHUB_USER}/${DOCKERHUB_REPO}:${DOCKER_TAG}"
+
+#docker pull "${DOCKER_IMAGE_SPEC}"
 
 Review comment:
   ```suggestion
   ```

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on a change in pull request #8400: Less frequent dockerhub builds

Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #8400: Less frequent dockerhub builds
URL: https://github.com/apache/airflow/pull/8400#discussion_r409506658
 
 

 ##########
 File path: scripts/ci/ci_build_dockerhub.sh
 ##########
 @@ -67,6 +67,26 @@ echo
 # shellcheck source=scripts/ci/_script_init.sh
 . "$( dirname "${BASH_SOURCE[0]}" )/_script_init.sh"
 
+DOCKER_IMAGE_SPEC="${DOCKERHUB_USER}/${DOCKERHUB_REPO}:${DOCKER_TAG}"
+
+#docker pull "${DOCKER_IMAGE_SPEC}"
 
 Review comment:
   aaaargh/// Testing :)

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on issue #8400: Less frequent dockerhub builds

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #8400: Less frequent dockerhub builds
URL: https://github.com/apache/airflow/pull/8400#issuecomment-615082154
 
 
   I think 2-3 hours is not enough. it looks like on dockerhub we are also competing with others for the queue - and quite often just queue time is 2-3 hours. Sometimes the build quue one after the other so 2-3 hours is often how much it takes for on complete set of images :(. My feeling now is that 10 hours is a "safe" period - it basically means 2-3 builds a day. And as long master and v1-10-test are a moving target, I do not think it has really bad side effects (for now). In the meantime we can figure out better ways (I would also like to see how it behaves with those 10 hours for now). We have stats and history so we will be able to make better decisions afterwards. I will merge it as is now - and we can always change. In the meantime I will wait for INFRA and maybe we will be able to figure out something with them. Maybe we can simply get a private Jenkins Apache regularly scheduled trigger  in the way that will not reveal the URL to anyone and we will be able to add regular builds.
   
   This way I think might be best - have 10 hours "build window" for regular builds + nightly external scheduled trigger should handle all cases nicely.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on a change in pull request #8400: Less frequent dockerhub builds

Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #8400: Less frequent dockerhub builds
URL: https://github.com/apache/airflow/pull/8400#discussion_r409677204
 
 

 ##########
 File path: scripts/ci/ci_build_dockerhub.sh
 ##########
 @@ -67,6 +67,25 @@ echo
 # shellcheck source=scripts/ci/_script_init.sh
 . "$( dirname "${BASH_SOURCE[0]}" )/_script_init.sh"
 
+DOCKER_IMAGE_SPEC="${DOCKERHUB_USER}/${DOCKERHUB_REPO}:${DOCKER_TAG}"
+
+IMAGE_CREATED_SINCE=$(docker image ls "${DOCKER_IMAGE_SPEC}" --format '{{ .CreatedSince }}')
+
+if [[ ${IMAGE_CREATED_SINCE} =~ .*days.* || ${IMAGE_CREATED_SINCE} =~ .*weeks.* ||
+      ${IMAGE_CREATED_SINCE} =~ .*months.* || ${IMAGE_CREATED_SINCE} =~ .*years.* ||
+      ${IMAGE_CREATED_SINCE} =~ [0-9][0-9]\ hours ]]; then
 
 Review comment:
   I am not sure how the 10 hours are calculated - i.e. I think 9h + 1 minute might be displayed as 10 hours 
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on a change in pull request #8400: Less frequent dockerhub builds

Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #8400: Less frequent dockerhub builds
URL: https://github.com/apache/airflow/pull/8400#discussion_r409659094
 
 

 ##########
 File path: scripts/ci/ci_build_dockerhub.sh
 ##########
 @@ -67,6 +67,25 @@ echo
 # shellcheck source=scripts/ci/_script_init.sh
 . "$( dirname "${BASH_SOURCE[0]}" )/_script_init.sh"
 
+DOCKER_IMAGE_SPEC="${DOCKERHUB_USER}/${DOCKERHUB_REPO}:${DOCKER_TAG}"
+
+IMAGE_CREATED_SINCE=$(docker image ls "${DOCKER_IMAGE_SPEC}" --format '{{ .CreatedSince }}')
+
+if [[ ${IMAGE_CREATED_SINCE} =~ .*days.* || ${IMAGE_CREATED_SINCE} =~ .*weeks.* ||
+      ${IMAGE_CREATED_SINCE} =~ .*months.* || ${IMAGE_CREATED_SINCE} =~ .*years.* ||
+      ${IMAGE_CREATED_SINCE} =~ [0-9][0-9]\ hours ]]; then
 
 Review comment:
   Nope. It's 9 hours :)

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on issue #8400: Less frequent dockerhub builds

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #8400: Less frequent dockerhub builds
URL: https://github.com/apache/airflow/pull/8400#issuecomment-614792771
 
 
   There is no such feature in Dockerhub :shrug: You can trigger it on 
   
   1) push to repo (Branch or Tag)
   2) externally called URL trigger - this can be scheduled but the URL is essentially a credential and can be called many times. We would need to have some external service calling the URL 
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on issue #8400: Less frequent dockerhub builds

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #8400: Less frequent dockerhub builds
URL: https://github.com/apache/airflow/pull/8400#issuecomment-614609797
 
 
   Currently we have quite a lot of builds happening in DockerHub (including production image). It takes a long time to build the images and they are queuing now. I have created an infra ticket to ask if we can push images from Github Actions (https://issues.apache.org/jira/projects/INFRA/issues/INFRA-20124) but I have no high hopes for it - it requires storing secrets in GA. That's why I propose that build scripts will check if the image is <10 hours old and only actually build it if it is.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] kaxil commented on a change in pull request #8400: Less frequent dockerhub builds

Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #8400: Less frequent dockerhub builds
URL: https://github.com/apache/airflow/pull/8400#discussion_r409503268
 
 

 ##########
 File path: scripts/ci/ci_build_dockerhub.sh
 ##########
 @@ -51,12 +51,12 @@ else
    echo "DOCKER_TAG=${DOCKER_TAG}"
 fi
 
-[[ ${DOCKER_TAG:=} =~ ${DEFAULT_BRANCH}-python([0-9.]*)(.*) ]] && export PYTHON_MAJOR_MINOR_VERSION=${BASH_REMATCH[1]}
+[[ ${DOCKER_TAG:=} =~ .*-python([0-9.]*)(.*) ]] && export PYTHON_MAJOR_MINOR_VERSION=${BASH_REMATCH[1]}
 
 Review comment:
   Why do we need to remove `${DEFAULT_BRANCH}`

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] dimberman commented on issue #8400: Less frequent dockerhub builds

Posted by GitBox <gi...@apache.org>.
dimberman commented on issue #8400: Less frequent dockerhub builds
URL: https://github.com/apache/airflow/pull/8400#issuecomment-614814456
 
 
   Hmmm so I think long term the INFRA ticket is our best hope. Would be great to be able to push from github actions. Really hoping apache lets us do that. 
   
   We could also do what this PR is proposing and just make it less time (like maybe if the build is 2-3 hours old?)

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk merged pull request #8400: Less frequent dockerhub builds

Posted by GitBox <gi...@apache.org>.
potiuk merged pull request #8400: Less frequent dockerhub builds
URL: https://github.com/apache/airflow/pull/8400
 
 
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services