You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/03/10 12:47:03 UTC
[GitHub] [airflow] ashb opened a new pull request #7678: [AIRFLOW-7029] Use
separate docker image for running license check
ashb opened a new pull request #7678: [AIRFLOW-7029] Use separate docker image for running license check
URL: https://github.com/apache/airflow/pull/7678
Each stage of the CI tests needs to pull our `ci` image. By removing
java from it we can save 1-2minutes from each test stage. This is part
of that work.
See AIRFLOW-7028 for reasoning.
---
Issue link: [AIRFLOW-7029](https://issues.apache.org/jira/browse/AIRFLOW-7029)
Make sure to mark the boxes below before creating PR: [x]
- [x] Description above provides context of the change
- [x] Commit message/PR title starts with `[AIRFLOW-NNNN]`. AIRFLOW-NNNN = JIRA ID<sup>*</sup>
- [x] Unit tests coverage for changes (not needed for documentation changes)
- [x] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)"
- [x] Relevant documentation is updated including usage instructions.
- [x] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
<sup>*</sup> For document-only changes commit message can start with `[AIRFLOW-XXXX]`.
---
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on issue #7678: [AIRFLOW-7029] Use
separate docker image for running license check
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #7678: [AIRFLOW-7029] Use separate docker image for running license check
URL: https://github.com/apache/airflow/pull/7678#issuecomment-597082345
Glad we are slimming down the CI image :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk edited a comment on issue #7678: [AIRFLOW-7029]
Use separate docker image for running license check
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #7678: [AIRFLOW-7029] Use separate docker image for running license check
URL: https://github.com/apache/airflow/pull/7678#issuecomment-597086982
Plus @ashb if you wait and rebase it on top of #7656 (after we merge it) , then people using breeze will be automatically pulling & building that new image rather than rebuilding them from RAT layer - this will be much faster.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk edited a comment on issue #7678: [AIRFLOW-7029]
Use separate docker image for running license check
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #7678: [AIRFLOW-7029] Use separate docker image for running license check
URL: https://github.com/apache/airflow/pull/7678#issuecomment-597086982
Plus @ashb if you wait and rebase it on top of #7656 (after we merge it) , then people using breeze will be automatically pulling & building that new image rather than rebuilding it from RAT layer - this will be much faster.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on issue #7678: [AIRFLOW-7029] Use
separate docker image for running license check
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #7678: [AIRFLOW-7029] Use separate docker image for running license check
URL: https://github.com/apache/airflow/pull/7678#issuecomment-597081999
https://hub.docker.com/u/potiuk
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on issue #7678: [AIRFLOW-7029] Use
separate docker image for running license check
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #7678: [AIRFLOW-7029] Use separate docker image for running license check
URL: https://github.com/apache/airflow/pull/7678#issuecomment-597226479
> @potiuk I don't see what rebasing on #7656 would do? Upon first look I thought it wouldn't matter which of the two are merged first...?
Actually you are right even better to merge it first!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] ashb commented on issue #7678: [AIRFLOW-7029] Use
separate docker image for running license check
Posted by GitBox <gi...@apache.org>.
ashb commented on issue #7678: [AIRFLOW-7029] Use separate docker image for running license check
URL: https://github.com/apache/airflow/pull/7678#issuecomment-597192834
@potiuk I don't see what rebasing on #7656 would do? Upon first look I thought it wouldn't matter which of the two are merged first...?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] ashb commented on issue #7678: [AIRFLOW-7029] Use
separate docker image for running license check
Posted by GitBox <gi...@apache.org>.
ashb commented on issue #7678: [AIRFLOW-7029] Use separate docker image for running license check
URL: https://github.com/apache/airflow/pull/7678#issuecomment-597067349
Uses https://github.com/ashb/apache-rat-docker (which is set up to auto build). Happy to give other people maintainer rights on the repo.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] ashb commented on a change in pull request #7678:
[AIRFLOW-7029] Use separate docker image for running license check
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #7678: [AIRFLOW-7029] Use separate docker image for running license check
URL: https://github.com/apache/airflow/pull/7678#discussion_r392412861
##########
File path: scripts/ci/ci_check_license.sh
##########
@@ -22,21 +22,39 @@ export PYTHON_VERSION=${PYTHON_VERSION:-3.6}
. "$( dirname "${BASH_SOURCE[0]}" )/_script_init.sh"
function run_check_license() {
- docker run "${EXTRA_DOCKER_FLAGS[@]}" -t \
- --entrypoint "/usr/local/bin/dumb-init" \
- --env PYTHONDONTWRITEBYTECODE \
- --env VERBOSE \
- --env VERBOSE_COMMANDS \
- --env HOST_USER_ID="$(id -ur)" \
- --env HOST_GROUP_ID="$(id -gr)" \
- --rm \
- "${AIRFLOW_CI_IMAGE}" \
- "--" "/opt/airflow/scripts/ci/in_container/run_check_licence.sh" \
- | tee -a "${OUTPUT_LOG}"
-}
+ echo
+ echo "Running Licence check"
+ echo
+
+ # This is the target of a symlink in airflow/www/static/docs -
+ # and rat exclude doesn't cope with the symlink target doesn't exist
+ mkdir -p docs/_build/html/
-prepare_build
+ echo "Running license checks. This can take a while."
-rebuild_ci_image_if_needed
+ if ! docker run "${EXTRA_DOCKER_FLAGS[@]}" -t \
+ --user "$(id -ur):$(id -gr)" \
+ --rm \
+ ashb/apache-rat:0.13-1 \
Review comment:
I'm not really sure which project it's part of, it seems like a side project?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] ashb merged pull request #7678: [AIRFLOW-7029] Use
separate docker image for running license check
Posted by GitBox <gi...@apache.org>.
ashb merged pull request #7678: [AIRFLOW-7029] Use separate docker image for running license check
URL: https://github.com/apache/airflow/pull/7678
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on issue #7678: [AIRFLOW-7029] Use
separate docker image for running license check
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #7678: [AIRFLOW-7029] Use separate docker image for running license check
URL: https://github.com/apache/airflow/pull/7678#issuecomment-597081727
> Uses https://github.com/ashb/apache-rat-docker (which is set up to auto build). Happy to give other people maintainer rights on the repo.
Please add me :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on issue #7678: [AIRFLOW-7029] Use
separate docker image for running license check
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #7678: [AIRFLOW-7029] Use separate docker image for running license check
URL: https://github.com/apache/airflow/pull/7678#issuecomment-597086982
Plus @ashb if you wait and rebase it on top of #7656 (after we merge it) , then people using breeze will be automatically pulling & building that new image rather than rebuilding them from scratch - this will be much faster.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #7678:
[AIRFLOW-7029] Use separate docker image for running license check
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #7678: [AIRFLOW-7029] Use separate docker image for running license check
URL: https://github.com/apache/airflow/pull/7678#discussion_r392342062
##########
File path: scripts/ci/ci_check_license.sh
##########
@@ -22,21 +22,39 @@ export PYTHON_VERSION=${PYTHON_VERSION:-3.6}
. "$( dirname "${BASH_SOURCE[0]}" )/_script_init.sh"
function run_check_license() {
- docker run "${EXTRA_DOCKER_FLAGS[@]}" -t \
- --entrypoint "/usr/local/bin/dumb-init" \
- --env PYTHONDONTWRITEBYTECODE \
- --env VERBOSE \
- --env VERBOSE_COMMANDS \
- --env HOST_USER_ID="$(id -ur)" \
- --env HOST_GROUP_ID="$(id -gr)" \
- --rm \
- "${AIRFLOW_CI_IMAGE}" \
- "--" "/opt/airflow/scripts/ci/in_container/run_check_licence.sh" \
- | tee -a "${OUTPUT_LOG}"
-}
+ echo
+ echo "Running Licence check"
+ echo
+
+ # This is the target of a symlink in airflow/www/static/docs -
+ # and rat exclude doesn't cope with the symlink target doesn't exist
+ mkdir -p docs/_build/html/
-prepare_build
+ echo "Running license checks. This can take a while."
-rebuild_ci_image_if_needed
+ if ! docker run "${EXTRA_DOCKER_FLAGS[@]}" -t \
+ --user "$(id -ur):$(id -gr)" \
+ --rm \
+ ashb/apache-rat:0.13-1 \
Review comment:
Have you ever asked the application developers if they didn't want to start publishing an official image?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #7678:
[AIRFLOW-7029] Use separate docker image for running license check
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #7678: [AIRFLOW-7029] Use separate docker image for running license check
URL: https://github.com/apache/airflow/pull/7678#discussion_r392449303
##########
File path: scripts/ci/ci_check_license.sh
##########
@@ -22,21 +22,39 @@ export PYTHON_VERSION=${PYTHON_VERSION:-3.6}
. "$( dirname "${BASH_SOURCE[0]}" )/_script_init.sh"
function run_check_license() {
- docker run "${EXTRA_DOCKER_FLAGS[@]}" -t \
- --entrypoint "/usr/local/bin/dumb-init" \
- --env PYTHONDONTWRITEBYTECODE \
- --env VERBOSE \
- --env VERBOSE_COMMANDS \
- --env HOST_USER_ID="$(id -ur)" \
- --env HOST_GROUP_ID="$(id -gr)" \
- --rm \
- "${AIRFLOW_CI_IMAGE}" \
- "--" "/opt/airflow/scripts/ci/in_container/run_check_licence.sh" \
- | tee -a "${OUTPUT_LOG}"
-}
+ echo
+ echo "Running Licence check"
+ echo
+
+ # This is the target of a symlink in airflow/www/static/docs -
+ # and rat exclude doesn't cope with the symlink target doesn't exist
+ mkdir -p docs/_build/html/
-prepare_build
+ echo "Running license checks. This can take a while."
-rebuild_ci_image_if_needed
+ if ! docker run "${EXTRA_DOCKER_FLAGS[@]}" -t \
+ --user "$(id -ur):$(id -gr)" \
+ --rm \
+ ashb/apache-rat:0.13-1 \
Review comment:
This is the official Apache project. Here is the Github repository: https://github.com/apache/creadur-rat/
I've already added Docker to Travis CI, so the Docker image is an interesting next step forward.
https://github.com/apache/creadur-rat/blob/master/.travis.yml
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services