You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/03/21 14:58:54 UTC
[GitHub] [airflow] potiuk opened a new pull request #7797: Test installation
of all provider packages
potiuk opened a new pull request #7797: Test installation of all provider packages
URL: https://github.com/apache/airflow/pull/7797
---
Issue link: WILL BE INSERTED BY [boring-cyborg](https://github.com/kaxil/boring-cyborg)
Make sure to mark the boxes below before creating PR: [x]
- [x] Description above provides context of the change
- [x] Unit tests coverage for changes (not needed for documentation changes)
- [x] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)"
- [x] Relevant documentation is updated including usage instructions.
- [x] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
---
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] turbaszek commented on a change in pull request #7797:
Test installation of all provider packages
Posted by GitBox <gi...@apache.org>.
turbaszek commented on a change in pull request #7797: Test installation of all provider packages
URL: https://github.com/apache/airflow/pull/7797#discussion_r396068387
##########
File path: scripts/ci/in_container/run_test_package_installation.sh
##########
@@ -0,0 +1,64 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# shellcheck source=scripts/ci/in_container/_in_container_script_init.sh
+. "$( dirname "${BASH_SOURCE[0]}" )/_in_container_script_init.sh"
+OUT_FILE=$(mktemp)
+
+if [[ ! ${INSTALL_AIRFLOW_VERSION:="current"} =~ 1.10* ]]; then
+ echo
+ echo "ERROR! You can only install providers package in 1.10. airflow series."
+ echo "You have: ${INSTALL_AIRFLOW_VERSION}"
+ echo
+ exit 1
+else
+ pip uninstall -y apache-airflow >>"${OUT_FILE}" 2>&1
+ pip install "apache-airflow==${INSTALL_AIRFLOW_VERSION}" >>"${OUT_FILE}" 2>&1
+fi
+
+for PACKAGE_FILE in /dist/*.whl
+do
+ if [[ ! ${PACKAGE_FILE} =~ /dist/(apache_airflow_providers_[^-]*)-.* ]]; then
+ echo
+ echo "ERROR: ${PACKAGE_FILE} does not match providers package regexp"
+ echo
+ else
+ PACKAGE_NAME_UNDERSCORE=${BASH_REMATCH[1]}
+ PACKAGE_NAME=$(echo "${PACKAGE_NAME_UNDERSCORE}" | tr "_" "-")
+ fi
+ echo "==================================================================================="
+ echo "Installing ${PACKAGE_NAME}"
+ echo "-----------------------------------------------------------------------------------"
+ pip install "${PACKAGE_FILE}" >>"${OUT_FILE}" 2>&1
+ echo "Installed ${PACKAGE_NAME}"
+ echo "-----------------------------------------------------------------------------------"
+ echo "Uninstalling ${PACKAGE_NAME}"
+ echo "-----------------------------------------------------------------------------------"
+ pip uninstall -y "${PACKAGE_NAME}" >>"${OUT_FILE}" 2>&1
+ echo "Uninstalled ${PACKAGE_NAME}"
+ echo "-----------------------------------------------------------------------------------"
+ AIRFLOW_VERSION=$(pip freeze | grep "apache-airflow==" | sed "s/apache-airflow==//")
+ echo "Airflow version after installation ${AIRFLOW_VERSION}"
+ if [[ ${AIRFLOW_VERSION} != "${INSTALL_AIRFLOW_VERSION}" ]]; then
+ echo
+ echo "ERROR! Installing ${PACKAGE_FILE} caused Airflow to upgrade to ${AIRFLOW_VERSION}"
+ echo
+ echo "Please fix dependencies in the package"
+ exit 1
Review comment:
Nice!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk merged pull request #7797: Test installation of
all provider packages
Posted by GitBox <gi...@apache.org>.
potiuk merged pull request #7797: Test installation of all provider packages
URL: https://github.com/apache/airflow/pull/7797
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #7797: Test
installation of all provider packages
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #7797: Test installation of all provider packages
URL: https://github.com/apache/airflow/pull/7797#discussion_r396088415
##########
File path: scripts/ci/ci_prepare_backport_packages.sh
##########
@@ -20,14 +20,47 @@ export PYTHON_VERSION=${PYTHON_VERSION:-3.6}
# shellcheck source=scripts/ci/_script_init.sh
. "$( dirname "${BASH_SOURCE[0]}" )/_script_init.sh"
+LIST_OF_DIRS_FILE=$(mktemp)
+
+cd "${MY_DIR}/../../airflow/providers" || exit 1
+
+find . -type d | sed 's/.\///' | sed 's/\//\./' | grep -E 'hooks|operators|sensors|secrets' \
Review comment:
```suggestion
find . -type d | sed 's/.\///; s/\//\./' | grep -E 'hooks|operators|sensors|secrets' \
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #7797: Test
installation of all provider packages
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #7797: Test installation of all provider packages
URL: https://github.com/apache/airflow/pull/7797#discussion_r396089607
##########
File path: scripts/ci/ci_prepare_backport_packages.sh
##########
@@ -20,14 +20,47 @@ export PYTHON_VERSION=${PYTHON_VERSION:-3.6}
# shellcheck source=scripts/ci/_script_init.sh
. "$( dirname "${BASH_SOURCE[0]}" )/_script_init.sh"
+LIST_OF_DIRS_FILE=$(mktemp)
+
+cd "${MY_DIR}/../../airflow/providers" || exit 1
+
+find . -type d | sed 's/.\///' | sed 's/\//\./' | grep -E 'hooks|operators|sensors|secrets' \
Review comment:
Yep
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on issue #7797: Test installation of all
provider packages
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #7797: Test installation of all provider packages
URL: https://github.com/apache/airflow/pull/7797#issuecomment-602168050
@turbaszek - see
https://lists.apache.org/thread.html/r19014fac5aa2aa15a86890aa438940da263e430cb535ca451ef451a4%40%3Cdev.airflow.apache.org%3E
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7797: Test
installation of all provider packages
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #7797: Test installation of all provider packages
URL: https://github.com/apache/airflow/pull/7797#issuecomment-602262866
# [Codecov](https://codecov.io/gh/apache/airflow/pull/7797?src=pr&el=h1) Report
> Merging [#7797](https://codecov.io/gh/apache/airflow/pull/7797?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/0d8b8465513acee608a97ab0715fab07d174dc41&el=desc) will **not change** coverage by `%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/airflow/pull/7797?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #7797 +/- ##
=======================================
Coverage 86.91% 86.91%
=======================================
Files 925 925
Lines 44870 44870
=======================================
Hits 39001 39001
Misses 5869 5869
```
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7797?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7797?src=pr&el=footer). Last update [0d8b846...72f9a67](https://codecov.io/gh/apache/airflow/pull/7797?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] turbaszek commented on issue #7797: Test installation of
all provider packages
Posted by GitBox <gi...@apache.org>.
turbaszek commented on issue #7797: Test installation of all provider packages
URL: https://github.com/apache/airflow/pull/7797#issuecomment-602166381
Just out of curiosity, this PR has no JIRA ticket and no Github Issue. Is it ok? How will we include it in the changelog? @potiuk @ashb
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk opened a new pull request #7797: Test installation
of all provider packages
Posted by GitBox <gi...@apache.org>.
potiuk opened a new pull request #7797: Test installation of all provider packages
URL: https://github.com/apache/airflow/pull/7797
---
Issue link: WILL BE INSERTED BY [boring-cyborg](https://github.com/kaxil/boring-cyborg)
Make sure to mark the boxes below before creating PR: [x]
- [x] Description above provides context of the change
- [x] Unit tests coverage for changes (not needed for documentation changes)
- [x] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)"
- [x] Relevant documentation is updated including usage instructions.
- [x] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
---
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #7797: Test
installation of all provider packages
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #7797: Test installation of all provider packages
URL: https://github.com/apache/airflow/pull/7797#discussion_r396089600
##########
File path: scripts/ci/ci_prepare_backport_packages.sh
##########
@@ -20,14 +20,47 @@ export PYTHON_VERSION=${PYTHON_VERSION:-3.6}
# shellcheck source=scripts/ci/_script_init.sh
. "$( dirname "${BASH_SOURCE[0]}" )/_script_init.sh"
+LIST_OF_DIRS_FILE=$(mktemp)
+
+cd "${MY_DIR}/../../airflow/providers" || exit 1
+
+find . -type d | sed 's/.\///' | sed 's/\//\./' | grep -E 'hooks|operators|sensors|secrets' \
Review comment:
Fixed. Was not intentional but it worked anyway ;)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on issue #7797: Test installation of all
provider packages
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #7797: Test installation of all provider packages
URL: https://github.com/apache/airflow/pull/7797#issuecomment-602167889
We discussed that not all PRs need to have issues. PR# is more than enough to identify changes and we are going to use PR# in changelog from now on.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on issue #7797: Test installation of all
provider packages
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #7797: Test installation of all provider packages
URL: https://github.com/apache/airflow/pull/7797#issuecomment-602194860
All Fixed and rebased @mik-laj
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7797: Test
installation of all provider packages
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #7797: Test installation of all provider packages
URL: https://github.com/apache/airflow/pull/7797#issuecomment-602262866
# [Codecov](https://codecov.io/gh/apache/airflow/pull/7797?src=pr&el=h1) Report
> Merging [#7797](https://codecov.io/gh/apache/airflow/pull/7797?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/0d8b8465513acee608a97ab0715fab07d174dc41&el=desc) will **decrease** coverage by `0.26%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/airflow/pull/7797?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #7797 +/- ##
==========================================
- Coverage 86.91% 86.65% -0.27%
==========================================
Files 925 925
Lines 44870 44870
==========================================
- Hits 39001 38882 -119
- Misses 5869 5988 +119
```
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7797?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7797?src=pr&el=footer). Last update [0d8b846...72f9a67](https://codecov.io/gh/apache/airflow/pull/7797?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] mik-laj commented on issue #7797: Test installation of
all provider packages
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #7797: Test installation of all provider packages
URL: https://github.com/apache/airflow/pull/7797#issuecomment-602192290
I started reviewwing.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk closed pull request #7797: Test installation of
all provider packages
Posted by GitBox <gi...@apache.org>.
potiuk closed pull request #7797: Test installation of all provider packages
URL: https://github.com/apache/airflow/pull/7797
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #7797: Test installation of
all provider packages
Posted by GitBox <gi...@apache.org>.
codecov-io commented on issue #7797: Test installation of all provider packages
URL: https://github.com/apache/airflow/pull/7797#issuecomment-602262866
# [Codecov](https://codecov.io/gh/apache/airflow/pull/7797?src=pr&el=h1) Report
> Merging [#7797](https://codecov.io/gh/apache/airflow/pull/7797?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/0d8b8465513acee608a97ab0715fab07d174dc41&el=desc) will **decrease** coverage by `0.26%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/airflow/pull/7797?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #7797 +/- ##
==========================================
- Coverage 86.91% 86.65% -0.27%
==========================================
Files 925 925
Lines 44870 44870
==========================================
- Hits 39001 38882 -119
- Misses 5869 5988 +119
```
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7797?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7797?src=pr&el=footer). Last update [0d8b846...72f9a67](https://codecov.io/gh/apache/airflow/pull/7797?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #7797: Test
installation of all provider packages
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #7797: Test installation of all provider packages
URL: https://github.com/apache/airflow/pull/7797#discussion_r396088360
##########
File path: scripts/ci/ci_prepare_backport_packages.sh
##########
@@ -20,14 +20,47 @@ export PYTHON_VERSION=${PYTHON_VERSION:-3.6}
# shellcheck source=scripts/ci/_script_init.sh
. "$( dirname "${BASH_SOURCE[0]}" )/_script_init.sh"
+LIST_OF_DIRS_FILE=$(mktemp)
+
+cd "${MY_DIR}/../../airflow/providers" || exit 1
+
+find . -type d | sed 's/.\///' | sed 's/\//\./' | grep -E 'hooks|operators|sensors|secrets' \
Review comment:
This command gives inconsistent results regarding the dot and slash. Is this expected?
```
redis.sensors
redis.operators
redis.hooks
google.ads/operators
google.ads/hooks
google.marketing_platform/sensors
google.marketing_platform/operators
google.marketing_platform/hooks
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services