You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/12/09 11:15:18 UTC
[GitHub] [airflow] potiuk opened a new pull request #12946: Fixed broken dependencies for apache-beam.
potiuk opened a new pull request #12946:
URL: https://github.com/apache/airflow/pull/12946
Apache beam is now also included in our extras.
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #12946: Fixed broken dependencies for apache-beam.
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #12946:
URL: https://github.com/apache/airflow/pull/12946#issuecomment-741867932
[The Workflow run](https://github.com/apache/airflow/actions/runs/410946614) is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #12946: Fixed broken dependencies for apache-beam.
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #12946:
URL: https://github.com/apache/airflow/pull/12946#issuecomment-741740664
FYI: The master PIP also does not work with those constraints (even if they are non-conflicting) everything installs fine in pip 20.2.44, but the new PIP has problem with it:
```
Collecting apache-beam[gcp]<=2.23.0
Downloading apache_beam-2.23.0-cp36-cp36m-manylinux2010_x86_64.whl (8.3 MB)
|████████████████████████████████| 8.3 MB 22.8 MB/s
INFO: pip is looking at multiple versions of analytics-python to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of amqp to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of alembic to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of sphinxcontrib-spelling to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of sphinx-autoapi to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of rich to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of apscheduler to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of python-telegram-bot to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of pylint to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of pinotdb to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of mypy to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of json-merge-patch to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of flask-swagger to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of elasticsearch-dbapi to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of colorlog to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of sphinx-airflow-theme to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of <Python from Requires-Python> to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of apache-airflow to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of apache-airflow[devel-ci] to determine which version is compatible with other requirements. This could take a while.
ERROR: Could not find a version that satisfies the requirement cachetools<4,>=3.1.0; extra == "gcp" (from apache-beam[gcp])
ERROR: No matching distribution found for cachetools<4,>=3.1.0; extra == "gcp"
```
It does not download all the many versions of packages, but it fails on cachetools this time.
So either those are imaginary dependencies by mistakenly using some extras, or 2.20.4 did not detect those conflicts.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #12946: Fixed broken dependencies for apache-beam.
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #12946:
URL: https://github.com/apache/airflow/pull/12946#discussion_r539260915
##########
File path: setup.cfg
##########
@@ -116,10 +117,12 @@ install_requires =
markdown>=2.5.2, <4.0
markupsafe>=1.1.1, <2.0
marshmallow-oneofschema>=2.0.1
+ mock<3.0.0 # required to keep apache-beam happy
pandas>=0.17.1, <2.0
pendulum~=2.0
pep562~=1.0;python_version<"3.7"
psutil>=4.2.0, <6.0.0
+ pyarrow<0.18.0
Review comment:
Yeah. I will experiment with moving those out of install_requires. I want to see if they will work for our tests first
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #12946: Fixed broken dependencies for apache-beam.
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #12946:
URL: https://github.com/apache/airflow/pull/12946#discussion_r539269102
##########
File path: setup.cfg
##########
@@ -116,10 +117,12 @@ install_requires =
markdown>=2.5.2, <4.0
markupsafe>=1.1.1, <2.0
marshmallow-oneofschema>=2.0.1
+ mock<3.0.0 # required to keep apache-beam happy
pandas>=0.17.1, <2.0
pendulum~=2.0
pep562~=1.0;python_version<"3.7"
psutil>=4.2.0, <6.0.0
+ pyarrow<0.18.0
Review comment:
I only need to constraint deps in "install_requires" if they are already there transitively. I "eagerly" added them here to check if it works but likely httplib can be moved out, let's see pyarrow as well
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #12946: Fixed broken dependencies for apache-beam.
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #12946:
URL: https://github.com/apache/airflow/pull/12946#issuecomment-741775778
Wohoo! Seems I've done that. Waiting for https://github.com/potiuk/airflow/actions/runs/410667474 but it looks like we have apache-beam included and no more exclusions @mik-laj and @turbaszek !
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on pull request #12946: Fixed broken dependencies for apache-beam.
Posted by GitBox <gi...@apache.org>.
ashb commented on pull request #12946:
URL: https://github.com/apache/airflow/pull/12946#issuecomment-741731792
My concern here is this pins a lot of things to old versions globally, to support a sub-provider (not even a provider) that is only used by a tiny subset of our users.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #12946: Fixed broken dependencies for apache-beam.
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #12946:
URL: https://github.com/apache/airflow/pull/12946#issuecomment-741737352
> > tiny subset of our users.
>
> Depends how we count, Google Dataflow operators are ones of most commonly used from Google provider.
>
> @potiuk have you try to upgrade beam? I think latest 2.X version has higher requirements
Higher versions of beam have other conflicts, unfortunately unsolvable with other providers (I think requests). We have to limit it to 2.23
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on a change in pull request #12946: Fixed broken dependencies for apache-beam.
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #12946:
URL: https://github.com/apache/airflow/pull/12946#discussion_r539253206
##########
File path: setup.cfg
##########
@@ -116,10 +117,12 @@ install_requires =
markdown>=2.5.2, <4.0
markupsafe>=1.1.1, <2.0
marshmallow-oneofschema>=2.0.1
+ mock<3.0.0 # required to keep apache-beam happy
pandas>=0.17.1, <2.0
pendulum~=2.0
pep562~=1.0;python_version<"3.7"
psutil>=4.2.0, <6.0.0
+ pyarrow<0.18.0
Review comment:
This is quite old, I'm not sure we _want_ this globally.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #12946: Fixed broken dependencies for apache-beam.
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #12946:
URL: https://github.com/apache/airflow/pull/12946#discussion_r539278151
##########
File path: setup.cfg
##########
@@ -116,10 +117,12 @@ install_requires =
markdown>=2.5.2, <4.0
markupsafe>=1.1.1, <2.0
marshmallow-oneofschema>=2.0.1
+ mock<3.0.0 # required to keep apache-beam happy
pandas>=0.17.1, <2.0
pendulum~=2.0
pep562~=1.0;python_version<"3.7"
psutil>=4.2.0, <6.0.0
+ pyarrow<0.18.0
Review comment:
Yep. All three are NOT transitive dependencies of airflow without extras, so It looks like we can move them where they belong. Working on it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #12946: Fixed broken dependencies for apache-beam.
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #12946:
URL: https://github.com/apache/airflow/pull/12946#issuecomment-741730239
Got it even more stable and likely even works with 20.3.1 (not even master)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #12946: Fixed broken dependencies for apache-beam.
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #12946:
URL: https://github.com/apache/airflow/pull/12946#issuecomment-741864784
[The Workflow run](https://github.com/apache/airflow/actions/runs/410667257) is cancelling this PR. Building images for the PR has failed. Follow the the workflow link to check the reason.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #12946: Fixed broken dependencies for apache-beam.
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #12946:
URL: https://github.com/apache/airflow/pull/12946#issuecomment-753415376
Need to wait for snowflake connector
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #12946: Fixed broken dependencies for apache-beam.
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #12946:
URL: https://github.com/apache/airflow/pull/12946#issuecomment-741865217
[The Workflow run](https://github.com/apache/airflow/actions/runs/410927702) is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] turbaszek edited a comment on pull request #12946: Fixed broken dependencies for apache-beam.
Posted by GitBox <gi...@apache.org>.
turbaszek edited a comment on pull request #12946:
URL: https://github.com/apache/airflow/pull/12946#issuecomment-741735924
> tiny subset of our users.
Depends how we count, Google Dataflow operators are ones of most commonly used from Google provider.
@potiuk have you tried to upgrade beam? I think latest 2.X version has higher reuirements
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #12946: Fixed broken dependencies for apache-beam.
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #12946:
URL: https://github.com/apache/airflow/pull/12946#issuecomment-741866294
[The Workflow run](https://github.com/apache/airflow/actions/runs/410938664) is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #12946: Fixed broken dependencies for apache-beam.
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #12946:
URL: https://github.com/apache/airflow/pull/12946#issuecomment-742686177
[The Workflow run](https://github.com/apache/airflow/actions/runs/413466811) is cancelling this PR. Building images for the PR has failed. Follow the the workflow link to check the reason.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on a change in pull request #12946: Fixed broken dependencies for apache-beam.
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #12946:
URL: https://github.com/apache/airflow/pull/12946#discussion_r539253487
##########
File path: setup.cfg
##########
@@ -116,10 +117,12 @@ install_requires =
markdown>=2.5.2, <4.0
markupsafe>=1.1.1, <2.0
marshmallow-oneofschema>=2.0.1
+ mock<3.0.0 # required to keep apache-beam happy
pandas>=0.17.1, <2.0
pendulum~=2.0
pep562~=1.0;python_version<"3.7"
psutil>=4.2.0, <6.0.0
+ pyarrow<0.18.0
Review comment:
(Pyarrow is now at 2.0.0 for instance.)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #12946: Fixed broken dependencies for apache-beam.
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #12946:
URL: https://github.com/apache/airflow/pull/12946#issuecomment-741866915
[The Workflow run](https://github.com/apache/airflow/actions/runs/410942837) is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed pull request #12946: Fixed broken dependencies for apache-beam.
Posted by GitBox <gi...@apache.org>.
potiuk closed pull request #12946:
URL: https://github.com/apache/airflow/pull/12946
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #12946: Fixed broken dependencies for apache-beam.
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #12946:
URL: https://github.com/apache/airflow/pull/12946#discussion_r539278488
##########
File path: setup.cfg
##########
@@ -116,10 +117,12 @@ install_requires =
markdown>=2.5.2, <4.0
markupsafe>=1.1.1, <2.0
marshmallow-oneofschema>=2.0.1
+ mock<3.0.0 # required to keep apache-beam happy
pandas>=0.17.1, <2.0
pendulum~=2.0
pep562~=1.0;python_version<"3.7"
psutil>=4.2.0, <6.0.0
+ pyarrow<0.18.0
Review comment:
The only one that will have to stay there is dill, but I think this is not a problem
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #12946: Fixed broken dependencies for apache-beam.
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #12946:
URL: https://github.com/apache/airflow/pull/12946#issuecomment-741867326
[The Workflow run](https://github.com/apache/airflow/actions/runs/410944571) is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #12946: Fixed broken dependencies for apache-beam.
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #12946:
URL: https://github.com/apache/airflow/pull/12946#issuecomment-741757417
Confirmed. It works when I move httplib/pyarrow/mock limitation to beam extr @ashb !
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on pull request #12946: Fixed broken dependencies for apache-beam.
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on pull request #12946:
URL: https://github.com/apache/airflow/pull/12946#issuecomment-741757417
Confirmed. It works when I move httplib/pyarrow/mock limitations to beam extra @ashb !
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #12946: Fixed broken dependencies for apache-beam.
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #12946:
URL: https://github.com/apache/airflow/pull/12946#discussion_r539269102
##########
File path: setup.cfg
##########
@@ -116,10 +117,12 @@ install_requires =
markdown>=2.5.2, <4.0
markupsafe>=1.1.1, <2.0
marshmallow-oneofschema>=2.0.1
+ mock<3.0.0 # required to keep apache-beam happy
pandas>=0.17.1, <2.0
pendulum~=2.0
pep562~=1.0;python_version<"3.7"
psutil>=4.2.0, <6.0.0
+ pyarrow<0.18.0
Review comment:
I only need to constraint deps in "install_requires" if they are already there transitively. I "eagerly" added them here to check if it works but likely httplib can be moved out
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] turbaszek commented on pull request #12946: Fixed broken dependencies for apache-beam.
Posted by GitBox <gi...@apache.org>.
turbaszek commented on pull request #12946:
URL: https://github.com/apache/airflow/pull/12946#issuecomment-741735924
> tiny subset of our users.
Depends how we count, Google Dataflow operators are ones of most commonly used from Google provider.
@potiuk have you try to upgrade beam? I think latest 2.X version has higher reuirements
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #12946: Fixed broken dependencies for apache-beam.
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #12946:
URL: https://github.com/apache/airflow/pull/12946#issuecomment-741759106
So our tests and constraints will be running old versions of those libs, but this is not at all a problem. Constraints will also contain those, but this is not a problem as well, maybe we need to add few words about "users are free to upgrade the libraries as they see fit after installation" .
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org