You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/12/05 17:33:54 UTC
[GitHub] [airflow] potiuk opened a new issue #12838: PIP 20.3 breaks Airflow installation!
potiuk opened a new issue #12838:
URL: https://github.com/apache/airflow/issues/12838
I am adding this issue to keep track of the on-going problems with new PIP 20.3 released 30th of November.
There are multiple issues with the new PIP that makes it breaks with Airflow's dependency set.
The first blocking issue is https://github.com/pypa/pip/issues/9203 but even when it is solved we already know we are affected by a few others:
* https://github.com/pypa/pip/issues/9217
* https://github.com/pypa/pip/issues/9215
* https://github.com/pypa/pip/issues/9187
* https://github.com/pypa/pip/issues/9223
We've raised the issue to the PIP team and they struggle with fixing a number of other teething problems.
We keep fingers crossed that they will manage to fix the issues promptly and that they will not be overwhelmed with putting out the fire.
There is no resolution yet, so for the time being downgrading PIP to `20.2.4` version is the best thing we can do:
```
pip install --upgrade pip==20.2.4
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #12838: ❗ PIP 20.3 breaks Airflow installation❗
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #12838:
URL: https://github.com/apache/airflow/issues/12838#issuecomment-740937367
```
# Those packages are excluded because they break tests (downgrading mock) and they are
# not needed to run our test suite.
PACKAGES_EXCLUDED_FOR_CI = [
'apache-beam',
]
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #12838: ❗ PIP 20.3 breaks Airflow installation❗
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #12838:
URL: https://github.com/apache/airflow/issues/12838#issuecomment-745385299
Yep. Confirmed it works for 2.0.
I need to do a few more checks and verify 1.10.14 as well and I will close that one,
Thanks A LOT @pradyunsg -> just in time for 2.0.0 of Airflow ;).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] notatallshaw commented on issue #12838: ❗ PIP 20.3 breaks Airflow installation❗
Posted by GitBox <gi...@apache.org>.
notatallshaw commented on issue #12838:
URL: https://github.com/apache/airflow/issues/12838#issuecomment-745049883
@pradyunsg I'm not on the Airflow team and I don't have as deep of an understanding as @potiuk but I gave installing Airflow 1.10.14 with all dependencies using the new resolver with pip 20.3.2.
I'm not sure how much is Airflow fixes and how much is 20.3.2s improvements but I am able to successfully run `pip install apache-airflow[all]` with no errors 😄. Thanks to both teams!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #12838: ❗ PIP 20.3 breaks Airflow installation❗
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #12838:
URL: https://github.com/apache/airflow/issues/12838#issuecomment-744717680
Sorry - I've been busy identifiying and testing some issue with Airlow 2.0RC2 ( which lead to RC3 sent today). I will take a closer look tomorrow!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #12838: PIP 20.3 might break Airflow installation
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #12838:
URL: https://github.com/apache/airflow/issues/12838#issuecomment-772684986
> Hello, does the installation fail with the latest version of pip 21.0.1?
Can you please try and tell us ?
We did not have time to make some checks with it - it is likely that main problems have been solved already.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #12838: ❗ PIP 20.3 breaks Airflow installation❗
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #12838:
URL: https://github.com/apache/airflow/issues/12838#issuecomment-740937722
> Which version and which constraints you are comparing?
I am building images on Github Action.
Here is the report:
https://github.com/mik-laj/airflow/runs/1519002616?check_suite_focus=true
> And which installation combination you are testing (which extras etc.?)
All extras required by CI.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #12838: ❗ PIP 20.3 breaks Airflow installation❗
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #12838:
URL: https://github.com/apache/airflow/issues/12838#issuecomment-740916028
Which version and which constraints you are comparing?
The constraints are automatically generated after installing the requirements using pip 20.2.4 do we have any error
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #12838: ❗ PIP 20.3 breaks Airflow installation❗
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #12838:
URL: https://github.com/apache/airflow/issues/12838#issuecomment-740925183
> This piece of code looks interesting to me. Should we also add a similar check to our project?
> https://github.com/apache/beam/blob/545db7386b69eb3c61690172c575dc025d91cca7/sdks/python/setup.py#L99-L107
From what I know PIP does not show those warnings without --verbose commands (And then it is lost in a sea of other messages). But if you would like to test - it feel free. There are some hacks people are doing like guessing terminal and writing directly to it (similarly what I do with pre-commit) but this is rather dangerous.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #12838: ❗ PIP 20.3 breaks Airflow installation❗
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #12838:
URL: https://github.com/apache/airflow/issues/12838#issuecomment-745357007
I also had the "dependency solving" session anda just discussed witth PIP team and experimented a bit and it seems we managed to pin-point the PIP 20.2.4 bug that generated bad pyarrow dependency. I updated it manually and seems that we are able to make it works for 2.0 as well. :crossed_fingers: for quick 20.3.3 release (20.3.2 was considered bad and removed in the meantime)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaustubhharapanahalli commented on issue #12838: PIP 20.3 might break Airflow installation
Posted by GitBox <gi...@apache.org>.
kaustubhharapanahalli commented on issue #12838:
URL: https://github.com/apache/airflow/issues/12838#issuecomment-772957808
> Can you please try and tell us ?
>
> We did not have time to make some checks with it - it is likely that main problems have been solved already and improve upon.
Hi @potiuk I ran this command `pip install apache-airflow`. And it worked. My setup:
- OS: Windows10
- Python version = 3.9.1
- pip version = 21.0.1
Is there any test that I can run to confirm proper installation?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #12838: ❗ PIP 20.3 breaks Airflow installation❗
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #12838:
URL: https://github.com/apache/airflow/issues/12838#issuecomment-745357007
I also had the "dependency solving" session anda just discussed witth PIP team and experimented a bit and it seems we managed to pin-point the PIP 20.2.4 bug that generated bad pyarrow dependency. I updated it manually and seems that we are able to make it works for 2.0 as well. :crossed_fingers: for quick 20.3.2 release.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #12838: ❗ PIP 20.3 breaks Airflow installation❗
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #12838:
URL: https://github.com/apache/airflow/issues/12838#issuecomment-739490505
FYI @vikramkoka (and @eladkal @paolaperaza) the "upgrade to newer dependencies" and "full tests needed" are special labels that can be added to PRs to change the scope of PR builds:
See: https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#step-4-prepare-pr
* "upgrade to newer dependencies" causes an automated upgrade to latest dependencies using the "eager" upgrade strategy:
* "full tests needed" cause that full "matrix" of tests is run for our tests rather than one combination.
Maybe we need some special prefixes for those to distinguish from "regular" labels. If we decide to do that we will have to update our workflows to handle the new names.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] pradyunsg commented on issue #12838: ❗ PIP 20.3 breaks Airflow installation❗
Posted by GitBox <gi...@apache.org>.
pradyunsg commented on issue #12838:
URL: https://github.com/apache/airflow/issues/12838#issuecomment-745380773
20.3.3 is out. I _think_ it solves all the issues that broke Airflow's installation mechanisms. If @potiuk can confirm that, I'm guessing we can go ahead and close this. ;)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #12838: ❗ PIP 20.3 breaks Airflow installation❗
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #12838:
URL: https://github.com/apache/airflow/issues/12838#issuecomment-740916028
Which version and which constraints you are comparing?
The constraints are automatically generated after installing the requirements using pip 20.2.4
Do you have any error reported by PIP check?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #12838: ❗ PIP 20.3 breaks Airflow installation❗
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #12838:
URL: https://github.com/apache/airflow/issues/12838#issuecomment-740987886
Looks like the problem is that somehow the new resolver adds extras to our transitive dependencies where we did not ask for them (or maybe they were dropped by the previous resolver):
In 2.20.4 I do not see `google-cloud-bigquery` has any extra (it is transitive dependency of pandas-gbq) but in the new PIp somehow we have `google-cloud-bigquery[bqstorage,pandas]
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #12838: ❗ PIP 20.3 breaks Airflow installation❗
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #12838:
URL: https://github.com/apache/airflow/issues/12838#issuecomment-740823463
According to this PR, this entry is not needed.
https://github.com/apache/airflow/pull/12683
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #12838: ❗ PIP 20.3 breaks Airflow installation❗
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #12838:
URL: https://github.com/apache/airflow/issues/12838#issuecomment-740771832
Pypi may have problems installing the master version because we have references to an unreleased package - `apache-airflow-providers-telegram`.
I haven't tested the extras below. They may or may not work.
```
# all
# all_dbs
# aws
# devel
# devel_all
# devel_ci
# devel_hadoop
# gcp_api
# google_auth
apache.hive
apache.webhdfs
gcp
hive
kerberos
mysql
odbc
s3
telegram
webhdfs
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on issue #12838: PIP 20.3 might break Airflow installation
Posted by GitBox <gi...@apache.org>.
ashb commented on issue #12838:
URL: https://github.com/apache/airflow/issues/12838#issuecomment-807598325
Closing as a new version of pip has been released.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] notatallshaw commented on issue #12838: ❗ PIP 20.3 breaks Airflow installation❗
Posted by GitBox <gi...@apache.org>.
notatallshaw commented on issue #12838:
URL: https://github.com/apache/airflow/issues/12838#issuecomment-741904848
>
>
> Looks like the problem is that somehow the new resolver adds extras to our transitive dependencies where we did not ask for them (or maybe they were dropped by the previous resolver):
> In 2.20.4 I do not see `google-cloud-bigquery` has any extra (it is transitive dependency of `pandas-gbq`) but in the new PIp somehow we have `google-cloud-bigquery[bqstorage,pandas]
Is this related to this comment here: https://github.com/pypa/pip/issues/8785#issuecomment-678885871 ?
Note that there are 2 different issues being solved in the linked github thread, the one that is described in the comment (not solved yet) and the workaround using "--no-deps" (solved recently on master)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] pradyunsg commented on issue #12838: ❗ PIP 20.3 breaks Airflow installation❗
Posted by GitBox <gi...@apache.org>.
pradyunsg commented on issue #12838:
URL: https://github.com/apache/airflow/issues/12838#issuecomment-744071490
Ahoy! If you have been able to test against pip's current master branch, let me know if there's any outstanding issues. :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #12838: ❗ PIP 20.3 breaks Airflow installation❗
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #12838:
URL: https://github.com/apache/airflow/issues/12838#issuecomment-740786850
I updated the pip version to the newest master and trigger the build on my CI. Cross fingers. 🤞🏻
https://github.com/mik-laj/airflow/commit/4bb280b1e8218414f18b7a97442e1a86d9ea5ac6
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] vikramkoka commented on issue #12838: ❗ PIP 20.3 breaks Airflow installation❗
Posted by GitBox <gi...@apache.org>.
vikramkoka commented on issue #12838:
URL: https://github.com/apache/airflow/issues/12838#issuecomment-739551596
> FYI @vikramkoka (and @eladkal @paolaperaza) the "upgrade to newer dependencies" and "full tests needed" are special labels that can be added to PRs to change the scope of PR builds:
>
> See: https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#step-4-prepare-pr
>
> * "upgrade to newer dependencies" causes an automated upgrade to latest dependencies using the "eager" upgrade strategy:
> * "full tests needed" cause that full "matrix" of tests is run for our tests rather than one combination.
>
> Maybe we need some special prefixes for those to distinguish from "regular" labels. If we decide to do that, we will have to update our workflows to handle the new names.
Sorry about that @potiuk . I did not know that. Will avoid using this "upgrade to new dependencies" label in the future.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #12838: ❗ PIP 20.3 breaks Airflow installation❗
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #12838:
URL: https://github.com/apache/airflow/issues/12838#issuecomment-740941867
All my works are available on Github: https://github.com/apache/airflow/compare/master...mik-laj:master
If you are interested, you can view the history of changes and bugs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #12838: ❗ PIP 20.3 breaks Airflow installation❗
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #12838:
URL: https://github.com/apache/airflow/issues/12838#issuecomment-739553218
> Sorry about that @potiuk . I did not know that. Will avoid using this "upgrade to new dependencies" label in the future.
No problem :). that was mainly to explain what they are and probably ad them as exclusions in the description of the triage process.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #12838: ❗ PIP 20.3 breaks Airflow installation❗
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #12838:
URL: https://github.com/apache/airflow/issues/12838#issuecomment-740928972
When I run (pip 20.2.4)
```
pip install ".[devel_ci]" --constraint https://raw.githubusercontent.com/apache/airflow/constraints-master/constraints-3.6.txt
root@e57e08a8d4bf:/opt/airflow# pip check
No broken requirements found.
root@e57e08a8d4bf:/opt/airflow# pip freeze | grep pyarrow
pyarrow==0.17.1
```
No conflicts reported by pip.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #12838: ❗ PIP 20.3 breaks Airflow installation❗
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #12838:
URL: https://github.com/apache/airflow/issues/12838#issuecomment-740652881
I confirmed that one problem was solved.
Now it is possible to install `.[google]`, but `.[google, devel]` still doesn't work.
https://github.com/pypa/pip/pull/9241
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #12838: ❗ PIP 20.3 breaks Airflow installation❗
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #12838:
URL: https://github.com/apache/airflow/issues/12838#issuecomment-740949991
Here you can see example run of the very same PIP install command: https://github.com/apache/airflow/runs/1519600008?check_suite_focus=true#step:9:1149
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] pradyunsg commented on issue #12838: ❗ PIP 20.3 breaks Airflow installation❗
Posted by GitBox <gi...@apache.org>.
pradyunsg commented on issue #12838:
URL: https://github.com/apache/airflow/issues/12838#issuecomment-745003277
Well, pip's master branch is now 20.3.2, so... test against that! :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #12838: ❗ PIP 20.3 breaks Airflow installation❗
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #12838:
URL: https://github.com/apache/airflow/issues/12838#issuecomment-740768425
I tried to install almost all extra packages with the above patch and it worked. I have the impression that when a new version of pip is released the problem will not occur or it will be marginal.
| Extra | Status |
|-------------------|----------|
| amazon | SUCCESS |
| apache.atlas | SUCCESS |
| apache.beam | SUCCESS |
| apache.cassandra | SUCCESS |
| apache.druid | SUCCESS |
| apache.hdfs | SUCCESS |
| apache.kylin | SUCCESS |
| apache.livy | SUCCESS |
| apache.pig | SUCCESS |
| apache.pinot | SUCCESS |
| apache.spark | SUCCESS |
| apache.sqoop | SUCCESS |
| async | SUCCESS |
| atlas | SUCCESS |
| azure | SUCCESS |
| cassandra | SUCCESS |
| celery | SUCCESS |
| cgroups | SUCCESS |
| cloudant | SUCCESS |
| cncf.kubernetes | SUCCESS |
| crypto | SUCCESS |
| dask | SUCCESS |
| databricks | SUCCESS |
| datadog | SUCCESS |
| dingding | SUCCESS |
| discord | SUCCESS |
| doc | SUCCESS |
| docker | SUCCESS |
| druid | SUCCESS |
| elasticsearch | SUCCESS |
| exasol | SUCCESS |
| facebook | SUCCESS |
| ftp | SUCCESS |
| gcp | SUCCESS |
| github_enterprise | SUCCESS |
| google | SUCCESS |
| grpc | SUCCESS |
| hashicorp | SUCCESS |
| hdfs | SUCCESS |
| http | SUCCESS |
| imap | SUCCESS |
| jdbc | SUCCESS |
| jenkins | SUCCESS |
| jira | SUCCESS |
| kubernetes | SUCCESS |
| ldap | SUCCESS |
| microsoft.azure | SUCCESS |
| microsoft.mssql | SUCCESS |
| microsoft.winrm | SUCCESS |
| mongo | SUCCESS |
| mssql | SUCCESS |
| openfaas | SUCCESS |
| opsgenie | SUCCESS |
| oracle | SUCCESS |
| pagerduty | SUCCESS |
| papermill | SUCCESS |
| password | SUCCESS |
| pinot | SUCCESS |
| plexus | SUCCESS |
| postgres | SUCCESS |
| presto | SUCCESS |
| qds | SUCCESS |
| qubole | SUCCESS |
| rabbitmq | SUCCESS |
| redis | SUCCESS |
| s3 | SUCCESS |
| salesforce | SUCCESS |
| samba | SUCCESS |
| segment | SUCCESS |
| sendgrid | SUCCESS |
| sentry | SUCCESS |
| sftp | SUCCESS |
| singularity | SUCCESS |
| slack | SUCCESS |
| snowflake | SUCCESS |
| spark | SUCCESS |
| sqlite | SUCCESS |
| ssh | SUCCESS |
| statsd | SUCCESS |
| tableau | SUCCESS |
| vertica | SUCCESS |
| virtualenv | SUCCESS |
| winrm | SUCCESS |
| yandex | SUCCESS |
| zendesk | SUCCESS |
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #12838: ❗ PIP 20.3 breaks Airflow installation❗
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #12838:
URL: https://github.com/apache/airflow/issues/12838#issuecomment-740987886
Looks like the problem is that somehow the new resolver adds extras to our transitive dependencies where we did not ask for them (or maybe they were dropped by the previous resolver):
In 2.20.4 I do not see `google-cloud-bigquery` has any extra (it is transitive dependency of `pandas-gbq`) but in the new PIp somehow we have `google-cloud-bigquery[bqstorage,pandas]
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] pradyunsg commented on issue #12838: PIP 20.3 might break Airflow installation
Posted by GitBox <gi...@apache.org>.
pradyunsg commented on issue #12838:
URL: https://github.com/apache/airflow/issues/12838#issuecomment-782578367
Closing based on earlier comments:
> I'm guessing we can go ahead and close this. ;)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #12838: ❗ PIP 20.3 breaks Airflow installation❗
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #12838:
URL: https://github.com/apache/airflow/issues/12838#issuecomment-739490505
FYI @vikramkoka (and @eladkal @paolaperaza) the "upgrade to newer dependencies" and "full tests needed" are special labels that can be added to PRs to change the scope of PR builds:
See: https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#step-4-prepare-pr
* "upgrade to newer dependencies" causes an automated upgrade to latest dependencies using the "eager" upgrade strategy:
* "full tests needed" cause that full "matrix" of tests is run for our tests rather than one combination.
Maybe we need some special prefixes for those to distinguish from "regular" labels. If we decide to do that, we will have to update our workflows to handle the new names.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] vikramkoka commented on issue #12838: ❗ PIP 20.3 breaks Airflow installation❗
Posted by GitBox <gi...@apache.org>.
vikramkoka commented on issue #12838:
URL: https://github.com/apache/airflow/issues/12838#issuecomment-739425513
Thanks @potiuk I am really glad you added this to the installation instructions too!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr edited a comment on issue #12838: PIP 20.3 might break Airflow installation
Posted by GitBox <gi...@apache.org>.
uranusjr edited a comment on issue #12838:
URL: https://github.com/apache/airflow/issues/12838#issuecomment-807703878
FWIW I tried `pip install apache-airflow[all]` and it works correctly. Or rather, it correctly did not work due to #14994.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on issue #12838: PIP 20.3 might break Airflow installation
Posted by GitBox <gi...@apache.org>.
uranusjr commented on issue #12838:
URL: https://github.com/apache/airflow/issues/12838#issuecomment-807703878
FWIW I tried `pip install apache-airflow[all]` and it works correctly. Or rather, it corretly did not work due to #14994.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #12838: ❗ PIP 20.3 breaks Airflow installation❗
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #12838:
URL: https://github.com/apache/airflow/issues/12838#issuecomment-740822619
I found the source of the problem. We have a conflicting constraints entry.
https://github.com/apache/airflow/blob/fbd525ac1dfa06e0e3eb9ea6ce6013c08e4f0f1f/constraints-3.8.txt#L294
> google-cloud-bigquery[bqstorage,pandas] 2.4.0 depends on pyarrow<3.0dev and >=1.0.0
Without this entry, the Airflow installation works.
https://github.com/mik-laj/airflow/runs/1519148881?check_suite_focus=true
https://github.com/mik-laj/airflow/commit/b573f2a493de82b083cc50e281f0b631a42b5c32
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #12838: ❗ PIP 20.3 breaks Airflow installation❗
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #12838:
URL: https://github.com/apache/airflow/issues/12838#issuecomment-740916028
Which version and which constraints you are comparing?
The constraints are automatically generated after installing the requirements using pip 20.2.4
Do you have any error reported by PIP check?
And which installation combination you are testing (which extras etc.?)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] pradyunsg commented on issue #12838: PIP 20.3 might break Airflow installation
Posted by GitBox <gi...@apache.org>.
pradyunsg commented on issue #12838:
URL: https://github.com/apache/airflow/issues/12838#issuecomment-782578440
Oh wait, this isn’t pypa/pip. Nvm me. I blame the lack of a breakfast. :P
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #12838: ❗ PIP 20.3 breaks Airflow installation❗
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #12838:
URL: https://github.com/apache/airflow/issues/12838#issuecomment-740939836
> If you are running those tests with 'apache-beam' then this is the reason we have conflict - but apache-beam is explicitly excluded in all the tests we do because we know it has unsolvable dependency conflicts.
Here is the error that I'm negotiating with.
https://github.com/mik-laj/airflow/runs/1519504904
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #12838: ❗ PIP 20.3 breaks Airflow installation❗
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #12838:
URL: https://github.com/apache/airflow/issues/12838#issuecomment-740775574
Cool. Good job @mik-laj !. I will take a look tomorrow as well and try to run all the automation we run on CI. Until this gets released in 20.3.2 we still keep the warning in our docs but this looks very promising !
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #12838: ❗ PIP 20.3 breaks Airflow installation❗
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #12838:
URL: https://github.com/apache/airflow/issues/12838#issuecomment-740951637
I will take much closer look later this week once we release 2.0rc, and once they hopefully release 20.3.2. But my hopes are much less than even an hour before.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #12838: ❗ PIP 20.3 breaks Airflow installation❗
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #12838:
URL: https://github.com/apache/airflow/issues/12838#issuecomment-740933822
If you are running those tests with 'apache-beam' then this is the reason we have conflict - but `apache-beam` is explicitly excluded in all the tests we do because we know it has unsolvable dependency conflicts.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaustubhharapanahalli commented on issue #12838: PIP 20.3 might break Airflow installation
Posted by GitBox <gi...@apache.org>.
kaustubhharapanahalli commented on issue #12838:
URL: https://github.com/apache/airflow/issues/12838#issuecomment-772465950
Hello, does the installation fail with the latest version of pip 21.0.1?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #12838: ❗ PIP 20.3 breaks Airflow installation❗
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #12838:
URL: https://github.com/apache/airflow/issues/12838#issuecomment-741909281
> Is this related to this comment here: [pypa/pip#8785 (comment)](https://github.com/pypa/pip/issues/8785#issuecomment-678885871) ?
>
Indeed. it looks like this might be the same root cause.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb closed issue #12838: PIP 20.3 might break Airflow installation
Posted by GitBox <gi...@apache.org>.
ashb closed issue #12838:
URL: https://github.com/apache/airflow/issues/12838
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #12838: ❗ PIP 20.3 breaks Airflow installation❗
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #12838:
URL: https://github.com/apache/airflow/issues/12838#issuecomment-740944825
So my hopes are much less now.
This looks like problem with new PIP (or a problem with old PIP that the new command reveals).
The same command with PIP 2.20.4 works flawlessly and shows no conflict.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #12838: PIP 20.3 might break Airflow installation
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #12838:
URL: https://github.com/apache/airflow/issues/12838#issuecomment-772684986
> Hello, does the installation fail with the latest version of pip 21.0.1?
Can you please try and tell us ?
We did not have time to make some checks with it - it is likely that main problems have been solved already and improve upon.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #12838: ❗ PIP 20.3 breaks Airflow installation❗
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #12838:
URL: https://github.com/apache/airflow/issues/12838#issuecomment-740912491
This piece of code looks interesting to me. Should we also add a similar check to our project?
https://github.com/apache/beam/blob/545db7386b69eb3c61690172c575dc025d91cca7/sdks/python/setup.py#L99-L107
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org