You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/04/20 21:40:42 UTC

[GitHub] [airflow] zachliu opened a new issue #15463: Inconsistency between the setup.py and the constraints file

zachliu opened a new issue #15463:
URL: https://github.com/apache/airflow/issues/15463


   **Apache Airflow version**: 2.0.2
   
   **What happened**:
   
   Airflow's 2.0.2's [constraints file](https://raw.githubusercontent.com/apache/airflow/constraints-2.0.2/constraints-3.8.txt) has used newer `oauthlib==3.1.0` and `request-oauthlib==1.3.0` than 2.0.1's [constraints file](https://raw.githubusercontent.com/apache/airflow/constraints-2.0.1/constraints-3.8.txt)
   
   However both 2.0.2's [setup.py](https://github.com/apache/airflow/blob/10023fdd65fa78033e7125d3d8103b63c127056e/setup.py#L282-L286) and 2.0.1's [setup.py](https://github.com/apache/airflow/blob/beb8af5ac6c438c29e2c186145115fb1334a3735/setup.py#L273) don't allow these new versions
   
   Image build with `google_auth` being an "extra"  will fail if using `pip==21.0.1` **without** the `--use-deprecated=legacy-resolver` flag.
   
   Another option is to use `pip==20.2.4`.
   
   **What you expected to happen**:
   
   The package versions in `setup.py` and `constraints-3.8.txt` should be consistent with each other.
   
   <!-- What do you think went wrong? -->
   
   **How to reproduce it**:
   
   `docker build` with the following in the `Dockerfile`:
   ```
   pip install apache-airflow[password,celery,redis,postgres,hive,jdbc,mysql,statsd,ssh,google_auth]==2.0.2 \
       --constraint https://raw.githubusercontent.com/apache/airflow/constraints-2.0.2/constraints-3.8.txt
   ```
   image build failed with
   
   ```
   ERROR: Could not find a version that satisfies the requirement oauthlib!=2.0.3,!=2.0.4,!=2.0.5,<3.0.0,>=1.1.2; extra == "google_auth" (from apache-airflow[celery,google-auth,hive,jdbc,mysql,password,postgres,redis,ssh,statsd])
   ERROR: No matching distribution found for oauthlib!=2.0.3,!=2.0.4,!=2.0.5,<3.0.0,>=1.1.2; extra == "google_auth"
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] uranusjr commented on issue #15463: Inconsistency between the setup.py and the constraints file

Posted by GitBox <gi...@apache.org>.
uranusjr commented on issue #15463:
URL: https://github.com/apache/airflow/issues/15463#issuecomment-823696425


   Your analysis is accurate.
   
   > Another option is to use `pip==20.2.4`.
   
   This allows the packages to be installed, but then Airflow will just break because flask-oauthlib is incompatible with oauth 3.1.0 an request-oauthlib 1.3.0.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk edited a comment on issue #15463: Inconsistency between the setup.py and the constraints file

Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #15463:
URL: https://github.com/apache/airflow/issues/15463#issuecomment-825747630


   > i'm confused, looks like `google-cloud-bigquery-storage` and `pyarrow` are both "children" of `bqstorage` right?
   > https://github.com/googleapis/python-bigquery/blob/c75e9cda85abdad440d87298b30e9d5291676064/setup.py#L39-L49
   
   Right. It is likely the "sibling" of the dependency that is printed wrongly, not the parent


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] zachliu commented on issue #15463: Inconsistency between the setup.py and the constraints file

Posted by GitBox <gi...@apache.org>.
zachliu commented on issue #15463:
URL: https://github.com/apache/airflow/issues/15463#issuecomment-825740098


   both errors are valid right? pip somehow changes evaluation order with the verbose option :thinking: 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] zachliu commented on issue #15463: Inconsistency between the setup.py and the constraints file

Posted by GitBox <gi...@apache.org>.
zachliu commented on issue #15463:
URL: https://github.com/apache/airflow/issues/15463#issuecomment-825026261


   @potiuk out of curiosity, i'm trying to wrap my head around the 2nd problem and that example. let's see if i understand it correctly:
   
   In Airflow 2.0.2's Python 3.8 [constraints file](https://raw.githubusercontent.com/apache/airflow/constraints-2.0.2/constraints-3.8.txt) there are:
   * google-cloud-bigquery-storage==2.4.0
   * google-cloud-bigquery==1.28.0
   * pandas-gbq==0.14.1
   
   In pandas-gbq 0.14.1's [setup.py](https://github.com/pydata/pandas-gbq/blob/ac2d2fe4ac0025109f8df3723e3f03a337face94/setup.py#L25) there is:
   * google-cloud-bigquery[bqstorage,pandas]>=1.11.1,<3.0.0dev
   
   and the version 1.28.0 in our constraints file satisfies this requirement :heavy_check_mark: 
   
   In google-cloud-bigquery 1.28.0's [setup.py](https://github.com/googleapis/python-bigquery/blob/c75e9cda85abdad440d87298b30e9d5291676064/setup.py#L40) there is:
   * google-cloud-bigquery-storage >= 1.0.0, <2.0.0dev
   
   which doesn't match with Airflow's constraint on `google-cloud-bigquery-storage` :x: and causes this error when using `pip 21.0.1`:
   ```
   pip install apache-airflow[google]==2.0.2 --constraint https://raw.githubusercontent.com/apache/airflow/constraints-2.0.2/constraints-3.8.txt
   ...
   ERROR: Could not find a version that satisfies the requirement google-cloud-bigquery-storage<2.0.0dev,>=1.0.0; extra == "bqstorage" (from google-cloud-bigquery[bqstorage,pandas])
   ERROR: No matching distribution found for google-cloud-bigquery-storage<2.0.0dev,>=1.0.0; extra == "bqstorage"
   ```
   
   and i guess for `google-cloud-bigquery-storage` we are unable to use any versions below 2.0.0 for some reason?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk commented on issue #15463: Inconsistency between the setup.py and the constraints file

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #15463:
URL: https://github.com/apache/airflow/issues/15463#issuecomment-825760936


   I am not 100% sure of what's going on though - and while It awfully looks like this - I'd love at least some way of debugging it (and with -vvvv I do not even get a single `google-cloud-bigquery-storage` reference, so maybe this is not it - maybe indeed it's just a different sequence, but I would at least like to know what's causing the problem :).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk closed issue #15463: Inconsistency between the setup.py and the constraints file

Posted by GitBox <gi...@apache.org>.
potiuk closed issue #15463:
URL: https://github.com/apache/airflow/issues/15463


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] uranusjr edited a comment on issue #15463: Inconsistency between the setup.py and the constraints file

Posted by GitBox <gi...@apache.org>.
uranusjr edited a comment on issue #15463:
URL: https://github.com/apache/airflow/issues/15463#issuecomment-823696425


   Your analysis is accurate. We need to fix the constraints file.
   
   > Another option is to use `pip==20.2.4`.
   
   This allows the packages to be installed, but then Airflow will just break because flask-oauthlib is incompatible with oauth 3.1.0 an request-oauthlib 1.3.0.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] zachliu commented on issue #15463: Inconsistency between the setup.py and the constraints file

Posted by GitBox <gi...@apache.org>.
zachliu commented on issue #15463:
URL: https://github.com/apache/airflow/issues/15463#issuecomment-825750458


   apologize... still confused :sweat_smile:  what about the conflict between
   ```
   google-cloud-bigquery-storage==2.4.0` --- from constraint
   ```
   and
   ```
   google-cloud-bigquery-storage >= 1.0.0, <2.0.0dev --- from bqstorage
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk commented on issue #15463: Inconsistency between the setup.py and the constraints file

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #15463:
URL: https://github.com/apache/airflow/issues/15463#issuecomment-824300116


   Just to explain the root cause of those problems and some complexities involved
   
   1) Root cause of the problem above:
   
   The original problem turned out to be a bug in recently changed mechanism of generating constraints. Fixed here:  https://github.com/apache/airflow/pull/15470 - the bug was, that when we prepared constraints for "production" release we use "pip-providers" version of constraints - we assume that we install all providers that are **currently** (at the time of constraint generation) available in PyPI and work out the version of constraints that will allow to install all of them. The problem was that when we calculated those, we did not include few extras that have no providers. 
   
   So rather than calculating the set of constraints that are "good" for `pip install apache-airflow[google_auth,ldap....] apache-airflow-providers-*` we calculated what is "good" for `pip install apache-airflow-providers-*`.  This is why oauhlib was bumped (wrongly).
   
   2) Now there are other problems that are more subtle, the  problem is on how PIP treats "extra" dependencies especially in transitive dependencies. Those are differently solved at "installation" time and differently "at rest" and differently when different PIP versions are used. This gets complex fast.
   
   Example (real that we have now)  : 
   
   `airflow[google] -> apache-airflow-providers-google -> pandas-gbq-> "google-cloud-bigquery[bqstorage,pandas]>=1.11.1,<3.0.0dev,!=2.4.*"` and then it gets funny... because what we have in `google-cloud-bigguery` in extras is: 
   
   ```
   `"bqstorage": [
           "google-cloud-bigquery-storage >= 2.0.0, <3.0.0dev",
   ```
   
   
   There are problems with that 
     * old pip does not complain about the "bqstorage"  limitation  - and will happily install  goog-le-coloud -bigiquery-storage  outside of the range
     * new pip > 20.3 does complain and will not install 
     * however when they are installed both pip's after installation when you run 'pip check` with non-matching google-cloud-biguery will show that everything is fine - because pandas-gbq -> google-cloud-bigquery[bqstorage,pandas] is transitive, only valid at installation time. It's not 'install-requires' 
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] uranusjr commented on issue #15463: Inconsistency between the setup.py and the constraints file

Posted by GitBox <gi...@apache.org>.
uranusjr commented on issue #15463:
URL: https://github.com/apache/airflow/issues/15463#issuecomment-824290669


   Python dependency conflict detection is complicated because you only see incompatibilites when the related code is actually accessed. Sometimes that’s easy to tell (e.g. when imports change), but more often it’s some function argument change or even semantic change, so there’s really no way to tell what is working and what is not without analysing call paths to the dependency. So if it works for you, way to go, but don’t count on it.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk edited a comment on issue #15463: Inconsistency between the setup.py and the constraints file

Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #15463:
URL: https://github.com/apache/airflow/issues/15463#issuecomment-825752577


   That's the thing. I cannot find `google-cloud-bigquery-storage >= 1.0.0, <2.0.0dev --- from bqstorage` anywhere in any of the packages. I have no idea where it comes from,
   
   Thats' why I think it is wrong error message.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] zachliu commented on issue #15463: Inconsistency between the setup.py and the constraints file

Posted by GitBox <gi...@apache.org>.
zachliu commented on issue #15463:
URL: https://github.com/apache/airflow/issues/15463#issuecomment-825849331


   so i was playing with `Poetry` and set 
   ```
   [tool.poetry.dependencies]
   python = "^3.8"
   apache-airflow = "2.0.2"
   apache-airflow-providers-google = "^2.2.0"
   ```
   in my `pyproject.toml`, then i installed everything without any errors. It turns out `google-cloud-bigquery` has been bumped to `2.6.1` just to make everything match up.
   ```
    • Installing pandas-gbq (0.14.1)
    • Installing google-cloud-bigquery (2.6.1)
    • Installing google-cloud-bigquery-storage (2.4.0)
   ```
   Here is the poetry output and lock file:
   https://gist.github.com/zachliu/cb8de028f5ce08c03d38f329b97794d9


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk commented on issue #15463: Inconsistency between the setup.py and the constraints file

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #15463:
URL: https://github.com/apache/airflow/issues/15463#issuecomment-826262336


   > lol, this culprit hides [deep](https://github.com/apache/beam/blob/4202efb5f01254e46fe421225a2ebbdba46da04d/sdks/python/setup.py#L192)
   
   It does. But with even more extras I think I managed to eat cake and have it too..
   
   The nice thing is that I am not impairing the whole "google" provider, or "apache.beam" provider for Airflow. They will install ok alongside each other and most of it will even work even if the 'gcp' extra from Bem is not installed (most of the functionality will work with the new google python packages, but not everything). But even if someone wants to have "conflicting" installation with old PIP, they will be able to:
   
   The "google" provider has an optional extra for beam already which I modified (we generate them automatically):
   
   ```
           extras_require={
               'amazon': ['apache-airflow-providers-amazon'],
               'apache.beam': ['apache-airflow-providers-apache-beam', 'apache-beam[gcp]'],
               'apache.cassandra': ['apache-airflow-providers-apache-cassandra'],
               ....
           },
   ```
   
   And likewise "apache.beam" provider:
   
   ```
           extras_require={'google': ['apache-airflow-providers-google', 'apache-beam[gcp]']},
   ```
   
   So if someone really wants to install all the mess with old PIP the will be able to do `pip install apache-airflow-providers-google['apache.beam']` or `pip install apache-airflow-providers-apache-beam['google']` and it will work (or not (?)) in exactly the same way it worked (or not (?)) before.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk edited a comment on issue #15463: Inconsistency between the setup.py and the constraints file

Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #15463:
URL: https://github.com/apache/airflow/issues/15463#issuecomment-825216717


   Happy to brainstorm/collaborate on fixing this one. I have never had time to go to the bottom of it - why PIP 21 is actually not able to downgrade bigquery-storage in this case. I think there is another dependency of 'google' providers (most likely a transitive one with another extra) that causes the bigquery-storage to be "limited" to >2.0.0. Unfortunately with transitive dependencies with extras pipdeptree and similar tools do not show the actual origin of limitation.
   
   If you look at 'pipdeptree` for example afterr installing airflow you will see that "google-cloud-bigqury-storage` is "top" level dependency. NOTHING requires it. Unfortuntely also PIP output will not tell you that (even when I tried verbose). Maybe there are some better tools we can use?, @uranusjr I see that you commented on related issue in PIP -> https://github.com/pypa/pip/issues/4824, so maybe you can help with that ? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] zachliu commented on issue #15463: Inconsistency between the setup.py and the constraints file

Posted by GitBox <gi...@apache.org>.
zachliu commented on issue #15463:
URL: https://github.com/apache/airflow/issues/15463#issuecomment-824253654


   > Your analysis is accurate. We need to fix the constraints file.
   > 
   > > Another option is to use `pip==20.2.4`.
   > 
   > This allows the packages to be installed, but then Airflow will just break because flask-oauthlib is incompatible with oauth 3.1.0 an request-oauthlib 1.3.0.
   
   @uranusjr actually i was going to ask about that. airflow should have failed due to the incompatibility but it didn't and i was really confused :confused:
   anyway, it doesn't matter now since the issue is solved


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] uranusjr commented on issue #15463: Inconsistency between the setup.py and the constraints file

Posted by GitBox <gi...@apache.org>.
uranusjr commented on issue #15463:
URL: https://github.com/apache/airflow/issues/15463#issuecomment-824363185


   > however when they are installed both pip's after installation when you run 'pip check` with non-matching google-cloud-biguery will show that everything is fine
   
   Just to clarify, “`pip check` will show that everything is fine” does not mean everything is actually fine. It still isn’t, but limitation in `pip check`’s design prevents to detection.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk commented on issue #15463: Inconsistency between the setup.py and the constraints file

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #15463:
URL: https://github.com/apache/airflow/issues/15463#issuecomment-826098727


   Interesting. I am trying it now today to see where the google-cloud-bigquery limitations come from in the constraints, but I agree - this is the root cause of the incompatibilities.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk commented on issue #15463: Inconsistency between the setup.py and the constraints file

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #15463:
URL: https://github.com/apache/airflow/issues/15463#issuecomment-825748815


   Issue created https://github.com/pypa/pip/issues/9823 @uranusjr -> would you mind taking a look maybe ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk edited a comment on issue #15463: Inconsistency between the setup.py and the constraints file

Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #15463:
URL: https://github.com/apache/airflow/issues/15463#issuecomment-825216717


   Happy to brainstorm/collaborate on fixing this one. I have never had time to go to the bottom of it - why PIP 21 is actually not able to downgrade bigquery-storage in this case. I think there is another dependency of 'google' providers (most likely a transitive one with another extra) that causes the bigquery-storage to be "limited" to >2.0.0. Unfortunately with transitive dependencies with extras pipdeptree and similar tools do not show the actual origin of limitation.
   
   If you look at `pipdeptree` for example after installing airflow you will see that "google-cloud-bigqury-storage` is "top" level dependency. NOTHING requires it. Unfortuntely also PIP output will not tell you that (even when I tried verbose). Maybe there are some better tools we can use?, @uranusjr I see that you commented on related issue in PIP -> https://github.com/pypa/pip/issues/4824, so maybe you can help with that ? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk commented on issue #15463: Inconsistency between the setup.py and the constraints file

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #15463:
URL: https://github.com/apache/airflow/issues/15463#issuecomment-826098727






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk edited a comment on issue #15463: Inconsistency between the setup.py and the constraints file

Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #15463:
URL: https://github.com/apache/airflow/issues/15463#issuecomment-825742921


   I think not. I believe we ONLY have problem with Pyarrow, and it is misreported. Looks like instead of actual dependency that is wrong (`pyarrow`), the sibling of that dependency (`google-cloud-bigquery-storage`) is printed out by PIP. Note that other than that - those are the very same limits which are problem (`<2.0.0dev,>=1.0.0; extra == "bqstorage"`). I think this is a bug in PIP.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] zachliu edited a comment on issue #15463: Inconsistency between the setup.py and the constraints file

Posted by GitBox <gi...@apache.org>.
zachliu edited a comment on issue #15463:
URL: https://github.com/apache/airflow/issues/15463#issuecomment-825849331


   so i was playing with `Poetry` and set 
   ```
   [tool.poetry.dependencies]
   python = "^3.8"
   apache-airflow = "2.0.2"
   apache-airflow-providers-google = "^2.2.0"
   ```
   in my `pyproject.toml`, then i installed everything without any errors. It turns out `google-cloud-bigquery` has been bumped to `2.6.1` just to make everything match up.
   ```
    • Installing pandas-gbq (0.14.1)
    • Installing google-cloud-bigquery (2.6.1)
    • Installing google-cloud-bigquery-storage (2.4.0)
    • Installing pyarrow (2.0.0)
   ```
   Here is the poetry output and lock file:
   https://gist.github.com/zachliu/cb8de028f5ce08c03d38f329b97794d9


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk commented on issue #15463: Inconsistency between the setup.py and the constraints file

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #15463:
URL: https://github.com/apache/airflow/issues/15463#issuecomment-826135456


   The build is Green :). Once we merge we will have PIP 21 official support for Airflow. Yay! 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk commented on issue #15463: Inconsistency between the setup.py and the constraints file

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #15463:
URL: https://github.com/apache/airflow/issues/15463#issuecomment-825742921


   I think not. I believe we ONLY have problem with Pyarrow, and it is misreported. Looks like instead of actual dependency that is wrong (`pyarrow`), the parent of that dependency (`google-cloud-bigquery-storage`) is printed out by PIP. Note that other than that - those are the very same limits which are problem (`<2.0.0dev,>=1.0.0; extra == "bqstorage"`). I think this is a bug in PIP.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] zachliu commented on issue #15463: Inconsistency between the setup.py and the constraints file

Posted by GitBox <gi...@apache.org>.
zachliu commented on issue #15463:
URL: https://github.com/apache/airflow/issues/15463#issuecomment-825758230


   got it, always forget the constraint is auto-generated :facepalm: 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] zachliu commented on issue #15463: Inconsistency between the setup.py and the constraints file

Posted by GitBox <gi...@apache.org>.
zachliu commented on issue #15463:
URL: https://github.com/apache/airflow/issues/15463#issuecomment-826238870


   lol, this culprit hides [deep](https://github.com/apache/beam/blob/4202efb5f01254e46fe421225a2ebbdba46da04d/sdks/python/setup.py#L192) :joy: 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk edited a comment on issue #15463: Inconsistency between the setup.py and the constraints file

Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #15463:
URL: https://github.com/apache/airflow/issues/15463#issuecomment-824300116


   Just to explain the root cause of those problems and some complexities involved
   
   1) Root cause of the problem above:
   
   The original problem turned out to be a bug in recently changed mechanism of generating constraints. Fixed here:  https://github.com/apache/airflow/pull/15470 - the bug was, that when we prepared constraints for "production" release we use "pip-providers" version of constraints - we assume that we install all providers that are **currently** (at the time of constraint generation) available in PyPI and work out the version of constraints that will allow to install all of them. The problem was that when we calculated those, we did not include few extras that have no providers. 
   
   So rather than calculating the set of constraints that are "good" for `pip install apache-airflow[google_auth,ldap....] apache-airflow-providers-*` we calculated what is "good" for `pip install apache-airflow-providers-*`.  This is why oauhlib was bumped (wrongly).
   
   2) Now there are other problems that are more subtle, the  problem is on how PIP treats "extra" dependencies especially in transitive dependencies. Those are differently solved at "installation" time and differently "at rest" and differently when different PIP versions are used. This gets complex fast.
   
   Example (real that we have now)  : 
   
   `airflow[google] -> apache-airflow-providers-google -> pandas-gbq-> "google-cloud-bigquery[bqstorage,pandas]>=1.11.1,<3.0.0dev,!=2.4.*"` and then it gets funny... because what we have in `google-cloud-bigguery` in extras is: 
   
   ```
   `"bqstorage": [
           "google-cloud-bigquery-storage >= 2.0.0, <3.0.0dev",
   ```
   
   
   There are problems with that 
     * old pip does not complain about the "bqstorage"  limitation  - and will happily install  google-coloud-bigquery-storage  outside of the range
     * new pip > 20.3 does complain and will not install 
     * however when they are installed both pip's after installation when you run 'pip check` with non-matching google-cloud-biguery will show that everything is fine - because pandas-gbq -> google-cloud-bigquery[bqstorage,pandas] is transitive, only valid at installation time. It's not 'install-requires' 
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk edited a comment on issue #15463: Inconsistency between the setup.py and the constraints file

Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #15463:
URL: https://github.com/apache/airflow/issues/15463#issuecomment-825760936


   I am not 100% sure of what's going on though - and while It awfully looks like this - I'd love at least some way of debugging it (and with -vvvv I do not even get a single `google-cloud-bigquery-storage` reference. So maybe this is not it - maybe indeed it's just a different sequence, but I would at least like to know what's causing the problem :).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk commented on issue #15463: Inconsistency between the setup.py and the constraints file

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #15463:
URL: https://github.com/apache/airflow/issues/15463#issuecomment-825732350


   Hey @uranusjr @zachliu 
   
   I am looking into it now, any help or observation much appreciated:
   
   The output of: `pip install apache-airflow[google]==2.0.2 --constraint https://raw.githubusercontent.com/apache/airflow/constraints-2.0.2/constraints-3.6.txt `: 
   
   Here: https://gist.github.com/potiuk/04f6127469a709e3e47be7585c9a863c
   
   Much more verbose output with -vvv : 
   
   https://gist.github.com/potiuk/17a3d591fb091bdd8a0e213f49b6b0af
   
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk commented on issue #15463: Inconsistency between the setup.py and the constraints file

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #15463:
URL: https://github.com/apache/airflow/issues/15463#issuecomment-823842965


   Yeah. Seems like moving to 21.1.* is inevitable as we have already a few packages for which the old PIP is simply wrong. I have few other things that require manual modification of constraints for now and I will work on finally switching to the new PIP that should make it works for the future.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk commented on issue #15463: Inconsistency between the setup.py and the constraints file

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #15463:
URL: https://github.com/apache/airflow/issues/15463#issuecomment-825747630


   > i'm confused, looks like `google-cloud-bigquery-storage` and `pyarrow` are both "children" of `bqstorage` right?
   > https://github.com/googleapis/python-bigquery/blob/c75e9cda85abdad440d87298b30e9d5291676064/setup.py#L39-L49
   
   Right. It is likely the "sibling" of the dependency that is printed, not the parent


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk edited a comment on issue #15463: Inconsistency between the setup.py and the constraints file

Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #15463:
URL: https://github.com/apache/airflow/issues/15463#issuecomment-825216717


   Happy to brainstorm/collaborate on fixing this one. I have never had time to go to the bottom of it - why PIP 21 is actually not able to downgrade bigquery-storage in this case. I think there is another dependency of 'google' providers (most likely a transitive one with another extra) that causes the bigquery-storage to be "limited" to >2.0.0. Unfortunately with transitive dependencies with extras pipdeptree and similar tools do not show the actual origin of limitation.
   
   If you look at `pipdeptree` for example after installing airflow you will see that "google-cloud-bigquery-storage` is "top" level dependency. NOTHING requires it. Unfortuntely also PIP output will not tell you what prevents it from being downloaded (even when I tried verbose). Maybe there are some better tools we can use?, @uranusjr I see that you commented on related issue in PIP -> https://github.com/pypa/pip/issues/4824, so maybe you can help with that ? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk commented on issue #15463: Inconsistency between the setup.py and the constraints file

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #15463:
URL: https://github.com/apache/airflow/issues/15463#issuecomment-829197269


   For all interested seems that there is an early attempt to implement constraints in Poetry: https://github.com/python-poetry/poetry/pull/4005 resulting from this discussion: https://github.com/python-poetry/poetry/issues/3225#issuecomment-829186310


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk commented on issue #15463: Inconsistency between the setup.py and the constraints file

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #15463:
URL: https://github.com/apache/airflow/issues/15463#issuecomment-825736963


   OK. So it seems the error produced by PIP is wrong. 
   The verbose option is producing a DIFFERENT error:
   
   ```
   ERROR: Could not find a version that satisfies the requirement pyarrow<2.0dev,>=1.0.0; extra == "bqstorage" (from google-cloud-bigquery[bqstorage,pandas])
   ERROR: No matching distribution found for pyarrow<2.0dev,>=1.0.0; extra == "bqstorage"
   ```
   
   Somehow it looks like when PIP is run without `-vvv` option it prints wrong error. It seems that it's the PYARROW version is wrong no the google-cloud-biquery-storage. @uranusjr I am going to open an issue in PIP. It is very easily reproducible.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk commented on issue #15463: Inconsistency between the setup.py and the constraints file

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #15463:
URL: https://github.com/apache/airflow/issues/15463#issuecomment-823949976


   OK. I found the root cause. That was in fact a bug in our constraint generation process. Fix is coming.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] zachliu commented on issue #15463: Inconsistency between the setup.py and the constraints file

Posted by GitBox <gi...@apache.org>.
zachliu commented on issue #15463:
URL: https://github.com/apache/airflow/issues/15463#issuecomment-825747090


   i'm confused, looks like `google-cloud-bigquery-storage` and `pyarrow` are both "children" of `bqstorage` right?
   https://github.com/googleapis/python-bigquery/blob/c75e9cda85abdad440d87298b30e9d5291676064/setup.py#L39-L49


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk edited a comment on issue #15463: Inconsistency between the setup.py and the constraints file

Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #15463:
URL: https://github.com/apache/airflow/issues/15463#issuecomment-825216717


   Happy to brainstorm/collaborate on fixing this one. I have never had time to go to the bottom of it - why PIP 21 is actually not able to downgrade bigquery-storage in this case. I think there is another dependency of 'google' providers (most likely a transitive one with another extra) that causes the bigquery-storage to be "limited" to >2.0.0. Unfortunately with transitive dependencies with extras pipdeptree and similar tools do not show the actual origin of limitation.
   
   If you look at 'pipdeptree` for example afterr installing airflow you will see that "google-cloud-bigqury-storage` is "top" level dependency. NOTHING requires it. Unfortuntely also PIP output will not tell you that (even when I tried verbose). Maybe there are some better tools we can use?, I see that you commented on related issue in PIP -> https://github.com/pypa/pip/issues/4824, so maybe you can help with that ? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk commented on issue #15463: Inconsistency between the setup.py and the constraints file

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #15463:
URL: https://github.com/apache/airflow/issues/15463#issuecomment-825216717


   Happy to brainstorm/collaborate on fixing this one. I have never had time to go to the bottom of it - why PIP 21 is actually not able to downgrade it. I think there is another dependency of 'google' providers (most likely a transitive one with another extra) that causes the bigquery-storage to be "limited" to >2.0.0. Unfortunately with transitive dependencies with extras pipdeptree and similar tools do not show the actual origin of limitation.
   
   If you look at 'pipdeptree` for example afterr installing airflow you will see that "google-cloud-bigqury-storage` is "top" level dependency. NOTHING requires it. Unfortuntely also PIP output will not tell you that (even when I tried verbose). Maybe there are some better tools we can use?, I see that you commented on related issue in PIP -> https://github.com/pypa/pip/issues/4824, so maybe you can help with that ? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk commented on issue #15463: Inconsistency between the setup.py and the constraints file

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #15463:
URL: https://github.com/apache/airflow/issues/15463#issuecomment-825752577


   That's the thing. I cannot find "google-cloud-bigquery-storage >= 1.0.0, <2.0.0dev --- from bqstorage" ANYWHERE. Thats' why I think it is wrong error message.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk commented on issue #15463: Inconsistency between the setup.py and the constraints file

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #15463:
URL: https://github.com/apache/airflow/issues/15463#issuecomment-824623098


   
   > Just to clarify, “`pip check` will show that everything is fine” does not mean everything is actually fine. It still isn’t, but limitation in `pip check`’s design prevents to detection.
   
   Correct. Even though theoretically it could be possible potentially (pip has all the information necessary to make this check really thoroughly). Unfortunately it is not quite obvious when the message printed in this case is literally 'No broken requirements found' :).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] zachliu commented on issue #15463: Inconsistency between the setup.py and the constraints file

Posted by GitBox <gi...@apache.org>.
zachliu commented on issue #15463:
URL: https://github.com/apache/airflow/issues/15463#issuecomment-826238870


   lol, this culprit hides [deep](https://github.com/apache/beam/blob/4202efb5f01254e46fe421225a2ebbdba46da04d/sdks/python/setup.py#L192) :joy: 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk commented on issue #15463: Inconsistency between the setup.py and the constraints file

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #15463:
URL: https://github.com/apache/airflow/issues/15463#issuecomment-826112107


   OK @zachliu @uranusjr -> I think I solved it. The PR to switch to PIP 21.* is here: https://github.com/apache/airflow/pull/15513
   
   The culprit was apache-beam and it's 'gcp' extra. We had "apache-beam[gcp]" specified in our requirements for apache-beam, but in fact the `gcp` is only needed for it if you want to use it together with `google` package. Which is already an extra on the 'apache-airflow-providers-beam` package. So I moved the `extra` from "airflow" level to "provider" level. 
   
   I believe  this will work fine by PIP automatically finding and installing one of the earlier version of `apache-airflow-providers-google` which should have compatible set of requirements (i will do more tests with it later). 
   
   Even if someone would like to use latest version of Google providers with Dataflow (Google's managed Apache Beam  service), this will still work fine, as google provider already has `Dataflow` operators. And It is anyhow a bit unsolvable problem until Apache Beam migrates to  google client > 2*, so I think this solution will allow to eat and have the biggest part of the cake we can  in this case. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] zachliu commented on issue #15463: Inconsistency between the setup.py and the constraints file

Posted by GitBox <gi...@apache.org>.
zachliu commented on issue #15463:
URL: https://github.com/apache/airflow/issues/15463#issuecomment-824299500


   > Python dependency conflict detection is complicated because you only see incompatibilites when the related code is actually accessed. Sometimes that’s easy to tell (e.g. when imports change), but more often it’s some function argument change or even semantic change, so there’s really no way to tell what is working and what is not without analysing call paths to the dependency. So if it works for you, way to go, but don’t count on it.
   
   yeah i forgot that not all functions all lines of code from the conflicting packages were called here :joy: 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org