You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/12/23 04:27:39 UTC
[GitHub] [airflow] mik-laj edited a comment on pull request #12188: Upgrade azure blob to v12
mik-laj edited a comment on pull request #12188:
URL: https://github.com/apache/airflow/pull/12188#issuecomment-749921653
> When I ran breeze locally, I found that azure-storage 0.36 is still getting installed
Yes. We have extra protection to prevent image damage. In most cases, the constraints file is used to build the image, which contains a set of tested libraries. But we would like to update this file automatically sometimes, so we need the normal pip install to install the correct dependencies.
More info:
http://apache-airflow-docs.s3-website.eu-central-1.amazonaws.com/docs/apache-airflow/latest/installation.html
https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pinned-constraint-files
Recently, snowflake-connector-python dropped support for old azure library, so now snowflake and azure can't be installed, because pip will have trouble finding dependencies.
See: https://github.com/apache/airflow/pull/13195
In this PR, I am trying to improve compatibility with newer versions of Pinot and the pinot-python library, so I added the label "upgrade to newer dependencies" which means the image will be built without constraint, so the dependency installation may fail and it happened now. Several libraries have been updated and unfortunately, these dependencies do not work.
Here I have compiled a list of the differences between installing with and without a constraints file. Most are safe and will be updated soon in constraints.txt as well, but we have a few changes we will add to setup.py/setup.cfg, so users will be limited in using these libraries.
```
diff <(pip freeze | sort) <(curl https://raw.githubusercontent.com/apache/airflow/constraints-master/constraints-3.6.txt | sort)
< -e /opt/airflow
19c18
< JPype1==1.2.0
---
> JPype1==1.1.2
30c29
< Pygments==2.7.3
---
> Pygments==2.7.2
33,35c32,34
< SQLAlchemy==1.3.22
< Sphinx==3.4.0
< Unidecode==1.1.2
---
> SQLAlchemy==1.3.20
> Sphinx==3.3.1
> Unidecode==1.1.1
56c55
< aws-sam-translator==1.33.0
---
> aws-sam-translator==1.31.0
58c57
< azure-batch==10.0.0
---
> azure-batch==9.0.0
65c64
< azure-keyvault-keys==4.3.1
---
> azure-keyvault-keys==4.3.0
90c89
< cachetools==4.2.0
---
> cachetools==4.1.1
94c93
< certifi==2020.12.5
---
> certifi==2020.11.8
97c96
< cfn-lint==0.44.1
---
> cfn-lint==0.42.0
109c108
< coverage==5.3.1
---
> coverage==5.3
111c110
< cryptography==3.3.1
---
> cryptography==3.2.1
113,114c112,113
< cx-Oracle==8.1.0
< dask==2020.12.0
---
> cx-Oracle==8.0.1
> dask==2.30.0
133c132
< eventlet==0.30.0
---
> eventlet==0.29.1
135,136c134,135
< facebook-business==9.0.1
< fastavro==1.2.1
---
> facebook-business==9.0.0
> fastavro==1.2.0
142c141,142
< flower==0.9.7
---
> flask-swagger==0.2.13
> flower==0.9.5
144c144,145
< fsspec==0.8.5
---
> fsspec==0.8.4
> funcsigs==1.0.2
152c153
< google-api-core==1.24.1
---
> google-api-core==1.23.0
156c157
< google-auth==1.24.0
---
> google-auth==1.23.0
160,161c161,162
< google-cloud-bigquery==2.6.1
< google-cloud-bigtable==1.6.1
---
> google-cloud-bigquery==2.4.0
> google-cloud-bigtable==1.6.0
163c164
< google-cloud-core==1.5.0
---
> google-cloud-core==1.4.3
178c179
< google-cloud-storage==1.35.0
---
> google-cloud-storage==1.33.0
184,185c185,186
< google-crc32c==1.1.0
< google-resumable-media==1.2.0
---
> google-crc32c==1.0.0
> google-resumable-media==1.1.0
191c192
< grpcio==1.34.0
---
> grpcio==1.33.2
196,197c197,198
< humanize==3.2.0
< hvac==0.10.6
---
> humanize==3.1.0
> hvac==0.10.5
212c213
< isort==5.6.4
---
> isort==4.3.21
219,220c220,221
< jsonpatch==1.28
< jsonpickle==1.4.2
---
> jsonpatch==1.27
> jsonpickle==1.4.1
232c233
< libcst==0.3.16
---
> libcst==0.3.14
237c238
< marshmallow==3.10.0
---
> marshmallow==3.9.1
239c240
< mock==4.0.3
---
> mock==4.0.2
245,246c246,247
< msal==1.8.0
< msgpack==1.0.2
---
> msal==1.6.0
> msgpack==1.0.0
250c251
< multidict==5.1.0
---
> multidict==5.0.2
267c268
< packaging==20.8
---
> packaging==20.4
269c270
< pandas==1.1.5
---
> pandas==1.1.4
281c282
< pinotdb==0.3.3
---
> pinotdb==0.1.1
285c286
< pre-commit==2.9.3
---
> pre-commit==2.9.2
290c291
< proto-plus==1.13.0
---
> proto-plus==1.11.0
292c293
< psutil==5.8.0
---
> psutil==5.7.3
296,297c297,298
< py==1.10.0
< pyOpenSSL==20.0.1
---
> py==1.9.0
> pyOpenSSL==19.1.0
306,307c307,308
< pydruid==0.6.2
< pyenchant==3.2.0
---
> pydruid==0.6.1
> pyenchant==3.1.1
311,312c312,313
< pylint==2.6.0
< pymongo==3.11.2
---
> pylint==2.5.3
> pymongo==3.11.1
324,325c325,326
< pytest-xdist==2.2.0
< pytest==6.2.1
---
> pytest-xdist==2.1.0
> pytest==6.1.2
355c356
< sendgrid==6.4.8
---
> sendgrid==6.4.7
357,358c358,359
< sentry-sdk==0.19.5
< setproctitle==1.2.1
---
> sentry-sdk==0.19.4
> setproctitle==1.2
364c365
< snowflake-connector-python==2.3.7
---
> snowflake-connector-python==2.3.6
367c368
< soupsieve==2.1
---
> soupsieve==2.0.1
391c392
< tableauserverclient==0.14.1
---
> tableauserverclient==0.14.0
396c397
< testfixtures==6.17.0
---
> testfixtures==6.15.0
404c405
< tqdm==4.54.1
---
> tqdm==4.54.0
414c415
< vertica-python==1.0.1
---
> vertica-python==1.0.0
416c417
< virtualenv==20.2.2
---
> virtualenv==20.2.1
424c425
< yandexcloud==0.64.0
---
> yandexcloud==0.61.0
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org