You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2023/01/11 21:59:03 UTC
[GitHub] [airflow] sfc-gh-madkins commented on a diff in pull request #24652: Add ``@task.snowpark`` decorator
sfc-gh-madkins commented on code in PR #24652:
URL: https://github.com/apache/airflow/pull/24652#discussion_r1067497268
##########
Dockerfile:
##########
@@ -1234,8 +1234,10 @@ ARG ADDITIONAL_PYTHON_DEPS=""
# https://developers.google.com/protocol-buffers/docs/news/2022-05-06#python-updates
# * authlib, gcloud_aio_auth, adal are needed to generate constraints for PyPI packages and can be removed after we release
# new google, azure providers
+# * cloudpickle==2.0.0 is required by snowfalke-snowpark-python, which conflicts with new versions of apache beam, so
+# we pin apache-beam also to the latest version which uses cloudpickle==2.0.0.
# !!! MAKE SURE YOU SYNCHRONIZE THE LIST BETWEEN: Dockerfile, Dockerfile.ci, find_newer_dependencies.py
-ARG EAGER_UPGRADE_ADDITIONAL_REQUIREMENTS="dill<0.3.3 pyarrow>=6.0.0 protobuf<4.21.0 authlib>=1.0.0 gcloud_aio_auth>=4.0.0 adal>=1.2.7"
+ARG EAGER_UPGRADE_ADDITIONAL_REQUIREMENTS="dill<0.3.3 pyarrow>=6.0.0 protobuf<4.21.0 authlib>=1.0.0 gcloud_aio_auth>=4.0.0 adal>=1.2.7 cloudpickle==2.0.0 apache-beam==2.39.0"
Review Comment:
@mik-laj where does teh dependency for apache beam come from?
Here is wha I see the snowpark dependency list is?
asn1crypto==1.5.1
certifi==2022.12.7
cffi==1.15.1
charset-normalizer==2.1.1
cloudpickle==2.0.0
cryptography==38.0.4
filelock==3.9.0
idna==3.4
oscrypto==1.3.0
pycparser==2.21
pycryptodomex==3.16.0
PyJWT==2.6.0
pyOpenSSL==22.1.0
pytz==2022.7
requests==2.28.1
snowflake-connector-python==2.9.0
snowflake-snowpark-python==1.0.0
typing_extensions==4.4.0
urllib3==1.26.14
@mpgreg cc
##########
Dockerfile:
##########
@@ -1234,8 +1234,10 @@ ARG ADDITIONAL_PYTHON_DEPS=""
# https://developers.google.com/protocol-buffers/docs/news/2022-05-06#python-updates
# * authlib, gcloud_aio_auth, adal are needed to generate constraints for PyPI packages and can be removed after we release
# new google, azure providers
+# * cloudpickle==2.0.0 is required by snowfalke-snowpark-python, which conflicts with new versions of apache beam, so
+# we pin apache-beam also to the latest version which uses cloudpickle==2.0.0.
# !!! MAKE SURE YOU SYNCHRONIZE THE LIST BETWEEN: Dockerfile, Dockerfile.ci, find_newer_dependencies.py
-ARG EAGER_UPGRADE_ADDITIONAL_REQUIREMENTS="dill<0.3.3 pyarrow>=6.0.0 protobuf<4.21.0 authlib>=1.0.0 gcloud_aio_auth>=4.0.0 adal>=1.2.7"
+ARG EAGER_UPGRADE_ADDITIONAL_REQUIREMENTS="dill<0.3.3 pyarrow>=6.0.0 protobuf<4.21.0 authlib>=1.0.0 gcloud_aio_auth>=4.0.0 adal>=1.2.7 cloudpickle==2.0.0 apache-beam==2.39.0"
Review Comment:
@mik-laj where does the dependency for apache beam come from?
Here is wha I see the snowpark dependency list is?
asn1crypto==1.5.1
certifi==2022.12.7
cffi==1.15.1
charset-normalizer==2.1.1
cloudpickle==2.0.0
cryptography==38.0.4
filelock==3.9.0
idna==3.4
oscrypto==1.3.0
pycparser==2.21
pycryptodomex==3.16.0
PyJWT==2.6.0
pyOpenSSL==22.1.0
pytz==2022.7
requests==2.28.1
snowflake-connector-python==2.9.0
snowflake-snowpark-python==1.0.0
typing_extensions==4.4.0
urllib3==1.26.14
@mpgreg cc
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org