You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/12/07 12:47:16 UTC
[GitHub] [airflow] potiuk opened a new issue #12881: Snowflake python connector monkeypatches urllib and makes presto ssl unusable
potiuk opened a new issue #12881:
URL: https://github.com/apache/airflow/issues/12881
Curreently wnen you run snowflke provider, it monkeypatches urlllb in a way that is not compatible with other libraries (for example presto SSL with kerberos).
This is not critical (as in 2.0 we have provider separation and snowflake code will not even be there until you choose [snowflake] extra or install provider manually, however it would be great to work with snowflake team to upgrade the python connector to not require monkeypatching such a common library.
Additional links:
* Issue: https://github.com/snowflakedb/snowflake-connector-python/issues/324
Offending code:
* https://github.com/snowflakedb/snowflake-connector-python/blob/133d6215f7920d304c5f2d466bae38127c1b836d/src/snowflake/connector/network.py#L89-L92
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #12881: Snowflake python connector monkeypatches urllib and makes presto ssl unusable
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #12881:
URL: https://github.com/apache/airflow/issues/12881#issuecomment-743714838
Yeah. OCSP support was the root cause, but the solution to monkeypatch a library that others use was, well.. offensive. They could have vendoered it in as we did with some other packages in the past
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] davido912 commented on issue #12881: Snowflake python connector monkeypatches urllib and makes many services unusable.
Posted by GitBox <gi...@apache.org>.
davido912 commented on issue #12881:
URL: https://github.com/apache/airflow/issues/12881#issuecomment-749570602
I would assume this https://github.com/apache/airflow/issues/13255 issue I raised is related (?)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #12881: Snowflake python connector monkeypatches urllib and makes many services unusable.
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #12881:
URL: https://github.com/apache/airflow/issues/12881#issuecomment-759496860
Thanks @eladkal for keeping an eye :+1:
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] lgwacker commented on issue #12881: Snowflake python connector monkeypatches urllib and makes presto ssl unusable
Posted by GitBox <gi...@apache.org>.
lgwacker commented on issue #12881:
URL: https://github.com/apache/airflow/issues/12881#issuecomment-743453393
I read all issues related to that there and never found why they did that...
The main issue was open since June. But with community backlash seems like they started working on it. :pray:
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #12881: Snowflake python connector monkeypatches urllib and makes many services unusable.
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #12881:
URL: https://github.com/apache/airflow/issues/12881#issuecomment-759519269
@eladkal - absolutelty https://github.com/apache/airflow/pull/13654 - I unmarked the test in presto that was marked as "expect to fail" because of this problem, this will tell us whether the problem is fixed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] manugarri commented on issue #12881: Snowflake python connector monkeypatches urllib and makes many services unusable.
Posted by GitBox <gi...@apache.org>.
manugarri commented on issue #12881:
URL: https://github.com/apache/airflow/issues/12881#issuecomment-745435165
agree @potiuk , but i dont work in a heavyweight company so i had to make our data pipelines and reports work.
I would definitely recommend a full rewrite of the python package, how it handles dependencies is an absolute mess.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #12881: Snowflake python connector monkeypatches urllib and makes many services unusable.
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #12881:
URL: https://github.com/apache/airflow/issues/12881#issuecomment-761566497
> @potiuk that is true, however it means snowflake operators cant be reliably used right?
Why? I think the dependencies that are left are pretty ok and we are also upgrading azure provider so that the last conflicts are removed. From what I see boto dependency in snowflake connector is:
```
boto3>=1.4.4,<2.0.0
```
So nothing extraodirinary - taking into account that latest released version is 1.16.56.
Do you KNOW that you have some other conflicts, or do you just suspect it ?
From the latest set of constraints generated for Airflow (I just merged the snowflake change and regenerated constraints) - we have those constraints 'blessed" for airflow (3.6 example):
https://github.com/apache/airflow/blob/constraints-master/constraints-3.6.txt
* boto3==1.15.18 (not latest but this is because amazon provider limits it (`boto3>=1.15.0,<1.16.0`) - if amazon provider is
upgraded, it will go up
* snowflake-connector-python==2.3.8
* requests==2.25.1 (latest)
* urllib3==1.25.11 (not latest but it is not limited by python-snowflake-connector) - actually one of my fixes to the connector
which I contributed to snowflake was getting rid of the urllib3 dependency altogether as they did not need it.
The < 1.26 comes from boto-core actually, so if amazon provider goes up with botocore, this one will go up
as well.
Do you think your problems are not solved by this?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #12881: Snowflake python connector monkeypatches urllib and makes many services unusable.
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #12881:
URL: https://github.com/apache/airflow/issues/12881
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #12881: Snowflake python connector monkeypatches urllib and makes many services unusable.
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #12881:
URL: https://github.com/apache/airflow/issues/12881#issuecomment-759489611
Woooho! Switching to it!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #12881: Snowflake python connector monkeypatches urllib and makes presto ssl unusable
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #12881:
URL: https://github.com/apache/airflow/issues/12881#issuecomment-743463304
@ashb OCSP support.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #12881: Snowflake python connector monkeypatches urllib and makes many services unusable.
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #12881:
URL: https://github.com/apache/airflow/issues/12881#issuecomment-744711310
Thanks. I think we are just trying to push snowflake (and we have some heavyweights helping us) to fix it, so let's wait and see :).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #12881: Snowflake python connector monkeypatches urllib and makes many services unusable.
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #12881:
URL: https://github.com/apache/airflow/issues/12881#issuecomment-760229356
> I dont think this PR fixes the strict dependencies issue though https://github.com/snowflakedb/snowflake-connector-python/blob/master/setup.py#L194 , that is usually what breaks builds.
Not really. You do not have to install the snowflake provider (Airflow in 2.0 is now split into separate core and providers). Even in 1.10 snowflake was an 'extra' that you did not have to install. But in the meantime we impleemented much better way of handling the requirement upgrade, so now we have less requirements in airflow's install_requires and those that are problematic are moved out of airflow 'install_requires' into test Dockerfiles when we could.
Thanks to that 2.0.1 and 1.10.15 should be much gentler w/regards to requirements:
See here: https://github.com/apache/airflow/blob/bfb7cb343d0adfdca92a77a273954d9f2dbcfdc9/Dockerfile.ci#L345 for those problematic requirements. Airflow on it's own does not depend any more on them, so as long as you are not installing any "problematic' providers you will be able to upgrade.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] lgwacker commented on issue #12881: Snowflake python connector monkeypatches urllib and makes presto ssl unusable
Posted by GitBox <gi...@apache.org>.
lgwacker commented on issue #12881:
URL: https://github.com/apache/airflow/issues/12881#issuecomment-743439963
Seems like a fix is finally on the way https://github.com/snowflakedb/snowflake-connector-python/pull/557
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #12881: Snowflake python connector monkeypatches urllib and makes many services unusable.
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #12881:
URL: https://github.com/apache/airflow/issues/12881#issuecomment-761566497
> @potiuk that is true, however it means snowflake operators cant be reliably used right?
Why? I think the dependencies that are left are pretty ok and we are also upgrading azure provider so that the last conflicts are removed. From what I see boto dependency in snowflake connector is:
```
boto3>=1.4.4,<2.0.0
```
So nothing extraodirinary - taking into account that latest released version is 1.16.56.
Do you KNOW that you have some other conflicts, or do you just suspect it ?
From the latest set of constraints generated for Airflow (I just merged the snowflake change and regenerated constraints) - we have those constraints 'blessed" for airflow (3.6 example):
https://github.com/apache/airflow/blob/constraints-master/constraints-3.6.txt
* boto3==1.15.18 (not latest but this is because amazon provider limits it (`boto3>=1.15.0,<1.16.0`) - if amazon provider is
upgraded, it will go up
* snowflake-connector-python==2.3.8 (latest)
* requests==2.25.1 (latest)
* urllib3==1.25.11 (not latest but it is not limited by python-snowflake-connector) - actually one of my fixes to the connector
which I contributed to snowflake was getting rid of the urllib3 dependency altogether as they did not need it.
The < 1.26 comes from boto-core actually, so if amazon provider goes up with botocore, this one will go up
as well.
Do you think your problems are not solved by this?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on issue #12881: Snowflake python connector monkeypatches urllib and makes presto ssl unusable
Posted by GitBox <gi...@apache.org>.
ashb commented on issue #12881:
URL: https://github.com/apache/airflow/issues/12881#issuecomment-743444388
I'm still not sure _why_ they are even doing such monkeypatching. Does anyone know why they do this?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on issue #12881: Snowflake python connector monkeypatches urllib and makes many services unusable.
Posted by GitBox <gi...@apache.org>.
eladkal commented on issue #12881:
URL: https://github.com/apache/airflow/issues/12881#issuecomment-745228965
**Update:** just heard back from Snowflake. The current estimation is that they will fix it by end of this year.
I'll keep track
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] davido912 commented on issue #12881: Snowflake python connector monkeypatches urllib and makes many services unusable.
Posted by GitBox <gi...@apache.org>.
davido912 commented on issue #12881:
URL: https://github.com/apache/airflow/issues/12881#issuecomment-759496168
what a day. let us know if this solved anything! :D
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #12881: Snowflake python connector monkeypatches urllib and makes many services unusable.
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #12881:
URL: https://github.com/apache/airflow/issues/12881#issuecomment-745465266
Let's see. I hope things will get better with Snowflke's integrations and ecosystem in general. They have a very, very cool product.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #12881: Snowflake python connector monkeypatches urllib and makes many services unusable.
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #12881:
URL: https://github.com/apache/airflow/issues/12881#issuecomment-749593616
Absolutely @davido912
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] manugarri commented on issue #12881: Snowflake python connector monkeypatches urllib and makes many services unusable.
Posted by GitBox <gi...@apache.org>.
manugarri commented on issue #12881:
URL: https://github.com/apache/airflow/issues/12881#issuecomment-760952492
@potiuk what that means is, if we install snowflake as an extra dependency in airflow, there might (probably?) be conflicts with other dependencies (boto3 being the usual suspect).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #12881: Snowflake python connector monkeypatches urllib and makes many services unusable.
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #12881:
URL: https://github.com/apache/airflow/issues/12881#issuecomment-761196739
> @potiuk what that means is, if we install snowflake as an extra dependency in airflow, there might (probably?) be conflicts with other dependencies (boto3 being the usual suspect).
It is not a problem for airflow - but if you have problem with that - I encourage you to open an issue for that, they seem to be responsive to community complaints
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on issue #12881: Snowflake python connector monkeypatches urllib and makes many services unusable.
Posted by GitBox <gi...@apache.org>.
eladkal commented on issue #12881:
URL: https://github.com/apache/airflow/issues/12881#issuecomment-759464036
Snowflake released new version of the connector
https://pypi.org/project/snowflake-connector-python/
```
v2.3.8
Arrow result conversion speed up.
Send all Python Connector exceptions to in-band or out-of-band telemetry.
Vendoring requests and urllib3 to contain OCSP monkey patching to our library only.
Declare dependency on setuptools.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] manugarri commented on issue #12881: Snowflake python connector monkeypatches urllib and makes many services unusable.
Posted by GitBox <gi...@apache.org>.
manugarri commented on issue #12881:
URL: https://github.com/apache/airflow/issues/12881#issuecomment-760364812
@potiuk by strict dependencies issue i meant snowflake-python-connector dependencies, not Airflow's
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #12881: Snowflake python connector monkeypatches urllib and makes presto ssl unusable
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #12881:
URL: https://github.com/apache/airflow/issues/12881#issuecomment-739897146
cc: @mik-laj
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on issue #12881: Snowflake python connector monkeypatches urllib and makes presto ssl unusable
Posted by GitBox <gi...@apache.org>.
ashb commented on issue #12881:
URL: https://github.com/apache/airflow/issues/12881#issuecomment-743732559
Or submitted the patch upstream :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #12881: Snowflake python connector monkeypatches urllib and makes many services unusable.
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #12881:
URL: https://github.com/apache/airflow/issues/12881#issuecomment-745284023
Thanks @eladkal !
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #12881: Snowflake python connector monkeypatches urllib and makes many services unusable.
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #12881:
URL: https://github.com/apache/airflow/issues/12881#issuecomment-745284413
Yeah, I think NOT switching to a slimmed down version of the lib is a better incentive for Snowflake to fix it :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] manugarri commented on issue #12881: Snowflake python connector monkeypatches urllib and makes many services unusable.
Posted by GitBox <gi...@apache.org>.
manugarri commented on issue #12881:
URL: https://github.com/apache/airflow/issues/12881#issuecomment-744641325
fyi, we have been trying to get snowflake to fix this for months.
As an alternative, I made a [lite version of snowflake-connector package](https://github.com/manugarri/snowflake-connector-python-lite) that i use in my airflow environments in case someone wants to check it out
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] manugarri commented on issue #12881: Snowflake python connector monkeypatches urllib and makes many services unusable.
Posted by GitBox <gi...@apache.org>.
manugarri commented on issue #12881:
URL: https://github.com/apache/airflow/issues/12881#issuecomment-745251152
great @eladkal . thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on issue #12881: Snowflake python connector monkeypatches urllib and makes presto ssl unusable
Posted by GitBox <gi...@apache.org>.
eladkal commented on issue #12881:
URL: https://github.com/apache/airflow/issues/12881#issuecomment-742389457
Raised the issue with my contact at Snowflake
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #12881: Snowflake python connector monkeypatches urllib and makes many services unusable.
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #12881:
URL: https://github.com/apache/airflow/issues/12881#issuecomment-760368576
> @potiuk by strict dependencies issue i meant snowflake-python-connector dependencies, not Airflow's
I see. They got improved but indeed could be more open. For me it is a bit less of an issue so far.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] manugarri commented on issue #12881: Snowflake python connector monkeypatches urllib and makes many services unusable.
Posted by GitBox <gi...@apache.org>.
manugarri commented on issue #12881:
URL: https://github.com/apache/airflow/issues/12881#issuecomment-760150763
I dont think this PR fixes the strict dependencies issue though https://github.com/snowflakedb/snowflake-connector-python/blob/master/setup.py#L194 , that is usually what breaks builds.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] manugarri commented on issue #12881: Snowflake python connector monkeypatches urllib and makes many services unusable.
Posted by GitBox <gi...@apache.org>.
manugarri commented on issue #12881:
URL: https://github.com/apache/airflow/issues/12881#issuecomment-761238626
@potiuk that is true, however it means snowflake operators cant be reliably used right?
And regarding opening issues and responsiveness, well, lets say I opened a [related issue almost a year ago](https://github.com/snowflakedb/snowflake-connector-python/issues/284). In fact i released [snowflake-connector-python-lite](https://pypi.org/project/snowflake-connector-python-lite/), a lite version of snowflake package just so my company's CI/CD builds would stop failing.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org