You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/12/19 16:36:08 UTC

[GitHub] [airflow] JavierLopezT opened a new issue #13185: Cannot import name 'BlobServiceClient'

JavierLopezT opened a new issue #13185:
URL: https://github.com/apache/airflow/issues/13185


   I am using Airflow with docker-compose and Dockerfile. I have just changed in the Dockerfile the version from 1.10.10 to 1.10.14, and I get this import error in all files which involve Snowflake. I am not using Azure at all, so I guess it is because of some kind of error in snowflake-connector library. 
   
   In one particular DAG I also get the error `An HTTP Client raised and unhandled exception: cannot import name 'BlobServiceClient'
   `
   Might be related to https://github.com/apache/airflow/issues/13093


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk edited a comment on issue #13185: Cannot import name 'BlobServiceClient'

Posted by GitBox <gi...@apache.org>.

potiuk edited a comment on issue #13185:
URL: https://github.com/apache/airflow/issues/13185#issuecomment-781337172

> This is still an issue for me with latest 1.10.x and latest backport providers release.

Hey @feluelle You likely have the old `azure-storage` package installed (or it was installed in the past and removed without reinstalling `azure-storage-*`. The problem is that there are two sets of libraries for azure in PyPI:

* legacy (very old) "azure-storage' (https://pypi.org/project/azure-storage/)
* new - set of "azure-storage-*" libraries that replace the storage one. For example (https://pypi.org/project/azure-storage-blob/ or https://pypi.org/project/azure-storage-logging/).

The problem was that azure did a very, very bad thing. Both libraries have overlapping packages and they can override each other when both legacy and new libraries are installed. If you have `azure-storage` and `azure-storage-blob`, the set of classes available in `azure.storage.blob` depend on which is the SEQUENCE of installing the two libraries and whethere they were installed with --force or not. Total mess.

So if you want to bring the situation under control - remove `azure-storage` and reinstall `azure-storage-*` packages you have. That should help with the error. If you install 2.0.1+ from scratch, the problem is gone (we have no more `azure-storage` dependency. However in 1.10 you need to install airflow without `[azure]` extra and then add `apache-airflow-backport-providers-azure`. Or remove `azure-storage` and reinstall the `azure-storage-blob`.

Let me know if either of those worked.

I hope it helps. Maybe also someone could describe it nicely somewhere else in the docs ;) ?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on issue #13185: Cannot import name 'BlobServiceClient'

Posted by GitBox <gi...@apache.org>.

potiuk commented on issue #13185:
URL: https://github.com/apache/airflow/issues/13185#issuecomment-781338977


   For now I am closing it with the information that you need to cleanup it in the way described above - I will re-open if I hear otherwise :)


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] JavierLopezT commented on issue #13185: Cannot import name 'BlobServiceClient'

Posted by GitBox <gi...@apache.org>.

JavierLopezT commented on issue #13185:
URL: https://github.com/apache/airflow/issues/13185#issuecomment-748499206


   > What version of snowflake connector do you use? I think I saw this error some time ago...
   
   2.3.7. I have downgraded it to 2.3.6 and now I don't have the error. Should I close the issue or it can still be airflow-related?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] feluelle commented on issue #13185: Cannot import name 'BlobServiceClient'

Posted by GitBox <gi...@apache.org>.

feluelle commented on issue #13185:
URL: https://github.com/apache/airflow/issues/13185#issuecomment-781327259


   This is still an issue for me with latest 1.10.x and latest backport providers release.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on issue #13185: Cannot import name 'BlobServiceClient'

Posted by GitBox <gi...@apache.org>.

potiuk commented on issue #13185:
URL: https://github.com/apache/airflow/issues/13185#issuecomment-781337172

> This is still an issue for me with latest 1.10.x and latest backport providers release.

Hey @feluelle You likely have the old `azure-storage` library installed. The problem was that there were two libraries:
* legacy (very old) "azure-storage' (https://pypi.org/project/azure-storage/)
* new - set of "azure-storage-*" libraries that replace the storage one. For example (https://pypi.org/project/azure-storage-blob/ or https://pypi.org/project/azure-storage-logging/).

The problem was that azure did a very, very bad thing. Both libraries have overlapping packages and they can override each other when both legacy and new libraries are installed. If you have 'azure-storge' and 'azure-storage-blob', the set of classes available in `azure.storage.blob` depend on which is the SEQUENCE of installing the two libraries and whethere they were installed with --force or not. Total mess.

So if you want to bring the situation under control - remove 'azure-storage' and reinstall 'azure-storage-*' packages you have. That should help with the error. If you install 2.0.1+ from scratch, the problem is gone (we have no more `azure-storage` dependency. However in 1.10 you need to install airflow without `[azure]` extra and then add 'apache-airflow-providers-backports-azure'. Or remove storage and reinstall the 'azure-storage-*'.

Let me know if either of those worked.

I hope it helps. Maybe also someone could describe it nicely somewhere else in the docs ;) ?

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] turbaszek commented on issue #13185: Cannot import name 'BlobServiceClient'

Posted by GitBox <gi...@apache.org>.

turbaszek commented on issue #13185:
URL: https://github.com/apache/airflow/issues/13185#issuecomment-748497774


   What version of snowflake connector do you use? I think I saw this error some time ago...


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk edited a comment on issue #13185: Cannot import name 'BlobServiceClient'

Posted by GitBox <gi...@apache.org>.

potiuk edited a comment on issue #13185:
URL: https://github.com/apache/airflow/issues/13185#issuecomment-781591730

> Why can't we ship the fix to 1.10.x release? :)

We could potentially. But it would require https://github.com/apache/airflow/pull/12188 to be back-ported (and possibly few other PRs. I am sure @ephraimbuddy would not be so happy because I think that was one of the most "troublesome" changes he went through ;)

Honestly - it is much simpler via backports, even if requires some of hassle with installed packages.

> I don't have azure as an explicit dependency. So I would have to remove azure-storage and reinstall the azure-storage-blob. Can this be automate somehow? :D I mean adding sth to requirements. Or do I have to write bash for that one?

We could potentially add `azure-storage==0.37.0`. One "good" thing Azure team did there was that they added 0.37.0 version of the deprecated package that removes the "azure.storage" folder. BUT. This is again tricky. Because depending on the order of installation we can first install `azure-storage-blob`, then `azure-storage==0.36.0` then `azure-storage== 0.37.0` and puff! whole azure.storage dir is gone.

It would have to be an external script. I think it's worth to add. such snippet of 'pip uninistall && pip install ` would be nice but I'd say it should be in documentation. We really want to limit our investment in 1.10 and if we can document workaround and tell people ("but in 2.0 you do not have to do it - so migrate") - this is the best we can do.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk edited a comment on issue #13185: Cannot import name 'BlobServiceClient'

Posted by GitBox <gi...@apache.org>.

potiuk edited a comment on issue #13185:
URL: https://github.com/apache/airflow/issues/13185#issuecomment-781337172

> This is still an issue for me with latest 1.10.x and latest backport providers release.

The problem was that Azure did a very, very bad thing. Both libraries have overlapping packages and they can override each other when both legacy and new libraries are installed. If you have `azure-storage` and `azure-storage-blob`, the set of classes available in `azure.storage.blob` depend on which is the SEQUENCE of installing the two libraries and whethere they were installed with --force or not. Total mess.

Let me know if either of those worked.

I hope it helps. Maybe also someone could describe it nicely somewhere else in the docs ;) ?

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk edited a comment on issue #13185: Cannot import name 'BlobServiceClient'

Posted by GitBox <gi...@apache.org>.

potiuk edited a comment on issue #13185:
URL: https://github.com/apache/airflow/issues/13185#issuecomment-781337172

> This is still an issue for me with latest 1.10.x and latest backport providers release.

The problem was that azure did a very, very bad thing. Both libraries have overlapping packages and they can override each other when both legacy and new libraries are installed. If you have 'azure-storage' and 'azure-storage-blob', the set of classes available in `azure.storage.blob` depend on which is the SEQUENCE of installing the two libraries and whethere they were installed with --force or not. Total mess.

Let me know if either of those worked.

I hope it helps. Maybe also someone could describe it nicely somewhere else in the docs ;) ?

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk edited a comment on issue #13185: Cannot import name 'BlobServiceClient'

Posted by GitBox <gi...@apache.org>.

potiuk edited a comment on issue #13185:
URL: https://github.com/apache/airflow/issues/13185#issuecomment-781337172

> This is still an issue for me with latest 1.10.x and latest backport providers release.

The problem was that azure did a very, very bad thing. Both libraries have overlapping packages and they can override each other when both legacy and new libraries are installed. If you have 'azure-storage' and 'azure-storage-blob', the set of classes available in `azure.storage.blob` depend on which is the SEQUENCE of installing the two libraries and whethere they were installed with --force or not. Total mess.

So if you want to bring the situation under control - remove 'azure-storage' and reinstall `azure-storage-*` packages you have. That should help with the error. If you install 2.0.1+ from scratch, the problem is gone (we have no more `azure-storage` dependency. However in 1.10 you need to install airflow without `[azure]` extra and then add 'apache-airflow-providers-backports-azure'. Or remove storage and reinstall the `azure-storage-*`.

Let me know if either of those worked.

I hope it helps. Maybe also someone could describe it nicely somewhere else in the docs ;) ?

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk closed issue #13185: Cannot import name 'BlobServiceClient'

Posted by GitBox <gi...@apache.org>.

potiuk closed issue #13185:
URL: https://github.com/apache/airflow/issues/13185


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] feluelle commented on issue #13185: Cannot import name 'BlobServiceClient'

Posted by GitBox <gi...@apache.org>.

feluelle commented on issue #13185:
URL: https://github.com/apache/airflow/issues/13185#issuecomment-781877050


   > It would have to be an external script. I think it's worth to add. such snippet of 'pip uninistall && pip install ` would be nice but I'd say it should be in documentation. We really want to limit our investment in 1.10 and if we can document workaround and tell people ("but in 2.0 you do not have to do it - so migrate") - this is the best we can do.
   
   SGTM 👍 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on issue #13185: Cannot import name 'BlobServiceClient'

Posted by GitBox <gi...@apache.org>.

potiuk commented on issue #13185:
URL: https://github.com/apache/airflow/issues/13185#issuecomment-781591730

> Why can't we ship the fix to 1.10.x release? :)

Honestly - it is much simpler via backports, even if requires some of hassle with installed packages.

It would have to be an external script. I think it's worth to add. such snipped of 'pip uninistall && pip install ` would be nice but I'd say it should be in documentation. We really want to limit our investment in 1.10 and if we can document workaround and tell people ("but in 2.0 you do not have to do it - so migrate") - this is the best we can do.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] arbass22 commented on issue #13185: Cannot import name 'BlobServiceClient'

Posted by GitBox <gi...@apache.org>.

arbass22 commented on issue #13185:
URL: https://github.com/apache/airflow/issues/13185#issuecomment-750471232


   What end up working for me was explicitly uninstall the older version of azure-storage
   `pip uninstall azure-storage`
   The newer version is split into multiple packages and it seemed like because of this the conflicting versions were not noticed by pip.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk edited a comment on issue #13185: Cannot import name 'BlobServiceClient'

Posted by GitBox <gi...@apache.org>.

potiuk edited a comment on issue #13185:
URL: https://github.com/apache/airflow/issues/13185#issuecomment-781337172

> This is still an issue for me with latest 1.10.x and latest backport providers release.

So if you want to bring the situation under control - remove 'azure-storage' and reinstall `azure-storage-*` packages you have. That should help with the error. If you install 2.0.1+ from scratch, the problem is gone (we have no more `azure-storage` dependency. However in 1.10 you need to install airflow without `[azure]` extra and then add `apache-airflow-backport-providers-azure`. Or remove `azure-storage` and reinstall the `azure-storage-blob`.

Let me know if either of those worked.

I hope it helps. Maybe also someone could describe it nicely somewhere else in the docs ;) ?

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] eladkal commented on issue #13185: Cannot import name 'BlobServiceClient'

Posted by GitBox <gi...@apache.org>.

eladkal commented on issue #13185:
URL: https://github.com/apache/airflow/issues/13185#issuecomment-766767389


   @JavierLopezT I believe this issue is fixed by  https://github.com/apache/airflow/pull/13654 , https://github.com/apache/airflow/pull/13286
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] HaloKo4 commented on issue #13185: Cannot import name 'BlobServiceClient'

Posted by GitBox <gi...@apache.org>.

HaloKo4 commented on issue #13185:
URL: https://github.com/apache/airflow/issues/13185#issuecomment-748584685


   You probably have conflicting versions of `azure-storage-blob`
   See:
   https://github.com/apache/airflow/blob/master/setup.py#L395
   
   snowflake-connector-python have`'azure-storage-blob>=12.0.0,<13.0.0'`
   https://github.com/snowflakedb/snowflake-connector-python/issues/225


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk edited a comment on issue #13185: Cannot import name 'BlobServiceClient'

Posted by GitBox <gi...@apache.org>.

potiuk edited a comment on issue #13185:
URL: https://github.com/apache/airflow/issues/13185#issuecomment-781337172

> This is still an issue for me with latest 1.10.x and latest backport providers release.

The problem was that azure did a very, very bad thing. Both libraries have overlapping packages and they can override each other when both legacy and new libraries are installed. If you have 'azure-storage' and 'azure-storage-blob', the set of classes available in `azure.storage.blob` depend on which is the SEQUENCE of installing the two libraries and whethere they were installed with --force or not. Total mess.

So if you want to bring the situation under control - remove 'azure-storage' and reinstall `azure-storage-*` packages you have. That should help with the error. If you install 2.0.1+ from scratch, the problem is gone (we have no more `azure-storage` dependency. However in 1.10 you need to install airflow without `[azure]` extra and then add `apache-airflow-backport-providers-azure`. Or remove `azure-storage` and reinstall the `azure-storage-blob`.

Let me know if either of those worked.

I hope it helps. Maybe also someone could describe it nicely somewhere else in the docs ;) ?

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk edited a comment on issue #13185: Cannot import name 'BlobServiceClient'

Posted by GitBox <gi...@apache.org>.

potiuk edited a comment on issue #13185:
URL: https://github.com/apache/airflow/issues/13185#issuecomment-781337172

> This is still an issue for me with latest 1.10.x and latest backport providers release.

The problem was that azure did a very, very bad thing. Both libraries have overlapping packages and they can override each other when both legacy and new libraries are installed. If you have 'azure-storage' and 'azure-storage-blob', the set of classes available in `azure.storage.blob` depend on which is the SEQUENCE of installing the two libraries and whethere they were installed with --force or not. Total mess.

So if you want to bring the situation under control - remove 'azure-storage' and reinstall `azure-storage-*` packages you have. That should help with the error. If you install 2.0.1+ from scratch, the problem is gone (we have no more `azure-storage` dependency. However in 1.10 you need to install airflow without `[azure]` extra and then add `apache-airflow-backport-providers-azure`. Or remove storage and reinstall the `azure-storage-*`.

Let me know if either of those worked.

I hope it helps. Maybe also someone could describe it nicely somewhere else in the docs ;) ?

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk edited a comment on issue #13185: Cannot import name 'BlobServiceClient'

Posted by GitBox <gi...@apache.org>.

potiuk edited a comment on issue #13185:
URL: https://github.com/apache/airflow/issues/13185#issuecomment-781337172

> This is still an issue for me with latest 1.10.x and latest backport providers release.

Hey @feluelle You likely have the old `azure-storage` package installed (or it was installed in the past and removed without reinstalling `azure-storage-*`. The problem was that there were two libraries:
* legacy (very old) "azure-storage' (https://pypi.org/project/azure-storage/)
* new - set of "azure-storage-*" libraries that replace the storage one. For example (https://pypi.org/project/azure-storage-blob/ or https://pypi.org/project/azure-storage-logging/).

Let me know if either of those worked.

I hope it helps. Maybe also someone could describe it nicely somewhere else in the docs ;) ?

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] eladkal commented on issue #13185: Cannot import name 'BlobServiceClient'

Posted by GitBox <gi...@apache.org>.

eladkal commented on issue #13185:
URL: https://github.com/apache/airflow/issues/13185#issuecomment-766767389


   @JavierLopezT I believe this issue is fixed by  https://github.com/apache/airflow/pull/13654 , https://github.com/apache/airflow/pull/13286
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] feluelle commented on issue #13185: Cannot import name 'BlobServiceClient'

Posted by GitBox <gi...@apache.org>.

feluelle commented on issue #13185:
URL: https://github.com/apache/airflow/issues/13185#issuecomment-781341709


   > However in 1.10 you need to install airflow without [azure] extra and then add apache-airflow-backport-providers-azure. Or remove azure-storage and reinstall the azure-storage-blob.
   
   Why can't we ship the fix to 1.10.x release? :)
   
   I don't have azure as an explicit dependency. So I would have to remove azure-storage and reinstall the azure-storage-blob. Can this be automate somehow? :D I mean adding sth to requirements. Or do I have to write bash for that one?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] feluelle edited a comment on issue #13185: Cannot import name 'BlobServiceClient'

Posted by GitBox <gi...@apache.org>.

feluelle edited a comment on issue #13185:
URL: https://github.com/apache/airflow/issues/13185#issuecomment-781877050


   > It would have to be an external script. I think it's worth to add. such snippet of 'pip uninistall && pip install ` would be nice but I'd say it should be in documentation. We really want to limit our investment in 1.10 and if we can document workaround and tell people ("but in 2.0 you do not have to do it - so migrate") - this is the best we can do.
   
   SGTM 👍  and thanks for elaborating :)


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk edited a comment on issue #13185: Cannot import name 'BlobServiceClient'

Posted by GitBox <gi...@apache.org>.

potiuk edited a comment on issue #13185:
URL: https://github.com/apache/airflow/issues/13185#issuecomment-781337172

> This is still an issue for me with latest 1.10.x and latest backport providers release.

Let me know if either of those worked.

I hope it helps. Maybe also someone could describe it nicely somewhere else in the docs ;) ?

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] mik-laj commented on issue #13185: Cannot import name 'BlobServiceClient'

Posted by GitBox <gi...@apache.org>.

mik-laj commented on issue #13185:
URL: https://github.com/apache/airflow/issues/13185#issuecomment-748637029


   Related: https://github.com/apache/airflow/issues/12508 https://github.com/apache/airflow/issues/11968 https://github.com/apache/airflow/issues/9745


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org