You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/05/10 21:46:19 UTC
[GitHub] [airflow] dstandish opened a new issue #15770: google provider requires leveldb in host
dstandish opened a new issue #15770:
URL: https://github.com/apache/airflow/issues/15770
google provider now requires plyvel, which is used by leveldb hook.
to install plyvel, the user needs leveldb headers installed in the system, otherwise it will fail:
```
plyvel/_plyvel.cpp:632:10: fatal error: 'leveldb/db.h' file not found
#include "leveldb/db.h"
^~~~~~~~~~~~~~
1 error generated.
error: command 'gcc' failed with exit status 1
----------------------------------------
ERROR: Failed building wheel for plyvel
```
the average GCP user may not use leveldb and it seems a little burdernsome to requirem its installation just to use GCP hooks.
perhaps there is a way to make this dependency optional?
or perhaps is there a way to install only certain components of a provider?
cc: @mik-laj
related: https://github.com/apache/airflow/pull/14105
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish commented on issue #15770: google provider requires leveldb installed in system
Posted by GitBox <gi...@apache.org>.
dstandish commented on issue #15770:
URL: https://github.com/apache/airflow/issues/15770#issuecomment-839004079
> Plyvel is really such a niche thing that it's worth making it optional if only for the convenience of developers who want to develop GCP provider..
Keep in mind this is not just about airflow developers
This is about any dag developer that uses mac and virtualenv, and simply has a repo that requires GCP
That's a lot of people.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #15770: google provider requires leveldb installed in system
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #15770:
URL: https://github.com/apache/airflow/issues/15770#issuecomment-837993482
It's mostly for development convenience. Plyvel is really such a `niche` thing that it's worth making it optional if only for the convenience of developers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish edited a comment on issue #15770: google provider requires leveldb installed in system
Posted by GitBox <gi...@apache.org>.
dstandish edited a comment on issue #15770:
URL: https://github.com/apache/airflow/issues/15770#issuecomment-839004079
> Plyvel is really such a niche thing that it's worth making it optional if only for the convenience of developers who want to develop GCP provider..
Keep in mind this is not just about _airflow_ developers
This is about any dag developer that uses mac and virtualenv, and simply has a repo that requires GCP
That's a lot of people.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #15770: google provider requires leveldb installed in system
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #15770:
URL: https://github.com/apache/airflow/issues/15770#issuecomment-938071072
Why not - will you make PR for that @javatarz ? I kept on using devel anyway.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on issue #15770: google provider requires leveldb installed in system
Posted by GitBox <gi...@apache.org>.
uranusjr commented on issue #15770:
URL: https://github.com/apache/airflow/issues/15770#issuecomment-837975911
But is macOS a supported production system? I’ve thought it’s only supported as a development platform…
Either way, it’s a good idea to split plyvel out anyway. Not every Linux is manylinux and there might be some people having problems with this e.g. on Alpine.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #15770: google provider requires leveldb installed in system
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #15770:
URL: https://github.com/apache/airflow/issues/15770#issuecomment-837968561
I think it's a good idea. We already have the possibility of adding optional dependencies in providers, so it should be rather easy thing to do maybe you would like to add it @dstandish ?
See: https://github.com/apache/airflow/blob/996965aad9874e9c6dad0a1f147d779adc462278/airflow/providers/apache/beam/provider.yaml#L46
And
https://github.com/apache/airflow/blob/996965aad9874e9c6dad0a1f147d779adc462278/airflow/providers/google/provider.yaml#L750
It will be just a matter of adding "plyvel" additional extra and moving plyvel out of the 'google' extra dependencies (+ some documentation)..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #15770: google provider requires leveldb installed in system
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #15770:
URL: https://github.com/apache/airflow/issues/15770
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] javatarz commented on issue #15770: google provider requires leveldb installed in system
Posted by GitBox <gi...@apache.org>.
javatarz commented on issue #15770:
URL: https://github.com/apache/airflow/issues/15770#issuecomment-938052247
Until this one's fixed on Mac, it might be worth updating `pip install -e ".[devel_all]"` to `pip install -e ".[devel]"` in [breeze](https://github.com/apache/airflow/blob/main/breeze#L261).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish edited a comment on issue #15770: google provider requires leveldb installed in system
Posted by GitBox <gi...@apache.org>.
dstandish edited a comment on issue #15770:
URL: https://github.com/apache/airflow/issues/15770#issuecomment-839007915
> I think it's a good idea. We already have the possibility of adding optional dependencies in providers, so it should be rather easy thing to do maybe you would like to add it @dstandish ?
Consider me interested. I will try to find a minute to do so.
To be clear, you recommend making `leveldb` an optional extra within the google provider? But not splitting google into google-gcp and google-leveldb?
It's not the case that there is some magic such that you can do various options including `google` and get all (minus the optional extras), and do `google-gcp` to only get gcp, or do `google-leveldb`and get only the leveldb extra?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #15770: google provider requires leveldb installed in system
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #15770:
URL: https://github.com/apache/airflow/issues/15770#issuecomment-938071072
Why not - will you make PR for that @javatarz ? I kept on using devel anyway.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish commented on issue #15770: google provider requires leveldb installed in system
Posted by GitBox <gi...@apache.org>.
dstandish commented on issue #15770:
URL: https://github.com/apache/airflow/issues/15770#issuecomment-839007915
> I think it's a good idea. We already have the possibility of adding optional dependencies in providers, so it should be rather easy thing to do maybe you would like to add it @dstandish ?
Consider me interested. I will try to find a minute to do so.
To be clear, you recommend making `leveldb` an optional extra within the google provider? But not splitting google into google-gcp and google-leveldb?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #15770: google provider requires leveldb installed in system
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #15770:
URL: https://github.com/apache/airflow/issues/15770#issuecomment-839559243
We SHOULD do it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish edited a comment on issue #15770: google provider requires leveldb installed in system
Posted by GitBox <gi...@apache.org>.
dstandish edited a comment on issue #15770:
URL: https://github.com/apache/airflow/issues/15770#issuecomment-839004079
> Plyvel is really such a niche thing that it's worth making it optional if only for the convenience of developers who want to develop GCP provider..
Keep in mind this is not just about _airflow_ developers
This is about any dag developer that uses mac and virtualenv, and simply has a repo that requires GCP
That's a lot of people.
As a cluster maintainer, you have to manage dev env setup scripts and documentation. While I might have steps for installing certain odbc drivers to connect to a specific resource, I am generally able to avoid having setup steps where you need to brew install this or that before airflow will even install. You want your readme ideally to be minimally `pip install -r requirements.txt`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #15770: google provider requires leveldb installed in system
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #15770:
URL: https://github.com/apache/airflow/issues/15770#issuecomment-839059303
> To be clear, you recommend making `leveldb` an optional extra within the google provider? But not splitting google into google-gcp and google-leveldb?
Correct.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] javatarz commented on issue #15770: google provider requires leveldb installed in system
Posted by GitBox <gi...@apache.org>.
javatarz commented on issue #15770:
URL: https://github.com/apache/airflow/issues/15770#issuecomment-938100595
I'd be happy to. Incoming PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #15770: google provider requires leveldb installed in system
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #15770:
URL: https://github.com/apache/airflow/issues/15770
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #15770: google provider requires leveldb installed in system
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #15770:
URL: https://github.com/apache/airflow/issues/15770#issuecomment-837969716
> Doesn’t `plyvel` publish manylinux wheels and does not require compilation on most system?
They do - but it does not work for MacOS.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish edited a comment on issue #15770: google provider requires leveldb installed in system
Posted by GitBox <gi...@apache.org>.
dstandish edited a comment on issue #15770:
URL: https://github.com/apache/airflow/issues/15770#issuecomment-839007915
> I think it's a good idea. We already have the possibility of adding optional dependencies in providers, so it should be rather easy thing to do maybe you would like to add it @dstandish ?
Consider me interested. I will try to find a minute to do so.
To be clear, you recommend making `leveldb` an optional extra within the google provider? But not splitting google into google-gcp and google-leveldb?
It's not the case, is it, that there is some magic such that you can do various options including `google` and get all (minus the optional extras), and do `google-gcp` to only get gcp, or do `google-leveldb`and get only the leveldb extra?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish commented on issue #15770: google provider requires leveldb installed in system
Posted by GitBox <gi...@apache.org>.
dstandish commented on issue #15770:
URL: https://github.com/apache/airflow/issues/15770#issuecomment-839949104
I may not be able to get to it this weekend ... Traveling... but will let you know
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #15770: google provider requires leveldb installed in system
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #15770:
URL: https://github.com/apache/airflow/issues/15770#issuecomment-837968561
I think it's a good idea. We already have the possibility of adding optional dependencies in providers, so it should be rather easy thing to do @dstandish maybe you would like to add it @dstandish ?
See: https://github.com/apache/airflow/blob/996965aad9874e9c6dad0a1f147d779adc462278/airflow/providers/apache/beam/provider.yaml#L46
And
https://github.com/apache/airflow/blob/996965aad9874e9c6dad0a1f147d779adc462278/airflow/providers/google/provider.yaml#L750
It will be just a matter of adding "plyvel" additional extra and moving plyvel out of the 'google' extra dependencies (+ some documentation)..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] javatarz commented on issue #15770: google provider requires leveldb installed in system
Posted by GitBox <gi...@apache.org>.
javatarz commented on issue #15770:
URL: https://github.com/apache/airflow/issues/15770#issuecomment-938052247
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish edited a comment on issue #15770: google provider requires leveldb installed in system
Posted by GitBox <gi...@apache.org>.
dstandish edited a comment on issue #15770:
URL: https://github.com/apache/airflow/issues/15770#issuecomment-839949104
I may not be able to get to it this weekend ... Traveling... but will let you know... anyone else please take if you have time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish edited a comment on issue #15770: google provider requires leveldb installed in system
Posted by GitBox <gi...@apache.org>.
dstandish edited a comment on issue #15770:
URL: https://github.com/apache/airflow/issues/15770#issuecomment-839004079
> Plyvel is really such a niche thing that it's worth making it optional if only for the convenience of developers who want to develop GCP provider..
Keep in mind this is not just about _airflow_ developers
This is about any dag developer that uses mac and virtualenv, and simply has a repo that requires GCP
That's a lot of people.
As a cluster maintainer, you have to manage dev env setup scripts and documentation. While I might have steps for installing certain odbc drivers to connect to a specific resource, I am generally able to avoid having setup steps where you need to brew install this or that before airflow will even install. You want your minimal setup ideally to be `pip install -r requirements.txt`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish edited a comment on issue #15770: google provider requires leveldb installed in system
Posted by GitBox <gi...@apache.org>.
dstandish edited a comment on issue #15770:
URL: https://github.com/apache/airflow/issues/15770#issuecomment-839007915
> I think it's a good idea. We already have the possibility of adding optional dependencies in providers, so it should be rather easy thing to do maybe you would like to add it @dstandish ?
Consider me interested. I will try to find a minute to do so.
To be clear, you recommend making `leveldb` an optional extra within the google provider? But not splitting google into google-gcp and google-leveldb?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #15770: google provider requires leveldb installed in system
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #15770:
URL: https://github.com/apache/airflow/issues/15770#issuecomment-839559018
I hit the same problem on linux today. When installing python 3.6 version of airflow for whatever reason plyvel failed to compile :(
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #15770: google provider requires leveldb installed in system
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #15770:
URL: https://github.com/apache/airflow/issues/15770#issuecomment-837993482
It's mostly for development convenience. Plyvel is really such a `niche` thing that it's worth making it optional if only for the convenience of developers who want to develop GCP provider..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on issue #15770: google provider requires leveldb installed in system
Posted by GitBox <gi...@apache.org>.
uranusjr commented on issue #15770:
URL: https://github.com/apache/airflow/issues/15770#issuecomment-837968951
Doesn’t `plyvel` publish manylinux wheels and does not require compilation on most system?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org