You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/03/26 00:03:47 UTC
[GitHub] [airflow] dstandish opened a new pull request #22539: Reduce verbosity of provider import failure logs & make configurable
dstandish opened a new pull request #22539:
URL: https://github.com/apache/airflow/pull/22539
When running airflow from sources we might not have all deps installed so provider imports fail.
Currently the traceback is logged at warning level which is too much; it obliterates your history.
Instead we make optional the simple reporting of an import failure in a provider, and additionally we move the traceback to DEBUG only (not configurable).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #22539: Reduce verbosity of provider import failure logs & make configurable
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #22539:
URL: https://github.com/apache/airflow/pull/22539#issuecomment-1081095515
I thinl we could ignore all import errors from providers found "via sources" rather than "via packages".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on pull request #22539: Reduce verbosity of provider import failure logs & make configurable
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on pull request #22539:
URL: https://github.com/apache/airflow/pull/22539#issuecomment-1081181094
Just to add more context- unfortunately there is no easy way to to determine if we are in "dev" (editable) environment or not. We could simply check if we are in "BREEZE" (but this is not a full-proof either (we can run `./breeze --use-airflow-version 2.1.4` (and we could add some env variables here too). But it does not cover the case when you run `pip install -e .`. - in this case when you want to modify providers in the env you need to get them available from sources rather than from provider packages.
Detecing whether the provider import comes from a "packaged" provider (i.e. should have all deps installed) or sources (where you get all providers no matter if you have all the deps installed) seems to be the best way to know whether you can ignore import errors or not.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish commented on pull request #22539: Reduce verbosity of provider import failure logs & make configurable
Posted by GitBox <gi...@apache.org>.
dstandish commented on pull request #22539:
URL: https://github.com/apache/airflow/pull/22539#issuecomment-1081174022
@potiuk i looked at that method, it doesn't actually import the provider there, but just stores in `_provider_dict`.
are you thinking we add more information for each provider to `_provider_dict` (e.g. where it comes from, and store in `ProviderInfo` or something) and then use that information to conditionally suppress when importing lazily, later?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] pingzh commented on pull request #22539: Reduce verbosity of provider import failure logs & make configurable
Posted by GitBox <gi...@apache.org>.
pingzh commented on pull request #22539:
URL: https://github.com/apache/airflow/pull/22539#issuecomment-1079610301
there was a discussion in the email group. you can see https://lists.apache.org/thread/m6lsnkn0xdwqopzpv4m7qcvdz6t77w3m for more context.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] pingzh edited a comment on pull request #22539: Reduce verbosity of provider import failure logs & make configurable
Posted by GitBox <gi...@apache.org>.
pingzh edited a comment on pull request #22539:
URL: https://github.com/apache/airflow/pull/22539#issuecomment-1079610301
there was a discussion in the email group. you can see https://lists.apache.org/thread/m6lsnkn0xdwqopzpv4m7qcvdz6t77w3m for more context.
cc @potiuk
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #22539: Reduce verbosity of provider import failure logs & make configurable
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #22539:
URL: https://github.com/apache/airflow/pull/22539#issuecomment-1081181094
Just to add more context- unfortunately there is no easy way to to determine if we are in "dev" (editable) environment or not. We could simply check if we are in "BREEZE" (but this is not a full-proof either (we can run `./breeze --use-airflow-version 2.1.4` (and we could add some env variables here too). But it does not cover the case when you run `pip install -e .`. - in this case when you want to modify providers in the env you need to get them available from sources rather than from provider packages.
Detecint whether the provider import comes from a "packaged" provider (i.e. should have all deps installed) or sources (where you get all providers no matter if you have all the deps installed) seems to be the surest way to know whether you can ignore import errors or not.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish closed pull request #22539: Reduce verbosity of provider import failure logs & make configurable
Posted by GitBox <gi...@apache.org>.
dstandish closed pull request #22539:
URL: https://github.com/apache/airflow/pull/22539
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #22539: Reduce verbosity of provider import failure logs & make configurable
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #22539:
URL: https://github.com/apache/airflow/pull/22539#issuecomment-1081175189
> @potiuk i looked at that method, it doesn't actually import the provider there, but just stores in `_provider_dict`.
>
> are you thinking we add more information for each provider to `_provider_dict` (e.g. where it comes from, and store in `ProviderInfo` or something) and then use that information to conditionally suppress when importing lazily, later?
Yep.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish commented on pull request #22539: Reduce verbosity of provider import failure logs & make configurable
Posted by GitBox <gi...@apache.org>.
dstandish commented on pull request #22539:
URL: https://github.com/apache/airflow/pull/22539#issuecomment-1081047257
> But I guess it's only when you are in "editable" mode, right? We should really suppres the errors in this case not by configuration. It can be actually easily detected (and there is a separate path in Provider's Manager to handle providers from sources, so it should be done there.
if you have a hint i'll take it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish commented on pull request #22539: Reduce verbosity of provider import failure logs & make configurable
Posted by GitBox <gi...@apache.org>.
dstandish commented on pull request #22539:
URL: https://github.com/apache/airflow/pull/22539#issuecomment-1082060478
resolved via other approach in https://github.com/apache/airflow/pull/22579
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on pull request #22539: Reduce verbosity of provider import failure logs & make configurable
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on pull request #22539:
URL: https://github.com/apache/airflow/pull/22539#issuecomment-1081095515
I think we could ignore all import errors from providers found "via sources" rather than "via packages".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #22539: Reduce verbosity of provider import failure logs & make configurable
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #22539:
URL: https://github.com/apache/airflow/pull/22539#issuecomment-1079763876
I think the problem here is that we hav duplicate warnings generated for some of the providers (am I right about it @dstandish ?) an I think it comes from the fact that 4 of the providers are both installed by default and also available in the sources - and we should solve this in Breeze/Dev environment rather than adding a config setting that is "development only". I think I saw it few times too.... But I am not sure if this is the error you try to deal with @dstandish :).
Do I get it right? I am happy to take a close look at this and fix it "better" @dstandish - can you briefly desribe (copy&paste) the scenario/output logs you experience?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #22539: Reduce verbosity of provider import failure logs & make configurable
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #22539:
URL: https://github.com/apache/airflow/pull/22539#issuecomment-1079904285
But I guess it's only when you are in "editable" mode, right? We should really suppres the errors in this case not by configuration. It can be actually easily detected (and there is a separate path in Provider's Manager to handle providers from sources, so it should be done there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on pull request #22539: Reduce verbosity of provider import failure logs & make configurable
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on pull request #22539:
URL: https://github.com/apache/airflow/pull/22539#issuecomment-1081181094
Just to add more context- unfortunately there is no easy way to to determine if we are in "dev" (editable) environment or not. We could simply check if we are in "BREEZE" (but this is not a full-proof either (we can run `./breeze --use-airflow-version 2.1.4` (and we could add some env variables here too). But it does not cover the case when you run `pip install -e .`. - in this case when you want to modify providers in the env you need to get them available from sources rather than from provider packages.
Detecing whether the provider import comes from a "packaged" provider (i.e. should have all deps installed) or sources (where you get all providers no matter if you have all the deps installed) seems to be the surest way to know whether you can ignore import errors or not.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish commented on pull request #22539: Reduce verbosity of provider import failure logs & make configurable
Posted by GitBox <gi...@apache.org>.
dstandish commented on pull request #22539:
URL: https://github.com/apache/airflow/pull/22539#issuecomment-1079698871
I see yeah that PR made it possible for providers to have optional components.
I think this PR still should be merged.
What it does is
(1) reduce verbosity by removing traceback from the warning (so now it's just one line)
(2) add DEBUG logging that includes the traceback (so users can get more detail by adding debug logging)
(3) make it optional to suppress the one-line import error warning by configuration.
This is a painful issue for developers who use virtualenv and it seems like a reasonable solution to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #22539: Reduce verbosity of provider import failure logs & make configurable
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #22539:
URL: https://github.com/apache/airflow/pull/22539#issuecomment-1081094040
https://github.com/apache/airflow/blob/e1a42c4fc8a634852dd5ac5b16cade620851477f/airflow/providers_manager.py#L390
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish commented on pull request #22539: Reduce verbosity of provider import failure logs & make configurable
Posted by GitBox <gi...@apache.org>.
dstandish commented on pull request #22539:
URL: https://github.com/apache/airflow/pull/22539#issuecomment-1079833098
well the scenario is, when using virtualenv, if you don't have some deps for providers, then when you run webserver you'll get error logging over and over and over.
at a minimum we should suppress traceback except when log level is debug
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org