You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/03/19 21:34:53 UTC
[GitHub] [airflow] potiuk opened a new pull request #14903: Turn provider's import warnings into debug logs
potiuk opened a new pull request #14903:
URL: https://github.com/apache/airflow/pull/14903
When only some providers are installed, cross-dependencies between
the providers might cause import erors. Those import errors are
repeated in webserver as it is reloaded every 30 seconds.
Since "only some providers" case is valid, it should not generate
warrnings if it is an ImportError. Those warnings are now
turned into debug messages.
Fixes: #14286
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #14903: Turn provider's import warnings into debug logs
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #14903:
URL: https://github.com/apache/airflow/pull/14903#discussion_r597991798
##########
File path: airflow/providers_manager.py
##########
@@ -281,7 +281,7 @@ def _add_hook(self, hook_class_name: str, provider_package: str) -> None:
self._add_customized_fields(provider_package, hook_class, field_behaviours)
except Exception as e: # noqa pylint: disable=broad-except
- if os.environ.get("AIRFLOW_INSTALLATION_METHOD") != ".":
+ if os.environ.get("AIRFLOW_INSTALLATION_METHOD") != "." and not isinstance(e, ImportError):
Review comment:
Yep. Good point.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #14903: Turn provider's import warnings into debug logs
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #14903:
URL: https://github.com/apache/airflow/pull/14903#issuecomment-803159898
I simplified it a it more and added comment @ashb -> i simply turn all the ImportErrors into debug.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #14903: Turn provider's import warnings into debug logs
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #14903:
URL: https://github.com/apache/airflow/pull/14903#issuecomment-803166633
Should be updaed now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk merged pull request #14903: Turn provider's import warnings into debug logs
Posted by GitBox <gi...@apache.org>.
potiuk merged pull request #14903:
URL: https://github.com/apache/airflow/pull/14903
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on a change in pull request #14903: Turn provider's import warnings into debug logs
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #14903:
URL: https://github.com/apache/airflow/pull/14903#discussion_r597988410
##########
File path: airflow/providers_manager.py
##########
@@ -281,7 +281,7 @@ def _add_hook(self, hook_class_name: str, provider_package: str) -> None:
self._add_customized_fields(provider_package, hook_class, field_behaviours)
except Exception as e: # noqa pylint: disable=broad-except
- if os.environ.get("AIRFLOW_INSTALLATION_METHOD") != ".":
+ if os.environ.get("AIRFLOW_INSTALLATION_METHOD") != "." and not isinstance(e, ImportError):
Review comment:
Could you update the commend in the else block to refer to this case/behaviour please?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #14903: Turn provider's import warnings into debug logs
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #14903:
URL: https://github.com/apache/airflow/pull/14903#discussion_r597999139
##########
File path: airflow/providers_manager.py
##########
@@ -281,22 +281,17 @@ def _add_hook(self, hook_class_name: str, provider_package: str) -> None:
self._add_customized_fields(provider_package, hook_class, field_behaviours)
except Exception as e: # noqa pylint: disable=broad-except
- if os.environ.get("AIRFLOW_INSTALLATION_METHOD") != ".":
- # print providers manager warning when airflow is not installed from sources in the
- # production image
- log.warning(
+ if isinstance(e, ImportError):
Review comment:
Yep. Can be this way as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #14903: Turn provider's import warnings into debug logs
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #14903:
URL: https://github.com/apache/airflow/pull/14903#issuecomment-803175353
The PR is likely OK to be merged with just subset of tests for default Python and Database versions without running the full matrix of tests, because it does not modify the core of Airflow. If the committers decide that the full tests matrix is needed, they will add the label 'full tests needed'. Then you should rebase to the latest master or amend the last commit of the PR, and push it with --force-with-lease.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on a change in pull request #14903: Turn provider's import warnings into debug logs
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #14903:
URL: https://github.com/apache/airflow/pull/14903#discussion_r597997999
##########
File path: airflow/providers_manager.py
##########
@@ -281,22 +281,17 @@ def _add_hook(self, hook_class_name: str, provider_package: str) -> None:
self._add_customized_fields(provider_package, hook_class, field_behaviours)
except Exception as e: # noqa pylint: disable=broad-except
- if os.environ.get("AIRFLOW_INSTALLATION_METHOD") != ".":
- # print providers manager warning when airflow is not installed from sources in the
- # production image
- log.warning(
+ if isinstance(e, ImportError):
Review comment:
In which case:
```
except ImportError as e:
# When there is an ImportError we turn it into debug warnings as this is
# an expected case when only some providers are installed
log.debug(
"Exception when importing '%s' from '%s' package: %s",
hook_class_name,
provider_package,
e,
)
except Exception as e: # noqa pylint: disable=broad-except
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org