You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/05/06 21:11:54 UTC
[GitHub] [airflow] kaxil opened a new pull request #8746: Remove old airflow logger causing side effects in tests
kaxil opened a new pull request #8746:
URL: https://github.com/apache/airflow/pull/8746
https://github.com/apache/airflow/pull/2793 updated the DEFAULT_CONFIG but missed changing it in tests_logging_config.py
That is the reason why `caplog` fixture was not working in https://github.com/apache/airflow/pull/8732
With the change in this PR the `caplog` fixture works, example:
```python
@mock.patch('airflow.plugins_manager.import_errors', return_value={})
@mock.patch('airflow.plugins_manager.plugins', return_value=[])
@mock.patch('airflow.plugins_manager.pkg_resources.iter_entry_points')
def test_entrypoint_plugin_errors_dont_raise_exceptions(
mock_ep_plugins, mock_plugins, mock_import_errors, caplog
):
"""
Test that Airflow does not raise an Error if there is any Exception because of the
Plugin.
"""
from airflow.plugins_manager import load_entrypoint_plugins, import_errors
mock_entrypoint = mock.Mock()
mock_entrypoint.name = 'test-entrypoint'
mock_entrypoint.module_name = 'test.plugins.test_plugins_manager'
mock_entrypoint.load.side_effect = Exception('Version Conflict')
mock_ep_plugins.return_value = [mock_entrypoint]
# Clear Logs
caplog.clear()
load_entrypoint_plugins()
# Assert Traceback is shown too
assert "Traceback (most recent call last):" in caplog.text
assert "Version Conflict" in caplog.text
assert "Failed to import plugin test-entrypoint" in caplog.text
assert "Version Conflict", "test.plugins.test_plugins_manager" in import_errors.items()
```
---
Make sure to mark the boxes below before creating PR: [x]
- [x] Description above provides context of the change
- [x] Unit tests coverage for changes (not needed for documentation changes)
- [x] Target Github ISSUE in description if exists
- [x] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)"
- [x] Relevant documentation is updated including usage instructions.
- [x] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
---
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on pull request #8746: Remove old airflow logger causing side effects in tests
Posted by GitBox <gi...@apache.org>.
kaxil commented on pull request #8746:
URL: https://github.com/apache/airflow/pull/8746#issuecomment-624906629
> This feels like a patch, not fixing the underlying problem (which is that something config is leaking from this test)
Add function to Reset logging
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on a change in pull request #8746: Remove old airflow logger causing side effects in tests
Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #8746:
URL: https://github.com/apache/airflow/pull/8746#discussion_r421625121
##########
File path: tests/test_logging_config.py
##########
@@ -97,6 +93,33 @@
SETTINGS_DEFAULT_NAME = 'custom_airflow_local_settings'
+def reset_logging():
+ """Reset Logging"""
+ manager = logging.root.manager
+ manager.disabled = logging.NOTSET
+ airflow_loggers = [
+ logger for logger_name, logger in manager.loggerDict.items() if logger_name.startswith('airflow')
+ ]
+ for logger in airflow_loggers: # pylint: disable=too-many-nested-blocks
+ if isinstance(logger, logging.Logger):
Review comment:
We already do that in `settings_context` but it does seem to clear/reset everything:
https://github.com/apache/airflow/blob/7285e61c42c89799e967ce9f0d391707ee02664d/tests/test_logging_config.py#L186-L187
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on a change in pull request #8746: Remove old airflow logger causing side effects in tests
Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #8746:
URL: https://github.com/apache/airflow/pull/8746#discussion_r421128887
##########
File path: tests/test_logging_config.py
##########
@@ -97,6 +93,30 @@
SETTINGS_DEFAULT_NAME = 'custom_airflow_local_settings'
+def reset_logging():
+ """Reset Logging"""
+ manager = logging.root.manager
+ manager.disabled = logging.NOTSET
+ for logger in manager.loggerDict.values(): # pylint: disable=too-many-nested-blocks
+ if isinstance(logger, logging.Logger):
+ logger.setLevel(logging.NOTSET)
+ logger.propagate = True
+ logger.disabled = False
+ logger.filters.clear()
+ handlers = logger.handlers.copy()
+ for handler in handlers:
+ # Copied from `logging.shutdown`.
+ try:
+ handler.acquire()
+ handler.flush()
+ handler.close()
+ except (OSError, ValueError):
+ pass
+ finally:
+ handler.release()
+ logger.removeHandler(handler)
Review comment:
We can also just use the below code if we want to only reset logging for 'airflow' Loggers:
```python
manager = logging.root.manager
manager.disabled = logging.NOTSET
airflow_loggers = [
logger for logger_name, logger in manager.loggerDict.items() if logger_name.startswith('airflow')
]
for logger in airflow_loggers: # pylint: disable=too-many-nested-blocks
if isinstance(logger, logging.Logger):
logger.setLevel(logging.NOTSET)
logger.propagate = True
logger.disabled = False
logger.filters.clear()
handlers = logger.handlers.copy()
for handler in handlers:
# Copied from `logging.shutdown`.
try:
handler.acquire()
handler.flush()
handler.close()
except (OSError, ValueError):
pass
finally:
handler.release()
logger.removeHandler(handler)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on pull request #8746: Remove old airflow logger causing side effects in tests
Posted by GitBox <gi...@apache.org>.
ashb commented on pull request #8746:
URL: https://github.com/apache/airflow/pull/8746#issuecomment-624902269
This feels like a patch, not fixing the underlying problem (which is that something config is leaking from this test)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on a change in pull request #8746: Remove old airflow logger causing side effects in tests
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #8746:
URL: https://github.com/apache/airflow/pull/8746#discussion_r421621414
##########
File path: tests/test_logging_config.py
##########
@@ -97,6 +93,33 @@
SETTINGS_DEFAULT_NAME = 'custom_airflow_local_settings'
+def reset_logging():
+ """Reset Logging"""
+ manager = logging.root.manager
+ manager.disabled = logging.NOTSET
+ airflow_loggers = [
+ logger for logger_name, logger in manager.loggerDict.items() if logger_name.startswith('airflow')
+ ]
+ for logger in airflow_loggers: # pylint: disable=too-many-nested-blocks
+ if isinstance(logger, logging.Logger):
Review comment:
could we not just reload the logging config?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil edited a comment on pull request #8746: Remove old airflow logger causing side effects in tests
Posted by GitBox <gi...@apache.org>.
kaxil edited a comment on pull request #8746:
URL: https://github.com/apache/airflow/pull/8746#issuecomment-624906629
> This feels like a patch, not fixing the underlying problem (which is that something config is leaking from this test)
Added function to Reset logging
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on a change in pull request #8746: Remove old airflow logger causing side effects in tests
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #8746:
URL: https://github.com/apache/airflow/pull/8746#discussion_r422640517
##########
File path: tests/test_logging_config.py
##########
@@ -97,6 +93,33 @@
SETTINGS_DEFAULT_NAME = 'custom_airflow_local_settings'
+def reset_logging():
+ """Reset Logging"""
+ manager = logging.root.manager
+ manager.disabled = logging.NOTSET
+ airflow_loggers = [
+ logger for logger_name, logger in manager.loggerDict.items() if logger_name.startswith('airflow')
+ ]
+ for logger in airflow_loggers: # pylint: disable=too-many-nested-blocks
+ if isinstance(logger, logging.Logger):
Review comment:
Probably because of https://github.com/apache/airflow/blob/7285e61c42c89799e967ce9f0d391707ee02664d/airflow/config_templates/airflow_local_settings.py#L59
But we probably want to keep that. Hm.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org