You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/05/06 21:11:54 UTC

[GitHub] [airflow] kaxil opened a new pull request #8746: Remove old airflow logger causing side effects in tests

kaxil opened a new pull request #8746:
URL: https://github.com/apache/airflow/pull/8746


   https://github.com/apache/airflow/pull/2793 updated the DEFAULT_CONFIG but missed changing it in tests_logging_config.py
   
   That is the reason why `caplog` fixture was not working in https://github.com/apache/airflow/pull/8732 
   
   With the change in this PR the `caplog` fixture works, example:
   
   ```python
   @mock.patch('airflow.plugins_manager.import_errors', return_value={})
   @mock.patch('airflow.plugins_manager.plugins', return_value=[])
   @mock.patch('airflow.plugins_manager.pkg_resources.iter_entry_points')
   def test_entrypoint_plugin_errors_dont_raise_exceptions(
       mock_ep_plugins, mock_plugins, mock_import_errors, caplog
   ):
       """
       Test that Airflow does not raise an Error if there is any Exception because of the
       Plugin.
       """
       from airflow.plugins_manager import load_entrypoint_plugins, import_errors
   
       mock_entrypoint = mock.Mock()
       mock_entrypoint.name = 'test-entrypoint'
       mock_entrypoint.module_name = 'test.plugins.test_plugins_manager'
       mock_entrypoint.load.side_effect = Exception('Version Conflict')
       mock_ep_plugins.return_value = [mock_entrypoint]
   
       # Clear Logs
       caplog.clear()
   
       load_entrypoint_plugins()
   
       # Assert Traceback is shown too
       assert "Traceback (most recent call last):" in caplog.text
       assert "Version Conflict" in caplog.text
       assert "Failed to import plugin test-entrypoint" in caplog.text
       assert "Version Conflict", "test.plugins.test_plugins_manager" in import_errors.items()
   ```
   
   ---
   Make sure to mark the boxes below before creating PR: [x]
   
   - [x] Description above provides context of the change
   - [x] Unit tests coverage for changes (not needed for documentation changes)
   - [x] Target Github ISSUE in description if exists
   - [x] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)"
   - [x] Relevant documentation is updated including usage instructions.
   - [x] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   ---
   In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information.
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] kaxil commented on pull request #8746: Remove old airflow logger causing side effects in tests

Posted by GitBox <gi...@apache.org>.
kaxil commented on pull request #8746:
URL: https://github.com/apache/airflow/pull/8746#issuecomment-624906629


   > This feels like a patch, not fixing the underlying problem (which is that something config is leaking from this test)
   
   Add function to Reset logging


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] kaxil commented on a change in pull request #8746: Remove old airflow logger causing side effects in tests

Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #8746:
URL: https://github.com/apache/airflow/pull/8746#discussion_r421625121



##########
File path: tests/test_logging_config.py
##########
@@ -97,6 +93,33 @@
 SETTINGS_DEFAULT_NAME = 'custom_airflow_local_settings'
 
 
+def reset_logging():
+    """Reset Logging"""
+    manager = logging.root.manager
+    manager.disabled = logging.NOTSET
+    airflow_loggers = [
+        logger for logger_name, logger in manager.loggerDict.items() if logger_name.startswith('airflow')
+    ]
+    for logger in airflow_loggers:  # pylint: disable=too-many-nested-blocks
+        if isinstance(logger, logging.Logger):

Review comment:
       We already do that in `settings_context` but it does seem to clear/reset everything:
   
   https://github.com/apache/airflow/blob/7285e61c42c89799e967ce9f0d391707ee02664d/tests/test_logging_config.py#L186-L187




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] kaxil commented on a change in pull request #8746: Remove old airflow logger causing side effects in tests

Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #8746:
URL: https://github.com/apache/airflow/pull/8746#discussion_r421128887



##########
File path: tests/test_logging_config.py
##########
@@ -97,6 +93,30 @@
 SETTINGS_DEFAULT_NAME = 'custom_airflow_local_settings'
 
 
+def reset_logging():
+    """Reset Logging"""
+    manager = logging.root.manager
+    manager.disabled = logging.NOTSET
+    for logger in manager.loggerDict.values():  # pylint: disable=too-many-nested-blocks
+        if isinstance(logger, logging.Logger):
+            logger.setLevel(logging.NOTSET)
+            logger.propagate = True
+            logger.disabled = False
+            logger.filters.clear()
+            handlers = logger.handlers.copy()
+            for handler in handlers:
+                # Copied from `logging.shutdown`.
+                try:
+                    handler.acquire()
+                    handler.flush()
+                    handler.close()
+                except (OSError, ValueError):
+                    pass
+                finally:
+                    handler.release()
+                logger.removeHandler(handler)

Review comment:
       We can also just use the below code if we want to only reset logging for 'airflow' Loggers:
   
   ```python
       manager = logging.root.manager
       manager.disabled = logging.NOTSET
       airflow_loggers = [
           logger for logger_name, logger in manager.loggerDict.items() if logger_name.startswith('airflow')
       ]
       for logger in airflow_loggers:  # pylint: disable=too-many-nested-blocks
           if isinstance(logger, logging.Logger):
               logger.setLevel(logging.NOTSET)
               logger.propagate = True
               logger.disabled = False
               logger.filters.clear()
               handlers = logger.handlers.copy()
               for handler in handlers:
                   # Copied from `logging.shutdown`.
                   try:
                       handler.acquire()
                       handler.flush()
                       handler.close()
                   except (OSError, ValueError):
                       pass
                   finally:
                       handler.release()
                   logger.removeHandler(handler)
   ```




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] ashb commented on pull request #8746: Remove old airflow logger causing side effects in tests

Posted by GitBox <gi...@apache.org>.
ashb commented on pull request #8746:
URL: https://github.com/apache/airflow/pull/8746#issuecomment-624902269


   This feels like a patch, not fixing the underlying problem (which is that something config is leaking from this test)


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] ashb commented on a change in pull request #8746: Remove old airflow logger causing side effects in tests

Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #8746:
URL: https://github.com/apache/airflow/pull/8746#discussion_r421621414



##########
File path: tests/test_logging_config.py
##########
@@ -97,6 +93,33 @@
 SETTINGS_DEFAULT_NAME = 'custom_airflow_local_settings'
 
 
+def reset_logging():
+    """Reset Logging"""
+    manager = logging.root.manager
+    manager.disabled = logging.NOTSET
+    airflow_loggers = [
+        logger for logger_name, logger in manager.loggerDict.items() if logger_name.startswith('airflow')
+    ]
+    for logger in airflow_loggers:  # pylint: disable=too-many-nested-blocks
+        if isinstance(logger, logging.Logger):

Review comment:
       could we not just reload the logging config?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] kaxil edited a comment on pull request #8746: Remove old airflow logger causing side effects in tests

Posted by GitBox <gi...@apache.org>.
kaxil edited a comment on pull request #8746:
URL: https://github.com/apache/airflow/pull/8746#issuecomment-624906629


   > This feels like a patch, not fixing the underlying problem (which is that something config is leaking from this test)
   
   Added function to Reset logging


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] ashb commented on a change in pull request #8746: Remove old airflow logger causing side effects in tests

Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #8746:
URL: https://github.com/apache/airflow/pull/8746#discussion_r422640517



##########
File path: tests/test_logging_config.py
##########
@@ -97,6 +93,33 @@
 SETTINGS_DEFAULT_NAME = 'custom_airflow_local_settings'
 
 
+def reset_logging():
+    """Reset Logging"""
+    manager = logging.root.manager
+    manager.disabled = logging.NOTSET
+    airflow_loggers = [
+        logger for logger_name, logger in manager.loggerDict.items() if logger_name.startswith('airflow')
+    ]
+    for logger in airflow_loggers:  # pylint: disable=too-many-nested-blocks
+        if isinstance(logger, logging.Logger):

Review comment:
       Probably because of https://github.com/apache/airflow/blob/7285e61c42c89799e967ce9f0d391707ee02664d/airflow/config_templates/airflow_local_settings.py#L59
   
   But we probably want to keep that. Hm.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org