You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/03/16 07:59:48 UTC

[GitHub] [airflow] suiting-young opened a new pull request #14823: `logging.exception` redundancy.

suiting-young opened a new pull request #14823:
URL: https://github.com/apache/airflow/pull/14823


   Having both `logging.exception` and any other `logging` methods side by side seems unnecessary, and may lead to misread.
   - Per [doc](https://docs.python.org/3/library/logging.html#logging.Logger.exception), the first argument of `logging.exception` is a string-like `msg`.
   - `logging.exception(msg)` ≡ `logging.error(msg, exc_info=True)`
   - `exc_info=True` will add `str(exception)` at the end of trace anyway.
   
   <!--
   Thank you for contributing! Please make sure that your code changes
   are covered with tests. And in case of new features or big changes
   remember to adjust the documentation.
   
   Feel free to ping committers for the review!
   
   In case of existing issue, reference it using one of the following:
   
   closes: #ISSUE
   related: #ISSUE
   
   How to write a good git commit message:
   http://chris.beams.io/posts/git-commit/
   -->
   
   ---
   **^ Add meaningful description above**
   
   Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
   In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] suiting-young commented on a change in pull request #14823: `logging.exception` redundancy.

Posted by GitBox <gi...@apache.org>.
suiting-young commented on a change in pull request #14823:
URL: https://github.com/apache/airflow/pull/14823#discussion_r594956730



##########
File path: airflow/models/taskinstance.py
##########
@@ -1297,9 +1297,10 @@ def signal_handler(signum, frame):  # pylint: disable=unused-argument
                     registered = task_copy.register_in_sensor_service(self, context)
                 except Exception as e:
                     self.log.warning(
-                        "Failed to register in sensor service.Continue to run task in non smart sensor mode."
+                        "Failed to register in sensor service."
+                        "Continue to run task in non smart sensor mode.",
+                        exc_info=True
                     )
-                    self.log.exception(e, exc_info=True)

Review comment:
       er, I may misunderstood.
   
   Do you mean that there is use case to set logging level to hide/skip the warning `msg` but keep the `exception` and its trace?
   
   I didn't tweak logging level before in my need, would like to hear that in more details.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] suiting-young commented on a change in pull request #14823: `logging.exception` redundancy.

Posted by GitBox <gi...@apache.org>.
suiting-young commented on a change in pull request #14823:
URL: https://github.com/apache/airflow/pull/14823#discussion_r594960662



##########
File path: airflow/models/connection.py
##########
@@ -333,9 +333,8 @@ def extra_dejson(self) -> Dict:
         if self.extra:
             try:
                 obj = json.loads(self.extra)
-            except JSONDecodeError as e:
-                self.log.exception(e)
-                self.log.error("Failed parsing the json for conn_id %s", self.conn_id)
+            except JSONDecodeError:

Review comment:
       That's why I say "may lead to misread".
   
   `logging.exception` is **NOT** going to log the `e` object you provided,
   but log the the `msg` (_`str(e)` here_), then fetch the trace (_as instructed by `exc_info`_) from the stack.
   
   I learnt this recently... LoL




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] suiting-young commented on a change in pull request #14823: `logging.exception` redundancy.

Posted by GitBox <gi...@apache.org>.
suiting-young commented on a change in pull request #14823:
URL: https://github.com/apache/airflow/pull/14823#discussion_r595733321



##########
File path: airflow/models/taskinstance.py
##########
@@ -1297,9 +1297,10 @@ def signal_handler(signum, frame):  # pylint: disable=unused-argument
                     registered = task_copy.register_in_sensor_service(self, context)
                 except Exception as e:
                     self.log.warning(
-                        "Failed to register in sensor service.Continue to run task in non smart sensor mode."
+                        "Failed to register in sensor service."
+                        "Continue to run task in non smart sensor mode.",
+                        exc_info=True
                     )
-                    self.log.exception(e, exc_info=True)

Review comment:
       Generally, I agree with you. It's not acceptable to drop critical information.
   ```py
   log.warning('descriptive message')
   log.exception('critical message')  # hide or downgrade this is unacceptable
   ```
   
   But in these specific situations (*total 6 cases in this PR*),
   all of them (*`exception`*) didn't provide useful (*than their `warning` partners*) information,
   but looks really like to **attach** stack trace. I'd believe they are a bit of misuse.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] kaxil merged pull request #14823: `logging.exception` redundancy.

Posted by GitBox <gi...@apache.org>.
kaxil merged pull request #14823:
URL: https://github.com/apache/airflow/pull/14823


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] uranusjr commented on a change in pull request #14823: `logging.exception` redundancy.

Posted by GitBox <gi...@apache.org>.
uranusjr commented on a change in pull request #14823:
URL: https://github.com/apache/airflow/pull/14823#discussion_r594936998



##########
File path: airflow/models/taskinstance.py
##########
@@ -1297,9 +1297,10 @@ def signal_handler(signum, frame):  # pylint: disable=unused-argument
                     registered = task_copy.register_in_sensor_service(self, context)
                 except Exception as e:
                     self.log.warning(
-                        "Failed to register in sensor service.Continue to run task in non smart sensor mode."
+                        "Failed to register in sensor service."
+                        "Continue to run task in non smart sensor mode.",
+                        exc_info=True
                     )
-                    self.log.exception(e, exc_info=True)

Review comment:
       This (and some more other changes) makes the message warning-only (previously there’d be one warning and one error message each) and may affect compatibility.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] ashb commented on a change in pull request #14823: `logging.exception` redundancy.

Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #14823:
URL: https://github.com/apache/airflow/pull/14823#discussion_r596032427



##########
File path: airflow/models/taskinstance.py
##########
@@ -1297,9 +1297,10 @@ def signal_handler(signum, frame):  # pylint: disable=unused-argument
                     registered = task_copy.register_in_sensor_service(self, context)
                 except Exception as e:
                     self.log.warning(
-                        "Failed to register in sensor service.Continue to run task in non smart sensor mode."
+                        "Failed to register in sensor service."
+                        "Continue to run task in non smart sensor mode.",
+                        exc_info=True
                     )
-                    self.log.exception(e, exc_info=True)

Review comment:
       This change as a warning makes sense. 👍🏻 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] github-actions[bot] commented on pull request #14823: `logging.exception` redundancy.

Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #14823:
URL: https://github.com/apache/airflow/pull/14823#issuecomment-822914389


   [The Workflow run](https://github.com/apache/airflow/actions/runs/765594357) is cancelling this PR. Building images for the PR has failed. Follow the workflow link to check the reason.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] github-actions[bot] commented on pull request #14823: `logging.exception` redundancy.

Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #14823:
URL: https://github.com/apache/airflow/pull/14823#issuecomment-800123509


   [The Workflow run](https://github.com/apache/airflow/actions/runs/657087957) is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] XD-DENG commented on a change in pull request #14823: `logging.exception` redundancy.

Posted by GitBox <gi...@apache.org>.
XD-DENG commented on a change in pull request #14823:
URL: https://github.com/apache/airflow/pull/14823#discussion_r594965410



##########
File path: airflow/models/connection.py
##########
@@ -333,9 +333,8 @@ def extra_dejson(self) -> Dict:
         if self.extra:
             try:
                 obj = json.loads(self.extra)
-            except JSONDecodeError as e:
-                self.log.exception(e)
-                self.log.error("Failed parsing the json for conn_id %s", self.conn_id)
+            except JSONDecodeError:

Review comment:
       I see what you mean, and it's a valid point.
   
   Please address @uranusjr 's comment though. It's relating to logging level given eventually.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] suiting-young commented on a change in pull request #14823: `logging.exception` redundancy.

Posted by GitBox <gi...@apache.org>.
suiting-young commented on a change in pull request #14823:
URL: https://github.com/apache/airflow/pull/14823#discussion_r594950498



##########
File path: airflow/models/taskinstance.py
##########
@@ -1297,9 +1297,10 @@ def signal_handler(signum, frame):  # pylint: disable=unused-argument
                     registered = task_copy.register_in_sensor_service(self, context)
                 except Exception as e:
                     self.log.warning(
-                        "Failed to register in sensor service.Continue to run task in non smart sensor mode."
+                        "Failed to register in sensor service."
+                        "Continue to run task in non smart sensor mode.",
+                        exc_info=True
                     )
-                    self.log.exception(e, exc_info=True)

Review comment:
       Removed `as` in aa94e4a556776daaa7c65df68abd0ea451e5ec2f. Thanks for your advice!
   
   All `logging` methods, include `warning`, supports `exc_info` keyword argument.
   So for those statements, I just added the argument.
   ```patch
   - log.warning(msg)
   + log.warning(msg, exc_info=True)
   ```




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] turbaszek commented on pull request #14823: `logging.exception` redundancy.

Posted by GitBox <gi...@apache.org>.
turbaszek commented on pull request #14823:
URL: https://github.com/apache/airflow/pull/14823#issuecomment-826049835


   @suiting-young would you mind rebasing your PR?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] github-actions[bot] commented on pull request #14823: `logging.exception` redundancy.

Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #14823:
URL: https://github.com/apache/airflow/pull/14823#issuecomment-811976194


   The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest master at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] github-actions[bot] commented on pull request #14823: `logging.exception` redundancy.

Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #14823:
URL: https://github.com/apache/airflow/pull/14823#issuecomment-800124103






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] eladkal commented on pull request #14823: `logging.exception` redundancy.

Posted by GitBox <gi...@apache.org>.
eladkal commented on pull request #14823:
URL: https://github.com/apache/airflow/pull/14823#issuecomment-816063620


   @suiting-young can you take a look at the failing test?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] github-actions[bot] commented on pull request #14823: `logging.exception` redundancy.

Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #14823:
URL: https://github.com/apache/airflow/pull/14823#issuecomment-822390181


   [The Workflow run](https://github.com/apache/airflow/actions/runs/763346391) is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] suiting-young commented on pull request #14823: `logging.exception` redundancy.

Posted by GitBox <gi...@apache.org>.
suiting-young commented on pull request #14823:
URL: https://github.com/apache/airflow/pull/14823#issuecomment-822327665


   > @suiting-young can you take a look at the failing test?
   
   @eladkal Sorry for the delay. I got connection problem recently (_from China_), and finally found a workaround. 😋 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] uranusjr commented on a change in pull request #14823: `logging.exception` redundancy.

Posted by GitBox <gi...@apache.org>.
uranusjr commented on a change in pull request #14823:
URL: https://github.com/apache/airflow/pull/14823#discussion_r595797570



##########
File path: airflow/models/taskinstance.py
##########
@@ -1297,9 +1297,10 @@ def signal_handler(signum, frame):  # pylint: disable=unused-argument
                     registered = task_copy.register_in_sensor_service(self, context)
                 except Exception as e:
                     self.log.warning(
-                        "Failed to register in sensor service.Continue to run task in non smart sensor mode."
+                        "Failed to register in sensor service."
+                        "Continue to run task in non smart sensor mode.",
+                        exc_info=True
                     )
-                    self.log.exception(e, exc_info=True)

Review comment:
       I don’t disagree the current approach is bad. But that does not mean it is appropriate to change it to
   
   ```python
   log.warning('descriptive message', exc_info=True)
   ```
   
   either (it *also* does not mean it is inappropriate). The correct fix may be
   
   ```python
   log.exception('descriptive message')
   ```
   
   instead.
   
   I don’t know the answer, so we need a more experienced reviewer on this one.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] XD-DENG commented on a change in pull request #14823: `logging.exception` redundancy.

Posted by GitBox <gi...@apache.org>.
XD-DENG commented on a change in pull request #14823:
URL: https://github.com/apache/airflow/pull/14823#discussion_r594954135



##########
File path: airflow/models/connection.py
##########
@@ -333,9 +333,8 @@ def extra_dejson(self) -> Dict:
         if self.extra:
             try:
                 obj = json.loads(self.extra)
-            except JSONDecodeError as e:
-                self.log.exception(e)
-                self.log.error("Failed parsing the json for conn_id %s", self.conn_id)
+            except JSONDecodeError:

Review comment:
       This change is not making sense to me. 
   
   `self.log.exception(e)` has its purpose here: to add Exception info/trackback into logs.
   
   Or to add `exc_info` properly in necessary manner.
   
   Overall, personally I find more clean-ups are needed in this PR.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] turbaszek commented on pull request #14823: `logging.exception` redundancy.

Posted by GitBox <gi...@apache.org>.
turbaszek commented on pull request #14823:
URL: https://github.com/apache/airflow/pull/14823#issuecomment-826049835


   @suiting-young would you mind rebasing your PR?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] ashb commented on a change in pull request #14823: `logging.exception` redundancy.

Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #14823:
URL: https://github.com/apache/airflow/pull/14823#discussion_r596034802



##########
File path: airflow/providers/amazon/aws/log/s3_task_handler.py
##########
@@ -120,8 +120,9 @@ def _read(self, ti, try_number, metadata=None):
         try:
             log_exists = self.s3_log_exists(remote_loc)
         except Exception as error:  # pylint: disable=broad-except
-            self.log.exception(error)
-            log = f'*** Failed to verify remote log exists {remote_loc}.\n{str(error)}\n'
+            log = f'*** Failed to verify remote log exists {remote_loc}.'
+            self.log.exception(log)
+            log += f'\n{error}\n'

Review comment:
       Slight tweak to this one please (just because the `log` object and the log message appear in different contexts.)
   
   ```suggestion
               self.log.exception("Failed to verify remote log exists %s.", remote_loc)
               log = f'*** Failed to verify remote log exists {remote_loc}.'
               log += f'\n{error}\n'
   ```




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] github-actions[bot] commented on pull request #14823: `logging.exception` redundancy.

Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #14823:
URL: https://github.com/apache/airflow/pull/14823#issuecomment-801784149


   [The Workflow run](https://github.com/apache/airflow/actions/runs/664096314) is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] uranusjr commented on a change in pull request #14823: `logging.exception` redundancy.

Posted by GitBox <gi...@apache.org>.
uranusjr commented on a change in pull request #14823:
URL: https://github.com/apache/airflow/pull/14823#discussion_r595115475



##########
File path: airflow/models/taskinstance.py
##########
@@ -1297,9 +1297,10 @@ def signal_handler(signum, frame):  # pylint: disable=unused-argument
                     registered = task_copy.register_in_sensor_service(self, context)
                 except Exception as e:
                     self.log.warning(
-                        "Failed to register in sensor service.Continue to run task in non smart sensor mode."
+                        "Failed to register in sensor service."
+                        "Continue to run task in non smart sensor mode.",
+                        exc_info=True
                     )
-                    self.log.exception(e, exc_info=True)

Review comment:
       Yeah, that’s what I was thinking. If a user (say) sets logging level to `ERROR`, previously they’d be able to see info associated with the exception. Not anymore with this patch.
   
   An obvious fix would be to change this to `log.error(msg, exc_info=True)` instead, but that’d of course also make the warning become an error, which also may be problematic.
   
   Personally I think it’s probably best to keep the exception info a warning (i.e. your current approach) since it doesn’t really make that much sense to me to only show the exception traceback but not the main warning message, but this probably should be a decision made by someone with more knowledge on Airflow’s logging configs.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org