You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/12/04 17:33:52 UTC
[GitHub] [airflow] potiuk opened a new pull request #20039: Fix infinite recursion on redact log
potiuk opened a new pull request #20039:
URL: https://github.com/apache/airflow/pull/20039
When redact warning log on "unredactable" item is printed, the
log entered an infinite recursion, because the item was attempted
to be redacted again in the log.
This PR converts the item to str() - in the worst case the str
converstion will fail and raise exception - but this will be about
right - but it will not attempt to redact the item again.
Fixws: #19816
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/main/UPDATING.md).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #20039: Fix infinite recursion on redact log
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #20039:
URL: https://github.com/apache/airflow/pull/20039#issuecomment-987846125
The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #20039: Fix infinite recursion on redact log
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #20039:
URL: https://github.com/apache/airflow/pull/20039#issuecomment-987182288
Shalll we ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #20039: Fix infinite recursion on redact log
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #20039:
URL: https://github.com/apache/airflow/pull/20039#discussion_r762573533
##########
File path: airflow/utils/log/secrets_masker.py
##########
@@ -213,11 +213,13 @@ def _redact(self, item: "RedactableItem", name: Optional[str], depth: int) -> "R
else:
return item
# I think this should never happen, but it does not hurt to leave it just in case
+ # Well. It happened (see https://github.com/apache/airflow/issues/19816#issuecomment-983311373)
+ # but it caused infinite recursion, so we need to cast it to str first.
except Exception as e:
log.warning(
- "Unable to redact %r, please report this via <https://github.com/apache/airflow/issues>. "
+ "Unable to redact %s, please report this via <https://github.com/apache/airflow/issues>. "
"Error was: %s: %s",
- item,
+ str(item),
type(e).__name__,
str(e),
)
Review comment:
Now - with repr() it should be even better.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #20039: Fix infinite recursion on redact log
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #20039:
URL: https://github.com/apache/airflow/pull/20039#discussion_r762573088
##########
File path: airflow/utils/log/secrets_masker.py
##########
@@ -213,11 +213,13 @@ def _redact(self, item: "RedactableItem", name: Optional[str], depth: int) -> "R
else:
return item
# I think this should never happen, but it does not hurt to leave it just in case
+ # Well. It happened (see https://github.com/apache/airflow/issues/19816#issuecomment-983311373)
+ # but it caused infinite recursion, so we need to cast it to str first.
except Exception as e:
log.warning(
- "Unable to redact %r, please report this via <https://github.com/apache/airflow/issues>. "
+ "Unable to redact %s, please report this via <https://github.com/apache/airflow/issues>. "
"Error was: %s: %s",
- item,
+ str(item),
type(e).__name__,
str(e),
)
Review comment:
Not really. Neither of those proposal is good. But I think mine was not perfect either.
1) If we use '%s' and pass the object, we might have the same recursion problem. The problem was that calling `repr()` of the object threw an exception which caused the recursion (inside the log.warning() message). But `str()` migh have exactly the same problem. So the only good solution to avoid the recursion is to run a conversion to string before the `warning` message is entered - i.e. before the counter is reset. Just in case so if it throws an exception, it won't reset the counter.
2) using '%r' when you pass string makes little sense. because it will call `repr()` method on a string. Which is unnecessary (though result is the same). By convention (and I think it is used everywhere in our code including two lines below when we do str(e) - we always use '%s' when we pass string, because it will simply check that the class is string and will print it without running extra conversion. I've never seen someone uses '%r` to print a string :).
However your comment made me think and slightly better solution would be to actually call _repr() manually and then use '%s' to log it. This would be a bit better equivalent of the previous version. Let me fix it quickly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on pull request #20039: Fix infinite recursion on redact log
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on pull request #20039:
URL: https://github.com/apache/airflow/pull/20039#issuecomment-986262606
Ok. That one should be ready to merge - the only failing test is another test_scheduler_job flakiness that is already addressed in #19860
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #20039: Fix infinite recursion on redact log
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #20039:
URL: https://github.com/apache/airflow/pull/20039#discussion_r762573088
##########
File path: airflow/utils/log/secrets_masker.py
##########
@@ -213,11 +213,13 @@ def _redact(self, item: "RedactableItem", name: Optional[str], depth: int) -> "R
else:
return item
# I think this should never happen, but it does not hurt to leave it just in case
+ # Well. It happened (see https://github.com/apache/airflow/issues/19816#issuecomment-983311373)
+ # but it caused infinite recursion, so we need to cast it to str first.
except Exception as e:
log.warning(
- "Unable to redact %r, please report this via <https://github.com/apache/airflow/issues>. "
+ "Unable to redact %s, please report this via <https://github.com/apache/airflow/issues>. "
"Error was: %s: %s",
- item,
+ str(item),
type(e).__name__,
str(e),
)
Review comment:
Not really. Neither of those proposal is good. But I think mine was not perfect either.
1) If we use '%s' and pass the object, we might have the same recursion problem. The problem was that calling `repr()` of the object threw an exception which caused the recursion (inside the log.warning() message). But `str()` migh have exactly the same problem. So the only good solution to avoid the recursion is to run a conversion to string before the `warning` message is entered - i.e. before the counter is reset. Just in case so if it throws an exception, it won't reset the counter.
2) using '%r' when you pass string makes little sense. because it will call `repr()` method on a string. Which is unnecessary (though result is the same). By convention (and I think it is used everywhere in our code including two lines below when we do str(e) - we always use '%s' when we pass string, because it will simply check that the object is string and will print it without running extra conversion. I've never seen someone uses '%r` to print a string :).
However your comment made me think and slightly better solution would be to actually call _repr() manually and then use '%s' to log it. This would be a bit better equivalent of the previous version. Let me fix it quickly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #20039: Fix infinite recursion on redact log
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #20039:
URL: https://github.com/apache/airflow/pull/20039#issuecomment-986262606
Ok. That one shoudl be ready to merge - the only failing test is another test_scheduler_job flakiness that is already addressed in #19860
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #20039: Fix infinite recursion on redact log
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #20039:
URL: https://github.com/apache/airflow/pull/20039#discussion_r762573088
##########
File path: airflow/utils/log/secrets_masker.py
##########
@@ -213,11 +213,13 @@ def _redact(self, item: "RedactableItem", name: Optional[str], depth: int) -> "R
else:
return item
# I think this should never happen, but it does not hurt to leave it just in case
+ # Well. It happened (see https://github.com/apache/airflow/issues/19816#issuecomment-983311373)
+ # but it caused infinite recursion, so we need to cast it to str first.
except Exception as e:
log.warning(
- "Unable to redact %r, please report this via <https://github.com/apache/airflow/issues>. "
+ "Unable to redact %s, please report this via <https://github.com/apache/airflow/issues>. "
"Error was: %s: %s",
- item,
+ str(item),
type(e).__name__,
str(e),
)
Review comment:
Not really. Neither of those proposal is good. But I think mine was not perfect either.
1) If we use '%s' and pass the object, we might have the same recursion problem. The problem was that calling `repr()` of the object threw an exception which caused the recursion (inside the log.warning() message). But `str()` migh have exactly the same problem. So the only good solution to avoid the recursion is to run a conversion to string before the `warning` message is entered - i.e. before the counter is reset. Just in case so if it throws an exception, it won't reset the counter.
2) using '%r' when you pass string makes little sense. because it will call a __repr() method on a string. Which is unnecessary (though result is the same). By convention (and I think it is used everywhere in our code including two lines below when we do str(e) - we always use '%s' when we pass string, because it will simply check that the class is string and will print it without running extra conversion. I've never seen someone uses '%r` to print a string :).
However your comment made me think and slightly better solution would be to actually call _repr() manually and then use '%s' to log it. This would be a bit better equivalent of the previous version. Let me fix it quickly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk merged pull request #20039: Fix infinite recursion on redact log
Posted by GitBox <gi...@apache.org>.
potiuk merged pull request #20039:
URL: https://github.com/apache/airflow/pull/20039
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on a change in pull request #20039: Fix infinite recursion on redact log
Posted by GitBox <gi...@apache.org>.
uranusjr commented on a change in pull request #20039:
URL: https://github.com/apache/airflow/pull/20039#discussion_r762536624
##########
File path: airflow/utils/log/secrets_masker.py
##########
@@ -213,11 +213,13 @@ def _redact(self, item: "RedactableItem", name: Optional[str], depth: int) -> "R
else:
return item
# I think this should never happen, but it does not hurt to leave it just in case
+ # Well. It happened (see https://github.com/apache/airflow/issues/19816#issuecomment-983311373)
+ # but it caused infinite recursion, so we need to cast it to str first.
except Exception as e:
log.warning(
- "Unable to redact %r, please report this via <https://github.com/apache/airflow/issues>. "
+ "Unable to redact %s, please report this via <https://github.com/apache/airflow/issues>. "
"Error was: %s: %s",
- item,
+ str(item),
type(e).__name__,
str(e),
)
Review comment:
Eithe you don’t need to `str()` (it is implied by `%s`), or you should `str()` and then use `%r`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on a change in pull request #20039: Fix infinite recursion on redact log
Posted by GitBox <gi...@apache.org>.
uranusjr commented on a change in pull request #20039:
URL: https://github.com/apache/airflow/pull/20039#discussion_r763746096
##########
File path: airflow/utils/log/secrets_masker.py
##########
@@ -213,11 +213,13 @@ def _redact(self, item: "RedactableItem", name: Optional[str], depth: int) -> "R
else:
return item
# I think this should never happen, but it does not hurt to leave it just in case
+ # Well. It happened (see https://github.com/apache/airflow/issues/19816#issuecomment-983311373)
+ # but it caused infinite recursion, so we need to cast it to str first.
except Exception as e:
log.warning(
- "Unable to redact %r, please report this via <https://github.com/apache/airflow/issues>. "
+ "Unable to redact %s, please report this via <https://github.com/apache/airflow/issues>. "
"Error was: %s: %s",
- item,
+ item.repr(),
Review comment:
```suggestion
repr(item),
```
?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org