You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/11/24 22:24:06 UTC
[GitHub] [airflow] aresabalo opened a new issue #19816: logging error
aresabalo opened a new issue #19816:
URL: https://github.com/apache/airflow/issues/19816
### Apache Airflow version
2.2.2 (latest released)
### Operating System
Linux-5.4.0-1056-azure-x86_64-with-glibc2.2.5
### Versions of Apache Airflow Providers
_No response_
### Deployment
Other Docker-based deployment
### Deployment details
kubernetes 1.21.2 (AKS)
### What happened
**DAG using logging class produce stack overflow and abort task.** It seems like a bug in /home/airflow/.local/lib/python3.8/site-packages/airflow/utils/log/secrets_masker.py
**ENV:**
AIRFLOW__CORE__HIDE_SENSITIVE_VAR_CONN_FIELDS="False"
**Dag Code fragment:**
log_stream = StringIO()
logging.basicConfig(stream=log_stream, level=logging.DEBUG)
logger = logging.getLogger("airflow.task")
logger.addHandler(logging.StreamHandler(log_stream))
...
logger.info("------------------------------ ------------------- ------------------------------")
#####
**Log in worker pod:**
Fatal Python error: Cannot recover from stack overflow.
Python runtime state: initialized
Current thread 0x00007fbf8d8190c0 (most recent call first):
File "/usr/local/lib/python3.8/posixpath.py", line 42 in _get_sep
File "/usr/local/lib/python3.8/posixpath.py", line 143 in basename
File "/usr/local/lib/python3.8/logging/__init__.py", line 322 in __init__
File "/usr/local/lib/python3.8/logging/__init__.py", line 1556 in makeRecord
File "/usr/local/lib/python3.8/logging/__init__.py", line 1587 in _log
File "/usr/local/lib/python3.8/logging/__init__.py", line 1458 in warning
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/log/secrets_masker.py", line 215 in _redact
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/log/secrets_masker.py", line 208 in <genexpr>
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/log/secrets_masker.py", line 208 in _redact
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/log/secrets_masker.py", line 231 in redact
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/log/secrets_masker.py", line 164 in filter
File "/usr/local/lib/python3.8/logging/__init__.py", line 811 in filter
File "/usr/local/lib/python3.8/logging/__init__.py", line 950 in handle
File "/usr/local/lib/python3.8/logging/__init__.py", line 1661 in callHandlers
File "/usr/local/lib/python3.8/logging/__init__.py", line 1599 in handle
File "/usr/local/lib/python3.8/logging/__init__.py", line 1589 in _log
File "/usr/local/lib/python3.8/logging/__init__.py", line 1458 in warning
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/log/secrets_masker.py", line 215 in _redact
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/log/secrets_masker.py", line 208 in <genexpr>
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/log/secrets_masker.py", line 208 in _redact
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/log/secrets_masker.py", line 231 in redact
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/log/secrets_masker.py", line 164 in filter
File "/usr/local/lib/python3.8/logging/__init__.py", line 811 in filter
File "/usr/local/lib/python3.8/logging/__init__.py", line 950 in handle
File "/usr/local/lib/python3.8/logging/__init__.py", line 1661 in callHandlers
File "/usr/local/lib/python3.8/logging/__init__.py", line 1599 in handle
File "/usr/local/lib/python3.8/logging/__init__.py", line 1589 in _log
File "/usr/local/lib/python3.8/logging/__init__.py", line 1458 in warning
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/log/secrets_masker.py", line 215 in _redact
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/log/secrets_masker.py", line 208 in <genexpr>
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/log/secrets_masker.py", line 208 in _redact
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/log/secrets_masker.py", line 231 in redact
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/log/secrets_masker.py", line 164 in filter
File "/usr/local/lib/python3.8/logging/__init__.py", line 811 in filter
File "/usr/local/lib/python3.8/logging/__init__.py", line 950 in handle
File "/usr/local/lib/python3.8/logging/__init__.py", line 1661 in callHandlers
File "/usr/local/lib/python3.8/logging/__init__.py", line 1599 in handle
File "/usr/local/lib/python3.8/logging/__init__.py", line 1589 in _log
File "/usr/local/lib/python3.8/logging/__init__.py", line 1458 in warning
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/log/secrets_masker.py", line 215 in _redact
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/log/secrets_masker.py", line 208 in <genexpr>
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/log/secrets_masker.py", line 208 in _redact
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/log/secrets_masker.py", line 208 in <genexpr>
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/log/secrets_masker.py", line 208 in _redact
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/log/secrets_masker.py", line 231 in redact
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/log/secrets_masker.py", line 164 in filter
File "/usr/local/lib/python3.8/logging/__init__.py", line 811 in filter
File "/usr/local/lib/python3.8/logging/__init__.py", line 950 in handle
File "/usr/local/lib/python3.8/logging/__init__.py", line 1661 in callHandlers
File "/usr/local/lib/python3.8/logging/__init__.py", line 1599 in handle
File "/usr/local/lib/python3.8/logging/__init__.py", line 1589 in _log
File "/usr/local/lib/python3.8/logging/__init__.py", line 1458 in warning
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/log/secrets_masker.py", line 215 in _redact
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/log/secrets_masker.py", line 231 in redact
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/log/secrets_masker.py", line 164 in filter
File "/usr/local/lib/python3.8/logging/__init__.py", line 811 in filter
File "/usr/local/lib/python3.8/logging/__init__.py", line 950 in handle
File "/usr/local/lib/python3.8/logging/__init__.py", line 1661 in callHandlers
File "/usr/local/lib/python3.8/logging/__init__.py", line 1599 in handle
File "/usr/local/lib/python3.8/logging/__init__.py", line 1589 in _log
File "/usr/local/lib/python3.8/logging/__init__.py", line 1458 in warning
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/log/secrets_masker.py", line 215 in _redact
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/log/secrets_masker.py", line 208 in <genexpr>
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/log/secrets_masker.py", line 208 in _redact
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/log/secrets_masker.py", line 231 in redact
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/log/secrets_masker.py", line 164 in filter
File "/usr/local/lib/python3.8/logging/__init__.py", line 811 in filter
File "/usr/local/lib/python3.8/logging/__init__.py", line 950 in handle
File "/usr/local/lib/python3.8/logging/__init__.py", line 1661 in callHandlers
File "/usr/local/lib/python3.8/logging/__init__.py", line 1599 in handle
File "/usr/local/lib/python3.8/logging/__init__.py", line 1589 in _log
File "/usr/local/lib/python3.8/logging/__init__.py", line 1458 in warning
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/log/secrets_masker.py", line 215 in _redact
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/log/secrets_masker.py", line 208 in <genexpr>
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/log/secrets_masker.py", line 208 in _redact
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/log/secrets_masker.py", line 231 in redact
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/log/secrets_masker.py", line 164 in filter
File "/usr/local/lib/python3.8/logging/__init__.py", line 811 in filter
File "/usr/local/lib/python3.8/logging/__init__.py", line 950 in handle
File "/usr/local/lib/python3.8/logging/__init__.py", line 1661 in callHandlers
File "/usr/local/lib/python3.8/logging/__init__.py", line 1599 in handle
File "/usr/local/lib/python3.8/logging/__init__.py", line 1589 in _log
File "/usr/local/lib/python3.8/logging/__init__.py", line 1458 in warning
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/log/secrets_masker.py", line 215 in _redact
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/log/secrets_masker.py", line 208 in <genexpr>
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/log/secrets_masker.py", line 208 in _redact
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/log/secrets_masker.py", line 231 in redact
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/log/secrets_masker.py", line 164 in filter
File "/usr/local/lib/python3.8/logging/__init__.py", line 811 in filter
File "/usr/local/lib/python3.8/logging/__init__.py", line 950 in handle
File "/usr/local/lib/python3.8/logging/__init__.py", line 1661 in callHandlers
File "/usr/local/lib/python3.8/logging/__init__.py", line 1599 in handle
File "/usr/local/lib/python3.8/logging/__init__.py", line 1589 in _log
File "/usr/local/lib/python3.8/logging/__init__.py", line 1458 in warning
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/log/secrets_masker.py", line 215 in _redact
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/log/secrets_masker.py", line 208 in <genexpr>
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/log/secrets_masker.py", line 208 in _redact
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/log/secrets_masker.py", line 231 in redact
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/log/secrets_masker.py", line 164 in filter
File "/usr/local/lib/python3.8/logging/__init__.py", line 811 in filter
### What you expected to happen
_No response_
### How to reproduce
_No response_
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on issue #19816: logging error
Posted by GitBox <gi...@apache.org>.
uranusjr commented on issue #19816:
URL: https://github.com/apache/airflow/issues/19816#issuecomment-978872230
The redact function needs to have a hard recursion limit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] aresabalo commented on issue #19816: logging error
Posted by GitBox <gi...@apache.org>.
aresabalo commented on issue #19816:
URL: https://github.com/apache/airflow/issues/19816#issuecomment-978354451
Bad workaround patch /home/airflow/.local/lib/python3.8/site-packages/airflow/utils/log/secrets_masker.py
def redact(self, item: "RedactableItem", name: Optional[str] = None) -> "RedactableItem":
"""Redact an any secrets found in ``item``, if it is a string.
If ``name`` is given, and it's a "sensitive" name (see
:func:`should_hide_value_for_key`) then all string values in the item
is redacted.
"""
# Bad workaround avoiding stack overflow
return item
# return self._redact(item, name, depth=0)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mitsos1os commented on issue #19816: logging error
Posted by GitBox <gi...@apache.org>.
mitsos1os commented on issue #19816:
URL: https://github.com/apache/airflow/issues/19816#issuecomment-1038776707
Just wanted to add that I was getting the same error on the latest MWAA setup on Airflow 2.2.2. Tasks were failing instantly once they were executed with the same stack overflow error!
**The problem finally was that LOG LEVEL!** We were deploying it through Terraform and the [**mwaa**](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/mwaa_environment) module allows you to use log-level configuration up to DEBUG.
However, MWAA UI seems not to like them and doesn't display the DEBUG selection. As soon as we changed it to INFO+ the tasks started running normally
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] djpate commented on issue #19816: logging error
Posted by GitBox <gi...@apache.org>.
djpate commented on issue #19816:
URL: https://github.com/apache/airflow/issues/19816#issuecomment-1067339023
@mitsos1os You just saved my life. Thanks for sharing. Been stuck on this for two days just because of DEBUG level.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] hterik commented on issue #19816: logging error
Posted by GitBox <gi...@apache.org>.
hterik commented on issue #19816:
URL: https://github.com/apache/airflow/issues/19816#issuecomment-983311373
I've seen this happen as well, it happens when the `_redact` function throws an exception, which triggers a new `log.warning("Unable to redact %r ...` call that resets the recursion-limit counter.
Since the failing item itself is embedded into the warning-log, it will trigger the same issue again and it repeats forever.
https://github.com/apache/airflow/blob/968952bb8d0b410e5cf013f3d0d074fc5783c6f1/airflow/utils/log/secrets_masker.py#L215
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #19816: logging error
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #19816:
URL: https://github.com/apache/airflow/issues/19816#issuecomment-983584159
AAAAARGH ! Really nice find @hterik - would you like to attempt to fix it ? I think one of the solution would be to pass current depth to the log() call with some custom name and have the log pass it as starting depth to redact method?
But possibly we could find a nicer solution :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mitsos1os commented on issue #19816: logging error
Posted by GitBox <gi...@apache.org>.
mitsos1os commented on issue #19816:
URL: https://github.com/apache/airflow/issues/19816#issuecomment-1068007060
> @mitsos1os You just saved my life. Thanks for sharing. Been stuck on this for two days just because of DEBUG level.
Glad I was helpful!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] aresabalo commented on issue #19816: logging error
Posted by GitBox <gi...@apache.org>.
aresabalo commented on issue #19816:
URL: https://github.com/apache/airflow/issues/19816#issuecomment-979381659
The redact function already has a hard recursion limit (by default 5) in secrets_masker.py
There must be another problem in that function.
Setting AIRFLOW__CORE__HIDE_SENSITIVE_VAR_CONN_FIELDS to "False" should avoid executing this buggy function, but it doesn't work either :-(
----
def _redact(self, item: "RedactableItem", name: Optional[str], depth: int) -> "RedactableItem":
# Avoid spending too much effort on redacting on deeply nested
# structures. This also avoid infinite recursion if a structure has
# reference to self.
if depth > self.MAX_RECURSION_DEPTH:
return item
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #19816: logging error
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #19816:
URL: https://github.com/apache/airflow/issues/19816#issuecomment-983584159
AAAAARGH ! Really nice find @hterik - would you like to attempt to fix it ? I think one of the solutions would be to pass current depth to the log() call with some custom name and have the log pass it as starting depth to redact method?
But possibly we could find a nicer solution :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #19816: logging error
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #19816:
URL: https://github.com/apache/airflow/issues/19816#issuecomment-983584159
AAAAARGH ! Really nice find @hterik - would you like to attempt to fix it ? I think one of the solution would be to pass current depth to the log() call with some cusom name and have the log pass it as starting depth to redact method?
But possibly we could find a nicer solution :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mitsos1os edited a comment on issue #19816: logging error
Posted by GitBox <gi...@apache.org>.
mitsos1os edited a comment on issue #19816:
URL: https://github.com/apache/airflow/issues/19816#issuecomment-1038776707
Just wanted to add some notes about getting the same error on the latest **MWAA setup on Airflow 2.2.2**. Tasks were failing instantly once they were executed with the same stack overflow error!
**The problem finally was that LOG LEVEL!** We were deploying it through Terraform and the [**mwaa**](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/mwaa_environment) module allows you to use log-level configuration up to DEBUG.
However, MWAA UI seems not to like them and doesn't display the DEBUG selection. As soon as we changed it to INFO+ the tasks started running normally.
I hope this saves someone all the frustration that I went through while searching for the cause of a fresh cluster not working
**UPDATE**
It is even stated in their CLI update utility and does not work: https://awscli.amazonaws.com/v2/documentation/api/latest/reference/mwaa/update-environment.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mitsos1os edited a comment on issue #19816: logging error
Posted by GitBox <gi...@apache.org>.
mitsos1os edited a comment on issue #19816:
URL: https://github.com/apache/airflow/issues/19816#issuecomment-1038776707
Just wanted to add some notes about getting the same error on the latest **MWAA setup on Airflow 2.2.2**. Tasks were failing instantly once they were executed with the same stack overflow error!
**The problem finally was that LOG LEVEL!** We were deploying it through Terraform and the [**mwaa**](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/mwaa_environment) module allows you to use log-level configuration up to DEBUG.
However, MWAA UI seems not to like them and doesn't display the DEBUG selection. As soon as we changed it to INFO+ the tasks started running normally.
I hope this saves someone all the frustration that I went through while searching for the cause of a fresh cluster not working
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #19816: logging error
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #19816:
URL: https://github.com/apache/airflow/issues/19816#issuecomment-978346208
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #19816: logging error
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #19816:
URL: https://github.com/apache/airflow/issues/19816
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org