You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/09/19 21:48:54 UTC
[GitHub] [airflow] lewismc opened a new issue, #26508: SchedulerJob error message fails to perform string substitution for externally killed tasks.
lewismc opened a new issue, #26508:
URL: https://github.com/apache/airflow/issues/26508
### Apache Airflow version
Other Airflow 2 version
### What happened
In this particular instance a task was killed due to an Airflow task timeout. The Kubernetes Pod took too long to initialize.
Sentry reported the message as follows..
```
Executor reports task instance %s finished (%s) although the task says its %s. (Info: %s) Was the task killed externally?
```
As you can see the string substitution is not being performed correctly. This is also shown in the following snippet.
Although the variables are displayed below the message this behaviour is not consistent with other errors displayed via Sentry. The snippet below provides a correctly working example.
Here you can see the message populated correctly.
### What you think should happen instead
The variables should be properly accommodated within the message.
### How to reproduce
You need to increment the `scheduler.tasks.killed_externally` `Stat` in [scheduler_job.py](https://github.com/apache/airflow/blob/main/airflow/jobs/scheduler_job.py#L668-L674). This will generate the error.
### Operating System
PRETTY_NAME="Debian GNU/Linux 10 (buster)" NAME="Debian GNU/Linux" VERSION_ID="10" VERSION="10 (buster)" VERSION_CODENAME=buster ID=debian HOME_URL="https://www.debian.org/" SUPPORT_URL="https://www.debian.org/support" BUG_REPORT_URL="https://bugs.debian.org/"
### Versions of Apache Airflow Providers
```
apache-airflow[ldap]
apache-airflow[sentry]
apache-airflow-providers-slack==4.1.0
apache-airflow-providers-amazon==4.1.0
apache-airflow-providers-elasticsearch==2.1.0
apache-airflow-providers-microsoft-mssql==2.0.1
atlassian-python-api==3.20.1
authlib==0.15.4
beautifulsoup4==4.9.3
blinker>=1.1
pytest==7.1.2
requests-ntlm==1.1.0
sentry-sdk>=1.7.1
webvtt-py==0.4.6
```
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
Vanilla Helm Chart airflow-1.5.0 installation (Airflow 2.2.4) into OLD Kubernetes version
```
Client Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.1", GitCommit:"e4d4e1ab7cf1bf15273ef97303551b279f0920a9", GitTreeState:"clean", BuildDate:"2022-09-14T19:40:59Z", GoVersion:"go1.19.1", Compiler:"gc", Platform:"darwin/arm64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"11+", GitVersion:"v1.11.9-dirty", GitCommit:"16236ce91790d4c75b79f6ce96841db1c843e7d2", GitTreeState:"dirty", BuildDate:"2019-04-08T18:22:05Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"linux/amd64"}
```
### Anything else
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] lewismc commented on issue #26508: SchedulerJob error message fails to perform string substitution for externally killed tasks.
Posted by GitBox <gi...@apache.org>.
lewismc commented on issue #26508:
URL: https://github.com/apache/airflow/issues/26508#issuecomment-1254387679
Thank you for the context on this one @potiuk i appreciate it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #26508: SchedulerJob error message fails to perform string substitution for externally killed tasks.
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #26508:
URL: https://github.com/apache/airflow/issues/26508#issuecomment-1253985090
This is problem with sentry not properly resolving message with python arguments. This is the recommended way in python to log an error message - pass it a format string and parameters separately. As you see the parameters are there. it's sentry that should render it properly. There is an open PR in sentry that explains the issue https://github.com/getsentry/sentry-python/issues/228 and until this one is not fixed, we cannot do much about it. We will not make any special exceptions for Sentry, they should fix it to allow not only airflow logs but also logs from any other 3rd-party libraries to follow standard Python conventions. You should likely comment in the sentry issue and raise awareness that you are also affected by it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #26508: SchedulerJob error message fails to perform string substitution for externally killed tasks.
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #26508: SchedulerJob error message fails to perform string substitution for externally killed tasks.
URL: https://github.com/apache/airflow/issues/26508
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org