You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/08/25 19:41:13 UTC
[GitHub] [airflow] patgarz opened a new issue, #25964: on_failure_callback does not pass full exception object
patgarz opened a new issue, #25964:
URL: https://github.com/apache/airflow/issues/25964
### Apache Airflow version
Other Airflow 2 version
### What happened
We're using `on_failure_callback` to trigger alerts when a TaskInstance fails. The exception details, including stack trace, were available in Airflow 1(.10.14), however, upon upgrading to Airflow 2(.2.5), this is no longer working. For example, in Airflow 1, we were able to do something like this:
```py
import traceback
...
exception = context.get('exception')
formatted_exception = ''.join(
traceback.format_exception(etype=type(exception),
value=exception, tb=exception.__traceback__
)
).strip()
```
However, when we attempt to do this in Airflow 2, this produces an error:
```text
{taskinstance.py:1607} ERROR - Error when executing on_failure_callback
Traceback (most recent call last):
File "/app/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1605, in _run_finished_callback
task.on_failure_callback(context)
File "[SCRUBBED]", line 381, in send_error_alert
formatted_exception = ''.join(traceback.format_exception(etype=type(exception), value=exception, tb=exception.__traceback__)).strip()
AttributeError: 'str' object has no attribute '__traceback__'
```
In this case the `context['exception']` is returning a string, but this isn't consistent in other tests (this is coming from an `EmrCreateOperator`, whereas other tests --- see below sections --- are coming from a `PythonOperator`). What is consistent, though, is that nothing besides the error description text seems to be accessible.
### What you think should happen instead
Testing in base Python:
```python
import logging
try:
print(undefined_var)
except BaseException as e:
logging.exception(e)
```
has results that include not only the error message, but the full trace and error type
```text
ERROR:root:name 'undefined_var' is not defined
Traceback (most recent call last):
File "<stdin>", line 2, in <module>
NameError: name 'undefined_var' is not defined
```
### How to reproduce
DAG Code:
```python
from datetime import datetime, timedelta
from airflow import DAG
from airflow.models import TaskInstance
from airflow.operators.python import PythonOperator
# or .python_operator for Airflow 1
def _alarm(context):
import logging
import traceback
logging.info("Task instance on_failure")
traceback.print_exc()
exception = context.get('exception')
logging.info(f"exception type :: {type(exception)}")
logging.info(f"exception :: {exception}")
logging.exception(exception)
logging.info(f"{exception.__traceback__}")
default_args = {
"owner": "xxxx",
"email_on_failure": False,
#"on_failure_callback": _alarm,
}
def bad_python():
print(undefined_var)
with DAG(
dag_id="failure_callback_testing",
start_date=datetime(2021, 9, 7),
schedule_interval=None,
default_args=default_args,
catchup=False,
dagrun_timeout=timedelta(seconds=15),
) as dag:
will_fail = PythonOperator(
task_id="will_fail",
python_callable=bad_python,
on_failure_callback=_alarm,
)
will_fail
```
has results (Airflow 2):
```text
[2022-08-25, 14:29:39 EDT] {taskinstance.py:1774} ERROR - Task failed with exception
Traceback (most recent call last):
File "/app/.local/lib/python3.8/site-packages/airflow/operators/python.py", line 174, in execute
return_value = self.execute_callable()
File "/app/.local/lib/python3.8/site-packages/airflow/operators/python.py", line 188, in execute_callable
return self.python_callable(*self.op_args, **self.op_kwargs)
File "[SCRUBBED]", line 28, in bad_python
print(undefined_var)
NameError: name 'undefined_var' is not defined
[2022-08-25, 14:29:39 EDT] {taskinstance.py:1278} INFO - Marking task as FAILED. dag_id=failure_callback_testing, task_id=will_fail, execution_date=20220825T182935, start_date=20220825T182939, end_date=20220825T182939
[2022-08-25, 14:29:39 EDT] {standard_task_runner.py:93} ERROR - Failed to execute job 79590 for task will_fail (name 'undefined_var' is not defined; 29411)
[2022-08-25, 14:29:39 EDT] {local_task_job.py:154} INFO - Task exited with return code 1
[2022-08-25, 14:29:39 EDT] {test_failure_callback.py:12} INFO - Task instance on_failure
[2022-08-25, 14:29:39 EDT] {logging_mixin.py:109} WARNING - NoneType: None
[2022-08-25, 14:29:39 EDT] {test_failure_callback.py:15} INFO - exception type :: <class 'NameError'>
[2022-08-25, 14:29:39 EDT] {test_failure_callback.py:16} ERROR - name 'undefined_var' is not defined
NoneType: None
[2022-08-25, 14:29:39 EDT] {test_failure_callback.py:17} INFO - None
[2022-08-25, 14:29:40 EDT] {local_task_job.py:264} INFO - 0 downstream tasks scheduled from follow-on schedule check
```
but prints the expected in Airflow 1:
```text
[2022-08-25 18:29:30,313] {taskinstance.py:1150} ERROR - name 'undefined_var' is not defined
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/airflow/models/taskinstance.py", line 984, in _run_raw_task
result = task_copy.execute(context=context)
File "/usr/local/lib/python3.8/dist-packages/airflow/operators/python_operator.py", line 113, in execute
return_value = self.execute_callable()
File "/usr/local/lib/python3.8/dist-packages/airflow/operators/python_operator.py", line 118, in execute_callable
return self.python_callable(*self.op_args, **self.op_kwargs)
File "[SCRUBBED]", line 29, in bad_python
print(undefined_var)
NameError: name 'undefined_var' is not defined
[2022-08-25 18:29:30,315] {taskinstance.py:1187} INFO - Marking task as FAILED. dag_id=failure_callback_testing, task_id=will_fail, execution_date=20220825T182922, start_date=20220825T182929, end_date=20220825T182930
[2022-08-25 18:29:30,315] {test_failure_callback.py:12} INFO - Task instance on_failure
[2022-08-25 18:29:30,315] {logging_mixin.py:112} WARNING - Traceback (most recent call last):
[2022-08-25 18:29:30,315] {logging_mixin.py:112} WARNING - File "/usr/local/lib/python3.8/dist-packages/airflow/models/taskinstance.py", line 984, in _run_raw_task
result = task_copy.execute(context=context)
[2022-08-25 18:29:30,315] {logging_mixin.py:112} WARNING - File "/usr/local/lib/python3.8/dist-packages/airflow/operators/python_operator.py", line 113, in execute
return_value = self.execute_callable()
[2022-08-25 18:29:30,316] {logging_mixin.py:112} WARNING - File "/usr/local/lib/python3.8/dist-packages/airflow/operators/python_operator.py", line 118, in execute_callable
return self.python_callable(*self.op_args, **self.op_kwargs)
[2022-08-25 18:29:30,316] {logging_mixin.py:112} WARNING - File "[SCRUBBED]", line 29, in bad_python
print(undefined_var)
[2022-08-25 18:29:30,316] {logging_mixin.py:112} WARNING - NameError: name 'undefined_var' is not defined
[2022-08-25 18:29:30,316] {test_failure_callback.py:15} INFO - exception type :: <class 'NameError'>
[2022-08-25 18:29:30,316] {test_failure_callback.py:16} INFO - exception :: name 'undefined_var' is not defined
[2022-08-25 18:29:30,316] {test_failure_callback.py:17} ERROR - name 'undefined_var' is not defined
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/airflow/models/taskinstance.py", line 984, in _run_raw_task
result = task_copy.execute(context=context)
File "/usr/local/lib/python3.8/dist-packages/airflow/operators/python_operator.py", line 113, in execute
return_value = self.execute_callable()
File "/usr/local/lib/python3.8/dist-packages/airflow/operators/python_operator.py", line 118, in execute_callable
return self.python_callable(*self.op_args, **self.op_kwargs)
File "[SCRUBBED]", line 29, in bad_python
print(undefined_var)
NameError: name 'undefined_var' is not defined
[2022-08-25 18:29:30,316] {test_failure_callback.py:18} INFO - <traceback object at 0x7f18b69e1100>
[2022-08-25 18:29:34,921] {local_task_job.py:102} INFO - Task exited with return code 1
```
### Operating System
Unbuntu 18.04.6
### Versions of Apache Airflow Providers
```text
apache-airflow-providers-amazon==3.2.0
apache-airflow-providers-celery==2.1.3
apache-airflow-providers-datadog==2.0.4
apache-airflow-providers-ftp==2.1.2
apache-airflow-providers-http==2.1.2
apache-airflow-providers-imap==2.2.3
apache-airflow-providers-postgres==4.1.0
apache-airflow-providers-redis==2.0.4
apache-airflow-providers-slack==4.2.3
apache-airflow-providers-snowflake==2.6.0
apache-airflow-providers-sqlite==2.1.3
apache-airflow-providers-ssh==2.4.3
```
### Deployment
Other Docker-based deployment
### Deployment details
* Pickling is disabled (default setting)
* PostgreSQL backend (12.8)
* LocalExecutors
### Anything else
Worked in Airflow 1, have a strong suspicion it's related to pickling being disabled, but being a not-Python-person, I'm not entirely sure what that means so I'm not confident that's the issue.
Would be willing to submit a PR but unfortunately:
1. I wouldn't really know where to start, and
2. My org unfortunately has fairly strict OSS contribution guidelines and I'm not sure how arduous the process to get something approved would be
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] patgarz commented on issue #25964: on_failure_callback does not pass full exception object
Posted by GitBox <gi...@apache.org>.
patgarz commented on issue #25964:
URL: https://github.com/apache/airflow/issues/25964#issuecomment-1229071544
Thanks for the reply!
Maybe I'm misunderstanding though, but I think that's missing the crux of the issue --- the returned object *is* of an Exception type (`NameError`, to be specific, in our tests). I believe there's a separate issue in that maybe the operator we're using at the top of the thread is returning an exception of type `str`, but in our tests below, we can see explicitly that `context['exception']` is a `NameError` and still does not contain the data we'd expect.
@potiuk unless you disagree, I believe this should be reopened? I'm happy to edit the top example since you are correct, if we attempt to read a `str` type `context['exception']` the top code should fail (and reading this a few days later I can see my description wasn't entirely clear), but the issue is different than that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #25964: on_failure_callback does not pass full exception object
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #25964:
URL: https://github.com/apache/airflow/issues/25964#issuecomment-1229082778
No, I think I think this is pretty expected stil. I do not think it's ever defined what will be in the exception - it depends on the circumstances. Sometimes it is string, sometims exception, and there is no guarantee about printing stacktrace mostly because callbacks can be executed in different context:
1) sometimes they are executed in Worker
2) sometimes they are executed in DagFileProcessor (and serialized to be send over the database sometimes)
3) sometimes they are executed in scheduler
4) sometiumes they are executed in mini-scheduler
There is lot more complexity in Airlfow 2 than 1 involved in running callbacks. Serialization is involved and I think the decision was made (and semi-documented in the types possible to return) that "exception" might be more than just excpetion. Side effect of changes in Airlfow 2 are that some of the things from 1 are not preserved. And to be perfectly honest relaying on having a stack-trace and its content by the logic of your DAG is super-brittle. What happens if the code change ? How do you keep track of it? How does it work when you change different versions? how do you react on different provider versions that might change every month and basically have the stack trace change every month?
I thinkt it is unreasonable to rely on it, so lack of it is not a bug, rally and should not hinder any "reasonable" logic you might implement there. The fact that is not "perfect" when even "perfect" solution is unusable realisticall makes me very reluctant on even attempting to make it consistent - because that might make other users like you atempt to rely on it - but this is a futile effort.
If you want to implement any logic based on the exceoption - tou will do much better if relay on str() coercion of anytning you get in the 'exception' field and check if it contain what you expect it to contain (exception or string message that you throw when you fail). this is far better strategy.
So I do not argue your "inconsistency" statement does not hold. It does. But it's more of a feature than a but , and I think it is highly unlikely it will turn into something you will be able to rely one.
I will convert it into a discussion rather than just closing - maybe others will have other opinion, or maybe you will have other arguments - why relying on stacktrace being present is important and useful.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #25964: on_failure_callback does not pass full exception object
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #25964: on_failure_callback does not pass full exception object
URL: https://github.com/apache/airflow/issues/25964
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #25964: on_failure_callback does not pass full exception object
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #25964: on_failure_callback does not pass full exception object
URL: https://github.com/apache/airflow/issues/25964
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #25964: on_failure_callback does not pass full exception object
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #25964:
URL: https://github.com/apache/airflow/issues/25964#issuecomment-1227686330
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] patgarz commented on issue #25964: on_failure_callback does not pass full exception object
Posted by GitBox <gi...@apache.org>.
patgarz commented on issue #25964:
URL: https://github.com/apache/airflow/issues/25964#issuecomment-1228770536
As an update -- the above was tested with py38 and Airflow 2.2.5. I've now tested with py39 and 2.3.4 and am seeing the same behavior.
As an aside, the reason I'm calling this a bug and not a feature request is that the passed object (which we test by doing a log of `type(context['exception'])` --> `<class 'NameError'>`) is correct, but we don't see the expected data passed with it. I'm pretty stumped, personally.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #25964: on_failure_callback does not pass full exception object
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #25964:
URL: https://github.com/apache/airflow/issues/25964#issuecomment-1228804928
This is as expected. Context type definition (airlfow/utils/context.pyi) is rather explicit that exception passed in context is possibly Exception, string, or KeyboardInterrupt depending on the (ehm) context.
```
class Context(TypedDict, total=False):
conf: AirflowConfigParser
conn: Any
dag: DAG
dag_run: DagRun
data_interval_end: DateTime
data_interval_start: DateTime
ds: str
ds_nodash: str
execution_date: DateTime
exception: Union[KeyboardInterrupt, Exception, str, None]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org