You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "NilsJPWerner (via GitHub)" <gi...@apache.org> on 2025/09/19 22:07:16 UTC
[I] Exceptions are not logged properly [airflow]
NilsJPWerner opened a new issue, #55910:
URL: https://github.com/apache/airflow/issues/55910
### Apache Airflow version
3.0.6
### If "Other Airflow 2 version" selected, which one?
_No response_
### What happened?
Airflow 3 no longer captures all the lines from an Exception. It seems like it displays part of the traceback but not the actual exception. This makes debugging very difficult.
Previously:
<img width="1364" height="932" alt="Image" src="https://github.com/user-attachments/assets/5b7a64e1-bffe-4cd1-b2d9-99a0c5daa85b" />
Now:
<img width="2102" height="452" alt="Image" src="https://github.com/user-attachments/assets/281940ee-36dd-4a7f-b673-6a32c44756ca" />
### What you think should happen instead?
_No response_
### How to reproduce
```
def make_it_fail():
assert False, "Intentional failure for reproduction"
with DAG(
dag_id="assert_fail_dag",
start_date=datetime(2024, 1, 1),
schedule=None,
catchup=False,
default_args={"retries": 0},
) as dag:
fail_task = PythonOperator(
task_id="assert_false",
python_callable=make_it_fail,
)
```
### Operating System
Debian
### Versions of Apache Airflow Providers
_No response_
### Deployment
Astronomer
### Deployment details
_No response_
### Anything else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Exceptions are not logged properly [airflow]
Posted by "jroachgolf84 (via GitHub)" <gi...@apache.org>.
jroachgolf84 commented on issue #55910:
URL: https://github.com/apache/airflow/issues/55910#issuecomment-3316243931
Hmmm, those two screenshots that you included have different exceptions. The first (Airflow 2.X) looks like it is failing to fetch a Git repository. I was NOT able to reproduce this exception, and have not seen anything similar before. I'll leave this open in case someone else is able to reproduce.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Exceptions in task logs no longer contain source code [airflow]
Posted by "Venture200 (via GitHub)" <gi...@apache.org>.
Venture200 commented on issue #55910:
URL: https://github.com/apache/airflow/issues/55910#issuecomment-4980852394
is there any update on it ? I am also experiencing same problem. No verbose content about Exception it shows -> ERROR - Task failed with exception. Version 3.1.8
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Exceptions are not logged properly [airflow]
Posted by "NilsJPWerner (via GitHub)" <gi...@apache.org>.
NilsJPWerner commented on issue #55910:
URL: https://github.com/apache/airflow/issues/55910#issuecomment-3319094939
I will try to create another replication.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Exceptions are not logged properly [airflow]
Posted by "j6takish (via GitHub)" <gi...@apache.org>.
j6takish commented on issue #55910:
URL: https://github.com/apache/airflow/issues/55910#issuecomment-3343483711
I have a bunch of unit tests pertaining to logging that started failing in 3.1. Was going to start my own issue but it seems related to this somehow. I have a third party provided not included in the main airflow repo but I was able to re-create the issue using the postgres provider.
This test will pass using airflow < 3.1.0 but fail using 3.1.0
` @mock.patch("airflow.providers.postgres.hooks.postgres.PostgresHook.get_conn")
def test_logging(self, mock_connect, caplog):
from airflow.providers.postgres.hooks.postgres import PostgresHook
from psycopg2.extensions import connection, cursor
mock_conn = mock.MagicMock(spec=connection)
monk_cursor = mock.MagicMock(spec=cursor, rowcount=-1)
mock_hook = PostgresHook(log_sql=True)
mock_connect.return_value = mock_conn
mock_conn.cursor.return_value = monk_cursor
with caplog.at_level(20):
mock_hook.get_records("SELECT 'RAVIOLI RAVIOLI GIVE ME THE FORMUOLI'")
print(caplog.text)
assert "SELECT 'RAVIOLI RAVIOLI GIVE ME THE FORMUOLI" in caplog.text`
caplog shows as an empty string
<img width="1398" height="559" alt="Image" src="https://github.com/user-attachments/assets/5505afb8-a14b-47da-8215-c6865a563916" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Exceptions are not logged properly [airflow]
Posted by "j6takish (via GitHub)" <gi...@apache.org>.
j6takish commented on issue #55910:
URL: https://github.com/apache/airflow/issues/55910#issuecomment-3344077284
My example seems somewhat related to #55903
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Exceptions in task logs no longer contain source code [airflow]
Posted by "arose26 (via GitHub)" <gi...@apache.org>.
arose26 commented on issue #55910:
URL: https://github.com/apache/airflow/issues/55910#issuecomment-5315481188
This reproduces, and the cause looks structural rather than a formatting slip — it only affects deployments using JSON log output, which may be why it hasn't reproduced for everyone in this thread.
**Where the source lines go missing**
`shared/logging/src/airflow_shared/logging/structlog.py` builds the JSON branch's exception renderer from structlog's `ExceptionDictTransformer`:
```python
dict_exc_formatter = structlog.tracebacks.ExceptionDictTransformer(
use_rich=False, show_locals=False, suppress=suppress
)
dict_tracebacks = structlog.processors.ExceptionRenderer(dict_exc_formatter)
```
That transformer emits one dict per frame containing `filename`, `lineno` and `name` — and nothing else. There is no field carrying the source text of the line, so the rendered traceback can only ever show the frame headers, never the indented code line beneath them. Checked against structlog 26.1.0:
```
use_rich=False -> frame keys = ['filename', 'lineno', 'name']
use_rich=True -> frame keys = ['filename', 'lineno', 'name']
```
`use_rich` makes no difference here; both paths produce the same three keys.
**Why it's specific to JSON output**
The non-JSON branch of the same function uses `structlog.dev.plain_traceback` (or `RichTracebackFormatter` when `DEV` is set), and those go through `traceback.print_exception`, which pulls the source line via `linecache`. Both retain the code line. So the same failing task shows full tracebacks in text output and header-only tracebacks in JSON output.
That matches the Airflow 2 → 3 change: the standard-library formatting that carried the source line is no longer on the path for structured logs.
**Question before anyone writes a patch**
Restoring the code line would mean carrying it per frame — for example a transformer that adds `linecache.getline(filename, lineno)` alongside the existing keys, with the UI rendering it under each frame. That does make every logged traceback bigger, and `show_locals=False` / `use_rich=False` suggest the current settings were chosen deliberately to keep structured logs compact.
Is restoring parity with the text output something the project wants, and if so is a per-frame source field the right shape — or would you rather this stayed as is and the docs noted the difference? Happy to put up a PR once there's a direction.
---
Drafted-by: Claude Code (Opus 5); reviewed by @arose26 before posting
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Exceptions are not logged properly [airflow]
Posted by "NilsJPWerner (via GitHub)" <gi...@apache.org>.
NilsJPWerner commented on issue #55910:
URL: https://github.com/apache/airflow/issues/55910#issuecomment-3319188850
It seems like airflow 3 does not show the indented part of the exception frame that shows the contents of the line of code.
<img width="838" height="470" alt="Image" src="https://github.com/user-attachments/assets/8119fad2-31ba-4645-a898-a063bbf16269" />
<img width="1301" height="453" alt="Image" src="https://github.com/user-attachments/assets/0c79e7e1-2e59-48fb-bd41-a1d90d956673" />
```python
from datetime import datetime
from airflow import DAG
from airflow.operators.python import PythonOperator
default_args = {
"owner": "airflow",
"depends_on_past": False,
"start_date": datetime(2021, 6, 18),
"retries": 0,
}
def failing_python_function():
print("Things are going well!")
raise Exception("Down bad rn")
def nested_function():
failing_python_function()
def nested_function_2():
nested_function()
with DAG(
"test_error",
default_args=default_args,
schedule=None,
catchup=False,
) as dag:
PythonOperator(
task_id="run_entrypoint",
python_callable=nested_function_2,
)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Exceptions in task logs no longer contain source code [airflow]
Posted by "bbovenzi (via GitHub)" <gi...@apache.org>.
bbovenzi closed issue #55910: Exceptions in task logs no longer contain source code
URL: https://github.com/apache/airflow/issues/55910
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Exceptions in task logs no longer contain source code [airflow]
Posted by "souradeep07 (via GitHub)" <gi...@apache.org>.
souradeep07 commented on issue #55910:
URL: https://github.com/apache/airflow/issues/55910#issuecomment-6269342869
I've opened #75100 for this, following the approach suggested above: each frame in the JSON traceback gets a `line` field (read via `linecache`, as the stdlib `traceback` module does), and the UI shows it under the frame.
On the open questions:
- **Scope:** this isn't limited to deployments that opt into JSON output. The task process always logs JSON to the supervisor, so every task log viewed in the UI is affected.
- **Size:** it adds one stripped source line per frame, only when an exception is logged. Frames without available source are unchanged, and plain log lines are unaffected (the PR has a small benchmark).
Happy to change the shape if maintainers would prefer something else.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org