Re: [I] Exceptions in task logs no longer contain source code [airflow]

["arose26 (via GitHub)" <gi...@apache.org>]

arose26 commented on issue #55910:
URL: https://github.com/apache/airflow/issues/55910#issuecomment-5315481188

   This reproduces, and the cause looks structural rather than a formatting slip — it only affects deployments using JSON log output, which may be why it hasn't reproduced for everyone in this thread.
   
   **Where the source lines go missing**
   
   `shared/logging/src/airflow_shared/logging/structlog.py` builds the JSON branch's exception renderer from structlog's `ExceptionDictTransformer`:
   
   ```python
   dict_exc_formatter = structlog.tracebacks.ExceptionDictTransformer(
       use_rich=False, show_locals=False, suppress=suppress
   )
   dict_tracebacks = structlog.processors.ExceptionRenderer(dict_exc_formatter)
   ```
   
   That transformer emits one dict per frame containing `filename`, `lineno` and `name` — and nothing else. There is no field carrying the source text of the line, so the rendered traceback can only ever show the frame headers, never the indented code line beneath them. Checked against structlog 26.1.0:
   
   ```
   use_rich=False -> frame keys = ['filename', 'lineno', 'name']
   use_rich=True  -> frame keys = ['filename', 'lineno', 'name']
   ```
   
   `use_rich` makes no difference here; both paths produce the same three keys.
   
   **Why it's specific to JSON output**
   
   The non-JSON branch of the same function uses `structlog.dev.plain_traceback` (or `RichTracebackFormatter` when `DEV` is set), and those go through `traceback.print_exception`, which pulls the source line via `linecache`. Both retain the code line. So the same failing task shows full tracebacks in text output and header-only tracebacks in JSON output.
   
   That matches the Airflow 2 → 3 change: the standard-library formatting that carried the source line is no longer on the path for structured logs.
   
   **Question before anyone writes a patch**
   
   Restoring the code line would mean carrying it per frame — for example a transformer that adds `linecache.getline(filename, lineno)` alongside the existing keys, with the UI rendering it under each frame. That does make every logged traceback bigger, and `show_locals=False` / `use_rich=False` suggest the current settings were chosen deliberately to keep structured logs compact.
   
   Is restoring parity with the text output something the project wants, and if so is a per-frame source field the right shape — or would you rather this stayed as is and the docs noted the difference? Happy to put up a PR once there's a direction.
   
   ---
   Drafted-by: Claude Code (Opus 5); reviewed by @arose26 before posting
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org