[I] Include `run_id` in TaskInstance Logging for Enhanced Traceability and Debugging [airflow]

["RyuSA (via GitHub)" <gi...@apache.org>]

RyuSA opened a new issue, #39272:
URL: https://github.com/apache/airflow/issues/39272

   ### Description
   
   Currently, the log outputs for TaskInstance state changes within `TaskInstance.py` do not include the `run_id`. 
   
   i.e https://github.com/apache/airflow/blob/2.9.0/airflow/models/taskinstance.py#L1200
   
   This omission makes it difficult to directly trace and verify the status of specific Task Instances without relying on inference from the `execution_date`. Including `run_id` in these logs would greatly enhance the traceability and debugging process, particularly in environments where multiple runs might be managed concurrently.
   
   I propose changing the log outputs to include `run_id` in  `TaskInstance._log_state` and `TaskInstance._run_raw_task#TaskDeferred` log entry. The `run_id` is one of the primary keys for TaskInstances and I think its inclusion in the logs should not introduce any issues.
   
   **Current Log Outputs**
   - Pausing task as DEFERRED: dag_id=Test1, task_id=task1, execution_date=..., start_date=...
   - Marking task as SUCCESS: dag_id=Test1, task_id=task1, execution_date=..., start_date=..., end_date=...
   
   **Proposed Log Outouts**
   - Pausing task as DEFERRED: dag_id=Test1, task_id=task1, run_id=RUN_ID, execution_date=..., start_date=...
   - Marking task as SUCCESS: dag_id=Test1, task_id=task1, run_id=RUN_ID, execution_date=..., start_date=..., end_date=...
   
   
   ### Use case/motivation
   
   Debugging and troubleshooting specific DAGRuns on Workers can be a little bit cumbersome without the ability to search logs by `run_id`. 
   Currently, we need infer `execution_date` from `run_id` to search worker log to find the final status of a TaskInstance on a Worker. This leads to unnecessary queries and steps. Including `run_id` in the logs would simplify log searches in distributed environments(like Airflow on Kubernetes) and likely assist other users facing similar challenges.
   
   ### Related issues
   
   _No response_
   
   ### Are you willing to submit a PR?
   
   - [X] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Include `run_id` in TaskInstance Logging for Enhanced Traceability and Debugging [airflow]

["eladkal (via GitHub)" <gi...@apache.org>]

eladkal closed issue #39272:  Include `run_id` in TaskInstance Logging for Enhanced Traceability and Debugging
URL: https://github.com/apache/airflow/issues/39272


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Include `run_id` in TaskInstance Logging for Enhanced Traceability and Debugging [airflow]

["boring-cyborg[bot] (via GitHub)" <gi...@apache.org>]

boring-cyborg[bot] commented on issue #39272:
URL: https://github.com/apache/airflow/issues/39272#issuecomment-2078642768

   Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org