You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Jungtaek Lim (Jira)" <ji...@apache.org> on 2026/04/16 21:03:00 UTC
[jira] [Assigned] (SPARK-56326) Add Streaming query id and batch id to task scheduling logs
[ https://issues.apache.org/jira/browse/SPARK-56326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jungtaek Lim reassigned SPARK-56326:
------------------------------------
Assignee: Brooks Walls
> Add Streaming query id and batch id to task scheduling logs
> -----------------------------------------------------------
>
> Key: SPARK-56326
> URL: https://issues.apache.org/jira/browse/SPARK-56326
> Project: Spark
> Issue Type: Improvement
> Components: Scheduler, Structured Streaming
> Affects Versions: 4.2.0
> Reporter: Brooks Walls
> Assignee: Brooks Walls
> Priority: Minor
> Labels: pull-request-available
>
> Currently, logs involving the scheduling of tasks do not contain information such as query id and batch id for streaming queries. This makes debugging streaming queries confusing, especially when there are multiple queries running. Lets add the query id and batch id to each log about task scheduling when processing a streaming query.
> The current logs involving task scheduling look like this:
> {code:java}
> 6/03/11 22:03:29 INFO FairSchedulableBuilder: Added task set TaskSet_486190.0 tasks to pool 1772179380933
> 6/03/11 22:03:29 INFO TaskSetManager: Starting task 0.0 in stage 486190.0 (TID 3075017) (10.68.141.175,executor 13, partition 0, PROCESS_LOCAL,
> 6/03/11 22:03:29 INFO TaskSetManager: Finished task 0.0 in stage 486190.0 (TID 3075017) in 52 ms on 10.68.141.175 (executor 13) (1/1){code}
> Lets add query and batch information:
> {code:java}
> 6/03/11 22:03:29 INFO FairSchedulableBuilder: [queryId = 71c67] [batchId = 685] Added task set TaskSet_486190.0 tasks to pool 1772179380933
> 6/03/11 22:03:29 INFO TaskSetManager: [queryId = 71c67] [batchId = 685] Starting task 0.0 in stage 486190.0 (TID 3075017) (10.68.141.175,executor 13, partition 0, PROCESS_LOCAL,
> 6/03/11 22:03:29 INFO TaskSetManager: [queryId = 71c67] [batchId = 685] Finished task 0.0 in stage 486190.0 (TID 3075017) in 52 ms on 10.68.141.175 (executor 13) (1/1){code}
> *Impact:* This change is strictly additive to logs and does not change any public APIs or internal scheduling logic.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org