You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Jungtaek Lim (Jira)" <ji...@apache.org> on 2025/03/25 01:36:00 UTC

[jira] [Assigned] (SPARK-51586) Kafka continuous stream may go into infinite loop of reconfiguring

     [ https://issues.apache.org/jira/browse/SPARK-51586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jungtaek Lim reassigned SPARK-51586:
------------------------------------

    Assignee: Vlad Rozov

> Kafka continuous stream may go into infinite loop of reconfiguring
> ------------------------------------------------------------------
>
>                 Key: SPARK-51586
>                 URL: https://issues.apache.org/jira/browse/SPARK-51586
>             Project: Spark
>          Issue Type: Bug
>          Components: Structured Streaming
>    Affects Versions: 4.0.0, 4.1.0
>            Reporter: Vlad Rozov
>            Assignee: Vlad Rozov
>            Priority: Major
>              Labels: pull-request-available
>
> After [-SPARK-45080-|https://issues.apache.org/jira/browse/SPARK-45080] and [-PR 42823-|https://github.com/apache/spark/pull/42823] Kafka continuous stream may go into infinite loop of reconfiguring. As {{KafkaScan}} columnar support mode is hardcoded to {{UNSUPPORTED}} {{ContinuousScanExec}} does not initialize {{inputPartitions}} in {{supportsColumnar}} as it did prior to the above PR. So, there is no call to {{KafkaContinuousStream.planInputPartitions}} during query planning. It leaves {{KafkaContinuousStream.knownPartitions}} uninitialized, so call to {{needsReconfiguration}} returns true. Now {{epochUpdateThread}} in {{ContinuousExecution}} requests interruption of {{queryExecutionThread}}, so that thread also can't initialize {{knownPartitions}} as it checks for interrupts in {{KafkaOffsetReaderAdmin.withRetries}} and exits before {{knownPartitions}}  is assigned.
> The problem can be identified in the logs by the following repeating pattern:
> {noformat}
> 25/03/21 09:38:09.709 stream execution thread for [id = 085613ee-b0b0-4864-8ac9-2bbd7c885e13, runId = 03a5bef3-f5e1-4d23-991d-a19873db9192] INFO CommitLog: BatchIds found from listing: 
> 25/03/21 09:38:09.709 stream execution thread for [id = 085613ee-b0b0-4864-8ac9-2bbd7c885e13, runId = 03a5bef3-f5e1-4d23-991d-a19873db9192] INFO ContinuousExecution: Starting new streaming query.
> 25/03/21 09:38:09.710 stream execution thread for [id = 085613ee-b0b0-4864-8ac9-2bbd7c885e13, runId = 03a5bef3-f5e1-4d23-991d-a19873db9192] INFO KafkaContinuousStream: Initial offsets: {"topic-0":{"0":1,"1":1,"2":1,"3":1,"4":1}}
> 25/03/21 09:38:09.710 stream execution thread for [id = 085613ee-b0b0-4864-8ac9-2bbd7c885e13, runId = 03a5bef3-f5e1-4d23-991d-a19873db9192] INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer.
> 25/03/21 09:38:09.710 stream execution thread for [id = 085613ee-b0b0-4864-8ac9-2bbd7c885e13, runId = 03a5bef3-f5e1-4d23-991d-a19873db9192] INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer.
> 25/03/21 09:38:09.711 stream execution thread for [id = 085613ee-b0b0-4864-8ac9-2bbd7c885e13, runId = 03a5bef3-f5e1-4d23-991d-a19873db9192] INFO EpochCoordinatorRef: Registered EpochCoordinator endpoint
> 25/03/21 09:38:09.711 stream execution thread for [id = 085613ee-b0b0-4864-8ac9-2bbd7c885e13, runId = 03a5bef3-f5e1-4d23-991d-a19873db9192] INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer.
> 25/03/21 09:38:09.711 stream execution thread for [id = 085613ee-b0b0-4864-8ac9-2bbd7c885e13, runId = 03a5bef3-f5e1-4d23-991d-a19873db9192] INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer.
> 25/03/21 09:38:09.713 stream execution thread for [id = 085613ee-b0b0-4864-8ac9-2bbd7c885e13, runId = 03a5bef3-f5e1-4d23-991d-a19873db9192] INFO ContinuousExecution: Query 085613ee-b0b0-4864-8ac9-2bbd7c885e13 ignoring exception from reconfiguring: java.lang.InterruptedException
> 25/03/21 09:38:09.713 stream execution thread for [id = 085613ee-b0b0-4864-8ac9-2bbd7c885e13, runId = 03a5bef3-f5e1-4d23-991d-a19873db9192] INFO DAGScheduler: Asked to cancel job group 03a5bef3-f5e1-4d23-991d-a19873db9192 with cancelFutureJobs=false
> 25/03/21 09:38:09.713 dag-scheduler-event-loop WARN DAGScheduler: Failed to cancel job group 03a5bef3-f5e1-4d23-991d-a19873db9192. Cannot find active jobs for it.
> {noformat}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org