[jira] [Created] (SPARK-33274) [SS] Fix job hang in cp mode when total cores less than total kafka partition

["Zhongwei Zhu (Jira)" <ji...@apache.org>]

Zhongwei Zhu created SPARK-33274:
------------------------------------

             Summary: [SS] Fix job hang in cp mode when total cores less than total kafka partition 
                 Key: SPARK-33274
                 URL: https://issues.apache.org/jira/browse/SPARK-33274
             Project: Spark
          Issue Type: Improvement
          Components: Structured Streaming
    Affects Versions: 3.0.1
            Reporter: Zhongwei Zhu


In continuous processing mode, EpochCoordinator won't add offsets to query until got ReportPartitionOffset from all partitions. Normally, each kafka topic partition will be handled by one core, if total cores is smaller than total kafka topic partition counts, the job will hang without any error message. 

Failing the job with error message is better solution.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org