You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Apache Spark (Jira)" <ji...@apache.org> on 2020/10/28 16:38:00 UTC

[jira] [Commented] (SPARK-33274) [SS] Fix job hang in cp mode when total cores less than total kafka partition

    [ https://issues.apache.org/jira/browse/SPARK-33274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17222263#comment-17222263 ] 

Apache Spark commented on SPARK-33274:
--------------------------------------

User 'warrenzhu25' has created a pull request for this issue:
https://github.com/apache/spark/pull/30175

> [SS] Fix job hang in cp mode when total cores less than total kafka partition 
> ------------------------------------------------------------------------------
>
>                 Key: SPARK-33274
>                 URL: https://issues.apache.org/jira/browse/SPARK-33274
>             Project: Spark
>          Issue Type: Improvement
>          Components: Structured Streaming
>    Affects Versions: 3.0.1
>            Reporter: Zhongwei Zhu
>            Priority: Minor
>
> In continuous processing mode, EpochCoordinator won't add offsets to query until got ReportPartitionOffset from all partitions. Normally, each kafka topic partition will be handled by one core, if total cores is smaller than total kafka topic partition counts, the job will hang without any error message. 
> Failing the job with error message is better solution.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org