You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Apache Spark (Jira)" <ji...@apache.org> on 2021/04/25 02:04:00 UTC
[jira] [Commented] (SPARK-35212) Spark Streaming LocationStrategy
should provide a random option that mapping kafka partitions randomly to
spark executors
[ https://issues.apache.org/jira/browse/SPARK-35212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17331352#comment-17331352 ]
Apache Spark commented on SPARK-35212:
--------------------------------------
User 'tiehexue' has created a pull request for this issue:
https://github.com/apache/spark/pull/32326
> Spark Streaming LocationStrategy should provide a random option that mapping kafka partitions randomly to spark executors
> -------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-35212
> URL: https://issues.apache.org/jira/browse/SPARK-35212
> Project: Spark
> Issue Type: New Feature
> Components: DStreams, Spark Core
> Affects Versions: 3.1.1
> Reporter: Wang Yuan
> Priority: Critical
> Labels: pull-request-available
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> There are three LocationStrategy: PreferBrokers, PreferConsistent, PreferFixed. I got a scenario that I need a random one. There are plenty of topic partitions that are varies from each other with different records inside. And I have a lot of executors. PreferBrokers does not help here. PreferConsistent will make things worse that some executor will always get heavy tasks. PreferFixed does not help too, because it is fixed, neither to say I have to create a mapping manually.
> A random LocationStrategy should dispatch a topic partition to different executors in different window. This would balance the load among spark executors.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org