You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Wang Yuan (Jira)" <ji...@apache.org> on 2021/04/24 16:35:00 UTC
[jira] [Created] (SPARK-35212) Spark Streaming LocationStrategy
should provide a random option that mapping kafka partitions randomly to
spark executors
Wang Yuan created SPARK-35212:
---------------------------------
Summary: Spark Streaming LocationStrategy should provide a random option that mapping kafka partitions randomly to spark executors
Key: SPARK-35212
URL: https://issues.apache.org/jira/browse/SPARK-35212
Project: Spark
Issue Type: New Feature
Components: DStreams
Affects Versions: 3.1.1
Reporter: Wang Yuan
There are three LocationStrategy: PreferBrokers, PreferConsistent, PreferFixed. I got a scenario that I need a random one. There are plenty of topic partitions that are varies from each other with different records inside. And I have a lot of executors. PreferBrokers does not help here. PreferConsistent will make things worse that some executor will always get heavy tasks. PreferFixed does not help too, because it is fixed, neither to say I have to create a mapping manually.
A random LocationStrategy should dispatch a topic partition to different executors in different window. This would balance the load among spark executors.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org