You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "Vinoth Chandar (JIRA)" <ji...@apache.org> on 2019/08/09 23:31:00 UTC
[jira] [Comment Edited] (KAFKA-7149) Reduce assignment data size to
improve kafka streams scalability
[ https://issues.apache.org/jira/browse/KAFKA-7149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16904245#comment-16904245 ]
Vinoth Chandar edited comment on KAFKA-7149 at 8/9/19 11:30 PM:
----------------------------------------------------------------
Pasting some size tests for old and new assignment information :
{code:java}
// Assumptions : Streams topology with 10 input topics, 4 sub topologies (2 topics per sub topology) = ~20 topics
// High number of hosts = 500; High number of partitions = 128
//topicPrefix = "streams_topic_name"; <- gains are very sensitive to topic name length ofc;
oldAssignmentInfoBytes : 77684 , newAssignmentInfoBytes: 42698{code}
Roughly ~45% savings.. For 500 hosts, we will be reducing from ~39MB -> ~21MB
its prett s
(NOTE: this is still a single object only, we do need the protocol change/compression on internal topics to ultimately fix the large message problem)
was (Author: vc):
Pasting some size tests for old and new assignment information :
{code:java}
// Assumptions : Streams topology with 10 input topics, 4 sub topologies (2 topics per sub topology) = ~20 topics
// High number of hosts = 500; High number of partitions = 128
oldAssignmentInfoBytes : 77684 , newAssignmentInfoBytes: 42698{code}
Roughly ~45% savings.. For 500 hosts, we will be reducing from ~39MB -> ~21MB
(NOTE: this is still a single object only, we do need the protocol change/compression on internal topics to ultimately fix the large message problem)
> Reduce assignment data size to improve kafka streams scalability
> ----------------------------------------------------------------
>
> Key: KAFKA-7149
> URL: https://issues.apache.org/jira/browse/KAFKA-7149
> Project: Kafka
> Issue Type: Improvement
> Components: streams
> Affects Versions: 2.0.0
> Reporter: Ashish Surana
> Assignee: Vinoth Chandar
> Priority: Major
>
> We observed that when we have high number of partitions, instances or stream-threads, assignment-data size grows too fast and we start getting below RecordTooLargeException at kafka-broker.
> Workaround of this issue is commented at: https://issues.apache.org/jira/browse/KAFKA-6976
> Still it limits the scalability of kafka streams as moving around 100MBs of assignment data for each rebalancing affects performance & reliability (timeout exceptions starts appearing) as well. Also this limits kafka streams scale even with high max.message.bytes setting as data size increases pretty quickly with number of partitions, instances or stream-threads.
>
> Solution:
> To address this issue in our cluster, we are sending the compressed assignment-data. We saw assignment-data size reduced by 8X-10X. This improved the kafka streams scalability drastically for us and we could now run it with more than 8,000 partitions.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)