[jira] [Commented] (KAFKA-5178) Potential Performance Degradation in Kafka Producer when using Multiple Threads

["Ismael Juma (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-5178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15998250#comment-15998250 ] 

Ismael Juma commented on KAFKA-5178:
------------------------------------

Hmm, is the issue about the number of threads or the number of partitions? Seems like some resource is exhausted, maybe the buffer pool?

> Potential Performance Degradation in Kafka Producer when using Multiple Threads
> -------------------------------------------------------------------------------
>
>                 Key: KAFKA-5178
>                 URL: https://issues.apache.org/jira/browse/KAFKA-5178
>             Project: Kafka
>          Issue Type: Bug
>            Reporter: Ben Stopford
>         Attachments: TestDetails.pdf
>
>
> There is evidence that the Kafka Producer drops performance as we increase the number of threads using it. 
> This is based on some benchmarking done in the community. I have not independently validated these results. Details of the test setup attached.
> ...
> *Effect of Shared KafkaProducer across threads*
>  
> Kafka documentation recommend using the KafkaProducer across multiple worker threads.
>  
> ||#Producers||#Consumers||#Topics||#Partitions per topic||RoundTrip Throughput (events/sec)||Approx Broker Events (Millions/sec)||
> |1|1|1|1|268,312|0.5|
> |4|4|4|4|759,186|1.5|
> |8|8|8|8|640,738|1.2|
> |8|8|8|16|847,314|1.7|
> |8|8|8|48|17,791|0.035|
> |16|16|16|64|5,997|0.01|
>  
> Something appears to be wrong here, with 48 and 64 partitions the shared KafkaProducer struggled to the point that performance became quite bad.
>  



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)