[jira] [Updated] (KAFKA-4141) 2x increase in cpu usage on new Producer API

["Andrew Jorgensen (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/KAFKA-4141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Andrew Jorgensen updated KAFKA-4141:
------------------------------------
    Summary: 2x increase in cpu usage on new Producer API  (was: 2x increase in cp usage on new Producer API)

> 2x increase in cpu usage on new Producer API
> --------------------------------------------
>
>                 Key: KAFKA-4141
>                 URL: https://issues.apache.org/jira/browse/KAFKA-4141
>             Project: Kafka
>          Issue Type: Bug
>            Reporter: Andrew Jorgensen
>
> We are seeing about a 2x increase in CPU usage for the new kafka producer compared to the 0.8.0.1 producer. We are currently using gzip compression.
> We recently upgraded our kafka server and producer to 0.10.0.1 from 0.8.1.1 and noticed that the cpu usage for the new producer had increased pretty significantly compared to the old producer. This has caused us to need more resources to do the same amount of work we were doing before. I did some quick profiling and it looks like during sending half of the cpu cycles are sped in org.apache.kafka.common.record.Compressor.putRecord and the other half is in org.apache.kafka.common.record.Record.computeChecksum (both are around 5.8% of the cpu cycles for that method). I know its not apples to apples but the old producer did not seem to have this overhead or at least it was greatly reduced.
> Is this a known performance degradation compared to the old producer? 
> Here is the trace from the old producer:
> !http://imgur.com/1xS34Dl.jpg!
> New producer:
> !http://imgur.com/0w0G5b1.jpg!



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)