You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by "Maxim Ivanov (JIRA)" <ji...@apache.org> on 2015/02/08 21:13:34 UTC
[jira] [Updated] (KAFKA-1933) Fine-grained locking in log append
[ https://issues.apache.org/jira/browse/KAFKA-1933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Maxim Ivanov updated KAFKA-1933:
--------------------------------
Attachment: KAFKA-1933.patch
> Fine-grained locking in log append
> ----------------------------------
>
> Key: KAFKA-1933
> URL: https://issues.apache.org/jira/browse/KAFKA-1933
> Project: Kafka
> Issue Type: Improvement
> Components: log
> Reporter: Maxim Ivanov
> Assignee: Jay Kreps
> Priority: Minor
> Fix For: 0.8.2
>
> Attachments: KAFKA-1933.patch
>
>
> This patch adds finer locking when appending to log. It breaks
> global append lock into 2 sequential and 1 parallel phase.
> Basic idea is to allow every thread to "reserve" offsets in non
> overlapping ranges, then do compression in parallel and then
> "commit" write to log in the same order offsets where reserved.
> On my Core i3 M370 @2.4Ghz (2 cores + HT) it resulted in following performance boost:
> LZ4: 7.2 sec -> 3.9 sec
> Gzip: 62.3 sec -> 24.8 sec
> Kafka was configured to run 4 IO threads, data was pushed using 5 netcat instances pushing in parallel batches of 200 msg 6.2 kb each (510 MB in total, 82180 messages in total)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)