You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@kafka.apache.org by "Maxim Ivanov (JIRA)" <ji...@apache.org> on 2015/02/08 21:01:34 UTC

[jira] [Created] (KAFKA-1933) Fine-grained locking in log append

Maxim Ivanov created KAFKA-1933:
-----------------------------------

             Summary: Fine-grained locking in log append
                 Key: KAFKA-1933
                 URL: https://issues.apache.org/jira/browse/KAFKA-1933
             Project: Kafka
          Issue Type: Improvement
          Components: log
            Reporter: Maxim Ivanov
            Assignee: Jay Kreps
            Priority: Minor
             Fix For: 0.8.2


This patch adds finer locking when appending to log. It breaks
global append lock into 2 sequential and 1 parallel phase.

Basic idea is to allow every thread to "reserve" offsets in non
overlapping ranges, then do compression in parallel and then
"commit" write to log in the same order offsets where reserved.

On my Core i3 M370 @2.4Ghz (2 cores + HT) it resulted in following performance boost:

LZ4: 7.2 sec -> 3.9 sec
Gzip: 62.3 sec -> 24.8 sec

Kafka was configured to run 4 IO threads, data was pushed using 5 netcat instances pushing in parallel batches of 200 msg 6.2 kb each (510 MB in total, 82180 messages in total)




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)