[jira] [Updated] (KAFKA-77) Implement "group commit" for kafka logs

["Jay Kreps (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/KAFKA-77?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jay Kreps updated KAFKA-77:
---------------------------

    Attachment: kafka-group-commit.patch

A patch that implements group commit for kafka. This implementation is a little complex, the append() method is now a little scary, maybe someone sees a way to do it more simply.

A couple of notes:

1. I don't use any separate threads, the actual write is done by one of the writing threads involved in the commit (essentially it is a race, whomever gets there first does it).

2. I only try to batch the flush, I don't try to batch the write() call though. Batching writes could be done as well but it would require either working around the MessageSet.writeTo interface since you now want to write multiple message sets at once in a single call, which breaks the current abstraction. Also the write call gives time for more writes to accumulate in the group so that might not help.

3. I try to limit the group size to some fixed upper limit (50) which I just hard code. In practice I could not produce groups of more than 3, but I want to guarantee that you can't block the commit forever by queuing up writes under high load.

This whole idea is really only worth it if there are non-pathological cases where performance gets significantly better and performance doesn't get worse anywhere else.

I haven't really done any performance testing yet as my laptop seems to get CPU bound by the producer perf test process which means I am having trouble producing an I/O-bound load on one machine. I think I need to run an experiment with more than one producer machine which is separate from the kafka machine and perhaps with more than one topic to force seeks when we do flushes (sequentially flushes should be much cheaper but that would only happen if you had one topic). I will update this bug when i have some real benchmarking.


> Implement "group commit" for kafka logs
> ---------------------------------------
>
>                 Key: KAFKA-77
>                 URL: https://issues.apache.org/jira/browse/KAFKA-77
>             Project: Kafka
>          Issue Type: Improvement
>    Affects Versions: 0.7
>            Reporter: Jay Kreps
>            Assignee: Jay Kreps
>             Fix For: 0.8
>
>         Attachments: kafka-group-commit.patch
>
>
> The most expensive operation for the server is usually going to be the fsync() call to sync data in a log to disk, if you don't flush your data is at greater risk of being lost in a crash. Currently we give two knobs to tune this trade--log.flush.interval and log.default.flush.interval.ms (no idea why one has default and the other doesn't since they are both defaults). However if you flush frequently, say on every write, then performance is not that great.
> One trick that can be used to improve this worst case of continual flushes is to allow a single fsync() to be used for multiple writes that occur at the same time. This is a lot like "group commit" in databases. It is unclear which cases this would improve and by how much but it might be worth a try.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira