[jira] [Updated] (KAFKA-309) Bug in FileMessageSet's append API can corrupt on disk log

["Neha Narkhede (Updated) (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/KAFKA-309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Neha Narkhede updated KAFKA-309:
--------------------------------

    Issue Type: Bug  (was: Sub-task)
        Parent:     (was: KAFKA-308)
    
> Bug in FileMessageSet's append API can corrupt on disk log
> ----------------------------------------------------------
>
>                 Key: KAFKA-309
>                 URL: https://issues.apache.org/jira/browse/KAFKA-309
>             Project: Kafka
>          Issue Type: Bug
>          Components: core
>    Affects Versions: 0.7
>            Reporter: Neha Narkhede
>            Assignee: Neha Narkhede
>            Priority: Critical
>         Attachments: kafka-309-test.patch
>
>
> In FileMessageSet's append API, we write a ByteBufferMessageSet to a log in the following manner -
>     while(written < messages.sizeInBytes)
>       written += messages.writeTo(channel, 0, messages.sizeInBytes)
> In ByteBufferMessageSet, the writeTo API uses buffer.duplicate() to append to a channel -
>   def writeTo(channel: GatheringByteChannel, offset: Long, size: Long): Long =
>     channel.write(buffer.duplicate)
> If the channel doesn't write the ByteBuffer in one call, then we call it again until sizeInBytes bytes are written. But the next call will use buffer.duplicate() to write to the FileChannel, which will write the entire ByteBufferMessageSet again to the file. 
> Effectively, we have a corrupted set of messages on disk. 
> Thinking about it, FileChannel is a blocking channel, so ideally, the entire ByteBuffer should be written to the FileChannel in one call. I wrote a test (attached here) and saw that it does. But I'm not aware if there are some corner cases when it doesn't do so. In those cases, Kafka will end up corrupting on disk log segment.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira