[jira] [Commented] (FLUME-1968) FileChannel new format while being backwards compatible

["Brock Noland (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLUME-1968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13622456#comment-13622456 ] 

Brock Noland commented on FLUME-1968:
-------------------------------------

1) I think we should introduce a "sync" record similar to sequence file/avro so we can do a binary search to a specific write order id.

2) We current write a byte at the start of a record to indicate it's a record. We should write a trailing byte (the event footer can probably be removed)

3) We should prefix each write with it's checksum so we can read the checksum, read the length, read the length and then verify the event if we choose.
                
> FileChannel new format while being backwards compatible
> -------------------------------------------------------
>
>                 Key: FLUME-1968
>                 URL: https://issues.apache.org/jira/browse/FLUME-1968
>             Project: Flume
>          Issue Type: Bug
>          Components: Channel, File Channel
>            Reporter: Brock Noland
>
> There are a couple issues with the current format:
> 1) We have to track the offset at checkpoint time and write the offset to a special location so we can seek to that offset during replay. In FLUME-1516 we are tracking two offsets.
> 2) We have no way to detect partial writes FLUME-1967
> 3) We can only checksum the body of the event, not the entire record FLUME-1485 and therefore cannot detect corruption outside an event body.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira