[jira] [Comment Edited] (KAFKA-1646) Improve consumer read performance for Windows

["Honghai Chen (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-1646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14348401#comment-14348401 ] 

Honghai Chen edited comment on KAFKA-1646 at 3/5/15 10:44 AM:
--------------------------------------------------------------

 Hey, [~jkreps]  just clarify, the 50MB/s you mentioned before is the checksum calculation on the machine, not copy replica data from other machine, right?

If that's true, seemly we need do 3 changes:
1, when call logManager.shutdown.   and os is windows ,  truncate active segment.
2, when start, if the os is windows,  add one new segment.
3, remove the change   " KAFKA-1646-truncate-off-trailing-zeros-on-broker-restart-if-bro.patch " made previously since it's unnecessary.   
Make sense?




was (Author: waldenchen):
Actually we want to add one more column to file  "recovery-point-offset-checkpoint", currently it only record offset, like below:
0
2
mvlogs 1 100
mvlogs 0 200
Change to below:
0
2
mvlogs 1 100 8000
mvlogs 0 200 16000

8000 is the start position of the data file for message with offset 100 . And 16000 is start position of the data file for message with offset 200.
Take first one as example, when recover the last segment (in function LogSegment.recover(maxMessageSize: Int) ,  ONLY recover  file to min(validBytes, 8000)  with offset 100 and rebuild index.   Make sense ?  [~jkreps]



> Improve consumer read performance for Windows
> ---------------------------------------------
>
>                 Key: KAFKA-1646
>                 URL: https://issues.apache.org/jira/browse/KAFKA-1646
>             Project: Kafka
>          Issue Type: Improvement
>          Components: log
>    Affects Versions: 0.8.1.1
>         Environment: Windows
>            Reporter: xueqiang wang
>            Assignee: xueqiang wang
>              Labels: newbie, patch
>         Attachments: Improve consumer read performance for Windows.patch, KAFKA-1646-truncate-off-trailing-zeros-on-broker-restart-if-bro.patch, KAFKA-1646_20141216_163008.patch
>
>
> This patch is for Window platform only. In Windows platform, if there are more than one replicas writing to disk, the segment log files will not be consistent in disk and then consumer reading performance will be dropped down greatly. This fix allocates more disk spaces when rolling a new segment, and then it will improve the consumer reading performance in NTFS file system.
> This patch doesn't affect file allocation of other filesystems, for it only adds statements like 'if(Os.iswindow)' or adds methods used on Windows.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)