[GitHub] [rocketmq] BeiKeJieDeLiuLangMao opened a new issue #1434: Question about CommitLog::recoverAbnormally

[GitBox <gi...@apache.org>]

BeiKeJieDeLiuLangMao opened a new issue #1434: Question about CommitLog::recoverAbnormally
URL: https://github.com/apache/rocketmq/issues/1434
 
 
   As i know, logic of recoverAbnormally is as follow:
   1. Find first valid CommitLog file from last one
   2. Check all message in last valid CommitLog file
   3. For these valid messages in last valid CommitLog file, rebuild ConsumeQueue and IndexFile
   4. Fix all kinds of pointers, and clear ConsumeQueue redundant data
   
   Now consider follow situation:
   1. A few messages been written to CommitLog file L2(last 2), and these message are last messages of this CommitLog file
   2. Create a new CommitLog file L1(last 1),then write a message to L1
   3. At this time, L2's last few messages and L1's first message are not been written to ConsumeQueue and IndexFile yet, then RocketMQ crash
   4. When RocketMQ restart, it will invoke recoverAbnormally, and only rebuild L1's message, L2's last few messages are lost.
   
   If my analysis is right, i think follow solution could resolve it:
   1. Find first valid CommitLog file from last one
   2. Check all message in last valid CommitLog file
   3. Fix all kinds of pointers, and clear ConsumeQueue redundant data
   4. In step3, we could get valid ConsumeQueue max physics offset, besides we could also get IndexFile's max physics offset from file header
   5. In the light of ConsumeQueue max physics offset and IndexFile max physics offset, we could rebuild lost content respectively
   
   This solution not only could resolve data lost issue what i described before, but also could reduce duplicate data of ConsumeQueue and IndexFile :).
   
   
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services