You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by "Jun Rao (JIRA)" <ji...@apache.org> on 2012/10/18 18:20:05 UTC
[jira] [Created] (KAFKA-577) extend DumpLogSegments to verify
consistency btw data and index
Jun Rao created KAFKA-577:
-----------------------------
Summary: extend DumpLogSegments to verify consistency btw data and index
Key: KAFKA-577
URL: https://issues.apache.org/jira/browse/KAFKA-577
Project: Kafka
Issue Type: Bug
Components: core
Affects Versions: 0.8
Reporter: Jun Rao
It would be good to extend DumpLogSegments to do the following verification:
1. The offsets stored in the index match those in the log data.
2. The offsets in the data log is consecutive.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (KAFKA-577) extend DumpLogSegments to verify
consistency btw data and index
Posted by "Jun Rao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jun Rao reopened KAFKA-577:
---------------------------
> extend DumpLogSegments to verify consistency btw data and index
> ---------------------------------------------------------------
>
> Key: KAFKA-577
> URL: https://issues.apache.org/jira/browse/KAFKA-577
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.8
> Reporter: Jun Rao
> Labels: newbie, tools
> Fix For: 0.8
>
> Attachments: kafka_577_v1.diff, kafka_577_v2.diff
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> It would be good to extend DumpLogSegments to do the following verification:
> 1. The offsets stored in the index match those in the log data.
> 2. The offsets in the data log is consecutive.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (KAFKA-577) extend DumpLogSegments to verify
consistency btw data and index
Posted by "Jun Rao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jun Rao resolved KAFKA-577.
---------------------------
Resolution: Fixed
Fix Version/s: 0.8
+1 on patch v6. Committed to 0.8.
> extend DumpLogSegments to verify consistency btw data and index
> ---------------------------------------------------------------
>
> Key: KAFKA-577
> URL: https://issues.apache.org/jira/browse/KAFKA-577
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.8
> Reporter: Jun Rao
> Labels: newbie, tools
> Fix For: 0.8
>
> Attachments: kafka_577_v1.diff, kafka_577_v2.diff
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> It would be good to extend DumpLogSegments to do the following verification:
> 1. The offsets stored in the index match those in the log data.
> 2. The offsets in the data log is consecutive.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-577) extend DumpLogSegments to verify
consistency btw data and index
Posted by "Yang Ye (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yang Ye updated KAFKA-577:
--------------------------
Attachment: kafka_577_v4.diff
1. also "do it in batch" for dumpLog() function
2. use scala HashMap instead of Pool
3. Use LinkedList in the right way
> extend DumpLogSegments to verify consistency btw data and index
> ---------------------------------------------------------------
>
> Key: KAFKA-577
> URL: https://issues.apache.org/jira/browse/KAFKA-577
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.8
> Reporter: Jun Rao
> Labels: newbie, tools
> Fix For: 0.8
>
> Attachments: kafka_577_v1.diff, kafka_577_v2.diff, kafka_577_v3.diff, kafka_577_v4.diff
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> It would be good to extend DumpLogSegments to do the following verification:
> 1. The offsets stored in the index match those in the log data.
> 2. The offsets in the data log is consecutive.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-577) extend DumpLogSegments to verify
consistency btw data and index
Posted by "Jun Rao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13489255#comment-13489255 ]
Jun Rao commented on KAFKA-577:
-------------------------------
A couple of more comments for patch v4.
40. LinkedList is supposed to be used for manipulating links directly. We should just use List (immutable) instead. Also, instead of the following,
misMatchesSeq = misMatchesSeq.:+((entry.offset + index.baseOffset, messageAndOffset.offset).asInstanceOf[(Int, Int)])
we should write
misMatchesSeq :+= ((entry.offset + index.baseOffset, messageAndOffset.offset).asInstanceOf[(Int, Int)])
> extend DumpLogSegments to verify consistency btw data and index
> ---------------------------------------------------------------
>
> Key: KAFKA-577
> URL: https://issues.apache.org/jira/browse/KAFKA-577
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.8
> Reporter: Jun Rao
> Labels: newbie, tools
> Fix For: 0.8
>
> Attachments: kafka_577_v1.diff, kafka_577_v2.diff, kafka_577_v3.diff, kafka_577_v4.diff
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> It would be good to extend DumpLogSegments to do the following verification:
> 1. The offsets stored in the index match those in the log data.
> 2. The offsets in the data log is consecutive.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (KAFKA-577) extend DumpLogSegments to verify
consistency btw data and index
Posted by "Jun Rao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jun Rao resolved KAFKA-577.
---------------------------
Resolution: Fixed
Thanks for patch v6. +1. Committed to 0.8 with a minor change: using ::= on a list instead of .:+=
> extend DumpLogSegments to verify consistency btw data and index
> ---------------------------------------------------------------
>
> Key: KAFKA-577
> URL: https://issues.apache.org/jira/browse/KAFKA-577
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.8
> Reporter: Jun Rao
> Labels: newbie, tools
> Fix For: 0.8
>
> Attachments: kafka_577_v1.diff, kafka_577_v2.diff, kafka_577_v3.diff, kafka_577_v4.diff, kafka_577_v5.diff, kafka_577_v6.diff
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> It would be good to extend DumpLogSegments to do the following verification:
> 1. The offsets stored in the index match those in the log data.
> 2. The offsets in the data log is consecutive.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-577) extend DumpLogSegments to verify
consistency btw data and index
Posted by "Jun Rao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13485306#comment-13485306 ]
Jun Rao commented on KAFKA-577:
-------------------------------
Thanks for the patch. Some comments:
1. Could you add a --verifyOnly option so that we only do the verification but not print out the content?
2. The text in the following statement is too verbose. It doesn't need to print the index file since it's provided in the command line. It can just say "Index position %d doesn't match log position at offset %d".
System.err.println(("The offset in index file [%s] does not match the offset stored in " +
"log file [%s], they're %d and %d separately").format(entry.offset + index.baseOffset, messageAndOffset.offset))
3. Shouldn't we use %d instead of %l in the following line?
System.err.println("The offset in the data log file [%s] is not consecutive, [%l] follows [%l]".format(file.getName, messageAndOffset.offset, lastOffset))
4. Could you add a shell script for DumpLogSegments in bin/ ?
> extend DumpLogSegments to verify consistency btw data and index
> ---------------------------------------------------------------
>
> Key: KAFKA-577
> URL: https://issues.apache.org/jira/browse/KAFKA-577
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.8
> Reporter: Jun Rao
> Labels: newbie, tools
> Attachments: kafka_577_v1.diff
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> It would be good to extend DumpLogSegments to do the following verification:
> 1. The offsets stored in the index match those in the log data.
> 2. The offsets in the data log is consecutive.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-577) extend DumpLogSegments to verify
consistency btw data and index
Posted by "Yang Ye (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yang Ye updated KAFKA-577:
--------------------------
Attachment: kafka_577_v6.diff
Using jOptSimple for option parsing
> extend DumpLogSegments to verify consistency btw data and index
> ---------------------------------------------------------------
>
> Key: KAFKA-577
> URL: https://issues.apache.org/jira/browse/KAFKA-577
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.8
> Reporter: Jun Rao
> Labels: newbie, tools
> Fix For: 0.8
>
> Attachments: kafka_577_v1.diff, kafka_577_v2.diff, kafka_577_v3.diff, kafka_577_v4.diff, kafka_577_v5.diff, kafka_577_v6.diff
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> It would be good to extend DumpLogSegments to do the following verification:
> 1. The offsets stored in the index match those in the log data.
> 2. The offsets in the data log is consecutive.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-577) extend DumpLogSegments to verify
consistency btw data and index
Posted by "Jun Rao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13486548#comment-13486548 ]
Jun Rao commented on KAFKA-577:
-------------------------------
Thanks for patch v2. Instead of exiting on first verification failure, it would be better if we dump the whole log and report all failed verification at the very end.
> extend DumpLogSegments to verify consistency btw data and index
> ---------------------------------------------------------------
>
> Key: KAFKA-577
> URL: https://issues.apache.org/jira/browse/KAFKA-577
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.8
> Reporter: Jun Rao
> Labels: newbie, tools
> Fix For: 0.8
>
> Attachments: kafka_577_v1.diff, kafka_577_v2.diff
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> It would be good to extend DumpLogSegments to do the following verification:
> 1. The offsets stored in the index match those in the log data.
> 2. The offsets in the data log is consecutive.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-577) extend DumpLogSegments to verify
consistency btw data and index
Posted by "Jun Rao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jun Rao updated KAFKA-577:
--------------------------
Comment: was deleted
(was: +1 on patch v6. Committed to 0.8.)
> extend DumpLogSegments to verify consistency btw data and index
> ---------------------------------------------------------------
>
> Key: KAFKA-577
> URL: https://issues.apache.org/jira/browse/KAFKA-577
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.8
> Reporter: Jun Rao
> Labels: newbie, tools
> Fix For: 0.8
>
> Attachments: kafka_577_v1.diff, kafka_577_v2.diff
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> It would be good to extend DumpLogSegments to do the following verification:
> 1. The offsets stored in the index match those in the log data.
> 2. The offsets in the data log is consecutive.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-577) extend DumpLogSegments to verify
consistency btw data and index
Posted by "Yang Ye (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yang Ye updated KAFKA-577:
--------------------------
Attachment: kafka_577_v2.diff
changed according to the comments
> extend DumpLogSegments to verify consistency btw data and index
> ---------------------------------------------------------------
>
> Key: KAFKA-577
> URL: https://issues.apache.org/jira/browse/KAFKA-577
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.8
> Reporter: Jun Rao
> Labels: newbie, tools
> Attachments: kafka_577_v1.diff, kafka_577_v2.diff
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> It would be good to extend DumpLogSegments to do the following verification:
> 1. The offsets stored in the index match those in the log data.
> 2. The offsets in the data log is consecutive.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-577) extend DumpLogSegments to verify
consistency btw data and index
Posted by "Yang Ye (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yang Ye updated KAFKA-577:
--------------------------
Attachment: kafka_577_v1.diff
> extend DumpLogSegments to verify consistency btw data and index
> ---------------------------------------------------------------
>
> Key: KAFKA-577
> URL: https://issues.apache.org/jira/browse/KAFKA-577
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.8
> Reporter: Jun Rao
> Labels: newbie, tools
> Attachments: kafka_577_v1.diff
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> It would be good to extend DumpLogSegments to do the following verification:
> 1. The offsets stored in the index match those in the log data.
> 2. The offsets in the data log is consecutive.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-577) extend DumpLogSegments to verify
consistency btw data and index
Posted by "Yang Ye (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yang Ye updated KAFKA-577:
--------------------------
Attachment: kafka_577_v3.diff
cache all verification failure and print out at the end, not verified by real data yet because cannot find proper data for that...., just check the logic of the code..
> extend DumpLogSegments to verify consistency btw data and index
> ---------------------------------------------------------------
>
> Key: KAFKA-577
> URL: https://issues.apache.org/jira/browse/KAFKA-577
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.8
> Reporter: Jun Rao
> Labels: newbie, tools
> Fix For: 0.8
>
> Attachments: kafka_577_v1.diff, kafka_577_v2.diff, kafka_577_v3.diff
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> It would be good to extend DumpLogSegments to do the following verification:
> 1. The offsets stored in the index match those in the log data.
> 2. The offsets in the data log is consecutive.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Closed] (KAFKA-577) extend DumpLogSegments to verify
consistency btw data and index
Posted by "Jun Rao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jun Rao closed KAFKA-577.
-------------------------
> extend DumpLogSegments to verify consistency btw data and index
> ---------------------------------------------------------------
>
> Key: KAFKA-577
> URL: https://issues.apache.org/jira/browse/KAFKA-577
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.8
> Reporter: Jun Rao
> Labels: newbie, tools
> Fix For: 0.8
>
> Attachments: kafka_577_v1.diff, kafka_577_v2.diff, kafka_577_v3.diff, kafka_577_v4.diff, kafka_577_v5.diff, kafka_577_v6.diff
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> It would be good to extend DumpLogSegments to do the following verification:
> 1. The offsets stored in the index match those in the log data.
> 2. The offsets in the data log is consecutive.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-577) extend DumpLogSegments to verify
consistency btw data and index
Posted by "Neha Narkhede (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13490305#comment-13490305 ]
Neha Narkhede commented on KAFKA-577:
-------------------------------------
Sorry for coming to this a little late. Patch v5 looks good, just one suggestion - Looks like DumpLogSegments is the only tool that doesn't quite follow the convention for argument parsing. It will be good to standardize on jopt parser since it also helps with a description for each option, a help command and convenient required argument checks. Just my 2c.
> extend DumpLogSegments to verify consistency btw data and index
> ---------------------------------------------------------------
>
> Key: KAFKA-577
> URL: https://issues.apache.org/jira/browse/KAFKA-577
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.8
> Reporter: Jun Rao
> Labels: newbie, tools
> Fix For: 0.8
>
> Attachments: kafka_577_v1.diff, kafka_577_v2.diff, kafka_577_v3.diff, kafka_577_v4.diff, kafka_577_v5.diff
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> It would be good to extend DumpLogSegments to do the following verification:
> 1. The offsets stored in the index match those in the log data.
> 2. The offsets in the data log is consecutive.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-577) extend DumpLogSegments to verify
consistency btw data and index
Posted by "Yang Ye (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yang Ye updated KAFKA-577:
--------------------------
Attachment: kafka_577_v5.diff
Thanks for the comments, and the LinkedList is replaced with List now
> extend DumpLogSegments to verify consistency btw data and index
> ---------------------------------------------------------------
>
> Key: KAFKA-577
> URL: https://issues.apache.org/jira/browse/KAFKA-577
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.8
> Reporter: Jun Rao
> Labels: newbie, tools
> Fix For: 0.8
>
> Attachments: kafka_577_v1.diff, kafka_577_v2.diff, kafka_577_v3.diff, kafka_577_v4.diff, kafka_577_v5.diff
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> It would be good to extend DumpLogSegments to do the following verification:
> 1. The offsets stored in the index match those in the log data.
> 2. The offsets in the data log is consecutive.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-577) extend DumpLogSegments to verify
consistency btw data and index
Posted by "Yang Ye (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13490834#comment-13490834 ]
Yang Ye commented on KAFKA-577:
-------------------------------
@Neha sure, we can do that
> extend DumpLogSegments to verify consistency btw data and index
> ---------------------------------------------------------------
>
> Key: KAFKA-577
> URL: https://issues.apache.org/jira/browse/KAFKA-577
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.8
> Reporter: Jun Rao
> Labels: newbie, tools
> Fix For: 0.8
>
> Attachments: kafka_577_v1.diff, kafka_577_v2.diff, kafka_577_v3.diff, kafka_577_v4.diff, kafka_577_v5.diff
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> It would be good to extend DumpLogSegments to do the following verification:
> 1. The offsets stored in the index match those in the log data.
> 2. The offsets in the data log is consecutive.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira