You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by "Jun Rao (JIRA)" <ji...@apache.org> on 2012/10/06 03:54:02 UTC
[jira] [Created] (KAFKA-551) Log.truncateTo() may need to trucate
immutable log segment
Jun Rao created KAFKA-551:
-----------------------------
Summary: Log.truncateTo() may need to trucate immutable log segment
Key: KAFKA-551
URL: https://issues.apache.org/jira/browse/KAFKA-551
Project: Kafka
Issue Type: Bug
Components: core
Affects Versions: 0.8
Reporter: Jun Rao
Priority: Blocker
In makeFollower, we need to first truncate the local log to high watermark. It's possible that we need to truncate into segments before the last one. The problem is that all segments except the last one are immutable. So the truncation will fail which prevents the replica fetcher from being started.
One solution is to reopen the segment as mutable during truncation, if it's not mutable already.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Closed] (KAFKA-551) Log.truncateTo() may need to trucate
immutable log segment
Posted by "Jun Rao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jun Rao closed KAFKA-551.
-------------------------
> Log.truncateTo() may need to trucate immutable log segment
> ----------------------------------------------------------
>
> Key: KAFKA-551
> URL: https://issues.apache.org/jira/browse/KAFKA-551
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.8
> Reporter: Jun Rao
> Assignee: Jay Kreps
> Priority: Blocker
> Labels: bugs
> Attachments: KAFKA-551.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> In makeFollower, we need to first truncate the local log to high watermark. It's possible that we need to truncate into segments before the last one. The problem is that all segments except the last one are immutable. So the truncation will fail which prevents the replica fetcher from being started.
> One solution is to reopen the segment as mutable during truncation, if it's not mutable already.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-551) Log.truncateTo() may need to trucate
immutable log segment
Posted by "Jay Kreps (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13470879#comment-13470879 ]
Jay Kreps commented on KAFKA-551:
---------------------------------
I think the simplest solution would just be to open all segments as r/w since that is effectively what they are now.
I can take this if you like since I was just mucking around on that stuff and it is possible we have the same bug in the OffsetIndex.
> Log.truncateTo() may need to trucate immutable log segment
> ----------------------------------------------------------
>
> Key: KAFKA-551
> URL: https://issues.apache.org/jira/browse/KAFKA-551
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.8
> Reporter: Jun Rao
> Priority: Blocker
> Labels: bugs
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> In makeFollower, we need to first truncate the local log to high watermark. It's possible that we need to truncate into segments before the last one. The problem is that all segments except the last one are immutable. So the truncation will fail which prevents the replica fetcher from being started.
> One solution is to reopen the segment as mutable during truncation, if it's not mutable already.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-551) Log.truncateTo() may need to trucate
immutable log segment
Posted by "Neha Narkhede (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13472480#comment-13472480 ]
Neha Narkhede commented on KAFKA-551:
-------------------------------------
+1. LGTM
> Log.truncateTo() may need to trucate immutable log segment
> ----------------------------------------------------------
>
> Key: KAFKA-551
> URL: https://issues.apache.org/jira/browse/KAFKA-551
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.8
> Reporter: Jun Rao
> Priority: Blocker
> Labels: bugs
> Attachments: KAFKA-551.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> In makeFollower, we need to first truncate the local log to high watermark. It's possible that we need to truncate into segments before the last one. The problem is that all segments except the last one are immutable. So the truncation will fail which prevents the replica fetcher from being started.
> One solution is to reopen the segment as mutable during truncation, if it's not mutable already.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-551) Log.truncateTo() may need to trucate
immutable log segment
Posted by "Jay Kreps (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jay Kreps updated KAFKA-551:
----------------------------
Attachment: KAFKA-551.patch
Attached patch removes mutable flag from offset index and log file. I think this actually turns out to simplify the code a bit since there was lots of casewise logic.
The downside is that it is now possible to append to the wrong log segment which would have previously thrown a helpful exception and now would silently succeed.
It would be possible to improve this situation in the future by (1) completing the refactor of log to only reference index and file message set through the LogSegment class, and (2) adding a mutable flag in the log segment and checking that on all mutations.
> Log.truncateTo() may need to trucate immutable log segment
> ----------------------------------------------------------
>
> Key: KAFKA-551
> URL: https://issues.apache.org/jira/browse/KAFKA-551
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.8
> Reporter: Jun Rao
> Priority: Blocker
> Labels: bugs
> Attachments: KAFKA-551.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> In makeFollower, we need to first truncate the local log to high watermark. It's possible that we need to truncate into segments before the last one. The problem is that all segments except the last one are immutable. So the truncation will fail which prevents the replica fetcher from being started.
> One solution is to reopen the segment as mutable during truncation, if it's not mutable already.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-551) Log.truncateTo() may need to trucate
immutable log segment
Posted by "Jun Rao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13472133#comment-13472133 ]
Jun Rao commented on KAFKA-551:
-------------------------------
Thanks for the patch. Looks good. Just one comment.
1. OffsetIndex.truncateTo(): If there is an index entry that equals to the target offset, shouldn't we delete that index entry too? TruncateTo will make target offset the offset for the next append and there shouldn't be an index entry for offset no available yet.
> Log.truncateTo() may need to trucate immutable log segment
> ----------------------------------------------------------
>
> Key: KAFKA-551
> URL: https://issues.apache.org/jira/browse/KAFKA-551
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.8
> Reporter: Jun Rao
> Priority: Blocker
> Labels: bugs
> Attachments: KAFKA-551.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> In makeFollower, we need to first truncate the local log to high watermark. It's possible that we need to truncate into segments before the last one. The problem is that all segments except the last one are immutable. So the truncation will fail which prevents the replica fetcher from being started.
> One solution is to reopen the segment as mutable during truncation, if it's not mutable already.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-551) Log.truncateTo() may need to trucate
immutable log segment
Posted by "Jun Rao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13472470#comment-13472470 ]
Jun Rao commented on KAFKA-551:
-------------------------------
Yes, you are right. The code looks good. The confusion for me was that slot is an index based off 0 and when using it to set entry size, it actually truncates the entry at slot. +1 on the patch.
> Log.truncateTo() may need to trucate immutable log segment
> ----------------------------------------------------------
>
> Key: KAFKA-551
> URL: https://issues.apache.org/jira/browse/KAFKA-551
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.8
> Reporter: Jun Rao
> Priority: Blocker
> Labels: bugs
> Attachments: KAFKA-551.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> In makeFollower, we need to first truncate the local log to high watermark. It's possible that we need to truncate into segments before the last one. The problem is that all segments except the last one are immutable. So the truncation will fail which prevents the replica fetcher from being started.
> One solution is to reopen the segment as mutable during truncation, if it's not mutable already.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-551) Log.truncateTo() may need to trucate
immutable log segment
Posted by "Jun Rao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13470900#comment-13470900 ]
Jun Rao commented on KAFKA-551:
-------------------------------
Currently, we mark the first n-1 segments as read-only channels on broker startup. It doesn't seem to hurt to open them as r/w. Yes, it would great if you can fix it in your patch to take care of the index part too.
> Log.truncateTo() may need to trucate immutable log segment
> ----------------------------------------------------------
>
> Key: KAFKA-551
> URL: https://issues.apache.org/jira/browse/KAFKA-551
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.8
> Reporter: Jun Rao
> Priority: Blocker
> Labels: bugs
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> In makeFollower, we need to first truncate the local log to high watermark. It's possible that we need to truncate into segments before the last one. The problem is that all segments except the last one are immutable. So the truncation will fail which prevents the replica fetcher from being started.
> One solution is to reopen the segment as mutable during truncation, if it's not mutable already.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-551) Log.truncateTo() may need to trucate
immutable log segment
Posted by "Jay Kreps (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13472460#comment-13472460 ]
Jay Kreps commented on KAFKA-551:
---------------------------------
I think (hope) that is what we are doing. Here is the logic in OffsetIndex.truncateTo:
/* There are 3 cases for choosing the new size
* 1) if there is no entry in the index <= the offset, delete everything
* 2) if there is an entry for this exact offset, delete it and everything larger than it
* 3) if there is no entry for this offset, delete everything larger than the next smallest
*/
val newEntries =
if(slot < 0)
0
else if(logical(idx, slot) == offset)
slot
else
slot + 1
The value of newEntries is the number of remaining entries after the truncation. If we truncate to an offset lower than what is contained in the index (say because there is no entry) we truncate to 0 entries. If we are given an offset which is in the index, we delete that entry and all greater entries. Otherwise if we find an entry smaller than the given entry we should not delete that but instead delete all entries larger than it.
For example in OffsetIndexTest.truncate:
idx.truncateTo(9)
assertEquals("Index should truncate off last entry", OffsetPosition(8, 8), idx.lookup(10))
This is asserting that in an index with an entry for every offset, if we truncate to 9, then the largest entry that remains will be 8.
I think this is right, what was the problem you saw?
> Log.truncateTo() may need to trucate immutable log segment
> ----------------------------------------------------------
>
> Key: KAFKA-551
> URL: https://issues.apache.org/jira/browse/KAFKA-551
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.8
> Reporter: Jun Rao
> Priority: Blocker
> Labels: bugs
> Attachments: KAFKA-551.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> In makeFollower, we need to first truncate the local log to high watermark. It's possible that we need to truncate into segments before the last one. The problem is that all segments except the last one are immutable. So the truncation will fail which prevents the replica fetcher from being started.
> One solution is to reopen the segment as mutable during truncation, if it's not mutable already.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (KAFKA-551) Log.truncateTo() may need to trucate
immutable log segment
Posted by "Jay Kreps (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jay Kreps resolved KAFKA-551.
-----------------------------
Resolution: Fixed
Assignee: Jay Kreps
Committed.
> Log.truncateTo() may need to trucate immutable log segment
> ----------------------------------------------------------
>
> Key: KAFKA-551
> URL: https://issues.apache.org/jira/browse/KAFKA-551
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.8
> Reporter: Jun Rao
> Assignee: Jay Kreps
> Priority: Blocker
> Labels: bugs
> Attachments: KAFKA-551.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> In makeFollower, we need to first truncate the local log to high watermark. It's possible that we need to truncate into segments before the last one. The problem is that all segments except the last one are immutable. So the truncation will fail which prevents the replica fetcher from being started.
> One solution is to reopen the segment as mutable during truncation, if it's not mutable already.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira