You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "stack (JIRA)" <ji...@apache.org> on 2010/06/15 01:40:13 UTC
[jira] Created: (HBASE-2727) Splits writing one file only is
untenable; need dir of recovered edits ordered by sequenceid.
Splits writing one file only is untenable; need dir of recovered edits ordered by sequenceid.
---------------------------------------------------------------------------------------------
Key: HBASE-2727
URL: https://issues.apache.org/jira/browse/HBASE-2727
Project: HBase
Issue Type: Bug
Reporter: stack
Assignee: stack
Priority: Blocker
Fix For: 0.21.0
This issue comes of tlipcon doing a bit of human unit testing. His speculation is:
Let a region X deploy to server A. Server A opens the region, then closes it.
Let region X now deploy to server B. Server B now crashes.
Both server A and server B now have edits for region X in their WALs.
The processing of server crashes is currently sequential.
If server A crashes before server B, server A will write out a file of recovered edits for region X but region X was not deployed on server A so, the file will just sit there unused. The processing of server B crash will overwrite the recovered edits file written by the split of server A wal. This is ok.
But if somehow, server B processing is done before server A's, then interesting issues will likely arise; in the main, there is danger that the server B's recovered edits could be overwritten.
Another issue comes up in the review of hbase-1025. During the replay of edits on region deploy, if the hosting regionserver crashes before we have processed all of the recovered edits, we could lose some (the recovery of the regionserver that is replaying the edits could overwrite the log of edits only partially replayed).
Discussing up on IRC, whats needed is a directory of edits to replay ordered by sequenceid. On recovery, we play the oldest through to the newest removing the edits only on successfully replay.
Making blocker on 0.21 since this is a correctness issue.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HBASE-2727) Splits writing one file only is
untenable; need dir of recovered edits ordered by sequenceid.
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack resolved HBASE-2727.
--------------------------
Hadoop Flags: [Reviewed]
Resolution: Fixed
committed.
> Splits writing one file only is untenable; need dir of recovered edits ordered by sequenceid.
> ---------------------------------------------------------------------------------------------
>
> Key: HBASE-2727
> URL: https://issues.apache.org/jira/browse/HBASE-2727
> Project: HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: stack
> Priority: Blocker
> Fix For: 0.90.0
>
> Attachments: 2727-v2.txt, 2727-v4.txt, 2727-v6.txt, 2727-v7.txt
>
>
> This issue comes of tlipcon doing a bit of human unit testing. His speculation is:
> Let a region X deploy to server A. Server A opens the region, then closes it.
> Let region X now deploy to server B. Server B now crashes.
> Both server A and server B now have edits for region X in their WALs.
> The processing of server crashes is currently sequential.
> If server A crashes before server B, server A will write out a file of recovered edits for region X but region X was not deployed on server A so, the file will just sit there unused. The processing of server B crash will overwrite the recovered edits file written by the split of server A wal. This is ok.
> But if somehow, server B processing is done before server A's, then interesting issues will likely arise; in the main, there is danger that the server B's recovered edits could be overwritten.
> Another issue comes up in the review of hbase-1025. During the replay of edits on region deploy, if the hosting regionserver crashes before we have processed all of the recovered edits, we could lose some (the recovery of the regionserver that is replaying the edits could overwrite the log of edits only partially replayed).
> Discussing up on IRC, whats needed is a directory of edits to replay ordered by sequenceid. On recovery, we play the oldest through to the newest removing the edits only on successfully replay.
> Making blocker on 0.21 since this is a correctness issue.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-2727) Splits writing one file only is
untenable; need dir of recovered edits ordered by sequenceid.
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-2727:
-------------------------
Attachment: 2727-v2.txt
Here's a start. Needs testing and unit tests.
> Splits writing one file only is untenable; need dir of recovered edits ordered by sequenceid.
> ---------------------------------------------------------------------------------------------
>
> Key: HBASE-2727
> URL: https://issues.apache.org/jira/browse/HBASE-2727
> Project: HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: stack
> Priority: Blocker
> Fix For: 0.90.0
>
> Attachments: 2727-v2.txt
>
>
> This issue comes of tlipcon doing a bit of human unit testing. His speculation is:
> Let a region X deploy to server A. Server A opens the region, then closes it.
> Let region X now deploy to server B. Server B now crashes.
> Both server A and server B now have edits for region X in their WALs.
> The processing of server crashes is currently sequential.
> If server A crashes before server B, server A will write out a file of recovered edits for region X but region X was not deployed on server A so, the file will just sit there unused. The processing of server B crash will overwrite the recovered edits file written by the split of server A wal. This is ok.
> But if somehow, server B processing is done before server A's, then interesting issues will likely arise; in the main, there is danger that the server B's recovered edits could be overwritten.
> Another issue comes up in the review of hbase-1025. During the replay of edits on region deploy, if the hosting regionserver crashes before we have processed all of the recovered edits, we could lose some (the recovery of the regionserver that is replaying the edits could overwrite the log of edits only partially replayed).
> Discussing up on IRC, whats needed is a directory of edits to replay ordered by sequenceid. On recovery, we play the oldest through to the newest removing the edits only on successfully replay.
> Making blocker on 0.21 since this is a correctness issue.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-2727) Splits writing one file only is
untenable; need dir of recovered edits ordered by sequenceid.
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-2727:
-------------------------
Attachment: 2727-v4.txt
This passes all tests. Still need to add tests that crash a RS that has a R that is replaying edits from a split to ensure we do not lose edits when the R is opened subsequently.
After study, the fix for HBASE-1025, "Reconstruction log playback has no bounds on memory used" turns out to be not so smart. It played recovered edits using the 'normal' Region put/delete paths with edits applied to the WAL. Notion was that should we need to flush -- because of global memory pressure or because a region was in excess of configure memstore size -- then we'd just use the default flush mechanism. Well, this won't work should we crash during a replay. The default flush uses the regionserver/HLog sequenceid. When replaying recovered edits, the effected region is not yet online; its contribution to the regionserver/hlog sequenceid has not yet been made. Therefore, the current regionserver/hlog sequenceid could mess us up. If it is far in excess of the recovering regions sequenceid and we crash during recovery, then on next replay, we'll skip all edits.
This patch does replay of recovered edits all in the scope of the recovering region; flushes, if they have to happen, are done using sequenceids that make sense in the context of this region only. We don't use the regionserver/hlog sequenceid. Should we crash during recovery, we'll go through same recovery again w/ initial sequenceid gotten from storefiles and this regions recovered edits rather than from hlog/regionserver.
> Splits writing one file only is untenable; need dir of recovered edits ordered by sequenceid.
> ---------------------------------------------------------------------------------------------
>
> Key: HBASE-2727
> URL: https://issues.apache.org/jira/browse/HBASE-2727
> Project: HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: stack
> Priority: Blocker
> Fix For: 0.90.0
>
> Attachments: 2727-v2.txt, 2727-v4.txt
>
>
> This issue comes of tlipcon doing a bit of human unit testing. His speculation is:
> Let a region X deploy to server A. Server A opens the region, then closes it.
> Let region X now deploy to server B. Server B now crashes.
> Both server A and server B now have edits for region X in their WALs.
> The processing of server crashes is currently sequential.
> If server A crashes before server B, server A will write out a file of recovered edits for region X but region X was not deployed on server A so, the file will just sit there unused. The processing of server B crash will overwrite the recovered edits file written by the split of server A wal. This is ok.
> But if somehow, server B processing is done before server A's, then interesting issues will likely arise; in the main, there is danger that the server B's recovered edits could be overwritten.
> Another issue comes up in the review of hbase-1025. During the replay of edits on region deploy, if the hosting regionserver crashes before we have processed all of the recovered edits, we could lose some (the recovery of the regionserver that is replaying the edits could overwrite the log of edits only partially replayed).
> Discussing up on IRC, whats needed is a directory of edits to replay ordered by sequenceid. On recovery, we play the oldest through to the newest removing the edits only on successfully replay.
> Making blocker on 0.21 since this is a correctness issue.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2727) Splits writing one file only is
untenable; need dir of recovered edits ordered by sequenceid.
Posted by "HBase Review Board (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12889296#action_12889296 ]
HBase Review Board commented on HBASE-2727:
-------------------------------------------
Message from: stack@duboce.net
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://review.hbase.org/r/329/
-----------------------------------------------------------
Review request for hbase and Ryan Rawson.
Summary
-------
See notes made over in hbase-2727
This addresses bug hbase-2727.
http://issues.apache.org/jira/browse/hbase-2727
Diffs
-----
src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 40205c4
src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java 7044891
src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogMethods.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogSplit.java 3fff2fa
src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java 9053d39
Diff: http://review.hbase.org/r/329/diff
Testing
-------
All related tests seem to pass. A few are failing for me but seem unrelated. Digging in while this review goes on.
Thanks,
stack
> Splits writing one file only is untenable; need dir of recovered edits ordered by sequenceid.
> ---------------------------------------------------------------------------------------------
>
> Key: HBASE-2727
> URL: https://issues.apache.org/jira/browse/HBASE-2727
> Project: HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: stack
> Priority: Blocker
> Fix For: 0.90.0
>
> Attachments: 2727-v2.txt, 2727-v4.txt, 2727-v6.txt
>
>
> This issue comes of tlipcon doing a bit of human unit testing. His speculation is:
> Let a region X deploy to server A. Server A opens the region, then closes it.
> Let region X now deploy to server B. Server B now crashes.
> Both server A and server B now have edits for region X in their WALs.
> The processing of server crashes is currently sequential.
> If server A crashes before server B, server A will write out a file of recovered edits for region X but region X was not deployed on server A so, the file will just sit there unused. The processing of server B crash will overwrite the recovered edits file written by the split of server A wal. This is ok.
> But if somehow, server B processing is done before server A's, then interesting issues will likely arise; in the main, there is danger that the server B's recovered edits could be overwritten.
> Another issue comes up in the review of hbase-1025. During the replay of edits on region deploy, if the hosting regionserver crashes before we have processed all of the recovered edits, we could lose some (the recovery of the regionserver that is replaying the edits could overwrite the log of edits only partially replayed).
> Discussing up on IRC, whats needed is a directory of edits to replay ordered by sequenceid. On recovery, we play the oldest through to the newest removing the edits only on successfully replay.
> Making blocker on 0.21 since this is a correctness issue.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-2727) Splits writing one file only is
untenable; need dir of recovered edits ordered by sequenceid.
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-2727:
-------------------------
Attachment: 2727-v7.txt
Here is what I'm going to commit. Includes consideration of Ryans' comments (thanks for great review Ryan).
> Splits writing one file only is untenable; need dir of recovered edits ordered by sequenceid.
> ---------------------------------------------------------------------------------------------
>
> Key: HBASE-2727
> URL: https://issues.apache.org/jira/browse/HBASE-2727
> Project: HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: stack
> Priority: Blocker
> Fix For: 0.90.0
>
> Attachments: 2727-v2.txt, 2727-v4.txt, 2727-v6.txt, 2727-v7.txt
>
>
> This issue comes of tlipcon doing a bit of human unit testing. His speculation is:
> Let a region X deploy to server A. Server A opens the region, then closes it.
> Let region X now deploy to server B. Server B now crashes.
> Both server A and server B now have edits for region X in their WALs.
> The processing of server crashes is currently sequential.
> If server A crashes before server B, server A will write out a file of recovered edits for region X but region X was not deployed on server A so, the file will just sit there unused. The processing of server B crash will overwrite the recovered edits file written by the split of server A wal. This is ok.
> But if somehow, server B processing is done before server A's, then interesting issues will likely arise; in the main, there is danger that the server B's recovered edits could be overwritten.
> Another issue comes up in the review of hbase-1025. During the replay of edits on region deploy, if the hosting regionserver crashes before we have processed all of the recovered edits, we could lose some (the recovery of the regionserver that is replaying the edits could overwrite the log of edits only partially replayed).
> Discussing up on IRC, whats needed is a directory of edits to replay ordered by sequenceid. On recovery, we play the oldest through to the newest removing the edits only on successfully replay.
> Making blocker on 0.21 since this is a correctness issue.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2727) Splits writing one file only is
untenable; need dir of recovered edits ordered by sequenceid.
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12886922#action_12886922 ]
Todd Lipcon commented on HBASE-2727:
------------------------------------
This is also really important for recovery performance.
I was doing load testing on a 5 node cluster overnight, and failure of one region server took nearly 30 minutes to recover, since replay is so much slowed by the fact that the recovering server is *rewriting* and syncing edits to a new log.
> Splits writing one file only is untenable; need dir of recovered edits ordered by sequenceid.
> ---------------------------------------------------------------------------------------------
>
> Key: HBASE-2727
> URL: https://issues.apache.org/jira/browse/HBASE-2727
> Project: HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: stack
> Priority: Blocker
> Fix For: 0.21.0
>
>
> This issue comes of tlipcon doing a bit of human unit testing. His speculation is:
> Let a region X deploy to server A. Server A opens the region, then closes it.
> Let region X now deploy to server B. Server B now crashes.
> Both server A and server B now have edits for region X in their WALs.
> The processing of server crashes is currently sequential.
> If server A crashes before server B, server A will write out a file of recovered edits for region X but region X was not deployed on server A so, the file will just sit there unused. The processing of server B crash will overwrite the recovered edits file written by the split of server A wal. This is ok.
> But if somehow, server B processing is done before server A's, then interesting issues will likely arise; in the main, there is danger that the server B's recovered edits could be overwritten.
> Another issue comes up in the review of hbase-1025. During the replay of edits on region deploy, if the hosting regionserver crashes before we have processed all of the recovered edits, we could lose some (the recovery of the regionserver that is replaying the edits could overwrite the log of edits only partially replayed).
> Discussing up on IRC, whats needed is a directory of edits to replay ordered by sequenceid. On recovery, we play the oldest through to the newest removing the edits only on successfully replay.
> Making blocker on 0.21 since this is a correctness issue.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2727) Splits writing one file only is
untenable; need dir of recovered edits ordered by sequenceid.
Posted by "HBase Review Board (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12889358#action_12889358 ]
HBase Review Board commented on HBASE-2727:
-------------------------------------------
Message from: "Ryan Rawson" <ry...@gmail.com>
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://review.hbase.org/r/329/#review419
-----------------------------------------------------------
Ship it!
Brief comments:
- radix sort in the list of files
- <= on the sequence id to discard
- some trailing whitespace
+1 lgtm
- Ryan
> Splits writing one file only is untenable; need dir of recovered edits ordered by sequenceid.
> ---------------------------------------------------------------------------------------------
>
> Key: HBASE-2727
> URL: https://issues.apache.org/jira/browse/HBASE-2727
> Project: HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: stack
> Priority: Blocker
> Fix For: 0.90.0
>
> Attachments: 2727-v2.txt, 2727-v4.txt, 2727-v6.txt
>
>
> This issue comes of tlipcon doing a bit of human unit testing. His speculation is:
> Let a region X deploy to server A. Server A opens the region, then closes it.
> Let region X now deploy to server B. Server B now crashes.
> Both server A and server B now have edits for region X in their WALs.
> The processing of server crashes is currently sequential.
> If server A crashes before server B, server A will write out a file of recovered edits for region X but region X was not deployed on server A so, the file will just sit there unused. The processing of server B crash will overwrite the recovered edits file written by the split of server A wal. This is ok.
> But if somehow, server B processing is done before server A's, then interesting issues will likely arise; in the main, there is danger that the server B's recovered edits could be overwritten.
> Another issue comes up in the review of hbase-1025. During the replay of edits on region deploy, if the hosting regionserver crashes before we have processed all of the recovered edits, we could lose some (the recovery of the regionserver that is replaying the edits could overwrite the log of edits only partially replayed).
> Discussing up on IRC, whats needed is a directory of edits to replay ordered by sequenceid. On recovery, we play the oldest through to the newest removing the edits only on successfully replay.
> Making blocker on 0.21 since this is a correctness issue.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-2727) Splits writing one file only is
untenable; need dir of recovered edits ordered by sequenceid.
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-2727:
-------------------------
Attachment: 2727-v6.txt
Added tests to demonstrate new facility whereby we can have more than one recovered edits file and that we'll replay the edits in the right order in the face of multiple edit files.
Regards the scenarios from the description above:
For 1, its highly unlikely but if for some reason we somehow process server shutdown B before we process server shutdown A and somehow, the assignment of the region in question does not happen until AFTER both server shutdowns have been processed, now its the case that wal splits will not overwrite since they'll be differently named -- named for the first sequenceid in the file -- and secondly, on replay of the edits on region deploy, we'll replay the edits from oldest to newest.
For 2, if we crash during replay of split edits, they'll be in place next time the region is deployed; we do not remove split replay edits until AFTER we've played them all and a flush has completed.
Here is commit message that gives overview on changes:
The replay of recovered edits has been changed again. We no longer replay by calling Region#get and Region#delete and no longer add the replays to the RS WAL. Instead we just add them to the memstore as we used to keeping account of the region memstore size. If the memstore grows too large, we'll flush -- but NOT by using the general flush mechanism. Instead we'll flush inline using sequenceids that make sense in the current replay context -- the sequenceids are from the regions storefiles and from its split recovered edits, NOT those of the hosting regionserver/hlog. We need to do this to avoid case where the hosting regionserver/hlog sequenceid is somehow in excess of ours. We don't want to add a storefile that has an inflated sequenceid in case we crash between the flush and the completion of the replay of edits (We'll miss edits on the second replay if we have a storefile w/ an excessive sequenceid).
Did other cleanup in HRegion. We don't need to monitor minimal sequenceids in families. That was silly.
Added ability to replay one or more recovered edits files.
In HLog, writes splts into a subdir of the region named recovered.edits rather than to a file named recovered.edits.
Added tests too.
> Splits writing one file only is untenable; need dir of recovered edits ordered by sequenceid.
> ---------------------------------------------------------------------------------------------
>
> Key: HBASE-2727
> URL: https://issues.apache.org/jira/browse/HBASE-2727
> Project: HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: stack
> Priority: Blocker
> Fix For: 0.90.0
>
> Attachments: 2727-v2.txt, 2727-v4.txt, 2727-v6.txt
>
>
> This issue comes of tlipcon doing a bit of human unit testing. His speculation is:
> Let a region X deploy to server A. Server A opens the region, then closes it.
> Let region X now deploy to server B. Server B now crashes.
> Both server A and server B now have edits for region X in their WALs.
> The processing of server crashes is currently sequential.
> If server A crashes before server B, server A will write out a file of recovered edits for region X but region X was not deployed on server A so, the file will just sit there unused. The processing of server B crash will overwrite the recovered edits file written by the split of server A wal. This is ok.
> But if somehow, server B processing is done before server A's, then interesting issues will likely arise; in the main, there is danger that the server B's recovered edits could be overwritten.
> Another issue comes up in the review of hbase-1025. During the replay of edits on region deploy, if the hosting regionserver crashes before we have processed all of the recovered edits, we could lose some (the recovery of the regionserver that is replaying the edits could overwrite the log of edits only partially replayed).
> Discussing up on IRC, whats needed is a directory of edits to replay ordered by sequenceid. On recovery, we play the oldest through to the newest removing the edits only on successfully replay.
> Making blocker on 0.21 since this is a correctness issue.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.