You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Prakash Khemani (JIRA)" <ji...@apache.org> on 2011/06/19 04:36:47 UTC
[jira] [Created] (HBASE-4007) distributed log splitting can get
indefinitely stuck
distributed log splitting can get indefinitely stuck
----------------------------------------------------
Key: HBASE-4007
URL: https://issues.apache.org/jira/browse/HBASE-4007
Project: HBase
Issue Type: Bug
Reporter: Prakash Khemani
Assignee: Prakash Khemani
After the configured number of retries SplitLogManager is not going to resubmit log-split tasks. In this situation even if the splitLogWorker that owns the task dies the task will not get resubmitted.
When a regionserver goes away then all the split-log tasks that it owned should be resubmitted by the SplitLogMaster.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4007) distributed log splitting can get
indefinitely stuck
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13052091#comment-13052091 ]
stack commented on HBASE-4007:
------------------------------
@Mingjian I have run into what you are describing (haven't dug in yet though).
> distributed log splitting can get indefinitely stuck
> ----------------------------------------------------
>
> Key: HBASE-4007
> URL: https://issues.apache.org/jira/browse/HBASE-4007
> Project: HBase
> Issue Type: Bug
> Reporter: Prakash Khemani
> Assignee: Prakash Khemani
>
> After the configured number of retries SplitLogManager is not going to resubmit log-split tasks. In this situation even if the splitLogWorker that owns the task dies the task will not get resubmitted.
> When a regionserver goes away then all the split-log tasks that it owned should be resubmitted by the SplitLogMaster.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4007) distributed log splitting can get
indefinitely stuck
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13100944#comment-13100944 ]
jiraposter@reviews.apache.org commented on HBASE-4007:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1747/#review1829
-----------------------------------------------------------
Ship it!
- Michael
On 2011-09-08 22:09:32, Prakash Khemani wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/1747/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2011-09-08 22:09:32)
bq.
bq.
bq. Review request for hbase, Ted Yu and Michael Stack.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. 1/ resubmit all tasks owned by a dead splitlog-worker
bq. 2/ prevent accumulation of /hbase/splitlog/RESCAN nodes
bq.
bq.
bq. This addresses bug HBASE-4007.
bq. https://issues.apache.org/jira/browse/HBASE-4007
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java 9a71fdf
bq. src/main/java/org/apache/hadoop/hbase/master/SplitLogManager.java 54b6d45
bq. src/main/java/org/apache/hadoop/hbase/zookeeper/ZKSplitLog.java 61e5c65
bq. src/test/java/org/apache/hadoop/hbase/master/TestSplitLogManager.java 9a88855
bq.
bq. Diff: https://reviews.apache.org/r/1747/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. 1/ resubmit all tasks owned by a dead splitlog-worker - only unit tested. will do cluster testing.
bq. 2/ prevent accumulation of /hbase/splitlog/RESCAN nodes - tested and deployed in production.
bq.
bq.
bq. Thanks,
bq.
bq. Prakash
bq.
bq.
> distributed log splitting can get indefinitely stuck
> ----------------------------------------------------
>
> Key: HBASE-4007
> URL: https://issues.apache.org/jira/browse/HBASE-4007
> Project: HBase
> Issue Type: Bug
> Reporter: Prakash Khemani
> Assignee: Prakash Khemani
> Priority: Critical
> Fix For: 0.92.0
>
> Attachments: 0001-HBASE-4007-distributed-log-splitting-can-get-indefin.patch
>
>
> After the configured number of retries SplitLogManager is not going to resubmit log-split tasks. In this situation even if the splitLogWorker that owns the task dies the task will not get resubmitted.
> When a regionserver goes away then all the split-log tasks that it owned should be resubmitted by the SplitLogMaster.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4007) distributed log splitting can get
indefinitely stuck
Posted by "Prakash Khemani (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Prakash Khemani updated HBASE-4007:
-----------------------------------
Status: Patch Available (was: Open)
fixes
(1) buildup of RESCAN zookeeper nodes in the event that all the regionservers are down
(2) a bug in tracking when the last RESCAN node was created which could lead to too frequent RESCAN node creation
(3) if master/splitlogmanager fails to complete the task handed over by a region-server/worker then keep retrying indefinitely
(4) if a regionserver/worker dies then ensure that all its tasks are resubmitted
> distributed log splitting can get indefinitely stuck
> ----------------------------------------------------
>
> Key: HBASE-4007
> URL: https://issues.apache.org/jira/browse/HBASE-4007
> Project: HBase
> Issue Type: Bug
> Reporter: Prakash Khemani
> Assignee: Prakash Khemani
>
> After the configured number of retries SplitLogManager is not going to resubmit log-split tasks. In this situation even if the splitLogWorker that owns the task dies the task will not get resubmitted.
> When a regionserver goes away then all the split-log tasks that it owned should be resubmitted by the SplitLogMaster.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4007) distributed log splitting can get
indefinitely stuck
Posted by "mingjian (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13051819#comment-13051819 ]
mingjian commented on HBASE-4007:
---------------------------------
@Prakash Khemani: Do you mean it will block hmaster start and write master logs like "DEBUG org.apache.hadoop.hbase.master.SplitLogManager: task not yet acquired /hbase/splitlog/RESCAN0000179042
ver = 0"?
It seems create RESCAN indefinitely in znodes and all of the regionservers done for SplitWorker.
> distributed log splitting can get indefinitely stuck
> ----------------------------------------------------
>
> Key: HBASE-4007
> URL: https://issues.apache.org/jira/browse/HBASE-4007
> Project: HBase
> Issue Type: Bug
> Reporter: Prakash Khemani
> Assignee: Prakash Khemani
>
> After the configured number of retries SplitLogManager is not going to resubmit log-split tasks. In this situation even if the splitLogWorker that owns the task dies the task will not get resubmitted.
> When a regionserver goes away then all the split-log tasks that it owned should be resubmitted by the SplitLogMaster.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4007) distributed log splitting can get
indefinitely stuck
Posted by "Prakash Khemani (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13091168#comment-13091168 ]
Prakash Khemani commented on HBASE-4007:
----------------------------------------
I am not running the patch yet. It is up internally for review.
> distributed log splitting can get indefinitely stuck
> ----------------------------------------------------
>
> Key: HBASE-4007
> URL: https://issues.apache.org/jira/browse/HBASE-4007
> Project: HBase
> Issue Type: Bug
> Reporter: Prakash Khemani
> Assignee: Prakash Khemani
> Priority: Critical
> Attachments: 0001-HBASE-4007-distributed-log-splitting-can-get-indefin.patch
>
>
> After the configured number of retries SplitLogManager is not going to resubmit log-split tasks. In this situation even if the splitLogWorker that owns the task dies the task will not get resubmitted.
> When a regionserver goes away then all the split-log tasks that it owned should be resubmitted by the SplitLogMaster.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4007) distributed log splitting can get
indefinitely stuck
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13091612#comment-13091612 ]
stack commented on HBASE-4007:
------------------------------
@Prakash Yeah, can be non-intuitive uploading patches to RB. Let me try one later or if you want to retry, dump in here steps and what you chose and I'll try what you did.
ConcurrentSet sounds fine. Gives you a 'view' at a particular time. Looks like it should be fine for deadworkers. If you go the ConcurrentSet route, it looks like it will make the patch a little simpler. You can get rid of the local copy you sometimes make as in here:
{code}
+ if (localDeadWorkers != null && localDeadWorkers.contains(cur_worker)) {
{code}
.. and just go against actual set (can remove the null check too?)
Otherwise, patch looks fine to me.
> distributed log splitting can get indefinitely stuck
> ----------------------------------------------------
>
> Key: HBASE-4007
> URL: https://issues.apache.org/jira/browse/HBASE-4007
> Project: HBase
> Issue Type: Bug
> Reporter: Prakash Khemani
> Assignee: Prakash Khemani
> Priority: Critical
> Attachments: 0001-HBASE-4007-distributed-log-splitting-can-get-indefin.patch
>
>
> After the configured number of retries SplitLogManager is not going to resubmit log-split tasks. In this situation even if the splitLogWorker that owns the task dies the task will not get resubmitted.
> When a regionserver goes away then all the split-log tasks that it owned should be resubmitted by the SplitLogMaster.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4007) distributed log splitting can get
indefinitely stuck
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13100092#comment-13100092 ]
jiraposter@reviews.apache.org commented on HBASE-4007:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1747/
-----------------------------------------------------------
Review request for hbase, Ted Yu and Michael Stack.
Summary
-------
1/ resubmit all tasks owned by a dead splitlog-worker
2/ prevent accumulation of /hbase/splitlog/RESCAN nodes
This addresses bug HBASE-4007.
https://issues.apache.org/jira/browse/HBASE-4007
Diffs
-----
src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java 9a71fdf
src/main/java/org/apache/hadoop/hbase/master/SplitLogManager.java 54b6d45
src/main/java/org/apache/hadoop/hbase/zookeeper/ZKSplitLog.java 61e5c65
src/test/java/org/apache/hadoop/hbase/master/TestSplitLogManager.java 9a88855
Diff: https://reviews.apache.org/r/1747/diff
Testing
-------
1/ resubmit all tasks owned by a dead splitlog-worker - only unit tested. will do cluster testing.
2/ prevent accumulation of /hbase/splitlog/RESCAN nodes - tested and deployed in production.
Thanks,
Prakash
> distributed log splitting can get indefinitely stuck
> ----------------------------------------------------
>
> Key: HBASE-4007
> URL: https://issues.apache.org/jira/browse/HBASE-4007
> Project: HBase
> Issue Type: Bug
> Reporter: Prakash Khemani
> Assignee: Prakash Khemani
> Priority: Critical
> Fix For: 0.92.0
>
> Attachments: 0001-HBASE-4007-distributed-log-splitting-can-get-indefin.patch
>
>
> After the configured number of retries SplitLogManager is not going to resubmit log-split tasks. In this situation even if the splitLogWorker that owns the task dies the task will not get resubmitted.
> When a regionserver goes away then all the split-log tasks that it owned should be resubmitted by the SplitLogMaster.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4007) distributed log splitting can get
indefinitely stuck
Posted by "Prakash Khemani (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13100085#comment-13100085 ]
Prakash Khemani commented on HBASE-4007:
----------------------------------------
Hi Stack, I have not pushed this out to production yet ... and the way things are it will be a while before we do the next push to the hbase-90 tiers.
I will try to get some cluster testing done and will update this thread.
Regarding the use of ConcurrentHashMap as opposed to HashSet + ObjectLock : I could not find any nice way to take a snapshot of a concurrent-hash-map. The way the code is written I need to take a snapshot of the deadWorkers set.
I have just rebased. I will try to put it up in the reviewboard one more time.
Thanks,
Prakash
> distributed log splitting can get indefinitely stuck
> ----------------------------------------------------
>
> Key: HBASE-4007
> URL: https://issues.apache.org/jira/browse/HBASE-4007
> Project: HBase
> Issue Type: Bug
> Reporter: Prakash Khemani
> Assignee: Prakash Khemani
> Priority: Critical
> Fix For: 0.92.0
>
> Attachments: 0001-HBASE-4007-distributed-log-splitting-can-get-indefin.patch
>
>
> After the configured number of retries SplitLogManager is not going to resubmit log-split tasks. In this situation even if the splitLogWorker that owns the task dies the task will not get resubmitted.
> When a regionserver goes away then all the split-log tasks that it owned should be resubmitted by the SplitLogMaster.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4007) distributed log splitting can get
indefinitely stuck
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-4007:
-------------------------
Resolution: Fixed
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
Committed to TRUNK. Thanks for the patch Prakash.
> distributed log splitting can get indefinitely stuck
> ----------------------------------------------------
>
> Key: HBASE-4007
> URL: https://issues.apache.org/jira/browse/HBASE-4007
> Project: HBase
> Issue Type: Bug
> Reporter: Prakash Khemani
> Assignee: Prakash Khemani
> Priority: Critical
> Fix For: 0.92.0
>
> Attachments: 0001-HBASE-4007-distributed-log-splitting-can-get-indefin.patch
>
>
> After the configured number of retries SplitLogManager is not going to resubmit log-split tasks. In this situation even if the splitLogWorker that owns the task dies the task will not get resubmitted.
> When a regionserver goes away then all the split-log tasks that it owned should be resubmitted by the SplitLogMaster.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4007) distributed log splitting can get
indefinitely stuck
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13100107#comment-13100107 ]
jiraposter@reviews.apache.org commented on HBASE-4007:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1747/#review1813
-----------------------------------------------------------
src/main/java/org/apache/hadoop/hbase/master/SplitLogManager.java
<https://reviews.apache.org/r/1747/#comment4164>
Typo for TimeoutMonitor
src/main/java/org/apache/hadoop/hbase/master/SplitLogManager.java
<https://reviews.apache.org/r/1747/#comment4165>
Since javadoc is added to explain what true means, would it make more sense to create an enum to replace this parameter ?
- Ted
On 2011-09-08 05:54:47, Prakash Khemani wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/1747/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2011-09-08 05:54:47)
bq.
bq.
bq. Review request for hbase, Ted Yu and Michael Stack.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. 1/ resubmit all tasks owned by a dead splitlog-worker
bq. 2/ prevent accumulation of /hbase/splitlog/RESCAN nodes
bq.
bq.
bq. This addresses bug HBASE-4007.
bq. https://issues.apache.org/jira/browse/HBASE-4007
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java 9a71fdf
bq. src/main/java/org/apache/hadoop/hbase/master/SplitLogManager.java 54b6d45
bq. src/main/java/org/apache/hadoop/hbase/zookeeper/ZKSplitLog.java 61e5c65
bq. src/test/java/org/apache/hadoop/hbase/master/TestSplitLogManager.java 9a88855
bq.
bq. Diff: https://reviews.apache.org/r/1747/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. 1/ resubmit all tasks owned by a dead splitlog-worker - only unit tested. will do cluster testing.
bq. 2/ prevent accumulation of /hbase/splitlog/RESCAN nodes - tested and deployed in production.
bq.
bq.
bq. Thanks,
bq.
bq. Prakash
bq.
bq.
> distributed log splitting can get indefinitely stuck
> ----------------------------------------------------
>
> Key: HBASE-4007
> URL: https://issues.apache.org/jira/browse/HBASE-4007
> Project: HBase
> Issue Type: Bug
> Reporter: Prakash Khemani
> Assignee: Prakash Khemani
> Priority: Critical
> Fix For: 0.92.0
>
> Attachments: 0001-HBASE-4007-distributed-log-splitting-can-get-indefin.patch
>
>
> After the configured number of retries SplitLogManager is not going to resubmit log-split tasks. In this situation even if the splitLogWorker that owns the task dies the task will not get resubmitted.
> When a regionserver goes away then all the split-log tasks that it owned should be resubmitted by the SplitLogMaster.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4007) distributed log splitting can get
indefinitely stuck
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13100741#comment-13100741 ]
jiraposter@reviews.apache.org commented on HBASE-4007:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1747/
-----------------------------------------------------------
(Updated 2011-09-08 22:09:32.715934)
Review request for hbase, Ted Yu and Michael Stack.
Changes
-------
implemented Ted's feedback.
Summary
-------
1/ resubmit all tasks owned by a dead splitlog-worker
2/ prevent accumulation of /hbase/splitlog/RESCAN nodes
This addresses bug HBASE-4007.
https://issues.apache.org/jira/browse/HBASE-4007
Diffs (updated)
-----
src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java 9a71fdf
src/main/java/org/apache/hadoop/hbase/master/SplitLogManager.java 54b6d45
src/main/java/org/apache/hadoop/hbase/zookeeper/ZKSplitLog.java 61e5c65
src/test/java/org/apache/hadoop/hbase/master/TestSplitLogManager.java 9a88855
Diff: https://reviews.apache.org/r/1747/diff
Testing
-------
1/ resubmit all tasks owned by a dead splitlog-worker - only unit tested. will do cluster testing.
2/ prevent accumulation of /hbase/splitlog/RESCAN nodes - tested and deployed in production.
Thanks,
Prakash
> distributed log splitting can get indefinitely stuck
> ----------------------------------------------------
>
> Key: HBASE-4007
> URL: https://issues.apache.org/jira/browse/HBASE-4007
> Project: HBase
> Issue Type: Bug
> Reporter: Prakash Khemani
> Assignee: Prakash Khemani
> Priority: Critical
> Fix For: 0.92.0
>
> Attachments: 0001-HBASE-4007-distributed-log-splitting-can-get-indefin.patch
>
>
> After the configured number of retries SplitLogManager is not going to resubmit log-split tasks. In this situation even if the splitLogWorker that owns the task dies the task will not get resubmitted.
> When a regionserver goes away then all the split-log tasks that it owned should be resubmitted by the SplitLogMaster.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4007) distributed log splitting can get
indefinitely stuck
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13100681#comment-13100681 ]
jiraposter@reviews.apache.org commented on HBASE-4007:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1747/#review1819
-----------------------------------------------------------
Ship it!
src/main/java/org/apache/hadoop/hbase/master/SplitLogManager.java
<https://reviews.apache.org/r/1747/#comment4181>
I know this code has been there.
But it follows a similar pattern where enum is more appropriate.
- Ted
On 2011-09-08 19:34:22, Prakash Khemani wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/1747/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2011-09-08 19:34:22)
bq.
bq.
bq. Review request for hbase, Ted Yu and Michael Stack.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. 1/ resubmit all tasks owned by a dead splitlog-worker
bq. 2/ prevent accumulation of /hbase/splitlog/RESCAN nodes
bq.
bq.
bq. This addresses bug HBASE-4007.
bq. https://issues.apache.org/jira/browse/HBASE-4007
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. src/main/java/org/apache/hadoop/hbase/master/SplitLogManager.java 54b6d45
bq. src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java 9a71fdf
bq. src/main/java/org/apache/hadoop/hbase/zookeeper/ZKSplitLog.java 61e5c65
bq. src/test/java/org/apache/hadoop/hbase/master/TestSplitLogManager.java 9a88855
bq.
bq. Diff: https://reviews.apache.org/r/1747/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. 1/ resubmit all tasks owned by a dead splitlog-worker - only unit tested. will do cluster testing.
bq. 2/ prevent accumulation of /hbase/splitlog/RESCAN nodes - tested and deployed in production.
bq.
bq.
bq. Thanks,
bq.
bq. Prakash
bq.
bq.
> distributed log splitting can get indefinitely stuck
> ----------------------------------------------------
>
> Key: HBASE-4007
> URL: https://issues.apache.org/jira/browse/HBASE-4007
> Project: HBase
> Issue Type: Bug
> Reporter: Prakash Khemani
> Assignee: Prakash Khemani
> Priority: Critical
> Fix For: 0.92.0
>
> Attachments: 0001-HBASE-4007-distributed-log-splitting-can-get-indefin.patch
>
>
> After the configured number of retries SplitLogManager is not going to resubmit log-split tasks. In this situation even if the splitLogWorker that owns the task dies the task will not get resubmitted.
> When a regionserver goes away then all the split-log tasks that it owned should be resubmitted by the SplitLogMaster.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4007) distributed log splitting can get
indefinitely stuck
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13099822#comment-13099822 ]
stack commented on HBASE-4007:
------------------------------
@Prakash Did you deploy this patch? Is it working for you? Good stuff.
> distributed log splitting can get indefinitely stuck
> ----------------------------------------------------
>
> Key: HBASE-4007
> URL: https://issues.apache.org/jira/browse/HBASE-4007
> Project: HBase
> Issue Type: Bug
> Reporter: Prakash Khemani
> Assignee: Prakash Khemani
> Priority: Critical
> Fix For: 0.92.0
>
> Attachments: 0001-HBASE-4007-distributed-log-splitting-can-get-indefin.patch
>
>
> After the configured number of retries SplitLogManager is not going to resubmit log-split tasks. In this situation even if the splitLogWorker that owns the task dies the task will not get resubmitted.
> When a regionserver goes away then all the split-log tasks that it owned should be resubmitted by the SplitLogMaster.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4007) distributed log splitting can get
indefinitely stuck
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-4007:
-------------------------
Status: Patch Available (was: Open)
Marking patch available.
> distributed log splitting can get indefinitely stuck
> ----------------------------------------------------
>
> Key: HBASE-4007
> URL: https://issues.apache.org/jira/browse/HBASE-4007
> Project: HBase
> Issue Type: Bug
> Reporter: Prakash Khemani
> Assignee: Prakash Khemani
> Attachments: 0001-HBASE-4007-distributed-log-splitting-can-get-indefin.patch
>
>
> After the configured number of retries SplitLogManager is not going to resubmit log-split tasks. In this situation even if the splitLogWorker that owns the task dies the task will not get resubmitted.
> When a regionserver goes away then all the split-log tasks that it owned should be resubmitted by the SplitLogMaster.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4007) distributed log splitting can get
indefinitely stuck
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13091295#comment-13091295 ]
stack commented on HBASE-4007:
------------------------------
OK. Here's a few comments if they'll help.
Why do this:
{code}
+ private Set<String> deadWorkers = null;
+ private Object deadWorkersLock = new Object();
{code}
Does deadWorkers have to be null? Can it not just be empty? (I see in the patch where you are setting it to null when work has been done -- could processed workers just be empty deadWorkers Set instead?) Then you could allocate deadWorkers when you declare it, make it final while you are at it, and then lock on deadWorkers rather than have this deadWorkersLock (This becomes unnecessary?).
Minor: call registerHeartbeat 'heartbeat' instead? We're handling the heartbeat updating data members.. we're not keeping track -- registering -- the event.
... more to follow..
> distributed log splitting can get indefinitely stuck
> ----------------------------------------------------
>
> Key: HBASE-4007
> URL: https://issues.apache.org/jira/browse/HBASE-4007
> Project: HBase
> Issue Type: Bug
> Reporter: Prakash Khemani
> Assignee: Prakash Khemani
> Priority: Critical
> Attachments: 0001-HBASE-4007-distributed-log-splitting-can-get-indefin.patch
>
>
> After the configured number of retries SplitLogManager is not going to resubmit log-split tasks. In this situation even if the splitLogWorker that owns the task dies the task will not get resubmitted.
> When a regionserver goes away then all the split-log tasks that it owned should be resubmitted by the SplitLogMaster.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4007) distributed log splitting can get
indefinitely stuck
Posted by "Prakash Khemani (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Prakash Khemani updated HBASE-4007:
-----------------------------------
Status: Open (was: Patch Available)
> distributed log splitting can get indefinitely stuck
> ----------------------------------------------------
>
> Key: HBASE-4007
> URL: https://issues.apache.org/jira/browse/HBASE-4007
> Project: HBase
> Issue Type: Bug
> Reporter: Prakash Khemani
> Assignee: Prakash Khemani
>
> After the configured number of retries SplitLogManager is not going to resubmit log-split tasks. In this situation even if the splitLogWorker that owns the task dies the task will not get resubmitted.
> When a regionserver goes away then all the split-log tasks that it owned should be resubmitted by the SplitLogMaster.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4007) distributed log splitting can get
indefinitely stuck
Posted by "Prakash Khemani (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13091345#comment-13091345 ]
Prakash Khemani commented on HBASE-4007:
----------------------------------------
(I had tried using reviewboard yesterday but it was fataling on me - Internal Server Error 500. Thanks for reviewing the patch the hard way)
==
For deadWorkers - one of the goals was to not block handleDeadWorkers() and worry about deadlocks etc. I could have used ConcurrentSet but I was not sure of the semantics - how an iterator behaves when another item is added.
I will change this to use ConcurrentSet ... please let me know.
==
I will change registerHeartbeat() to hearbeat().
> distributed log splitting can get indefinitely stuck
> ----------------------------------------------------
>
> Key: HBASE-4007
> URL: https://issues.apache.org/jira/browse/HBASE-4007
> Project: HBase
> Issue Type: Bug
> Reporter: Prakash Khemani
> Assignee: Prakash Khemani
> Priority: Critical
> Attachments: 0001-HBASE-4007-distributed-log-splitting-can-get-indefin.patch
>
>
> After the configured number of retries SplitLogManager is not going to resubmit log-split tasks. In this situation even if the splitLogWorker that owns the task dies the task will not get resubmitted.
> When a regionserver goes away then all the split-log tasks that it owned should be resubmitted by the SplitLogMaster.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4007) distributed log splitting can get
indefinitely stuck
Posted by "Prakash Khemani (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13052067#comment-13052067 ]
Prakash Khemani commented on HBASE-4007:
----------------------------------------
@mingjian What you are talking about is probably a different issue. The scenario you have described can happen when 1. Master puts up a task. 2. No one acquires the task. 3. Master puts up a RESCAN node asking everyone to re-look at the zk splitlog task list.
The bug described in this jira will happen in the following way (I have not encountered it yet but should be easy to reproduce)
a/ A splitlog task is slow. Master has already moved the task from one worker to another 3 times. It is with the 4th worker now. Even if the 4th worker takes too long doing this task the master is not going to do anything about it.
b/ the 4th worker dies.
c/ the task will hang.
Master has to resubmit the task when the 4th worker dies.
> distributed log splitting can get indefinitely stuck
> ----------------------------------------------------
>
> Key: HBASE-4007
> URL: https://issues.apache.org/jira/browse/HBASE-4007
> Project: HBase
> Issue Type: Bug
> Reporter: Prakash Khemani
> Assignee: Prakash Khemani
>
> After the configured number of retries SplitLogManager is not going to resubmit log-split tasks. In this situation even if the splitLogWorker that owns the task dies the task will not get resubmitted.
> When a regionserver goes away then all the split-log tasks that it owned should be resubmitted by the SplitLogMaster.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4007) distributed log splitting can get
indefinitely stuck
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-4007:
-------------------------
Priority: Critical (was: Major)
Marking critical (Hung split logging is pretty critical). Adding to 0.92.0.
> distributed log splitting can get indefinitely stuck
> ----------------------------------------------------
>
> Key: HBASE-4007
> URL: https://issues.apache.org/jira/browse/HBASE-4007
> Project: HBase
> Issue Type: Bug
> Reporter: Prakash Khemani
> Assignee: Prakash Khemani
> Priority: Critical
> Attachments: 0001-HBASE-4007-distributed-log-splitting-can-get-indefin.patch
>
>
> After the configured number of retries SplitLogManager is not going to resubmit log-split tasks. In this situation even if the splitLogWorker that owns the task dies the task will not get resubmitted.
> When a regionserver goes away then all the split-log tasks that it owned should be resubmitted by the SplitLogMaster.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4007) distributed log splitting can get
indefinitely stuck
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ted Yu updated HBASE-4007:
--------------------------
Fix Version/s: 0.92.0
I think I hit this bug in the course of troubleshooting HBASE-4301
> distributed log splitting can get indefinitely stuck
> ----------------------------------------------------
>
> Key: HBASE-4007
> URL: https://issues.apache.org/jira/browse/HBASE-4007
> Project: HBase
> Issue Type: Bug
> Reporter: Prakash Khemani
> Assignee: Prakash Khemani
> Priority: Critical
> Fix For: 0.92.0
>
> Attachments: 0001-HBASE-4007-distributed-log-splitting-can-get-indefin.patch
>
>
> After the configured number of retries SplitLogManager is not going to resubmit log-split tasks. In this situation even if the splitLogWorker that owns the task dies the task will not get resubmitted.
> When a regionserver goes away then all the split-log tasks that it owned should be resubmitted by the SplitLogMaster.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4007) distributed log splitting can get
indefinitely stuck
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13090801#comment-13090801 ]
stack commented on HBASE-4007:
------------------------------
Patch looks good Prakash. Are you running it?
> distributed log splitting can get indefinitely stuck
> ----------------------------------------------------
>
> Key: HBASE-4007
> URL: https://issues.apache.org/jira/browse/HBASE-4007
> Project: HBase
> Issue Type: Bug
> Reporter: Prakash Khemani
> Assignee: Prakash Khemani
> Attachments: 0001-HBASE-4007-distributed-log-splitting-can-get-indefin.patch
>
>
> After the configured number of retries SplitLogManager is not going to resubmit log-split tasks. In this situation even if the splitLogWorker that owns the task dies the task will not get resubmitted.
> When a regionserver goes away then all the split-log tasks that it owned should be resubmitted by the SplitLogMaster.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4007) distributed log splitting can get
indefinitely stuck
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13101014#comment-13101014 ]
Hudson commented on HBASE-4007:
-------------------------------
Integrated in HBase-TRUNK #2192 (See [https://builds.apache.org/job/HBase-TRUNK/2192/])
HBASE-4007 distributed log splitting can get indefinitely stuck
stack :
Files :
* /hbase/trunk/CHANGES.txt
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/SplitLogManager.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKSplitLog.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/master/TestSplitLogManager.java
> distributed log splitting can get indefinitely stuck
> ----------------------------------------------------
>
> Key: HBASE-4007
> URL: https://issues.apache.org/jira/browse/HBASE-4007
> Project: HBase
> Issue Type: Bug
> Reporter: Prakash Khemani
> Assignee: Prakash Khemani
> Priority: Critical
> Fix For: 0.92.0
>
> Attachments: 0001-HBASE-4007-distributed-log-splitting-can-get-indefin.patch
>
>
> After the configured number of retries SplitLogManager is not going to resubmit log-split tasks. In this situation even if the splitLogWorker that owns the task dies the task will not get resubmitted.
> When a regionserver goes away then all the split-log tasks that it owned should be resubmitted by the SplitLogMaster.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4007) distributed log splitting can get
indefinitely stuck
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13100745#comment-13100745 ]
jiraposter@reviews.apache.org commented on HBASE-4007:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1747/#review1824
-----------------------------------------------------------
Ship it!
- Ted
On 2011-09-08 22:09:32, Prakash Khemani wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/1747/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2011-09-08 22:09:32)
bq.
bq.
bq. Review request for hbase, Ted Yu and Michael Stack.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. 1/ resubmit all tasks owned by a dead splitlog-worker
bq. 2/ prevent accumulation of /hbase/splitlog/RESCAN nodes
bq.
bq.
bq. This addresses bug HBASE-4007.
bq. https://issues.apache.org/jira/browse/HBASE-4007
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java 9a71fdf
bq. src/main/java/org/apache/hadoop/hbase/master/SplitLogManager.java 54b6d45
bq. src/main/java/org/apache/hadoop/hbase/zookeeper/ZKSplitLog.java 61e5c65
bq. src/test/java/org/apache/hadoop/hbase/master/TestSplitLogManager.java 9a88855
bq.
bq. Diff: https://reviews.apache.org/r/1747/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. 1/ resubmit all tasks owned by a dead splitlog-worker - only unit tested. will do cluster testing.
bq. 2/ prevent accumulation of /hbase/splitlog/RESCAN nodes - tested and deployed in production.
bq.
bq.
bq. Thanks,
bq.
bq. Prakash
bq.
bq.
> distributed log splitting can get indefinitely stuck
> ----------------------------------------------------
>
> Key: HBASE-4007
> URL: https://issues.apache.org/jira/browse/HBASE-4007
> Project: HBase
> Issue Type: Bug
> Reporter: Prakash Khemani
> Assignee: Prakash Khemani
> Priority: Critical
> Fix For: 0.92.0
>
> Attachments: 0001-HBASE-4007-distributed-log-splitting-can-get-indefin.patch
>
>
> After the configured number of retries SplitLogManager is not going to resubmit log-split tasks. In this situation even if the splitLogWorker that owns the task dies the task will not get resubmitted.
> When a regionserver goes away then all the split-log tasks that it owned should be resubmitted by the SplitLogMaster.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4007) distributed log splitting can get
indefinitely stuck
Posted by "Prakash Khemani (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Prakash Khemani updated HBASE-4007:
-----------------------------------
Attachment: 0001-HBASE-4007-distributed-log-splitting-can-get-indefin.patch
patch
> distributed log splitting can get indefinitely stuck
> ----------------------------------------------------
>
> Key: HBASE-4007
> URL: https://issues.apache.org/jira/browse/HBASE-4007
> Project: HBase
> Issue Type: Bug
> Reporter: Prakash Khemani
> Assignee: Prakash Khemani
> Attachments: 0001-HBASE-4007-distributed-log-splitting-can-get-indefin.patch
>
>
> After the configured number of retries SplitLogManager is not going to resubmit log-split tasks. In this situation even if the splitLogWorker that owns the task dies the task will not get resubmitted.
> When a regionserver goes away then all the split-log tasks that it owned should be resubmitted by the SplitLogMaster.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4007) distributed log splitting can get
indefinitely stuck
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13100628#comment-13100628 ]
jiraposter@reviews.apache.org commented on HBASE-4007:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1747/
-----------------------------------------------------------
(Updated 2011-09-08 19:34:22.528153)
Review request for hbase, Ted Yu and Michael Stack.
Changes
-------
Implemented Ted's feedback
tested on a cluster that when a regionserver (splitlog-worker) dies then the notification reaches SplitLogManager.
Summary
-------
1/ resubmit all tasks owned by a dead splitlog-worker
2/ prevent accumulation of /hbase/splitlog/RESCAN nodes
This addresses bug HBASE-4007.
https://issues.apache.org/jira/browse/HBASE-4007
Diffs (updated)
-----
src/main/java/org/apache/hadoop/hbase/master/SplitLogManager.java 54b6d45
src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java 9a71fdf
src/main/java/org/apache/hadoop/hbase/zookeeper/ZKSplitLog.java 61e5c65
src/test/java/org/apache/hadoop/hbase/master/TestSplitLogManager.java 9a88855
Diff: https://reviews.apache.org/r/1747/diff
Testing
-------
1/ resubmit all tasks owned by a dead splitlog-worker - only unit tested. will do cluster testing.
2/ prevent accumulation of /hbase/splitlog/RESCAN nodes - tested and deployed in production.
Thanks,
Prakash
> distributed log splitting can get indefinitely stuck
> ----------------------------------------------------
>
> Key: HBASE-4007
> URL: https://issues.apache.org/jira/browse/HBASE-4007
> Project: HBase
> Issue Type: Bug
> Reporter: Prakash Khemani
> Assignee: Prakash Khemani
> Priority: Critical
> Fix For: 0.92.0
>
> Attachments: 0001-HBASE-4007-distributed-log-splitting-can-get-indefin.patch
>
>
> After the configured number of retries SplitLogManager is not going to resubmit log-split tasks. In this situation even if the splitLogWorker that owns the task dies the task will not get resubmitted.
> When a regionserver goes away then all the split-log tasks that it owned should be resubmitted by the SplitLogMaster.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira