You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "stack (JIRA)" <ji...@apache.org> on 2011/09/08 19:10:08 UTC
[jira] [Created] (HBASE-4352) Apply version of hbase-4015 to branch
Apply version of hbase-4015 to branch
-------------------------------------
Key: HBASE-4352
URL: https://issues.apache.org/jira/browse/HBASE-4352
Project: HBase
Issue Type: Bug
Reporter: stack
Fix For: 0.90.5
Consider adding a version of hbase-4015 to 0.90. It changes HRegionInterface so would need move change to end of the Interface and then test that it doesn't break rolling restart.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4352) Apply version of hbase-4015 to
branch
Posted by "stack (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13165873#comment-13165873 ]
stack commented on HBASE-4352:
------------------------------
It seemed like 5d0a564b65a53b1c86f7c02407c82231 was opened fine (I'd just done clean cluster start, killed master, started 0.90.5 master, then had tried the rolling restart --- no load on cluster). I didn't dig in too deep. Was going to start adding logging... debugging... but had spent enough time on it.
Can I roll a 0.90.5RC Ram?
> Apply version of hbase-4015 to branch
> -------------------------------------
>
> Key: HBASE-4352
> URL: https://issues.apache.org/jira/browse/HBASE-4352
> Project: HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: ramkrishna.s.vasudevan
> Priority: Blocker
> Attachments: HBASE-4352_0.90.patch, HBASE-4352_0.90_1.patch
>
>
> Consider adding a version of hbase-4015 to 0.90. It changes HRegionInterface so would need move change to end of the Interface and then test that it doesn't break rolling restart.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-4352) Apply version of hbase-4015 to
branch
Posted by "stack (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack resolved HBASE-4352.
--------------------------
Resolution: Fixed
Hadoop Flags: Reviewed
Applied to 0.90 branch. Need to test that it doesn't break rolling restarts. Committing so can put up a 0.90.5 RC.
> Apply version of hbase-4015 to branch
> -------------------------------------
>
> Key: HBASE-4352
> URL: https://issues.apache.org/jira/browse/HBASE-4352
> Project: HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: ramkrishna.s.vasudevan
> Priority: Blocker
> Fix For: 0.90.5
>
> Attachments: HBASE-4352_0.90.patch, HBASE-4352_0.90_1.patch
>
>
> Consider adding a version of hbase-4015 to 0.90. It changes HRegionInterface so would need move change to end of the Interface and then test that it doesn't break rolling restart.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4352) Apply version of hbase-4015 to
branch
Posted by "ramkrishna.s.vasudevan (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13165872#comment-13165872 ]
ramkrishna.s.vasudevan commented on HBASE-4352:
-----------------------------------------------
@Stack
5d0a564b65a53b1c86f7c02407c82231 - Did this region get opened up correctly? Then the second set of logs is not a problem.
First set of logs am not very clear as why it came up.
Thanks Stack for your time.:)
> Apply version of hbase-4015 to branch
> -------------------------------------
>
> Key: HBASE-4352
> URL: https://issues.apache.org/jira/browse/HBASE-4352
> Project: HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: ramkrishna.s.vasudevan
> Priority: Blocker
> Attachments: HBASE-4352_0.90.patch, HBASE-4352_0.90_1.patch
>
>
> Consider adding a version of hbase-4015 to 0.90. It changes HRegionInterface so would need move change to end of the Interface and then test that it doesn't break rolling restart.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4352) Apply version of hbase-4015 to
branch
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13108629#comment-13108629 ]
Ted Yu commented on HBASE-4352:
-------------------------------
TestZKBasedOpenCloseRegion hangs based on patch.
> Apply version of hbase-4015 to branch
> -------------------------------------
>
> Key: HBASE-4352
> URL: https://issues.apache.org/jira/browse/HBASE-4352
> Project: HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: ramkrishna.s.vasudevan
> Fix For: 0.90.5
>
> Attachments: HBASE-4352_0.90.patch
>
>
> Consider adding a version of hbase-4015 to 0.90. It changes HRegionInterface so would need move change to end of the Interface and then test that it doesn't break rolling restart.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4352) Apply version of hbase-4015 to
branch
Posted by "ramkrishna.s.vasudevan (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13165724#comment-13165724 ]
ramkrishna.s.vasudevan commented on HBASE-4352:
-----------------------------------------------
@Stack
Where is the NPE coming in the client side? Could you paste the stack trace to chk it out.
> Apply version of hbase-4015 to branch
> -------------------------------------
>
> Key: HBASE-4352
> URL: https://issues.apache.org/jira/browse/HBASE-4352
> Project: HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: ramkrishna.s.vasudevan
> Priority: Blocker
> Attachments: HBASE-4352_0.90.patch, HBASE-4352_0.90_1.patch
>
>
> Consider adding a version of hbase-4015 to 0.90. It changes HRegionInterface so would need move change to end of the Interface and then test that it doesn't break rolling restart.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4352) Apply version of hbase-4015 to
branch
Posted by "stack (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13165007#comment-13165007 ]
stack commented on HBASE-4352:
------------------------------
Playing around w/ a cluster under load upgrading it from 0.90.4 to 0.90.5, all basically works except stuff like the following:
{code}
2011-12-08 05:47:42,226 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Assigning region TestTable,0006971712,1323323148059.2e97f25eefbd77765d64b0900d94e5ba. to sv4r12s38,7003,1323321454788
2011-12-08 05:47:42,226 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Handling transition=M_ZK_REGION_OFFLINE, server=sv4r11s38:7001, region=2e97f25eefbd77765d64b0900d94e5ba
2011-12-08 05:47:42,229 WARN org.apache.hadoop.hbase.master.AssignmentManager: Failed assignment of TestTable,0006971712,1323323148059.2e97f25eefbd77765d64b0900d94e5ba. to serverName=sv4r12s38,7003,1323321454788, load=(requests=29558, regions=35, usedHeap=295, maxHeap=493), trying to assign elsewhere instead; retry=9
org.apache.hadoop.ipc.RemoteException: java.io.IOException: java.lang.NoSuchMethodException: org.apache.hadoop.hbase.regionserver.HRegionServer.openRegion(org.apache.hadoop.hbase.HRegionInfo, int)
at java.lang.Class.getMethod(Class.java:1605)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:565)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)
at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:771)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
at $Proxy7.openRegion(Unknown Source)
at org.apache.hadoop.hbase.master.ServerManager.sendRegionOpen(ServerManager.java:574)
at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1132)
at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:894)
at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:876)
at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:872)
at org.apache.hadoop.hbase.master.handler.ClosedRegionHandler.process(ClosedRegionHandler.java:92)
at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:156)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{code}
I see this when balancer tries to run (Its a 0.90.5 master running against a 0.90.4 cluster).
I don't think this too bad. Its transient situation: i.e. you can't do a balancer till all servers go up to 0.90.5.
> Apply version of hbase-4015 to branch
> -------------------------------------
>
> Key: HBASE-4352
> URL: https://issues.apache.org/jira/browse/HBASE-4352
> Project: HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: ramkrishna.s.vasudevan
> Priority: Blocker
> Fix For: 0.90.5
>
> Attachments: HBASE-4352_0.90.patch, HBASE-4352_0.90_1.patch
>
>
> Consider adding a version of hbase-4015 to 0.90. It changes HRegionInterface so would need move change to end of the Interface and then test that it doesn't break rolling restart.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HBASE-4352) Apply version of hbase-4015 to
branch
Posted by "stack (Reopened) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack reopened HBASE-4352:
--------------------------
Reopened after reverting patch from 0.90 branch.
> Apply version of hbase-4015 to branch
> -------------------------------------
>
> Key: HBASE-4352
> URL: https://issues.apache.org/jira/browse/HBASE-4352
> Project: HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: ramkrishna.s.vasudevan
> Priority: Blocker
> Attachments: HBASE-4352_0.90.patch, HBASE-4352_0.90_1.patch
>
>
> Consider adding a version of hbase-4015 to 0.90. It changes HRegionInterface so would need move change to end of the Interface and then test that it doesn't break rolling restart.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4352) Apply version of hbase-4015 to branch
Posted by "stack (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-4352:
-------------------------
Fix Version/s: (was: 0.90.5)
> Apply version of hbase-4015 to branch
> -------------------------------------
>
> Key: HBASE-4352
> URL: https://issues.apache.org/jira/browse/HBASE-4352
> Project: HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: ramkrishna.s.vasudevan
> Priority: Blocker
> Attachments: HBASE-4352_0.90.patch, HBASE-4352_0.90_1.patch
>
>
> Consider adding a version of hbase-4015 to 0.90. It changes HRegionInterface so would need move change to end of the Interface and then test that it doesn't break rolling restart.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4352) Apply version of hbase-4015 to
branch
Posted by "Ted Yu (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13114907#comment-13114907 ]
Ted Yu commented on HBASE-4352:
-------------------------------
+1 on latest patch.
No surprise in test suite.
> Apply version of hbase-4015 to branch
> -------------------------------------
>
> Key: HBASE-4352
> URL: https://issues.apache.org/jira/browse/HBASE-4352
> Project: HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: ramkrishna.s.vasudevan
> Fix For: 0.90.5
>
> Attachments: HBASE-4352_0.90.patch, HBASE-4352_0.90_1.patch
>
>
> Consider adding a version of hbase-4015 to 0.90. It changes HRegionInterface so would need move change to end of the Interface and then test that it doesn't break rolling restart.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4352) Apply version of hbase-4015 to
branch
Posted by "ramkrishna.s.vasudevan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13105322#comment-13105322 ]
ramkrishna.s.vasudevan commented on HBASE-4352:
-----------------------------------------------
I have prepared the patch. need to test as we did for trunk. once done will upload the patch.
Thanks
> Apply version of hbase-4015 to branch
> -------------------------------------
>
> Key: HBASE-4352
> URL: https://issues.apache.org/jira/browse/HBASE-4352
> Project: HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: ramkrishna.s.vasudevan
> Fix For: 0.90.5
>
>
> Consider adding a version of hbase-4015 to 0.90. It changes HRegionInterface so would need move change to end of the Interface and then test that it doesn't break rolling restart.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4352) Apply version of hbase-4015 to
branch
Posted by "stack (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13115822#comment-13115822 ]
stack commented on HBASE-4352:
------------------------------
I signed up to check we don't break rolling restart. Let me get that done first. Will report back if can do what you ask above (I'm trying to test 205 as background task, could do two things at once).
> Apply version of hbase-4015 to branch
> -------------------------------------
>
> Key: HBASE-4352
> URL: https://issues.apache.org/jira/browse/HBASE-4352
> Project: HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: ramkrishna.s.vasudevan
> Fix For: 0.90.5
>
> Attachments: HBASE-4352_0.90.patch, HBASE-4352_0.90_1.patch
>
>
> Consider adding a version of hbase-4015 to 0.90. It changes HRegionInterface so would need move change to end of the Interface and then test that it doesn't break rolling restart.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4352) Apply version of hbase-4015 to
branch
Posted by "ramkrishna.s.vasudevan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13108738#comment-13108738 ]
ramkrishna.s.vasudevan commented on HBASE-4352:
-----------------------------------------------
@Ted
Not yet done with the testing of this patch. Just a first cut version so that you can review the changes done as it involves interface changes. I will run all the testcases and ensure that the patch is clean and all testcases and scenarios pass.
> Apply version of hbase-4015 to branch
> -------------------------------------
>
> Key: HBASE-4352
> URL: https://issues.apache.org/jira/browse/HBASE-4352
> Project: HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: ramkrishna.s.vasudevan
> Fix For: 0.90.5
>
> Attachments: HBASE-4352_0.90.patch
>
>
> Consider adding a version of hbase-4015 to 0.90. It changes HRegionInterface so would need move change to end of the Interface and then test that it doesn't break rolling restart.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4352) Apply version of hbase-4015 to branch
Posted by "stack (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-4352:
-------------------------
Release Note: Refactors timeout monitor to make it less racy. Beware: changes the HRegionInterface. Stuff should keep working generally though you'll see nosuchmethod exceptions during the period where you have a 0.90.5 master and regionservers are still 0.90.4 (harmless).
> Apply version of hbase-4015 to branch
> -------------------------------------
>
> Key: HBASE-4352
> URL: https://issues.apache.org/jira/browse/HBASE-4352
> Project: HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: ramkrishna.s.vasudevan
> Priority: Blocker
> Fix For: 0.90.5
>
> Attachments: HBASE-4352_0.90.patch, HBASE-4352_0.90_1.patch
>
>
> Consider adding a version of hbase-4015 to 0.90. It changes HRegionInterface so would need move change to end of the Interface and then test that it doesn't break rolling restart.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4352) Apply version of hbase-4015 to
branch
Posted by "ramkrishna.s.vasudevan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13103615#comment-13103615 ]
ramkrishna.s.vasudevan commented on HBASE-4352:
-----------------------------------------------
@Stack
As part of this HBASE-4083 fix also needs to be applied to 0.90.x. HBASE-4083 fix has been checked into trunk version. If you can remember, you had told that once rolling restart is tested we can take it to 0.90.x version.
> Apply version of hbase-4015 to branch
> -------------------------------------
>
> Key: HBASE-4352
> URL: https://issues.apache.org/jira/browse/HBASE-4352
> Project: HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: ramkrishna.s.vasudevan
> Fix For: 0.90.5
>
>
> Consider adding a version of hbase-4015 to 0.90. It changes HRegionInterface so would need move change to end of the Interface and then test that it doesn't break rolling restart.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4352) Apply version of hbase-4015 to
branch
Posted by "ramkrishna.s.vasudevan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13114773#comment-13114773 ]
ramkrishna.s.vasudevan commented on HBASE-4352:
-----------------------------------------------
@Ted
All testcases are passing.
TestZKBasedOpenCloseRegion is also changed now.
I tested in cluster with 3 nodes and did the testing as in HBASE-4015.
There were no regions in RIT and the HBCK reporting was ok.
@Stack
How to do the rolling restart testing. This defect involves changes in the interface.
So if we have the master installation with the patch and the RS is without patch then it may not be able to find the sendRegionOpen() added newly. So how can we handle this?
> Apply version of hbase-4015 to branch
> -------------------------------------
>
> Key: HBASE-4352
> URL: https://issues.apache.org/jira/browse/HBASE-4352
> Project: HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: ramkrishna.s.vasudevan
> Fix For: 0.90.5
>
> Attachments: HBASE-4352_0.90.patch, HBASE-4352_0.90_1.patch
>
>
> Consider adding a version of hbase-4015 to 0.90. It changes HRegionInterface so would need move change to end of the Interface and then test that it doesn't break rolling restart.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-4352) Apply version of hbase-4015 to
branch
Posted by "ramkrishna.s.vasudevan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ramkrishna.s.vasudevan reassigned HBASE-4352:
---------------------------------------------
Assignee: ramkrishna.s.vasudevan
> Apply version of hbase-4015 to branch
> -------------------------------------
>
> Key: HBASE-4352
> URL: https://issues.apache.org/jira/browse/HBASE-4352
> Project: HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: ramkrishna.s.vasudevan
> Fix For: 0.90.5
>
>
> Consider adding a version of hbase-4015 to 0.90. It changes HRegionInterface so would need move change to end of the Interface and then test that it doesn't break rolling restart.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4352) Apply version of hbase-4015 to
branch
Posted by "stack (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13165828#comment-13165828 ]
stack commented on HBASE-4352:
------------------------------
Here is a 0.90.5 master trying to reload a regionserver that has been restarted as a 0.90.5 (it had been a 0.90.4):
{code}
2011-12-08 22:56:53,807 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Assigning region TestTable,0013681953,1323325588029.5d0a564b65a53b1c86f7c02407c82231. to sv4r9s38,7003,1323384867368
2011-12-08 22:56:53,809 WARN org.apache.hadoop.ipc.HBaseClient: Unexpected exception receiving call responses
java.lang.NullPointerException
2011-12-08 22:56:53,810 WARN org.apache.hadoop.hbase.master.AssignmentManager: Failed assignment of TestTable,0013681953,1323325588029.5d0a564b65a53b1c86f7c02407c82231. to serverName=sv4r9s38,7003,1323384867368, load=(requests=0, regions=0, usedHeap=23, maxHeap=493), trying to assign elsewhere instead; retry=0
java.io.IOException: Call to /10.4.9.38:7003 failed on local exception: java.io.IOException: Unexpected exception receiving call responses
at org.apache.hadoop.hbase.ipc.HBaseClient.wrapException(HBaseClient.java:806)
at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:775)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
at $Proxy7.openRegion(Unknown Source)
at org.apache.hadoop.hbase.master.ServerManager.sendRegionOpen(ServerManager.java:561)
at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1043)
at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:858)
at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:838)
at org.apache.hadoop.hbase.master.handler.ClosedRegionHandler.process(ClosedRegionHandler.java:92)
at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:156)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOException: Unexpected exception receiving call responses
at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:481)
Caused by: java.lang.NullPointerException
{code}
It looks like call is null as per another issue that just got committed. Shouldn't be.
Was getting bunches of this on regionserver side, though it don't seem directly related:
{code}
2011-12-08 22:56:53,809 DEBUG org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Processing open of TestTable,0013681953,1323325588029.5d0a564b65a53b1c86f7c02407c82231.
2011-12-08 22:56:53,810 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:7003-0x1341fa8d95400cc Attempting to transition node 5d0a564b65a53b1c86f7c02407c82231 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
2011-12-08 22:56:53,811 WARN org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:7003-0x1341fa8d95400cc Attempt to transition the unassigned node for 5d0a564b65a53b1c86f7c02407c82231 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING failed, the node existed but was version 2 not the expected version -1
2011-12-08 22:56:53,811 WARN org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Failed transition from OFFLINE to OPENING for region=5d0a564b65a53b1c86f7c02407c82231
2011-12-08 22:56:53,811 WARN org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Region was hijacked? It no longer exists, encodedName=5d0a564b65a53b1c86f7c02407c82231
{code}
I was hoping to put up a 0.90.5RC. If you want me to hold a while, I can. Else, I'll go ahead and cut an RC w/o this.
> Apply version of hbase-4015 to branch
> -------------------------------------
>
> Key: HBASE-4352
> URL: https://issues.apache.org/jira/browse/HBASE-4352
> Project: HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: ramkrishna.s.vasudevan
> Priority: Blocker
> Attachments: HBASE-4352_0.90.patch, HBASE-4352_0.90_1.patch
>
>
> Consider adding a version of hbase-4015 to 0.90. It changes HRegionInterface so would need move change to end of the Interface and then test that it doesn't break rolling restart.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4352) Apply version of hbase-4015 to
branch
Posted by "stack (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13165718#comment-13165718 ]
stack commented on HBASE-4352:
------------------------------
Backing this out. Can't do rolling restart from 0.90.4 to 0.90.5. I get NPEs client-side trying to receive response even after putting up a 0.90.5 master first and while trying to rolling restart a 0.90.4 to 0.90.5 regionserver; the offloading works but then the onloading fails.
Without this patch I can rolling restart from 0.90.4 to 0.90.5 smoothly.
> Apply version of hbase-4015 to branch
> -------------------------------------
>
> Key: HBASE-4352
> URL: https://issues.apache.org/jira/browse/HBASE-4352
> Project: HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: ramkrishna.s.vasudevan
> Priority: Blocker
> Fix For: 0.90.5
>
> Attachments: HBASE-4352_0.90.patch, HBASE-4352_0.90_1.patch
>
>
> Consider adding a version of hbase-4015 to 0.90. It changes HRegionInterface so would need move change to end of the Interface and then test that it doesn't break rolling restart.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4352) Apply version of hbase-4015 to branch
Posted by "ramkrishna.s.vasudevan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ramkrishna.s.vasudevan updated HBASE-4352:
------------------------------------------
Attachment: HBASE-4352_0.90_1.patch
> Apply version of hbase-4015 to branch
> -------------------------------------
>
> Key: HBASE-4352
> URL: https://issues.apache.org/jira/browse/HBASE-4352
> Project: HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: ramkrishna.s.vasudevan
> Fix For: 0.90.5
>
> Attachments: HBASE-4352_0.90.patch, HBASE-4352_0.90_1.patch
>
>
> Consider adding a version of hbase-4015 to 0.90. It changes HRegionInterface so would need move change to end of the Interface and then test that it doesn't break rolling restart.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4352) Apply version of hbase-4015 to branch
Posted by "ramkrishna.s.vasudevan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ramkrishna.s.vasudevan updated HBASE-4352:
------------------------------------------
Attachment: HBASE-4352_0.90.patch
First cut version. It has the fix of HBASE-4083 also without which this patch may be incomplete. Testing is going on. Test case results will publish sooner.
> Apply version of hbase-4015 to branch
> -------------------------------------
>
> Key: HBASE-4352
> URL: https://issues.apache.org/jira/browse/HBASE-4352
> Project: HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: ramkrishna.s.vasudevan
> Fix For: 0.90.5
>
> Attachments: HBASE-4352_0.90.patch
>
>
> Consider adding a version of hbase-4015 to 0.90. It changes HRegionInterface so would need move change to end of the Interface and then test that it doesn't break rolling restart.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4352) Apply version of hbase-4015 to
branch
Posted by "ramkrishna.s.vasudevan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13108906#comment-13108906 ]
ramkrishna.s.vasudevan commented on HBASE-4352:
-----------------------------------------------
Testing was done today. Out of4000 regions 5 regions had inconsistencies reported by HBCK. Trying to figure out the reason. But things may not be due to timeoutmonitor changes.
Out of 5 one is double assignment.
and 4 are like the RS hosting them are actually different from the one in META.
So tomorrow will dig in deeper and find if timeoutmonitor changes were the root cause or some existing flow is causing this inconsistency. But no regions are in RIT which is assured. :)
> Apply version of hbase-4015 to branch
> -------------------------------------
>
> Key: HBASE-4352
> URL: https://issues.apache.org/jira/browse/HBASE-4352
> Project: HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: ramkrishna.s.vasudevan
> Fix For: 0.90.5
>
> Attachments: HBASE-4352_0.90.patch
>
>
> Consider adding a version of hbase-4015 to 0.90. It changes HRegionInterface so would need move change to end of the Interface and then test that it doesn't break rolling restart.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4352) Apply version of hbase-4015 to
branch
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13114822#comment-13114822 ]
Ted Yu commented on HBASE-4352:
-------------------------------
I am running test suite on 0.90 branch. Will report back if any test failure comes up.
> Apply version of hbase-4015 to branch
> -------------------------------------
>
> Key: HBASE-4352
> URL: https://issues.apache.org/jira/browse/HBASE-4352
> Project: HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: ramkrishna.s.vasudevan
> Fix For: 0.90.5
>
> Attachments: HBASE-4352_0.90.patch, HBASE-4352_0.90_1.patch
>
>
> Consider adding a version of hbase-4015 to 0.90. It changes HRegionInterface so would need move change to end of the Interface and then test that it doesn't break rolling restart.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4352) Apply version of hbase-4015 to
branch
Posted by "Todd Lipcon (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13115816#comment-13115816 ]
Todd Lipcon commented on HBASE-4352:
------------------------------------
Can anyone volunteer to do some serious cluster testing with this patch? eg load up 1000 regions per server on 5+ nodes, and do rolling restarts or rolling crashes?
> Apply version of hbase-4015 to branch
> -------------------------------------
>
> Key: HBASE-4352
> URL: https://issues.apache.org/jira/browse/HBASE-4352
> Project: HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: ramkrishna.s.vasudevan
> Fix For: 0.90.5
>
> Attachments: HBASE-4352_0.90.patch, HBASE-4352_0.90_1.patch
>
>
> Consider adding a version of hbase-4015 to 0.90. It changes HRegionInterface so would need move change to end of the Interface and then test that it doesn't break rolling restart.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4352) Apply version of hbase-4015 to branch
Posted by "stack (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-4352:
-------------------------
Priority: Blocker (was: Major)
Making blocker on 0.90.5
> Apply version of hbase-4015 to branch
> -------------------------------------
>
> Key: HBASE-4352
> URL: https://issues.apache.org/jira/browse/HBASE-4352
> Project: HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: ramkrishna.s.vasudevan
> Priority: Blocker
> Fix For: 0.90.5
>
> Attachments: HBASE-4352_0.90.patch, HBASE-4352_0.90_1.patch
>
>
> Consider adding a version of hbase-4015 to 0.90. It changes HRegionInterface so would need move change to end of the Interface and then test that it doesn't break rolling restart.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira