You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "ramkrishna.s.vasudevan (JIRA)" <ji...@apache.org> on 2011/07/01 06:49:28 UTC
[jira] [Created] (HBASE-4052) Enabling a table after master switch
is not allowing table scan. Throws NotServingRegionException
Enabling a table after master switch is not allowing table scan. Throws NotServingRegionException
--------------------------------------------------------------------------------------------------
Key: HBASE-4052
URL: https://issues.apache.org/jira/browse/HBASE-4052
Project: HBase
Issue Type: Bug
Environment: Linux
Reporter: ramkrishna.s.vasudevan
Following is the scenario:
Start RS and Active and standby masters
Create table and insert data.
Disable the table.
Stop the active master and switch to the standby master.
Now enable the table.
Do a scan on the enabled table.
NotServingRegionException is Thrown.
But the same works well when we dont switch the master.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4052) Enabling a table after master
switch is not allowing table scan. Throws NotServingRegionException
Posted by "ramkrishna.s.vasudevan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13063336#comment-13063336 ]
ramkrishna.s.vasudevan commented on HBASE-4052:
-----------------------------------------------
The patch HBASE-4052-1-0.90_1.patch is for 0.90 branch.
The patch HBASE-4052-1-trunk_1.patch is for trunk branch.
The patch HBASE-4052-1-_TestCode.patch is the test code.
> Enabling a table after master switch is not allowing table scan. Throws NotServingRegionException
> --------------------------------------------------------------------------------------------------
>
> Key: HBASE-4052
> URL: https://issues.apache.org/jira/browse/HBASE-4052
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.90.3
> Environment: Linux
> Reporter: ramkrishna.s.vasudevan
> Assignee: ramkrishna.s.vasudevan
> Fix For: 0.90.4
>
> Attachments: 4052.txt, Disabled.bmp, Disabling-1.bmp, Disabling-2.bmp, HBASE-4052-1-0.90_1.patch, HBASE-4052-1-_TestCode.patch, HBASE-4052-1-trunk_1.patch, HBASE-4052.patch, TestMasterRestartAfterDisablingTable.java
>
>
> Following is the scenario:
> Start RS and Active and standby masters
> Create table and insert data.
> Disable the table.
> Stop the active master and switch to the standby master.
> Now enable the table.
> Do a scan on the enabled table.
> NotServingRegionException is Thrown.
> But the same works well when we dont switch the master.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4052) Enabling a table after master
switch does not allow table scan, throwing NotServingRegionException
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13066367#comment-13066367 ]
Hudson commented on HBASE-4052:
-------------------------------
Integrated in HBase-TRUNK #2036 (See [https://builds.apache.org/job/HBase-TRUNK/2036/])
HBASE-4052 Use same TestMasterRestartAfterDisablingTable in branch and TRUNK
HBASE-4052 Enabling a table after master switch does not allow table scan,
throwing NotServingRegionException. Add unit test
HBASE-4052 Enabling a table after master switch does not allow table scan,
throwing NotServingRegionException (ramkrishna via Ted Yu)
tedyu :
Files :
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/master/TestMasterRestartAfterDisablingTable.java
tedyu :
Files :
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/master/TestMasterRestartAfterDisablingTable.java
tedyu :
Files :
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java
* /hbase/trunk/CHANGES.txt
> Enabling a table after master switch does not allow table scan, throwing NotServingRegionException
> --------------------------------------------------------------------------------------------------
>
> Key: HBASE-4052
> URL: https://issues.apache.org/jira/browse/HBASE-4052
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.90.3
> Environment: Linux
> Reporter: ramkrishna.s.vasudevan
> Assignee: ramkrishna.s.vasudevan
> Fix For: 0.90.4
>
> Attachments: 4052.txt, Disabled.bmp, Disabling-1.bmp, Disabling-2.bmp, HBASE-4052-1-0.90_1.patch, HBASE-4052-1-_TestCode.patch, HBASE-4052-1-trunk_1.patch, HBASE-4052.patch, TestMasterRestartAfterDisablingTable.java
>
>
> Following is the scenario:
> Start RS and Active and standby masters
> Create table and insert data.
> Disable the table.
> Stop the active master and switch to the standby master.
> Now enable the table.
> Do a scan on the enabled table.
> NotServingRegionException is Thrown.
> But the same works well when we dont switch the master.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4052) Enabling a table after master
switch is not allowing table scan. Throws NotServingRegionException
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13062166#comment-13062166 ]
Ted Yu commented on HBASE-4052:
-------------------------------
Running your unit test, I got:
{code}
testForCheckingIfEnableAndDisableWorksFineAfterSwitch(org.apache.hadoop.hbase.master.TestMasterRestartAfterDisablingTable) Time elapsed: 24.594 sec <<< ERROR!
java.lang.reflect.UndeclaredThrowableException
at $Proxy12.isMasterRunning(Unknown Source)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:553)
at org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:101)
at org.apache.hadoop.hbase.HBaseTestingUtility.getHBaseAdmin(HBaseTestingUtility.java:1162)
at org.apache.hadoop.hbase.master.TestMasterRestartAfterDisablingTable.testForCheckingIfEnableAndDisableWorksFineAfterSwitch(TestMasterRestartAfterDisablingTable.java:105)
{code}
I think we should wait for backup master to come up before doing:
{code}
log("Enabling table\n");
TEST_UTIL.getHBaseAdmin().enableTable(table);
{code}
> Enabling a table after master switch is not allowing table scan. Throws NotServingRegionException
> --------------------------------------------------------------------------------------------------
>
> Key: HBASE-4052
> URL: https://issues.apache.org/jira/browse/HBASE-4052
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.90.3
> Environment: Linux
> Reporter: ramkrishna.s.vasudevan
> Assignee: ramkrishna.s.vasudevan
> Fix For: 0.90.4
>
> Attachments: Disabled.bmp, Disabling-1.bmp, Disabling-2.bmp, HBASE-4052.patch, TestMasterRestartAfterDisablingTable.java
>
>
> Following is the scenario:
> Start RS and Active and standby masters
> Create table and insert data.
> Disable the table.
> Stop the active master and switch to the standby master.
> Now enable the table.
> Do a scan on the enabled table.
> NotServingRegionException is Thrown.
> But the same works well when we dont switch the master.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4052) Enabling a table after master switch
is not allowing table scan. Throws NotServingRegionException
Posted by "ramkrishna.s.vasudevan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ramkrishna.s.vasudevan updated HBASE-4052:
------------------------------------------
Attachment: HBASE-4052-1-_TestCode.patch
> Enabling a table after master switch is not allowing table scan. Throws NotServingRegionException
> --------------------------------------------------------------------------------------------------
>
> Key: HBASE-4052
> URL: https://issues.apache.org/jira/browse/HBASE-4052
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.90.3
> Environment: Linux
> Reporter: ramkrishna.s.vasudevan
> Assignee: ramkrishna.s.vasudevan
> Fix For: 0.90.4
>
> Attachments: 4052.txt, Disabled.bmp, Disabling-1.bmp, Disabling-2.bmp, HBASE-4052-1-0.90_1.patch, HBASE-4052-1-_TestCode.patch, HBASE-4052-1-trunk_1.patch, HBASE-4052.patch, TestMasterRestartAfterDisablingTable.java
>
>
> Following is the scenario:
> Start RS and Active and standby masters
> Create table and insert data.
> Disable the table.
> Stop the active master and switch to the standby master.
> Now enable the table.
> Do a scan on the enabled table.
> NotServingRegionException is Thrown.
> But the same works well when we dont switch the master.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4052) Enabling a table after master switch
is not allowing table scan. Throws NotServingRegionException
Posted by "ramkrishna.s.vasudevan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ramkrishna.s.vasudevan updated HBASE-4052:
------------------------------------------
Attachment: Disabled.bmp
> Enabling a table after master switch is not allowing table scan. Throws NotServingRegionException
> --------------------------------------------------------------------------------------------------
>
> Key: HBASE-4052
> URL: https://issues.apache.org/jira/browse/HBASE-4052
> Project: HBase
> Issue Type: Bug
> Environment: Linux
> Reporter: ramkrishna.s.vasudevan
> Attachments: Disabled.bmp, TestMasterRestartAfterDisablingTable.java
>
>
> Following is the scenario:
> Start RS and Active and standby masters
> Create table and insert data.
> Disable the table.
> Stop the active master and switch to the standby master.
> Now enable the table.
> Do a scan on the enabled table.
> NotServingRegionException is Thrown.
> But the same works well when we dont switch the master.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4052) Enabling a table after master
switch is not allowing table scan. Throws NotServingRegionException
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13063428#comment-13063428 ]
Ted Yu commented on HBASE-4052:
-------------------------------
Or the test hangs:
{noformat}
"main" prio=5 tid=103000800 nid=0x100601000 in Object.wait() [1005fe000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <7a46c9828> (a org.apache.hadoop.hbase.ipc.HBaseClient$Call)
at java.lang.Object.wait(Object.java:485)
at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:835)
- locked <7a46c9828> (a org.apache.hadoop.hbase.ipc.HBaseClient$Call)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:142)
at $Proxy12.isMasterRunning(Unknown Source)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:553)
at org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:101)
at org.apache.hadoop.hbase.HBaseTestingUtility.getHBaseAdmin(HBaseTestingUtility.java:1162)
at org.apache.hadoop.hbase.master.TestMasterRestartAfterDisablingTable.testForCheckingIfEnableAndDisableWorksFineAfterSwitch(TestMasterRestartAfterDisablingTable.java:100)
{noformat}
> Enabling a table after master switch is not allowing table scan. Throws NotServingRegionException
> --------------------------------------------------------------------------------------------------
>
> Key: HBASE-4052
> URL: https://issues.apache.org/jira/browse/HBASE-4052
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.90.3
> Environment: Linux
> Reporter: ramkrishna.s.vasudevan
> Assignee: ramkrishna.s.vasudevan
> Fix For: 0.90.4
>
> Attachments: 4052.txt, Disabled.bmp, Disabling-1.bmp, Disabling-2.bmp, HBASE-4052-1-0.90_1.patch, HBASE-4052-1-_TestCode.patch, HBASE-4052-1-trunk_1.patch, HBASE-4052.patch, TestMasterRestartAfterDisablingTable.java
>
>
> Following is the scenario:
> Start RS and Active and standby masters
> Create table and insert data.
> Disable the table.
> Stop the active master and switch to the standby master.
> Now enable the table.
> Do a scan on the enabled table.
> NotServingRegionException is Thrown.
> But the same works well when we dont switch the master.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4052) Enabling a table after master
switch does not allow table scan, throwing NotServingRegionException
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13067056#comment-13067056 ]
Hudson commented on HBASE-4052:
-------------------------------
Integrated in HBase-TRUNK #2037 (See [https://builds.apache.org/job/HBase-TRUNK/2037/])
HBASE-4052 reapply code in unassign() which handles NotServingRegionException
tedyu :
Files :
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java
> Enabling a table after master switch does not allow table scan, throwing NotServingRegionException
> --------------------------------------------------------------------------------------------------
>
> Key: HBASE-4052
> URL: https://issues.apache.org/jira/browse/HBASE-4052
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.90.3
> Environment: Linux
> Reporter: ramkrishna.s.vasudevan
> Assignee: ramkrishna.s.vasudevan
> Fix For: 0.90.4
>
> Attachments: 4052.txt, Disabled.bmp, Disabling-1.bmp, Disabling-2.bmp, HBASE-4052-1-0.90_1.patch, HBASE-4052-1-_TestCode.patch, HBASE-4052-1-trunk_1.patch, HBASE-4052.patch, TestMasterRestartAfterDisablingTable.java
>
>
> Following is the scenario:
> Start RS and Active and standby masters
> Create table and insert data.
> Disable the table.
> Stop the active master and switch to the standby master.
> Now enable the table.
> Do a scan on the enabled table.
> NotServingRegionException is Thrown.
> But the same works well when we dont switch the master.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4052) Enabling a table after master switch
is not allowing table scan. Throws NotServingRegionException
Posted by "ramkrishna.s.vasudevan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ramkrishna.s.vasudevan updated HBASE-4052:
------------------------------------------
Status: Patch Available (was: Open)
> Enabling a table after master switch is not allowing table scan. Throws NotServingRegionException
> --------------------------------------------------------------------------------------------------
>
> Key: HBASE-4052
> URL: https://issues.apache.org/jira/browse/HBASE-4052
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.90.3
> Environment: Linux
> Reporter: ramkrishna.s.vasudevan
> Assignee: ramkrishna.s.vasudevan
> Fix For: 0.90.4
>
> Attachments: 4052.txt, Disabled.bmp, Disabling-1.bmp, Disabling-2.bmp, HBASE-4052-1-0.90_1.patch, HBASE-4052-1-_TestCode.patch, HBASE-4052-1-trunk_1.patch, HBASE-4052.patch, TestMasterRestartAfterDisablingTable.java
>
>
> Following is the scenario:
> Start RS and Active and standby masters
> Create table and insert data.
> Disable the table.
> Stop the active master and switch to the standby master.
> Now enable the table.
> Do a scan on the enabled table.
> NotServingRegionException is Thrown.
> But the same works well when we dont switch the master.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4052) Enabling a table after master switch
is not allowing table scan. Throws NotServingRegionException
Posted by "ramkrishna.s.vasudevan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ramkrishna.s.vasudevan updated HBASE-4052:
------------------------------------------
Attachment: HBASE-4052.patch
> Enabling a table after master switch is not allowing table scan. Throws NotServingRegionException
> --------------------------------------------------------------------------------------------------
>
> Key: HBASE-4052
> URL: https://issues.apache.org/jira/browse/HBASE-4052
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.90.3
> Environment: Linux
> Reporter: ramkrishna.s.vasudevan
> Fix For: 0.90.4
>
> Attachments: Disabled.bmp, Disabling-1.bmp, Disabling-2.bmp, HBASE-4052.patch, TestMasterRestartAfterDisablingTable.java
>
>
> Following is the scenario:
> Start RS and Active and standby masters
> Create table and insert data.
> Disable the table.
> Stop the active master and switch to the standby master.
> Now enable the table.
> Do a scan on the enabled table.
> NotServingRegionException is Thrown.
> But the same works well when we dont switch the master.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4052) Enabling a table after master
switch is not allowing table scan. Throws NotServingRegionException
Posted by "ramkrishna.s.vasudevan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13063741#comment-13063741 ]
ramkrishna.s.vasudevan commented on HBASE-4052:
-----------------------------------------------
Thanks Ted.
I will try looking into the issue in trunk why the restart or switch is not working.
Is there any other issue in the patch or solution? or can it be committed ?
> Enabling a table after master switch is not allowing table scan. Throws NotServingRegionException
> --------------------------------------------------------------------------------------------------
>
> Key: HBASE-4052
> URL: https://issues.apache.org/jira/browse/HBASE-4052
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.90.3
> Environment: Linux
> Reporter: ramkrishna.s.vasudevan
> Assignee: ramkrishna.s.vasudevan
> Fix For: 0.90.4
>
> Attachments: 4052.txt, Disabled.bmp, Disabling-1.bmp, Disabling-2.bmp, HBASE-4052-1-0.90_1.patch, HBASE-4052-1-_TestCode.patch, HBASE-4052-1-trunk_1.patch, HBASE-4052.patch, TestMasterRestartAfterDisablingTable.java
>
>
> Following is the scenario:
> Start RS and Active and standby masters
> Create table and insert data.
> Disable the table.
> Stop the active master and switch to the standby master.
> Now enable the table.
> Do a scan on the enabled table.
> NotServingRegionException is Thrown.
> But the same works well when we dont switch the master.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4052) Enabling a table after master
switch is not allowing table scan. Throws NotServingRegionException
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13063736#comment-13063736 ]
Ted Yu commented on HBASE-4052:
-------------------------------
TestMasterRestartAfterDisablingTable passed for 0.90 branch.
I think in TRUNK, the failure may be related to hung TestRestartCluster. See https://builds.apache.org/view/G-L/view/HBase/job/HBase-TRUNK/lastCompletedBuild/artifact/trunk/target/surefire-reports/org.apache.hadoop.hbase.master.TestRestartCluster-output.txt
> Enabling a table after master switch is not allowing table scan. Throws NotServingRegionException
> --------------------------------------------------------------------------------------------------
>
> Key: HBASE-4052
> URL: https://issues.apache.org/jira/browse/HBASE-4052
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.90.3
> Environment: Linux
> Reporter: ramkrishna.s.vasudevan
> Assignee: ramkrishna.s.vasudevan
> Fix For: 0.90.4
>
> Attachments: 4052.txt, Disabled.bmp, Disabling-1.bmp, Disabling-2.bmp, HBASE-4052-1-0.90_1.patch, HBASE-4052-1-_TestCode.patch, HBASE-4052-1-trunk_1.patch, HBASE-4052.patch, TestMasterRestartAfterDisablingTable.java
>
>
> Following is the scenario:
> Start RS and Active and standby masters
> Create table and insert data.
> Disable the table.
> Stop the active master and switch to the standby master.
> Now enable the table.
> Do a scan on the enabled table.
> NotServingRegionException is Thrown.
> But the same works well when we dont switch the master.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4052) Enabling a table after master
switch is not allowing table scan. Throws NotServingRegionException
Posted by "ramkrishna.s.vasudevan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13063842#comment-13063842 ]
ramkrishna.s.vasudevan commented on HBASE-4052:
-----------------------------------------------
Hi Ted,
I got the reason why we get the following error while executing the test case
{noformat}
testForCheckingIfEnableAndDisableWorksFineAfterSwitch(org.apache.hadoop.hbase.master.TestMasterRestartAfterDisablingTable) Time elapsed: 23.153 sec <<< ERROR!
java.lang.reflect.UndeclaredThrowableException
at $Proxy12.isMasterRunning(Unknown Source)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:553)
at org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:101)
at org.apache.hadoop.hbase.HBaseTestingUtility.getHBaseAdmin(HBaseTestingUtility.java:1162)
at org.apache.hadoop.hbase.master.TestMasterRestartAfterDisablingTab
{noformat}
I think this is also a bug in HBaseAdmin.getConnection().
The reason is
In HConnectionManager the connection object is cached based on the HConnectionKey.
The equals method checks the value of the CONNECTION PROPERTIES.
Suppose if we do a restart/switch of the master and again try to do an enable table operation then in the test code we will create a new HBaseAdmin object.
But the connection that the Admin creates to the Master is taken from the cache though it is a new connection.
Here none of the values in the CONNECTION PROPERTIES is changed so we get the same connection object when the previous master was active and hence though the master has been restarted we get the old active master address and hence an exception is thrown.
Workaround:
==========
So in order to pass the test case we change the value of one of the CONNECTION PROPERTIES so that the cached connection object is not returned.
I reverted the HBASE-4003 and this test case passed with the above change.
Not sure of the reason why RS doesnot checkin.
> Enabling a table after master switch is not allowing table scan. Throws NotServingRegionException
> --------------------------------------------------------------------------------------------------
>
> Key: HBASE-4052
> URL: https://issues.apache.org/jira/browse/HBASE-4052
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.90.3
> Environment: Linux
> Reporter: ramkrishna.s.vasudevan
> Assignee: ramkrishna.s.vasudevan
> Fix For: 0.90.4
>
> Attachments: 4052.txt, Disabled.bmp, Disabling-1.bmp, Disabling-2.bmp, HBASE-4052-1-0.90_1.patch, HBASE-4052-1-_TestCode.patch, HBASE-4052-1-trunk_1.patch, HBASE-4052.patch, TestMasterRestartAfterDisablingTable.java
>
>
> Following is the scenario:
> Start RS and Active and standby masters
> Create table and insert data.
> Disable the table.
> Stop the active master and switch to the standby master.
> Now enable the table.
> Do a scan on the enabled table.
> NotServingRegionException is Thrown.
> But the same works well when we dont switch the master.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4052) Enabling a table after master
switch is not allowing table scan. Throws NotServingRegionException
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13061659#comment-13061659 ]
stack commented on HBASE-4052:
------------------------------
The above looks good to me Ramkrishna.
> Enabling a table after master switch is not allowing table scan. Throws NotServingRegionException
> --------------------------------------------------------------------------------------------------
>
> Key: HBASE-4052
> URL: https://issues.apache.org/jira/browse/HBASE-4052
> Project: HBase
> Issue Type: Bug
> Environment: Linux
> Reporter: ramkrishna.s.vasudevan
>
> Following is the scenario:
> Start RS and Active and standby masters
> Create table and insert data.
> Disable the table.
> Stop the active master and switch to the standby master.
> Now enable the table.
> Do a scan on the enabled table.
> NotServingRegionException is Thrown.
> But the same works well when we dont switch the master.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4052) Enabling a table after master switch
is not allowing table scan. Throws NotServingRegionException
Posted by "ramkrishna.s.vasudevan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ramkrishna.s.vasudevan updated HBASE-4052:
------------------------------------------
Attachment: TestMasterRestartAfterDisablingTable.java
This is the test for the scenario 1
where the table state is DISABLED
> Enabling a table after master switch is not allowing table scan. Throws NotServingRegionException
> --------------------------------------------------------------------------------------------------
>
> Key: HBASE-4052
> URL: https://issues.apache.org/jira/browse/HBASE-4052
> Project: HBase
> Issue Type: Bug
> Environment: Linux
> Reporter: ramkrishna.s.vasudevan
> Attachments: Disabled.bmp, TestMasterRestartAfterDisablingTable.java
>
>
> Following is the scenario:
> Start RS and Active and standby masters
> Create table and insert data.
> Disable the table.
> Stop the active master and switch to the standby master.
> Now enable the table.
> Do a scan on the enabled table.
> NotServingRegionException is Thrown.
> But the same works well when we dont switch the master.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4052) Enabling a table after master
switch is not allowing table scan. Throws NotServingRegionException
Posted by "ramkrishna.s.vasudevan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13058215#comment-13058215 ]
ramkrishna.s.vasudevan commented on HBASE-4052:
-----------------------------------------------
As per my analysis the problem is that,
When we do a remove all the online regions are closed.
In the Enable table flow
{noformat}
private List<HRegionInfo> regionsToAssign(final List<HRegionInfo> regionsInMeta)
throws IOException {
final List<HRegionInfo> onlineRegions =
this.assignmentManager.getRegionsOfTable(tableName);
regionsInMeta.removeAll(onlineRegions);
return regionsInMeta;
}
{noformat}
We remove the regions if it is already online.
But as per the bug, enable is called after switching,
So while the standby master becomes active, in the rebuildUserRegion api,
we add all the regions from the Meta and we dont check if it is already disabled.
So when the flow comes to enable (regionsToAssign()) we consider the region to be onlined already.
Finally when we try to scan the table, we get NotServingRegionException.
Correct me if am wrong in my analysis.
> Enabling a table after master switch is not allowing table scan. Throws NotServingRegionException
> --------------------------------------------------------------------------------------------------
>
> Key: HBASE-4052
> URL: https://issues.apache.org/jira/browse/HBASE-4052
> Project: HBase
> Issue Type: Bug
> Environment: Linux
> Reporter: ramkrishna.s.vasudevan
>
> Following is the scenario:
> Start RS and Active and standby masters
> Create table and insert data.
> Disable the table.
> Stop the active master and switch to the standby master.
> Now enable the table.
> Do a scan on the enabled table.
> NotServingRegionException is Thrown.
> But the same works well when we dont switch the master.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4052) Enabling a table after master
switch does not allow table scan, throwing NotServingRegionException
Posted by "ramkrishna.s.vasudevan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13066775#comment-13066775 ]
ramkrishna.s.vasudevan commented on HBASE-4052:
-----------------------------------------------
Thanks for the review Ted and Stack.
> Enabling a table after master switch does not allow table scan, throwing NotServingRegionException
> --------------------------------------------------------------------------------------------------
>
> Key: HBASE-4052
> URL: https://issues.apache.org/jira/browse/HBASE-4052
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.90.3
> Environment: Linux
> Reporter: ramkrishna.s.vasudevan
> Assignee: ramkrishna.s.vasudevan
> Fix For: 0.90.4
>
> Attachments: 4052.txt, Disabled.bmp, Disabling-1.bmp, Disabling-2.bmp, HBASE-4052-1-0.90_1.patch, HBASE-4052-1-_TestCode.patch, HBASE-4052-1-trunk_1.patch, HBASE-4052.patch, TestMasterRestartAfterDisablingTable.java
>
>
> Following is the scenario:
> Start RS and Active and standby masters
> Create table and insert data.
> Disable the table.
> Stop the active master and switch to the standby master.
> Now enable the table.
> Do a scan on the enabled table.
> NotServingRegionException is Thrown.
> But the same works well when we dont switch the master.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4052) Enabling a table after master
switch is not allowing table scan. Throws NotServingRegionException
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13058860#comment-13058860 ]
stack commented on HBASE-4052:
------------------------------
What you say seems plausible Ramkrishna. I traced your reasoning above and yes, because rebuildUserRegion adds all regions without regard to whether table is enabled/disabled, when it comes to run the enable, when it asks what regions to enable, they all look as though they are already online because of what getRegionsOfTable returns (all regions that make up the table pulled from this.regions over in AM).
Good stuff.
Do you have a patch Ramkrishna?
> Enabling a table after master switch is not allowing table scan. Throws NotServingRegionException
> --------------------------------------------------------------------------------------------------
>
> Key: HBASE-4052
> URL: https://issues.apache.org/jira/browse/HBASE-4052
> Project: HBase
> Issue Type: Bug
> Environment: Linux
> Reporter: ramkrishna.s.vasudevan
>
> Following is the scenario:
> Start RS and Active and standby masters
> Create table and insert data.
> Disable the table.
> Stop the active master and switch to the standby master.
> Now enable the table.
> Do a scan on the enabled table.
> NotServingRegionException is Thrown.
> But the same works well when we dont switch the master.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4052) Enabling a table after master
switch does not allow table scan, throwing NotServingRegionException
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13066344#comment-13066344 ]
Ted Yu commented on HBASE-4052:
-------------------------------
Integrated to branch and TRUNK.
Thanks for the review Stack.
@ramkrishna:
In the future, please concatenate test file patch to main patch.
> Enabling a table after master switch does not allow table scan, throwing NotServingRegionException
> --------------------------------------------------------------------------------------------------
>
> Key: HBASE-4052
> URL: https://issues.apache.org/jira/browse/HBASE-4052
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.90.3
> Environment: Linux
> Reporter: ramkrishna.s.vasudevan
> Assignee: ramkrishna.s.vasudevan
> Fix For: 0.90.4
>
> Attachments: 4052.txt, Disabled.bmp, Disabling-1.bmp, Disabling-2.bmp, HBASE-4052-1-0.90_1.patch, HBASE-4052-1-_TestCode.patch, HBASE-4052-1-trunk_1.patch, HBASE-4052.patch, TestMasterRestartAfterDisablingTable.java
>
>
> Following is the scenario:
> Start RS and Active and standby masters
> Create table and insert data.
> Disable the table.
> Stop the active master and switch to the standby master.
> Now enable the table.
> Do a scan on the enabled table.
> NotServingRegionException is Thrown.
> But the same works well when we dont switch the master.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4052) Enabling a table after master
switch is not allowing table scan. Throws NotServingRegionException
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13064374#comment-13064374 ]
stack commented on HBASE-4052:
------------------------------
Patch looks good to me Ramkrishna. I tried to run your test but was getting NPE on trunk unrelated seemingly to your patch; I think this breaks tests 'HEAD is now at 46924a1... HBASE-3904 Addendum that fixes number of retries (Ita Pai)'
> Enabling a table after master switch is not allowing table scan. Throws NotServingRegionException
> --------------------------------------------------------------------------------------------------
>
> Key: HBASE-4052
> URL: https://issues.apache.org/jira/browse/HBASE-4052
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.90.3
> Environment: Linux
> Reporter: ramkrishna.s.vasudevan
> Assignee: ramkrishna.s.vasudevan
> Fix For: 0.90.4
>
> Attachments: 4052.txt, Disabled.bmp, Disabling-1.bmp, Disabling-2.bmp, HBASE-4052-1-0.90_1.patch, HBASE-4052-1-_TestCode.patch, HBASE-4052-1-trunk_1.patch, HBASE-4052.patch, TestMasterRestartAfterDisablingTable.java
>
>
> Following is the scenario:
> Start RS and Active and standby masters
> Create table and insert data.
> Disable the table.
> Stop the active master and switch to the standby master.
> Now enable the table.
> Do a scan on the enabled table.
> NotServingRegionException is Thrown.
> But the same works well when we dont switch the master.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4052) Enabling a table after master
switch is not allowing table scan. Throws NotServingRegionException
Posted by "ramkrishna.s.vasudevan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13060474#comment-13060474 ]
ramkrishna.s.vasudevan commented on HBASE-4052:
-----------------------------------------------
Hi,
Sorry for using wrong terminologies. I was aware that the Tables are only disabled and regions are only onlined or offlined.
I would like to attach the scenarios that needs to be addressed as part of this bug
{noformat}
If a table T1 has three regions R1, R2 and R3
We issue a disable command for T1
Scenario 1:
===========
All the regions R1, R2 and R3 are offlined and the state in zookeeper for the table is DISABLED and then the Active master went down,
R1- Offlined
R2- Offlined T1-DISABLED
R3-Offlined
This is straight forward scenario and the handling is also simple.
Scenario:2
==========
All the regions R1, R2 and R3 are offlined but the Active Master went down when the zookeeper state for the table was in DISABLING.
R1- Offlined
R2- Offlined T1-DISABLING
R3-Offlined
Scenario:3
========
The regions R1, R2 and R3 are not yet offlined and the Active Master went down when the zookeeper state was DISABLING.
R1- Online
R2- Online T1-DISABLING
R3-Online
Scenario:4
========
The regions R1, R2 are offlined and R3 are not yet offlined and the Active Master went down when the zookeeper state was DISABLING.
R1- offlined
R2- offlined T1-DISABLING
R3-Online
{noformat}
> Enabling a table after master switch is not allowing table scan. Throws NotServingRegionException
> --------------------------------------------------------------------------------------------------
>
> Key: HBASE-4052
> URL: https://issues.apache.org/jira/browse/HBASE-4052
> Project: HBase
> Issue Type: Bug
> Environment: Linux
> Reporter: ramkrishna.s.vasudevan
>
> Following is the scenario:
> Start RS and Active and standby masters
> Create table and insert data.
> Disable the table.
> Stop the active master and switch to the standby master.
> Now enable the table.
> Do a scan on the enabled table.
> NotServingRegionException is Thrown.
> But the same works well when we dont switch the master.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4052) Enabling a table after master switch
is not allowing table scan. Throws NotServingRegionException
Posted by "ramkrishna.s.vasudevan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ramkrishna.s.vasudevan updated HBASE-4052:
------------------------------------------
Attachment: HBASE-4052-1-trunk_1.patch
> Enabling a table after master switch is not allowing table scan. Throws NotServingRegionException
> --------------------------------------------------------------------------------------------------
>
> Key: HBASE-4052
> URL: https://issues.apache.org/jira/browse/HBASE-4052
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.90.3
> Environment: Linux
> Reporter: ramkrishna.s.vasudevan
> Assignee: ramkrishna.s.vasudevan
> Fix For: 0.90.4
>
> Attachments: 4052.txt, Disabled.bmp, Disabling-1.bmp, Disabling-2.bmp, HBASE-4052-1-0.90_1.patch, HBASE-4052-1-_TestCode.patch, HBASE-4052-1-trunk_1.patch, HBASE-4052.patch, TestMasterRestartAfterDisablingTable.java
>
>
> Following is the scenario:
> Start RS and Active and standby masters
> Create table and insert data.
> Disable the table.
> Stop the active master and switch to the standby master.
> Now enable the table.
> Do a scan on the enabled table.
> NotServingRegionException is Thrown.
> But the same works well when we dont switch the master.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4052) Enabling a table after master switch
does not allow table scan, throwing NotServingRegionException
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ted Yu updated HBASE-4052:
--------------------------
Summary: Enabling a table after master switch does not allow table scan, throwing NotServingRegionException (was: Enabling a table after master switch is not allowing table scan. Throws NotServingRegionException)
> Enabling a table after master switch does not allow table scan, throwing NotServingRegionException
> --------------------------------------------------------------------------------------------------
>
> Key: HBASE-4052
> URL: https://issues.apache.org/jira/browse/HBASE-4052
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.90.3
> Environment: Linux
> Reporter: ramkrishna.s.vasudevan
> Assignee: ramkrishna.s.vasudevan
> Fix For: 0.90.4
>
> Attachments: 4052.txt, Disabled.bmp, Disabling-1.bmp, Disabling-2.bmp, HBASE-4052-1-0.90_1.patch, HBASE-4052-1-_TestCode.patch, HBASE-4052-1-trunk_1.patch, HBASE-4052.patch, TestMasterRestartAfterDisablingTable.java
>
>
> Following is the scenario:
> Start RS and Active and standby masters
> Create table and insert data.
> Disable the table.
> Stop the active master and switch to the standby master.
> Now enable the table.
> Do a scan on the enabled table.
> NotServingRegionException is Thrown.
> But the same works well when we dont switch the master.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4052) Enabling a table after master
switch is not allowing table scan. Throws NotServingRegionException
Posted by "ramkrishna.s.vasudevan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13059088#comment-13059088 ]
ramkrishna.s.vasudevan commented on HBASE-4052:
-----------------------------------------------
Thanks Stack for your comments.
I am working on the patch.
I have 2 scenarios to consider
1. The AM may get killed when the state is in DISABLING but still the regions are not closed
2. The AM may get killed when the state is in DISABLING but the regions are closed.
So can we check both for DISABLING and DISABLED state.
I will provide my patch ASAP.
Thanks
> Enabling a table after master switch is not allowing table scan. Throws NotServingRegionException
> --------------------------------------------------------------------------------------------------
>
> Key: HBASE-4052
> URL: https://issues.apache.org/jira/browse/HBASE-4052
> Project: HBase
> Issue Type: Bug
> Environment: Linux
> Reporter: ramkrishna.s.vasudevan
>
> Following is the scenario:
> Start RS and Active and standby masters
> Create table and insert data.
> Disable the table.
> Stop the active master and switch to the standby master.
> Now enable the table.
> Do a scan on the enabled table.
> NotServingRegionException is Thrown.
> But the same works well when we dont switch the master.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4052) Enabling a table after master
switch is not allowing table scan. Throws NotServingRegionException
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13062014#comment-13062014 ]
Ted Yu commented on HBASE-4052:
-------------------------------
{code}
+ disabledTableRegions.add(disablingTableName);
{code}
Looks like disabledTableRegions should be renamed disablingTables.
> Enabling a table after master switch is not allowing table scan. Throws NotServingRegionException
> --------------------------------------------------------------------------------------------------
>
> Key: HBASE-4052
> URL: https://issues.apache.org/jira/browse/HBASE-4052
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.90.3
> Environment: Linux
> Reporter: ramkrishna.s.vasudevan
> Fix For: 0.90.4
>
> Attachments: Disabled.bmp, Disabling-1.bmp, Disabling-2.bmp, HBASE-4052.patch, TestMasterRestartAfterDisablingTable.java
>
>
> Following is the scenario:
> Start RS and Active and standby masters
> Create table and insert data.
> Disable the table.
> Stop the active master and switch to the standby master.
> Now enable the table.
> Do a scan on the enabled table.
> NotServingRegionException is Thrown.
> But the same works well when we dont switch the master.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4052) Enabling a table after master switch
is not allowing table scan. Throws NotServingRegionException
Posted by "ramkrishna.s.vasudevan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ramkrishna.s.vasudevan updated HBASE-4052:
------------------------------------------
Attachment: Disabling-2.bmp
> Enabling a table after master switch is not allowing table scan. Throws NotServingRegionException
> --------------------------------------------------------------------------------------------------
>
> Key: HBASE-4052
> URL: https://issues.apache.org/jira/browse/HBASE-4052
> Project: HBase
> Issue Type: Bug
> Environment: Linux
> Reporter: ramkrishna.s.vasudevan
> Attachments: Disabled.bmp, Disabling-1.bmp, Disabling-2.bmp, TestMasterRestartAfterDisablingTable.java
>
>
> Following is the scenario:
> Start RS and Active and standby masters
> Create table and insert data.
> Disable the table.
> Stop the active master and switch to the standby master.
> Now enable the table.
> Do a scan on the enabled table.
> NotServingRegionException is Thrown.
> But the same works well when we dont switch the master.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4052) Enabling a table after master switch
is not allowing table scan. Throws NotServingRegionException
Posted by "ramkrishna.s.vasudevan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ramkrishna.s.vasudevan updated HBASE-4052:
------------------------------------------
Attachment: Disabling-1.bmp
> Enabling a table after master switch is not allowing table scan. Throws NotServingRegionException
> --------------------------------------------------------------------------------------------------
>
> Key: HBASE-4052
> URL: https://issues.apache.org/jira/browse/HBASE-4052
> Project: HBase
> Issue Type: Bug
> Environment: Linux
> Reporter: ramkrishna.s.vasudevan
> Attachments: Disabled.bmp, Disabling-1.bmp, TestMasterRestartAfterDisablingTable.java
>
>
> Following is the scenario:
> Start RS and Active and standby masters
> Create table and insert data.
> Disable the table.
> Stop the active master and switch to the standby master.
> Now enable the table.
> Do a scan on the enabled table.
> NotServingRegionException is Thrown.
> But the same works well when we dont switch the master.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4052) Enabling a table after master switch
is not allowing table scan. Throws NotServingRegionException
Posted by "ramkrishna.s.vasudevan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ramkrishna.s.vasudevan updated HBASE-4052:
------------------------------------------
Status: Open (was: Patch Available)
> Enabling a table after master switch is not allowing table scan. Throws NotServingRegionException
> --------------------------------------------------------------------------------------------------
>
> Key: HBASE-4052
> URL: https://issues.apache.org/jira/browse/HBASE-4052
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.90.3
> Environment: Linux
> Reporter: ramkrishna.s.vasudevan
> Assignee: ramkrishna.s.vasudevan
> Fix For: 0.90.4
>
> Attachments: 4052.txt, Disabled.bmp, Disabling-1.bmp, Disabling-2.bmp, HBASE-4052-1-0.90_1.patch, HBASE-4052.patch, TestMasterRestartAfterDisablingTable.java
>
>
> Following is the scenario:
> Start RS and Active and standby masters
> Create table and insert data.
> Disable the table.
> Stop the active master and switch to the standby master.
> Now enable the table.
> Do a scan on the enabled table.
> NotServingRegionException is Thrown.
> But the same works well when we dont switch the master.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4052) Enabling a table after master
switch is not allowing table scan. Throws NotServingRegionException
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13063422#comment-13063422 ]
Ted Yu commented on HBASE-4052:
-------------------------------
I still got the following error (in TRUNK):
{noformat}
testForCheckingIfEnableAndDisableWorksFineAfterSwitch(org.apache.hadoop.hbase.master.TestMasterRestartAfterDisablingTable) Time elapsed: 23.153 sec <<< ERROR!
java.lang.reflect.UndeclaredThrowableException
at $Proxy12.isMasterRunning(Unknown Source)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:553)
at org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:101)
at org.apache.hadoop.hbase.HBaseTestingUtility.getHBaseAdmin(HBaseTestingUtility.java:1162)
at org.apache.hadoop.hbase.master.TestMasterRestartAfterDisablingTable.testForCheckingIfEnableAndDisableWorksFineAfterSwitch(TestMasterRestartAfterDisablingTable.java:100)
...
Caused by: java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcher.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:202)
at sun.nio.ch.IOUtil.read(IOUtil.java:175)
{noformat}
> Enabling a table after master switch is not allowing table scan. Throws NotServingRegionException
> --------------------------------------------------------------------------------------------------
>
> Key: HBASE-4052
> URL: https://issues.apache.org/jira/browse/HBASE-4052
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.90.3
> Environment: Linux
> Reporter: ramkrishna.s.vasudevan
> Assignee: ramkrishna.s.vasudevan
> Fix For: 0.90.4
>
> Attachments: 4052.txt, Disabled.bmp, Disabling-1.bmp, Disabling-2.bmp, HBASE-4052-1-0.90_1.patch, HBASE-4052-1-_TestCode.patch, HBASE-4052-1-trunk_1.patch, HBASE-4052.patch, TestMasterRestartAfterDisablingTable.java
>
>
> Following is the scenario:
> Start RS and Active and standby masters
> Create table and insert data.
> Disable the table.
> Stop the active master and switch to the standby master.
> Now enable the table.
> Do a scan on the enabled table.
> NotServingRegionException is Thrown.
> But the same works well when we dont switch the master.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-4052) Enabling a table after master switch
is not allowing table scan. Throws NotServingRegionException
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ted Yu reassigned HBASE-4052:
-----------------------------
Assignee: ramkrishna.s.vasudevan
> Enabling a table after master switch is not allowing table scan. Throws NotServingRegionException
> --------------------------------------------------------------------------------------------------
>
> Key: HBASE-4052
> URL: https://issues.apache.org/jira/browse/HBASE-4052
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.90.3
> Environment: Linux
> Reporter: ramkrishna.s.vasudevan
> Assignee: ramkrishna.s.vasudevan
> Fix For: 0.90.4
>
> Attachments: Disabled.bmp, Disabling-1.bmp, Disabling-2.bmp, HBASE-4052.patch, TestMasterRestartAfterDisablingTable.java
>
>
> Following is the scenario:
> Start RS and Active and standby masters
> Create table and insert data.
> Disable the table.
> Stop the active master and switch to the standby master.
> Now enable the table.
> Do a scan on the enabled table.
> NotServingRegionException is Thrown.
> But the same works well when we dont switch the master.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4052) Enabling a table after master
switch does not allow table scan, throwing NotServingRegionException
Posted by "ramkrishna.s.vasudevan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13066834#comment-13066834 ]
ramkrishna.s.vasudevan commented on HBASE-4052:
-----------------------------------------------
@Ted, one small correction in the patch that was applied
Actually the patch catches the NotServingRegionException as RemoteException and then
checks for the instanceof NotServingRegionException .
{noformat}
catch (Throwable t) {
if (t instanceof RemoteException) {
t = ((RemoteException)t).unwrapRemoteException();
}
{noformat}
But the latest AssignmentManager.java in trunk shows that it was applied in
{noformat}
catch (NotServingRegionException nsre) {
LOG.info("Server " + server + " returned " + nsre + " for " +
region.getEncodedName());
// Presume that master has stale data. Presume remote side just split.
// Presume that the split message when it comes in will fix up the master's
// in memory cluster state.
if (checkIfRegionBelongsToDisabling(region)) {
{noformat}
So the scenario of partial disabling is failing to recover back to DISABLED state.
Can you please check and reapply the patch.
> Enabling a table after master switch does not allow table scan, throwing NotServingRegionException
> --------------------------------------------------------------------------------------------------
>
> Key: HBASE-4052
> URL: https://issues.apache.org/jira/browse/HBASE-4052
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.90.3
> Environment: Linux
> Reporter: ramkrishna.s.vasudevan
> Assignee: ramkrishna.s.vasudevan
> Fix For: 0.90.4
>
> Attachments: 4052.txt, Disabled.bmp, Disabling-1.bmp, Disabling-2.bmp, HBASE-4052-1-0.90_1.patch, HBASE-4052-1-_TestCode.patch, HBASE-4052-1-trunk_1.patch, HBASE-4052.patch, TestMasterRestartAfterDisablingTable.java
>
>
> Following is the scenario:
> Start RS and Active and standby masters
> Create table and insert data.
> Disable the table.
> Stop the active master and switch to the standby master.
> Now enable the table.
> Do a scan on the enabled table.
> NotServingRegionException is Thrown.
> But the same works well when we dont switch the master.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4052) Enabling a table after master switch
is not allowing table scan. Throws NotServingRegionException
Posted by "ramkrishna.s.vasudevan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ramkrishna.s.vasudevan updated HBASE-4052:
------------------------------------------
Fix Version/s: 0.90.4
Affects Version/s: 0.90.3
Status: Patch Available (was: Open)
> Enabling a table after master switch is not allowing table scan. Throws NotServingRegionException
> --------------------------------------------------------------------------------------------------
>
> Key: HBASE-4052
> URL: https://issues.apache.org/jira/browse/HBASE-4052
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.90.3
> Environment: Linux
> Reporter: ramkrishna.s.vasudevan
> Fix For: 0.90.4
>
> Attachments: Disabled.bmp, Disabling-1.bmp, Disabling-2.bmp, HBASE-4052.patch, TestMasterRestartAfterDisablingTable.java
>
>
> Following is the scenario:
> Start RS and Active and standby masters
> Create table and insert data.
> Disable the table.
> Stop the active master and switch to the standby master.
> Now enable the table.
> Do a scan on the enabled table.
> NotServingRegionException is Thrown.
> But the same works well when we dont switch the master.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4052) Enabling a table after master
switch is not allowing table scan. Throws NotServingRegionException
Posted by "ramkrishna.s.vasudevan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13063728#comment-13063728 ]
ramkrishna.s.vasudevan commented on HBASE-4052:
-----------------------------------------------
Hi Ted,
I think the test case should work fine in the 0.90 branch.
In the trunk when we do a switch the RS is not able to connect to the new Master and we get
2011-07-12 11:01:22,602 INFO org.apache.hadoop.hbase.master.ServerManager: Waiting on regionserver(s) to checkin
2011-07-12 11:01:25,602 INFO org.apache.hadoop.hbase.master.ServerManager: Waiting on regionserver(s) to checkin
2011-07-12 11:01:28,602 INFO org.apache.hadoop.hbase.master.ServerManager: Waiting on regionserver(s) to checkin
So when this test case is executing also we get the same problem.
I will try to find why this behaviour is found in trunk and not in 0.90 branch. Can you please tell me if the
test case is passing in 0.90 branch. Because i have verified locally by running all the test cases in 0.90 branch and the testcases were passing.
{noformat}
Running org.apache.hadoop.hbase.master.TestMasterRestartAfterDisablingTable
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 26.148 sec
Running org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 45.383 sec
Running org.apache.hadoop.hbase.client.replication.TestReplicationAdmin
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.044 sec
Running org.apache.hadoop.hbase.regionserver.TestScanDeleteTracker
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.234 sec
Running org.apache.hadoop.hbase.client.TestMetaScanner
{noformat}
> Enabling a table after master switch is not allowing table scan. Throws NotServingRegionException
> --------------------------------------------------------------------------------------------------
>
> Key: HBASE-4052
> URL: https://issues.apache.org/jira/browse/HBASE-4052
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.90.3
> Environment: Linux
> Reporter: ramkrishna.s.vasudevan
> Assignee: ramkrishna.s.vasudevan
> Fix For: 0.90.4
>
> Attachments: 4052.txt, Disabled.bmp, Disabling-1.bmp, Disabling-2.bmp, HBASE-4052-1-0.90_1.patch, HBASE-4052-1-_TestCode.patch, HBASE-4052-1-trunk_1.patch, HBASE-4052.patch, TestMasterRestartAfterDisablingTable.java
>
>
> Following is the scenario:
> Start RS and Active and standby masters
> Create table and insert data.
> Disable the table.
> Stop the active master and switch to the standby master.
> Now enable the table.
> Do a scan on the enabled table.
> NotServingRegionException is Thrown.
> But the same works well when we dont switch the master.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4052) Enabling a table after master
switch is not allowing table scan. Throws NotServingRegionException
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13063744#comment-13063744 ]
Ted Yu commented on HBASE-4052:
-------------------------------
Patch for 0.90 looks good.
I think patch for TRUNK should be applied around same time as patch for 0.90 is applied.
Let's spend some more time on TRUNK although the cause for region server checkin problem may be somewhere else.
> Enabling a table after master switch is not allowing table scan. Throws NotServingRegionException
> --------------------------------------------------------------------------------------------------
>
> Key: HBASE-4052
> URL: https://issues.apache.org/jira/browse/HBASE-4052
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.90.3
> Environment: Linux
> Reporter: ramkrishna.s.vasudevan
> Assignee: ramkrishna.s.vasudevan
> Fix For: 0.90.4
>
> Attachments: 4052.txt, Disabled.bmp, Disabling-1.bmp, Disabling-2.bmp, HBASE-4052-1-0.90_1.patch, HBASE-4052-1-_TestCode.patch, HBASE-4052-1-trunk_1.patch, HBASE-4052.patch, TestMasterRestartAfterDisablingTable.java
>
>
> Following is the scenario:
> Start RS and Active and standby masters
> Create table and insert data.
> Disable the table.
> Stop the active master and switch to the standby master.
> Now enable the table.
> Do a scan on the enabled table.
> NotServingRegionException is Thrown.
> But the same works well when we dont switch the master.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4052) Enabling a table after master
switch is not allowing table scan. Throws NotServingRegionException
Posted by "ramkrishna.s.vasudevan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13061813#comment-13061813 ]
ramkrishna.s.vasudevan commented on HBASE-4052:
-----------------------------------------------
The .bmp files has the sequence of changes that needs to be done.
The other 3 scenarios is difficult to reproduce through testcase.
Please let me know if the solution is fine.
Provide your comments and any scenarios needs to be verified.
I am working on the patch and testing it. Will upload the patch sooner.
> Enabling a table after master switch is not allowing table scan. Throws NotServingRegionException
> --------------------------------------------------------------------------------------------------
>
> Key: HBASE-4052
> URL: https://issues.apache.org/jira/browse/HBASE-4052
> Project: HBase
> Issue Type: Bug
> Environment: Linux
> Reporter: ramkrishna.s.vasudevan
> Attachments: Disabled.bmp, Disabling-1.bmp, Disabling-2.bmp, TestMasterRestartAfterDisablingTable.java
>
>
> Following is the scenario:
> Start RS and Active and standby masters
> Create table and insert data.
> Disable the table.
> Stop the active master and switch to the standby master.
> Now enable the table.
> Do a scan on the enabled table.
> NotServingRegionException is Thrown.
> But the same works well when we dont switch the master.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4052) Enabling a table after master
switch is not allowing table scan. Throws NotServingRegionException
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13064438#comment-13064438 ]
Ted Yu commented on HBASE-4052:
-------------------------------
I applied an addendum for HBASE-3904 v6 to TRUNK. NPE is fixed.
However, HBASE-4087 is required for the new unit test to pass.
> Enabling a table after master switch is not allowing table scan. Throws NotServingRegionException
> --------------------------------------------------------------------------------------------------
>
> Key: HBASE-4052
> URL: https://issues.apache.org/jira/browse/HBASE-4052
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.90.3
> Environment: Linux
> Reporter: ramkrishna.s.vasudevan
> Assignee: ramkrishna.s.vasudevan
> Fix For: 0.90.4
>
> Attachments: 4052.txt, Disabled.bmp, Disabling-1.bmp, Disabling-2.bmp, HBASE-4052-1-0.90_1.patch, HBASE-4052-1-_TestCode.patch, HBASE-4052-1-trunk_1.patch, HBASE-4052.patch, TestMasterRestartAfterDisablingTable.java
>
>
> Following is the scenario:
> Start RS and Active and standby masters
> Create table and insert data.
> Disable the table.
> Stop the active master and switch to the standby master.
> Now enable the table.
> Do a scan on the enabled table.
> NotServingRegionException is Thrown.
> But the same works well when we dont switch the master.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4052) Enabling a table after master
switch is not allowing table scan. Throws NotServingRegionException
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13062297#comment-13062297 ]
Ted Yu commented on HBASE-4052:
-------------------------------
When I tried to apply your patch on 0.90 branch:
{noformat}
tyu-mbp:90hbase tyu$ patch -p0 -i HBASE-4052.patch
(Stripping trailing CRs from patch.)
patching file src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java
Hunk #1 succeeded at 59 with fuzz 1 (offset -1 lines).
Hunk #2 FAILED at 1152.
Hunk #3 FAILED at 1445.
Hunk #4 FAILED at 1460.
Hunk #5 succeeded at 1570 (offset 86 lines).
3 out of 5 hunks FAILED -- saving rejects to file src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java.rej
(Stripping trailing CRs from patch.)
patching file src/test/java/org/apache/hadoop/hbase/master/TestMasterRestartAfterDisablingTable.java
{noformat}
Please come up with clean patch for 0.90 branch. There have been 39 bug fixes after release of 0.90.3
Since this is a bug, the fix will go to both 0.90 and TRUNK.
> Enabling a table after master switch is not allowing table scan. Throws NotServingRegionException
> --------------------------------------------------------------------------------------------------
>
> Key: HBASE-4052
> URL: https://issues.apache.org/jira/browse/HBASE-4052
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.90.3
> Environment: Linux
> Reporter: ramkrishna.s.vasudevan
> Assignee: ramkrishna.s.vasudevan
> Fix For: 0.90.4
>
> Attachments: 4052.txt, Disabled.bmp, Disabling-1.bmp, Disabling-2.bmp, HBASE-4052.patch, TestMasterRestartAfterDisablingTable.java
>
>
> Following is the scenario:
> Start RS and Active and standby masters
> Create table and insert data.
> Disable the table.
> Stop the active master and switch to the standby master.
> Now enable the table.
> Do a scan on the enabled table.
> NotServingRegionException is Thrown.
> But the same works well when we dont switch the master.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4052) Enabling a table after master
switch is not allowing table scan. Throws NotServingRegionException
Posted by "ramkrishna.s.vasudevan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13059951#comment-13059951 ]
ramkrishna.s.vasudevan commented on HBASE-4052:
-----------------------------------------------
I have some doubts and like to get some suggestion before proceeding.
Following scenarios needs to be considered.
Scenario 1:
===========
All the regions are disabled and the state in zookeeper is DISABLED.
Scenario:2
==========
The regions are offlined but the AM went down when the zookeeper state was DISABLING.
Scenario:3
=========
The regions are not yet offlined(or only few regions are offlined) and the AM went down when the zookeeper state was DISABLING.
Now when we do a switch of the master or on restart scenario of master,
how can we decide which regions were offlined and which are not.
Though we can get the state of the table as either DISABLED or DISABLING, region wise i am not able to infer in what state the region is.
So what brings me to get this info is
The soln should be like we need to check for the state of the table while populating the regions map in master startup.
Checking only for DISABLED state:
==========================
Check for disabled state and those regions that are not in the DISABLED state add it into the regions map in master startup.
If i check only for the DISABLED state and if the table is in DISABLING state and
after master retry (or switch) if i try to enable then we will not be able to scan the table because while enabling none of the regions will be enabled
as the regions in META table and the regions that i have populated in the regions map are same.
So I will be getting the same issue as in the description of the defect.
Checking for DISABLED and DISABLING state:
===================================
if i check the state of the zookeeper for DISABLED and DISABLING and while restart of master(switch) only those regions which are not in DISABLED or DISABLING state is populated.
When i again try to enable the region if the region was not offlined as part of disable flow(Scenario:3), the waitUntilDone in BulkAssigner is not aware that the region was
already onlined and keeps on waiting as the waitUntilDone() sees for the number of regions to become online from the regions map and the actual count it gets from the meta table.
This makes enable to go in a loop.
Am i clear with the problem? so is it like before enabling any table do we need to check the state of the table and if it is DISABLING make all those regions to go to
offline mode.
> Enabling a table after master switch is not allowing table scan. Throws NotServingRegionException
> --------------------------------------------------------------------------------------------------
>
> Key: HBASE-4052
> URL: https://issues.apache.org/jira/browse/HBASE-4052
> Project: HBase
> Issue Type: Bug
> Environment: Linux
> Reporter: ramkrishna.s.vasudevan
>
> Following is the scenario:
> Start RS and Active and standby masters
> Create table and insert data.
> Disable the table.
> Stop the active master and switch to the standby master.
> Now enable the table.
> Do a scan on the enabled table.
> NotServingRegionException is Thrown.
> But the same works well when we dont switch the master.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4052) Enabling a table after master
switch is not allowing table scan. Throws NotServingRegionException
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13062259#comment-13062259 ]
Ted Yu commented on HBASE-4052:
-------------------------------
This test failure is new:
{code}
testRPCException(org.apache.hadoop.hbase.master.TestHMasterRPCException) Time elapsed: 1.543 sec <<< FAILURE!
java.lang.AssertionError: Unexpected throwable: java.net.SocketTimeoutException: Call to us01-ciqps1-grid06.carrieriq.com/10.202.50.106:19386 failed on socket timeout exception: java.net.SocketTimeoutException: 100 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/10.202.50.106:48181 remote=us01-ciqps1-grid06.carrieriq.com/10.202.50.106:19386]
at org.junit.Assert.fail(Assert.java:91)
at org.apache.hadoop.hbase.master.TestHMasterRPCException.testRPCException(TestHMasterRPCException.java:59)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
{code}
> Enabling a table after master switch is not allowing table scan. Throws NotServingRegionException
> --------------------------------------------------------------------------------------------------
>
> Key: HBASE-4052
> URL: https://issues.apache.org/jira/browse/HBASE-4052
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.90.3
> Environment: Linux
> Reporter: ramkrishna.s.vasudevan
> Assignee: ramkrishna.s.vasudevan
> Fix For: 0.90.4
>
> Attachments: 4052.txt, Disabled.bmp, Disabling-1.bmp, Disabling-2.bmp, HBASE-4052.patch, TestMasterRestartAfterDisablingTable.java
>
>
> Following is the scenario:
> Start RS and Active and standby masters
> Create table and insert data.
> Disable the table.
> Stop the active master and switch to the standby master.
> Now enable the table.
> Do a scan on the enabled table.
> NotServingRegionException is Thrown.
> But the same works well when we dont switch the master.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4052) Enabling a table after master
switch is not allowing table scan. Throws NotServingRegionException
Posted by "ramkrishna.s.vasudevan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13062097#comment-13062097 ]
ramkrishna.s.vasudevan commented on HBASE-4052:
-----------------------------------------------
Yes that can be renamed. I will change it.
Any scenarios to be verified Ted?
> Enabling a table after master switch is not allowing table scan. Throws NotServingRegionException
> --------------------------------------------------------------------------------------------------
>
> Key: HBASE-4052
> URL: https://issues.apache.org/jira/browse/HBASE-4052
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.90.3
> Environment: Linux
> Reporter: ramkrishna.s.vasudevan
> Assignee: ramkrishna.s.vasudevan
> Fix For: 0.90.4
>
> Attachments: Disabled.bmp, Disabling-1.bmp, Disabling-2.bmp, HBASE-4052.patch, TestMasterRestartAfterDisablingTable.java
>
>
> Following is the scenario:
> Start RS and Active and standby masters
> Create table and insert data.
> Disable the table.
> Stop the active master and switch to the standby master.
> Now enable the table.
> Do a scan on the enabled table.
> NotServingRegionException is Thrown.
> But the same works well when we dont switch the master.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4052) Enabling a table after master
switch is not allowing table scan. Throws NotServingRegionException
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13060052#comment-13060052 ]
Ted Yu commented on HBASE-4052:
-------------------------------
I think one source of confusion in the above description is that DISABLED and DISABLING states are TableState.
There is no concept of disabling/disabled region.
I would opt for Checking for DISABLED and DISABLING TableState.
I think the BulkAssigner in this case refers to EnableTableHandler.BulkEnabler which depends on assignmentManager.getRegionsOfTable() for the number of online regions.
You need to write a new method, e.g. assignmentManager.getOnlineRegionsOfTable(), and call it in place of assignmentManager.getRegionsOfTable() in EnableTableHandler.regionsToAssign()
> Enabling a table after master switch is not allowing table scan. Throws NotServingRegionException
> --------------------------------------------------------------------------------------------------
>
> Key: HBASE-4052
> URL: https://issues.apache.org/jira/browse/HBASE-4052
> Project: HBase
> Issue Type: Bug
> Environment: Linux
> Reporter: ramkrishna.s.vasudevan
>
> Following is the scenario:
> Start RS and Active and standby masters
> Create table and insert data.
> Disable the table.
> Stop the active master and switch to the standby master.
> Now enable the table.
> Do a scan on the enabled table.
> NotServingRegionException is Thrown.
> But the same works well when we dont switch the master.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4052) Enabling a table after master switch
does not allow table scan, throwing NotServingRegionException
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ted Yu updated HBASE-4052:
--------------------------
Resolution: Fixed
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
> Enabling a table after master switch does not allow table scan, throwing NotServingRegionException
> --------------------------------------------------------------------------------------------------
>
> Key: HBASE-4052
> URL: https://issues.apache.org/jira/browse/HBASE-4052
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.90.3
> Environment: Linux
> Reporter: ramkrishna.s.vasudevan
> Assignee: ramkrishna.s.vasudevan
> Fix For: 0.90.4
>
> Attachments: 4052.txt, Disabled.bmp, Disabling-1.bmp, Disabling-2.bmp, HBASE-4052-1-0.90_1.patch, HBASE-4052-1-_TestCode.patch, HBASE-4052-1-trunk_1.patch, HBASE-4052.patch, TestMasterRestartAfterDisablingTable.java
>
>
> Following is the scenario:
> Start RS and Active and standby masters
> Create table and insert data.
> Disable the table.
> Stop the active master and switch to the standby master.
> Now enable the table.
> Do a scan on the enabled table.
> NotServingRegionException is Thrown.
> But the same works well when we dont switch the master.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4052) Enabling a table after master
switch is not allowing table scan. Throws NotServingRegionException
Posted by "ramkrishna.s.vasudevan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13062295#comment-13062295 ]
ramkrishna.s.vasudevan commented on HBASE-4052:
-----------------------------------------------
Hi ted
I had taken the 0.90 branch code
and applied the patch on that.
So do I need to apply patch only
on the trunk?
Also the test case ran cleanly in
my setup anyway will check it
Sorry for the mistakes
I will rectify them .
> Enabling a table after master switch is not allowing table scan. Throws NotServingRegionException
> --------------------------------------------------------------------------------------------------
>
> Key: HBASE-4052
> URL: https://issues.apache.org/jira/browse/HBASE-4052
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.90.3
> Environment: Linux
> Reporter: ramkrishna.s.vasudevan
> Assignee: ramkrishna.s.vasudevan
> Fix For: 0.90.4
>
> Attachments: 4052.txt, Disabled.bmp, Disabling-1.bmp, Disabling-2.bmp, HBASE-4052.patch, TestMasterRestartAfterDisablingTable.java
>
>
> Following is the scenario:
> Start RS and Active and standby masters
> Create table and insert data.
> Disable the table.
> Stop the active master and switch to the standby master.
> Now enable the table.
> Do a scan on the enabled table.
> NotServingRegionException is Thrown.
> But the same works well when we dont switch the master.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4052) Enabling a table after master
switch does not allow table scan, throwing NotServingRegionException
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13066907#comment-13066907 ]
Ted Yu commented on HBASE-4052:
-------------------------------
Handling of NotServingRegionException has been moved to the last catch block.
> Enabling a table after master switch does not allow table scan, throwing NotServingRegionException
> --------------------------------------------------------------------------------------------------
>
> Key: HBASE-4052
> URL: https://issues.apache.org/jira/browse/HBASE-4052
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.90.3
> Environment: Linux
> Reporter: ramkrishna.s.vasudevan
> Assignee: ramkrishna.s.vasudevan
> Fix For: 0.90.4
>
> Attachments: 4052.txt, Disabled.bmp, Disabling-1.bmp, Disabling-2.bmp, HBASE-4052-1-0.90_1.patch, HBASE-4052-1-_TestCode.patch, HBASE-4052-1-trunk_1.patch, HBASE-4052.patch, TestMasterRestartAfterDisablingTable.java
>
>
> Following is the scenario:
> Start RS and Active and standby masters
> Create table and insert data.
> Disable the table.
> Stop the active master and switch to the standby master.
> Now enable the table.
> Do a scan on the enabled table.
> NotServingRegionException is Thrown.
> But the same works well when we dont switch the master.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4052) Enabling a table after master
switch is not allowing table scan. Throws NotServingRegionException
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13064080#comment-13064080 ]
Ted Yu commented on HBASE-4052:
-------------------------------
The following test consistently failed on Linux with patch for 0.90:
{noformat}
queueFailover(org.apache.hadoop.hbase.replication.TestReplication) Time elapsed: 85.997 sec <<< FAILURE!
java.lang.AssertionError: Waited too much time for queueFailover replication
at org.junit.Assert.fail(Assert.java:91)
at org.apache.hadoop.hbase.replication.TestReplication.queueFailover(TestReplication.java:572)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
{noformat}
> Enabling a table after master switch is not allowing table scan. Throws NotServingRegionException
> --------------------------------------------------------------------------------------------------
>
> Key: HBASE-4052
> URL: https://issues.apache.org/jira/browse/HBASE-4052
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.90.3
> Environment: Linux
> Reporter: ramkrishna.s.vasudevan
> Assignee: ramkrishna.s.vasudevan
> Fix For: 0.90.4
>
> Attachments: 4052.txt, Disabled.bmp, Disabling-1.bmp, Disabling-2.bmp, HBASE-4052-1-0.90_1.patch, HBASE-4052-1-_TestCode.patch, HBASE-4052-1-trunk_1.patch, HBASE-4052.patch, TestMasterRestartAfterDisablingTable.java
>
>
> Following is the scenario:
> Start RS and Active and standby masters
> Create table and insert data.
> Disable the table.
> Stop the active master and switch to the standby master.
> Now enable the table.
> Do a scan on the enabled table.
> NotServingRegionException is Thrown.
> But the same works well when we dont switch the master.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4052) Enabling a table after master switch
is not allowing table scan. Throws NotServingRegionException
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ted Yu updated HBASE-4052:
--------------------------
Attachment: 4052.txt
Patch for TRUNK.
ramkrishna's patch seems to be based on an old version of 0.90 which wouldn't apply cleanly on 0.90 branch
Also the indentation was off on several lines.
Please use the following for new files added:
{code}
+ * Copyright 2011 The Apache Software Foundation
{code}
> Enabling a table after master switch is not allowing table scan. Throws NotServingRegionException
> --------------------------------------------------------------------------------------------------
>
> Key: HBASE-4052
> URL: https://issues.apache.org/jira/browse/HBASE-4052
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.90.3
> Environment: Linux
> Reporter: ramkrishna.s.vasudevan
> Assignee: ramkrishna.s.vasudevan
> Fix For: 0.90.4
>
> Attachments: 4052.txt, Disabled.bmp, Disabling-1.bmp, Disabling-2.bmp, HBASE-4052.patch, TestMasterRestartAfterDisablingTable.java
>
>
> Following is the scenario:
> Start RS and Active and standby masters
> Create table and insert data.
> Disable the table.
> Stop the active master and switch to the standby master.
> Now enable the table.
> Do a scan on the enabled table.
> NotServingRegionException is Thrown.
> But the same works well when we dont switch the master.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4052) Enabling a table after master switch
is not allowing table scan. Throws NotServingRegionException
Posted by "ramkrishna.s.vasudevan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ramkrishna.s.vasudevan updated HBASE-4052:
------------------------------------------
Attachment: HBASE-4052-1-0.90_1.patch
> Enabling a table after master switch is not allowing table scan. Throws NotServingRegionException
> --------------------------------------------------------------------------------------------------
>
> Key: HBASE-4052
> URL: https://issues.apache.org/jira/browse/HBASE-4052
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.90.3
> Environment: Linux
> Reporter: ramkrishna.s.vasudevan
> Assignee: ramkrishna.s.vasudevan
> Fix For: 0.90.4
>
> Attachments: 4052.txt, Disabled.bmp, Disabling-1.bmp, Disabling-2.bmp, HBASE-4052-1-0.90_1.patch, HBASE-4052.patch, TestMasterRestartAfterDisablingTable.java
>
>
> Following is the scenario:
> Start RS and Active and standby masters
> Create table and insert data.
> Disable the table.
> Stop the active master and switch to the standby master.
> Now enable the table.
> Do a scan on the enabled table.
> NotServingRegionException is Thrown.
> But the same works well when we dont switch the master.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4052) Enabling a table after master
switch is not allowing table scan. Throws NotServingRegionException
Posted by "ramkrishna.s.vasudevan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13064338#comment-13064338 ]
ramkrishna.s.vasudevan commented on HBASE-4052:
-----------------------------------------------
Ted, will look into the failure of org.apache.hadoop.hbase.replication.TestReplication.
But my local test bed shows success for the same testcase..
{noformat}
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.031 sec
Running org.apache.hadoop.hbase.rest.TestStatusResource
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 23.785 sec
Running org.apache.hadoop.hbase.executor.TestExecutorService
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.324 sec
Running org.apache.hadoop.hbase.client.TestFromClientSide
Tests run: 42, Failures: 0, Errors: 0, Skipped: 3, Time elapsed: 279.871 sec
Running org.apache.hadoop.hbase.replication.TestReplication
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 213.828 sec
{noformat}
Anyways will look into it if this patch is the root cause.
> Enabling a table after master switch is not allowing table scan. Throws NotServingRegionException
> --------------------------------------------------------------------------------------------------
>
> Key: HBASE-4052
> URL: https://issues.apache.org/jira/browse/HBASE-4052
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.90.3
> Environment: Linux
> Reporter: ramkrishna.s.vasudevan
> Assignee: ramkrishna.s.vasudevan
> Fix For: 0.90.4
>
> Attachments: 4052.txt, Disabled.bmp, Disabling-1.bmp, Disabling-2.bmp, HBASE-4052-1-0.90_1.patch, HBASE-4052-1-_TestCode.patch, HBASE-4052-1-trunk_1.patch, HBASE-4052.patch, TestMasterRestartAfterDisablingTable.java
>
>
> Following is the scenario:
> Start RS and Active and standby masters
> Create table and insert data.
> Disable the table.
> Stop the active master and switch to the standby master.
> Now enable the table.
> Do a scan on the enabled table.
> NotServingRegionException is Thrown.
> But the same works well when we dont switch the master.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira