You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org> on 2009/12/22 03:54:18 UTC
[jira] Created: (HBASE-2065) Cannot disable a table if any of its
region is opening at the same
Cannot disable a table if any of its region is opening at the same
------------------------------------------------------------------
Key: HBASE-2065
URL: https://issues.apache.org/jira/browse/HBASE-2065
Project: Hadoop HBase
Issue Type: Sub-task
Affects Versions: 0.20.2
Reporter: Jean-Daniel Cryans
Assignee: Jean-Daniel Cryans
Fix For: 0.20.3, 0.21.0
Also found with the test in the parent jira:
{code}
2009-12-21 18:31:44,411 INFO [IPC Server handler 0 on 60000] master.RegionManager(331): Assigning region table113,,1261449026166 to 10.10.1.54,60853,1261448823301
2009-12-21 18:31:44,411 INFO [IPC Server handler 0 on 60000] master.RegionManager(331): Assigning region table121,,1261449041385 to 10.10.1.54,60853,1261448823301
2009-12-21 18:31:44,411 INFO [RegionServer:1] regionserver.HRegionServer(475): MSG_REGION_OPEN: table113,,1261449026166
2009-12-21 18:31:44,411 INFO [RegionServer:1] regionserver.HRegionServer(475): MSG_REGION_OPEN: table121,,1261449041385
...
2009-12-21 18:31:44,418 INFO [RegionServer:1.worker] regionserver.HRegion(343): region table113,,1261449026166/21044806 available; sequence id is 0
...
2009-12-21 18:31:44,445 DEBUG [IPC Server handler 4 on 60000] master.ChangeTableState(121): Adding region table113,,1261449026166 to setClosing list
2009-12-21 18:31:44,446 DEBUG [main] zookeeper.ZooKeeperWrapper(392): Read ZNode /hbase/root-region-server got 10.10.1.54:608532009-12-21 18:31:44,447 DEBUG [main] client.HConnectionManager$TableServers(990): Found ROOT at 10.10.1.54:60853
2009-12-21 18:31:44,447 DEBUG [main] client.HConnectionManager$TableServers(899): Cached location for .META.,,1 is 10.10.1.54:608552009-12-21 18:31:44,453 DEBUG [main] client.HConnectionManager$TableServers(554): Rowscanned=1, rowsOffline=0
2009-12-21 18:31:44,454 DEBUG [main] client.HBaseAdmin(397): Sleep. Waiting for all regions to be disabled from table1132009-12-21 18:31:44,554 DEBUG [main] client.HBaseAdmin(406): Wake. Waiting for all regions to be disabled from table113
...
2009-12-21 18:31:44,642 INFO [RegionServer:0] regionserver.HRegionServer(475): MSG_REGION_CLOSE: table113,,1261449026166
...
2009-12-21 18:31:44,642 INFO [RegionServer:0.worker] regionserver.HRegionServer$Worker(1332): Worker: MSG_REGION_CLOSE: table113,,1261449026166
...
2009-12-21 18:31:44,664 INFO [IPC Server handler 0 on 60000] master.ServerManager(421): Processing MSG_REPORT_PROCESS_OPEN: table113,,1261449026166 from 10.10.1.54,60853,1261448823301; 1 of 4
...
2009-12-21 18:31:44,664 INFO [IPC Server handler 0 on 60000] master.ServerManager(421): Processing MSG_REPORT_OPEN: table113,,1261449026166 from 10.10.1.54,60853,1261448823301; 3 of 4
2009-12-21 18:31:44,664 DEBUG [IPC Server handler 0 on 60000] master.ServerManager(562): region server 10.10.1.54:60853 should not have opened region table113,,1261449026166
2009-12-21 18:31:44,666 INFO [RegionServer:1] regionserver.HRegionServer(475): MSG_REGION_CLOSE_WITHOUT_REPORT: table113,,1261449026166: Duplicate assignment
2009-12-21 18:31:44,666 INFO [RegionServer:1.worker] regionserver.HRegionServer$Worker(1332): Worker: MSG_REGION_CLOSE_WITHOUT_REPORT: table113,,1261449026166: Duplicate assignment
{code}
Here the master reassigned table13 and told the old region server to close the region before the new one was able to report that it opened it. At the end the new region server (good one) is also told to close it After that my test times out, table13 is not disabled neither it is deployed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-2065) Cannot disable a table if any of its
region is opening at the same time
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jean-Daniel Cryans updated HBASE-2065:
--------------------------------------
Attachment: HBASE-2065-2.patch
This new patch attempts to fix the issue seen in Hudson (a region being opened while table is disabled was falling in a hole) as well as a new issue found with more testing: a region closed and set unassigned by the master is seen by TableOperation as an assigned region since the server info is still there so it's then switched from "unassigned" to "closing" by ChangeTableState. The rever is never reassigned nor seen as closed.
More comments:
TestAdmin:
- I added a more difficult test where we enable and disable so root out more issues.
HRegion:
- Refactored offlineRegionInMETA to add a new method called removeServerInfoInMETA which does pretty much that.
ProcessRegionClose:
- Calls the new HRegion method in order to clean the .META. entry.
ChangeTableState:
- Added that if we are disabling a table and we see a pending open region, that we do not attempt to mark it as "closing" since the master will be confused when the region server reports opening the region. We will rely now on the next modif in HBA...
HBaseAdmin:
- Changed that when enabling/disabling, instead of calling the Master's method and wait, we call it on every iteration to take into account regions that are moving like a pending open.
> Cannot disable a table if any of its region is opening at the same time
> -----------------------------------------------------------------------
>
> Key: HBASE-2065
> URL: https://issues.apache.org/jira/browse/HBASE-2065
> Project: Hadoop HBase
> Issue Type: Sub-task
> Affects Versions: 0.20.2
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.20.3, 0.21.0
>
> Attachments: HBASE-2065-2.patch, HBASE-2065-branch.patch, HBASE-2065.patch
>
>
> Also found with the test in the parent jira:
> {code}
> 2009-12-21 18:31:44,411 INFO [IPC Server handler 0 on 60000] master.RegionManager(331): Assigning region table113,,1261449026166 to 10.10.1.54,60853,1261448823301
> 2009-12-21 18:31:44,411 INFO [IPC Server handler 0 on 60000] master.RegionManager(331): Assigning region table121,,1261449041385 to 10.10.1.54,60853,1261448823301
> 2009-12-21 18:31:44,411 INFO [RegionServer:1] regionserver.HRegionServer(475): MSG_REGION_OPEN: table113,,1261449026166
> 2009-12-21 18:31:44,411 INFO [RegionServer:1] regionserver.HRegionServer(475): MSG_REGION_OPEN: table121,,1261449041385
> ...
> 2009-12-21 18:31:44,418 INFO [RegionServer:1.worker] regionserver.HRegion(343): region table113,,1261449026166/21044806 available; sequence id is 0
> ...
> 2009-12-21 18:31:44,445 DEBUG [IPC Server handler 4 on 60000] master.ChangeTableState(121): Adding region table113,,1261449026166 to setClosing list
> 2009-12-21 18:31:44,446 DEBUG [main] zookeeper.ZooKeeperWrapper(392): Read ZNode /hbase/root-region-server got 10.10.1.54:608532009-12-21 18:31:44,447 DEBUG [main] client.HConnectionManager$TableServers(990): Found ROOT at 10.10.1.54:60853
> 2009-12-21 18:31:44,447 DEBUG [main] client.HConnectionManager$TableServers(899): Cached location for .META.,,1 is 10.10.1.54:608552009-12-21 18:31:44,453 DEBUG [main] client.HConnectionManager$TableServers(554): Rowscanned=1, rowsOffline=0
> 2009-12-21 18:31:44,454 DEBUG [main] client.HBaseAdmin(397): Sleep. Waiting for all regions to be disabled from table1132009-12-21 18:31:44,554 DEBUG [main] client.HBaseAdmin(406): Wake. Waiting for all regions to be disabled from table113
> ...
> 2009-12-21 18:31:44,642 INFO [RegionServer:0] regionserver.HRegionServer(475): MSG_REGION_CLOSE: table113,,1261449026166
> ...
> 2009-12-21 18:31:44,642 INFO [RegionServer:0.worker] regionserver.HRegionServer$Worker(1332): Worker: MSG_REGION_CLOSE: table113,,1261449026166
> ...
> 2009-12-21 18:31:44,664 INFO [IPC Server handler 0 on 60000] master.ServerManager(421): Processing MSG_REPORT_PROCESS_OPEN: table113,,1261449026166 from 10.10.1.54,60853,1261448823301; 1 of 4
> ...
> 2009-12-21 18:31:44,664 INFO [IPC Server handler 0 on 60000] master.ServerManager(421): Processing MSG_REPORT_OPEN: table113,,1261449026166 from 10.10.1.54,60853,1261448823301; 3 of 4
> 2009-12-21 18:31:44,664 DEBUG [IPC Server handler 0 on 60000] master.ServerManager(562): region server 10.10.1.54:60853 should not have opened region table113,,1261449026166
> 2009-12-21 18:31:44,666 INFO [RegionServer:1] regionserver.HRegionServer(475): MSG_REGION_CLOSE_WITHOUT_REPORT: table113,,1261449026166: Duplicate assignment
> 2009-12-21 18:31:44,666 INFO [RegionServer:1.worker] regionserver.HRegionServer$Worker(1332): Worker: MSG_REGION_CLOSE_WITHOUT_REPORT: table113,,1261449026166: Duplicate assignment
> {code}
> Here the master reassigned table13 and told the old region server to close the region before the new one was able to report that it opened it. At the end the new region server (good one) is also told to close it After that my test times out, table13 is not disabled neither it is deployed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HBASE-2065) Cannot disable a table if any of its
region is opening at the same time
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jean-Daniel Cryans resolved HBASE-2065.
---------------------------------------
Resolution: Fixed
Hadoop Flags: [Reviewed]
The test passes now and others too. Closing.
> Cannot disable a table if any of its region is opening at the same time
> -----------------------------------------------------------------------
>
> Key: HBASE-2065
> URL: https://issues.apache.org/jira/browse/HBASE-2065
> Project: Hadoop HBase
> Issue Type: Sub-task
> Affects Versions: 0.20.2
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.20.3, 0.21.0
>
> Attachments: HBASE-2065-2-branch.patch, HBASE-2065-2.patch, HBASE-2065-branch.patch, HBASE-2065.patch
>
>
> Also found with the test in the parent jira:
> {code}
> 2009-12-21 18:31:44,411 INFO [IPC Server handler 0 on 60000] master.RegionManager(331): Assigning region table113,,1261449026166 to 10.10.1.54,60853,1261448823301
> 2009-12-21 18:31:44,411 INFO [IPC Server handler 0 on 60000] master.RegionManager(331): Assigning region table121,,1261449041385 to 10.10.1.54,60853,1261448823301
> 2009-12-21 18:31:44,411 INFO [RegionServer:1] regionserver.HRegionServer(475): MSG_REGION_OPEN: table113,,1261449026166
> 2009-12-21 18:31:44,411 INFO [RegionServer:1] regionserver.HRegionServer(475): MSG_REGION_OPEN: table121,,1261449041385
> ...
> 2009-12-21 18:31:44,418 INFO [RegionServer:1.worker] regionserver.HRegion(343): region table113,,1261449026166/21044806 available; sequence id is 0
> ...
> 2009-12-21 18:31:44,445 DEBUG [IPC Server handler 4 on 60000] master.ChangeTableState(121): Adding region table113,,1261449026166 to setClosing list
> 2009-12-21 18:31:44,446 DEBUG [main] zookeeper.ZooKeeperWrapper(392): Read ZNode /hbase/root-region-server got 10.10.1.54:608532009-12-21 18:31:44,447 DEBUG [main] client.HConnectionManager$TableServers(990): Found ROOT at 10.10.1.54:60853
> 2009-12-21 18:31:44,447 DEBUG [main] client.HConnectionManager$TableServers(899): Cached location for .META.,,1 is 10.10.1.54:608552009-12-21 18:31:44,453 DEBUG [main] client.HConnectionManager$TableServers(554): Rowscanned=1, rowsOffline=0
> 2009-12-21 18:31:44,454 DEBUG [main] client.HBaseAdmin(397): Sleep. Waiting for all regions to be disabled from table1132009-12-21 18:31:44,554 DEBUG [main] client.HBaseAdmin(406): Wake. Waiting for all regions to be disabled from table113
> ...
> 2009-12-21 18:31:44,642 INFO [RegionServer:0] regionserver.HRegionServer(475): MSG_REGION_CLOSE: table113,,1261449026166
> ...
> 2009-12-21 18:31:44,642 INFO [RegionServer:0.worker] regionserver.HRegionServer$Worker(1332): Worker: MSG_REGION_CLOSE: table113,,1261449026166
> ...
> 2009-12-21 18:31:44,664 INFO [IPC Server handler 0 on 60000] master.ServerManager(421): Processing MSG_REPORT_PROCESS_OPEN: table113,,1261449026166 from 10.10.1.54,60853,1261448823301; 1 of 4
> ...
> 2009-12-21 18:31:44,664 INFO [IPC Server handler 0 on 60000] master.ServerManager(421): Processing MSG_REPORT_OPEN: table113,,1261449026166 from 10.10.1.54,60853,1261448823301; 3 of 4
> 2009-12-21 18:31:44,664 DEBUG [IPC Server handler 0 on 60000] master.ServerManager(562): region server 10.10.1.54:60853 should not have opened region table113,,1261449026166
> 2009-12-21 18:31:44,666 INFO [RegionServer:1] regionserver.HRegionServer(475): MSG_REGION_CLOSE_WITHOUT_REPORT: table113,,1261449026166: Duplicate assignment
> 2009-12-21 18:31:44,666 INFO [RegionServer:1.worker] regionserver.HRegionServer$Worker(1332): Worker: MSG_REGION_CLOSE_WITHOUT_REPORT: table113,,1261449026166: Duplicate assignment
> {code}
> Here the master reassigned table13 and told the old region server to close the region before the new one was able to report that it opened it. At the end the new region server (good one) is also told to close it After that my test times out, table13 is not disabled neither it is deployed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2065) Cannot disable a table if any of its
region is opening at the same time
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12793485#action_12793485 ]
Jean-Daniel Cryans commented on HBASE-2065:
-------------------------------------------
Different instance of the bug:
{code}
2009-12-21 19:29:15,425 INFO [IPC Server handler 4 on 60000] master.ServerManager(421): Processing MSG_REPORT_OPEN: table122,,1261452520421 from 10.10.1.54,49932,1261452340816; 2 of 2
2009-12-21 19:29:15,426 INFO [main] client.HBaseAdmin(302): Deleted table121
2009-12-21 19:29:15,427 DEBUG [master] master.HMaster(479): Processing todo: PendingOpenOperation from 10.10.1.54,49932,1261452340816
2009-12-21 19:29:15,427 INFO [master] master.ProcessRegionOpen(70): table122,,1261452520421 open on 10.10.1.54:49932
2009-12-21 19:29:15,427 DEBUG [IPC Server handler 2 on 60000] master.ChangeTableState(70): Processing unserved regions
2009-12-21 19:29:15,427 DEBUG [IPC Server handler 2 on 60000] master.ChangeTableState(107): Processing regions currently being served
2009-12-21 19:29:15,428 DEBUG [IPC Server handler 2 on 60000] master.ChangeTableState(121): Adding region table122,,1261452520421 to setClosing list
2009-12-21 19:29:15,428 INFO [master] master.ProcessRegionOpen(80): Updated row table122,,1261452520421 in region .META.,,1 with startcode=1261452340816, server=10.10.1.54:49932
{code}
Under 3 milliseconds the region was opening, it was disabled, and then the master removed it from regions in transition (after that last line). Table was hosed.
> Cannot disable a table if any of its region is opening at the same time
> -----------------------------------------------------------------------
>
> Key: HBASE-2065
> URL: https://issues.apache.org/jira/browse/HBASE-2065
> Project: Hadoop HBase
> Issue Type: Sub-task
> Affects Versions: 0.20.2
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.20.3, 0.21.0
>
>
> Also found with the test in the parent jira:
> {code}
> 2009-12-21 18:31:44,411 INFO [IPC Server handler 0 on 60000] master.RegionManager(331): Assigning region table113,,1261449026166 to 10.10.1.54,60853,1261448823301
> 2009-12-21 18:31:44,411 INFO [IPC Server handler 0 on 60000] master.RegionManager(331): Assigning region table121,,1261449041385 to 10.10.1.54,60853,1261448823301
> 2009-12-21 18:31:44,411 INFO [RegionServer:1] regionserver.HRegionServer(475): MSG_REGION_OPEN: table113,,1261449026166
> 2009-12-21 18:31:44,411 INFO [RegionServer:1] regionserver.HRegionServer(475): MSG_REGION_OPEN: table121,,1261449041385
> ...
> 2009-12-21 18:31:44,418 INFO [RegionServer:1.worker] regionserver.HRegion(343): region table113,,1261449026166/21044806 available; sequence id is 0
> ...
> 2009-12-21 18:31:44,445 DEBUG [IPC Server handler 4 on 60000] master.ChangeTableState(121): Adding region table113,,1261449026166 to setClosing list
> 2009-12-21 18:31:44,446 DEBUG [main] zookeeper.ZooKeeperWrapper(392): Read ZNode /hbase/root-region-server got 10.10.1.54:608532009-12-21 18:31:44,447 DEBUG [main] client.HConnectionManager$TableServers(990): Found ROOT at 10.10.1.54:60853
> 2009-12-21 18:31:44,447 DEBUG [main] client.HConnectionManager$TableServers(899): Cached location for .META.,,1 is 10.10.1.54:608552009-12-21 18:31:44,453 DEBUG [main] client.HConnectionManager$TableServers(554): Rowscanned=1, rowsOffline=0
> 2009-12-21 18:31:44,454 DEBUG [main] client.HBaseAdmin(397): Sleep. Waiting for all regions to be disabled from table1132009-12-21 18:31:44,554 DEBUG [main] client.HBaseAdmin(406): Wake. Waiting for all regions to be disabled from table113
> ...
> 2009-12-21 18:31:44,642 INFO [RegionServer:0] regionserver.HRegionServer(475): MSG_REGION_CLOSE: table113,,1261449026166
> ...
> 2009-12-21 18:31:44,642 INFO [RegionServer:0.worker] regionserver.HRegionServer$Worker(1332): Worker: MSG_REGION_CLOSE: table113,,1261449026166
> ...
> 2009-12-21 18:31:44,664 INFO [IPC Server handler 0 on 60000] master.ServerManager(421): Processing MSG_REPORT_PROCESS_OPEN: table113,,1261449026166 from 10.10.1.54,60853,1261448823301; 1 of 4
> ...
> 2009-12-21 18:31:44,664 INFO [IPC Server handler 0 on 60000] master.ServerManager(421): Processing MSG_REPORT_OPEN: table113,,1261449026166 from 10.10.1.54,60853,1261448823301; 3 of 4
> 2009-12-21 18:31:44,664 DEBUG [IPC Server handler 0 on 60000] master.ServerManager(562): region server 10.10.1.54:60853 should not have opened region table113,,1261449026166
> 2009-12-21 18:31:44,666 INFO [RegionServer:1] regionserver.HRegionServer(475): MSG_REGION_CLOSE_WITHOUT_REPORT: table113,,1261449026166: Duplicate assignment
> 2009-12-21 18:31:44,666 INFO [RegionServer:1.worker] regionserver.HRegionServer$Worker(1332): Worker: MSG_REGION_CLOSE_WITHOUT_REPORT: table113,,1261449026166: Duplicate assignment
> {code}
> Here the master reassigned table13 and told the old region server to close the region before the new one was able to report that it opened it. At the end the new region server (good one) is also told to close it After that my test times out, table13 is not disabled neither it is deployed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2065) Cannot disable a table if any of its
region is opening at the same time
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12794251#action_12794251 ]
Jean-Daniel Cryans commented on HBASE-2065:
-------------------------------------------
I tested my new patch exclusively with TestAdmin 10-15 times in a row and it didn't fail.
It passes all the unit tests.
I tried it on a cluster during an import job that created 400 regions on 17 region servers and then played with disables and enables without any failure.
> Cannot disable a table if any of its region is opening at the same time
> -----------------------------------------------------------------------
>
> Key: HBASE-2065
> URL: https://issues.apache.org/jira/browse/HBASE-2065
> Project: Hadoop HBase
> Issue Type: Sub-task
> Affects Versions: 0.20.2
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.20.3, 0.21.0
>
> Attachments: HBASE-2065-2.patch, HBASE-2065-branch.patch, HBASE-2065.patch
>
>
> Also found with the test in the parent jira:
> {code}
> 2009-12-21 18:31:44,411 INFO [IPC Server handler 0 on 60000] master.RegionManager(331): Assigning region table113,,1261449026166 to 10.10.1.54,60853,1261448823301
> 2009-12-21 18:31:44,411 INFO [IPC Server handler 0 on 60000] master.RegionManager(331): Assigning region table121,,1261449041385 to 10.10.1.54,60853,1261448823301
> 2009-12-21 18:31:44,411 INFO [RegionServer:1] regionserver.HRegionServer(475): MSG_REGION_OPEN: table113,,1261449026166
> 2009-12-21 18:31:44,411 INFO [RegionServer:1] regionserver.HRegionServer(475): MSG_REGION_OPEN: table121,,1261449041385
> ...
> 2009-12-21 18:31:44,418 INFO [RegionServer:1.worker] regionserver.HRegion(343): region table113,,1261449026166/21044806 available; sequence id is 0
> ...
> 2009-12-21 18:31:44,445 DEBUG [IPC Server handler 4 on 60000] master.ChangeTableState(121): Adding region table113,,1261449026166 to setClosing list
> 2009-12-21 18:31:44,446 DEBUG [main] zookeeper.ZooKeeperWrapper(392): Read ZNode /hbase/root-region-server got 10.10.1.54:608532009-12-21 18:31:44,447 DEBUG [main] client.HConnectionManager$TableServers(990): Found ROOT at 10.10.1.54:60853
> 2009-12-21 18:31:44,447 DEBUG [main] client.HConnectionManager$TableServers(899): Cached location for .META.,,1 is 10.10.1.54:608552009-12-21 18:31:44,453 DEBUG [main] client.HConnectionManager$TableServers(554): Rowscanned=1, rowsOffline=0
> 2009-12-21 18:31:44,454 DEBUG [main] client.HBaseAdmin(397): Sleep. Waiting for all regions to be disabled from table1132009-12-21 18:31:44,554 DEBUG [main] client.HBaseAdmin(406): Wake. Waiting for all regions to be disabled from table113
> ...
> 2009-12-21 18:31:44,642 INFO [RegionServer:0] regionserver.HRegionServer(475): MSG_REGION_CLOSE: table113,,1261449026166
> ...
> 2009-12-21 18:31:44,642 INFO [RegionServer:0.worker] regionserver.HRegionServer$Worker(1332): Worker: MSG_REGION_CLOSE: table113,,1261449026166
> ...
> 2009-12-21 18:31:44,664 INFO [IPC Server handler 0 on 60000] master.ServerManager(421): Processing MSG_REPORT_PROCESS_OPEN: table113,,1261449026166 from 10.10.1.54,60853,1261448823301; 1 of 4
> ...
> 2009-12-21 18:31:44,664 INFO [IPC Server handler 0 on 60000] master.ServerManager(421): Processing MSG_REPORT_OPEN: table113,,1261449026166 from 10.10.1.54,60853,1261448823301; 3 of 4
> 2009-12-21 18:31:44,664 DEBUG [IPC Server handler 0 on 60000] master.ServerManager(562): region server 10.10.1.54:60853 should not have opened region table113,,1261449026166
> 2009-12-21 18:31:44,666 INFO [RegionServer:1] regionserver.HRegionServer(475): MSG_REGION_CLOSE_WITHOUT_REPORT: table113,,1261449026166: Duplicate assignment
> 2009-12-21 18:31:44,666 INFO [RegionServer:1.worker] regionserver.HRegionServer$Worker(1332): Worker: MSG_REGION_CLOSE_WITHOUT_REPORT: table113,,1261449026166: Duplicate assignment
> {code}
> Here the master reassigned table13 and told the old region server to close the region before the new one was able to report that it opened it. At the end the new region server (good one) is also told to close it After that my test times out, table13 is not disabled neither it is deployed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2065) Cannot disable a table if any of its
region is opening at the same time
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12793789#action_12793789 ]
Jean-Daniel Cryans commented on HBASE-2065:
-------------------------------------------
Yes I will make sure the tests passes and also I'm currently testing disabling/enabling on a cluster.
> Cannot disable a table if any of its region is opening at the same time
> -----------------------------------------------------------------------
>
> Key: HBASE-2065
> URL: https://issues.apache.org/jira/browse/HBASE-2065
> Project: Hadoop HBase
> Issue Type: Sub-task
> Affects Versions: 0.20.2
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.20.3, 0.21.0
>
> Attachments: HBASE-2065.patch
>
>
> Also found with the test in the parent jira:
> {code}
> 2009-12-21 18:31:44,411 INFO [IPC Server handler 0 on 60000] master.RegionManager(331): Assigning region table113,,1261449026166 to 10.10.1.54,60853,1261448823301
> 2009-12-21 18:31:44,411 INFO [IPC Server handler 0 on 60000] master.RegionManager(331): Assigning region table121,,1261449041385 to 10.10.1.54,60853,1261448823301
> 2009-12-21 18:31:44,411 INFO [RegionServer:1] regionserver.HRegionServer(475): MSG_REGION_OPEN: table113,,1261449026166
> 2009-12-21 18:31:44,411 INFO [RegionServer:1] regionserver.HRegionServer(475): MSG_REGION_OPEN: table121,,1261449041385
> ...
> 2009-12-21 18:31:44,418 INFO [RegionServer:1.worker] regionserver.HRegion(343): region table113,,1261449026166/21044806 available; sequence id is 0
> ...
> 2009-12-21 18:31:44,445 DEBUG [IPC Server handler 4 on 60000] master.ChangeTableState(121): Adding region table113,,1261449026166 to setClosing list
> 2009-12-21 18:31:44,446 DEBUG [main] zookeeper.ZooKeeperWrapper(392): Read ZNode /hbase/root-region-server got 10.10.1.54:608532009-12-21 18:31:44,447 DEBUG [main] client.HConnectionManager$TableServers(990): Found ROOT at 10.10.1.54:60853
> 2009-12-21 18:31:44,447 DEBUG [main] client.HConnectionManager$TableServers(899): Cached location for .META.,,1 is 10.10.1.54:608552009-12-21 18:31:44,453 DEBUG [main] client.HConnectionManager$TableServers(554): Rowscanned=1, rowsOffline=0
> 2009-12-21 18:31:44,454 DEBUG [main] client.HBaseAdmin(397): Sleep. Waiting for all regions to be disabled from table1132009-12-21 18:31:44,554 DEBUG [main] client.HBaseAdmin(406): Wake. Waiting for all regions to be disabled from table113
> ...
> 2009-12-21 18:31:44,642 INFO [RegionServer:0] regionserver.HRegionServer(475): MSG_REGION_CLOSE: table113,,1261449026166
> ...
> 2009-12-21 18:31:44,642 INFO [RegionServer:0.worker] regionserver.HRegionServer$Worker(1332): Worker: MSG_REGION_CLOSE: table113,,1261449026166
> ...
> 2009-12-21 18:31:44,664 INFO [IPC Server handler 0 on 60000] master.ServerManager(421): Processing MSG_REPORT_PROCESS_OPEN: table113,,1261449026166 from 10.10.1.54,60853,1261448823301; 1 of 4
> ...
> 2009-12-21 18:31:44,664 INFO [IPC Server handler 0 on 60000] master.ServerManager(421): Processing MSG_REPORT_OPEN: table113,,1261449026166 from 10.10.1.54,60853,1261448823301; 3 of 4
> 2009-12-21 18:31:44,664 DEBUG [IPC Server handler 0 on 60000] master.ServerManager(562): region server 10.10.1.54:60853 should not have opened region table113,,1261449026166
> 2009-12-21 18:31:44,666 INFO [RegionServer:1] regionserver.HRegionServer(475): MSG_REGION_CLOSE_WITHOUT_REPORT: table113,,1261449026166: Duplicate assignment
> 2009-12-21 18:31:44,666 INFO [RegionServer:1.worker] regionserver.HRegionServer$Worker(1332): Worker: MSG_REGION_CLOSE_WITHOUT_REPORT: table113,,1261449026166: Duplicate assignment
> {code}
> Here the master reassigned table13 and told the old region server to close the region before the new one was able to report that it opened it. At the end the new region server (good one) is also told to close it After that my test times out, table13 is not disabled neither it is deployed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2065) Cannot disable a table if any of its
region is opening at the same time
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12794289#action_12794289 ]
stack commented on HBASE-2065:
------------------------------
Patch looks good J-D.
> Cannot disable a table if any of its region is opening at the same time
> -----------------------------------------------------------------------
>
> Key: HBASE-2065
> URL: https://issues.apache.org/jira/browse/HBASE-2065
> Project: Hadoop HBase
> Issue Type: Sub-task
> Affects Versions: 0.20.2
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.20.3, 0.21.0
>
> Attachments: HBASE-2065-2-branch.patch, HBASE-2065-2.patch, HBASE-2065-branch.patch, HBASE-2065.patch
>
>
> Also found with the test in the parent jira:
> {code}
> 2009-12-21 18:31:44,411 INFO [IPC Server handler 0 on 60000] master.RegionManager(331): Assigning region table113,,1261449026166 to 10.10.1.54,60853,1261448823301
> 2009-12-21 18:31:44,411 INFO [IPC Server handler 0 on 60000] master.RegionManager(331): Assigning region table121,,1261449041385 to 10.10.1.54,60853,1261448823301
> 2009-12-21 18:31:44,411 INFO [RegionServer:1] regionserver.HRegionServer(475): MSG_REGION_OPEN: table113,,1261449026166
> 2009-12-21 18:31:44,411 INFO [RegionServer:1] regionserver.HRegionServer(475): MSG_REGION_OPEN: table121,,1261449041385
> ...
> 2009-12-21 18:31:44,418 INFO [RegionServer:1.worker] regionserver.HRegion(343): region table113,,1261449026166/21044806 available; sequence id is 0
> ...
> 2009-12-21 18:31:44,445 DEBUG [IPC Server handler 4 on 60000] master.ChangeTableState(121): Adding region table113,,1261449026166 to setClosing list
> 2009-12-21 18:31:44,446 DEBUG [main] zookeeper.ZooKeeperWrapper(392): Read ZNode /hbase/root-region-server got 10.10.1.54:608532009-12-21 18:31:44,447 DEBUG [main] client.HConnectionManager$TableServers(990): Found ROOT at 10.10.1.54:60853
> 2009-12-21 18:31:44,447 DEBUG [main] client.HConnectionManager$TableServers(899): Cached location for .META.,,1 is 10.10.1.54:608552009-12-21 18:31:44,453 DEBUG [main] client.HConnectionManager$TableServers(554): Rowscanned=1, rowsOffline=0
> 2009-12-21 18:31:44,454 DEBUG [main] client.HBaseAdmin(397): Sleep. Waiting for all regions to be disabled from table1132009-12-21 18:31:44,554 DEBUG [main] client.HBaseAdmin(406): Wake. Waiting for all regions to be disabled from table113
> ...
> 2009-12-21 18:31:44,642 INFO [RegionServer:0] regionserver.HRegionServer(475): MSG_REGION_CLOSE: table113,,1261449026166
> ...
> 2009-12-21 18:31:44,642 INFO [RegionServer:0.worker] regionserver.HRegionServer$Worker(1332): Worker: MSG_REGION_CLOSE: table113,,1261449026166
> ...
> 2009-12-21 18:31:44,664 INFO [IPC Server handler 0 on 60000] master.ServerManager(421): Processing MSG_REPORT_PROCESS_OPEN: table113,,1261449026166 from 10.10.1.54,60853,1261448823301; 1 of 4
> ...
> 2009-12-21 18:31:44,664 INFO [IPC Server handler 0 on 60000] master.ServerManager(421): Processing MSG_REPORT_OPEN: table113,,1261449026166 from 10.10.1.54,60853,1261448823301; 3 of 4
> 2009-12-21 18:31:44,664 DEBUG [IPC Server handler 0 on 60000] master.ServerManager(562): region server 10.10.1.54:60853 should not have opened region table113,,1261449026166
> 2009-12-21 18:31:44,666 INFO [RegionServer:1] regionserver.HRegionServer(475): MSG_REGION_CLOSE_WITHOUT_REPORT: table113,,1261449026166: Duplicate assignment
> 2009-12-21 18:31:44,666 INFO [RegionServer:1.worker] regionserver.HRegionServer$Worker(1332): Worker: MSG_REGION_CLOSE_WITHOUT_REPORT: table113,,1261449026166: Duplicate assignment
> {code}
> Here the master reassigned table13 and told the old region server to close the region before the new one was able to report that it opened it. At the end the new region server (good one) is also told to close it After that my test times out, table13 is not disabled neither it is deployed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-2065) Cannot disable a table if any of its
region is opening at the same time
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jean-Daniel Cryans updated HBASE-2065:
--------------------------------------
Attachment: HBASE-2065-2-branch.patch
Patch for branch so that others can test.
> Cannot disable a table if any of its region is opening at the same time
> -----------------------------------------------------------------------
>
> Key: HBASE-2065
> URL: https://issues.apache.org/jira/browse/HBASE-2065
> Project: Hadoop HBase
> Issue Type: Sub-task
> Affects Versions: 0.20.2
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.20.3, 0.21.0
>
> Attachments: HBASE-2065-2-branch.patch, HBASE-2065-2.patch, HBASE-2065-branch.patch, HBASE-2065.patch
>
>
> Also found with the test in the parent jira:
> {code}
> 2009-12-21 18:31:44,411 INFO [IPC Server handler 0 on 60000] master.RegionManager(331): Assigning region table113,,1261449026166 to 10.10.1.54,60853,1261448823301
> 2009-12-21 18:31:44,411 INFO [IPC Server handler 0 on 60000] master.RegionManager(331): Assigning region table121,,1261449041385 to 10.10.1.54,60853,1261448823301
> 2009-12-21 18:31:44,411 INFO [RegionServer:1] regionserver.HRegionServer(475): MSG_REGION_OPEN: table113,,1261449026166
> 2009-12-21 18:31:44,411 INFO [RegionServer:1] regionserver.HRegionServer(475): MSG_REGION_OPEN: table121,,1261449041385
> ...
> 2009-12-21 18:31:44,418 INFO [RegionServer:1.worker] regionserver.HRegion(343): region table113,,1261449026166/21044806 available; sequence id is 0
> ...
> 2009-12-21 18:31:44,445 DEBUG [IPC Server handler 4 on 60000] master.ChangeTableState(121): Adding region table113,,1261449026166 to setClosing list
> 2009-12-21 18:31:44,446 DEBUG [main] zookeeper.ZooKeeperWrapper(392): Read ZNode /hbase/root-region-server got 10.10.1.54:608532009-12-21 18:31:44,447 DEBUG [main] client.HConnectionManager$TableServers(990): Found ROOT at 10.10.1.54:60853
> 2009-12-21 18:31:44,447 DEBUG [main] client.HConnectionManager$TableServers(899): Cached location for .META.,,1 is 10.10.1.54:608552009-12-21 18:31:44,453 DEBUG [main] client.HConnectionManager$TableServers(554): Rowscanned=1, rowsOffline=0
> 2009-12-21 18:31:44,454 DEBUG [main] client.HBaseAdmin(397): Sleep. Waiting for all regions to be disabled from table1132009-12-21 18:31:44,554 DEBUG [main] client.HBaseAdmin(406): Wake. Waiting for all regions to be disabled from table113
> ...
> 2009-12-21 18:31:44,642 INFO [RegionServer:0] regionserver.HRegionServer(475): MSG_REGION_CLOSE: table113,,1261449026166
> ...
> 2009-12-21 18:31:44,642 INFO [RegionServer:0.worker] regionserver.HRegionServer$Worker(1332): Worker: MSG_REGION_CLOSE: table113,,1261449026166
> ...
> 2009-12-21 18:31:44,664 INFO [IPC Server handler 0 on 60000] master.ServerManager(421): Processing MSG_REPORT_PROCESS_OPEN: table113,,1261449026166 from 10.10.1.54,60853,1261448823301; 1 of 4
> ...
> 2009-12-21 18:31:44,664 INFO [IPC Server handler 0 on 60000] master.ServerManager(421): Processing MSG_REPORT_OPEN: table113,,1261449026166 from 10.10.1.54,60853,1261448823301; 3 of 4
> 2009-12-21 18:31:44,664 DEBUG [IPC Server handler 0 on 60000] master.ServerManager(562): region server 10.10.1.54:60853 should not have opened region table113,,1261449026166
> 2009-12-21 18:31:44,666 INFO [RegionServer:1] regionserver.HRegionServer(475): MSG_REGION_CLOSE_WITHOUT_REPORT: table113,,1261449026166: Duplicate assignment
> 2009-12-21 18:31:44,666 INFO [RegionServer:1.worker] regionserver.HRegionServer$Worker(1332): Worker: MSG_REGION_CLOSE_WITHOUT_REPORT: table113,,1261449026166: Duplicate assignment
> {code}
> Here the master reassigned table13 and told the old region server to close the region before the new one was able to report that it opened it. At the end the new region server (good one) is also told to close it After that my test times out, table13 is not disabled neither it is deployed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-2065) Cannot disable a table if any of its
region is opening at the same time
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jean-Daniel Cryans updated HBASE-2065:
--------------------------------------
Attachment: HBASE-2065.patch
Path for both HBASE-2064 and HBASE-2065.
> Cannot disable a table if any of its region is opening at the same time
> -----------------------------------------------------------------------
>
> Key: HBASE-2065
> URL: https://issues.apache.org/jira/browse/HBASE-2065
> Project: Hadoop HBase
> Issue Type: Sub-task
> Affects Versions: 0.20.2
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.20.3, 0.21.0
>
> Attachments: HBASE-2065.patch
>
>
> Also found with the test in the parent jira:
> {code}
> 2009-12-21 18:31:44,411 INFO [IPC Server handler 0 on 60000] master.RegionManager(331): Assigning region table113,,1261449026166 to 10.10.1.54,60853,1261448823301
> 2009-12-21 18:31:44,411 INFO [IPC Server handler 0 on 60000] master.RegionManager(331): Assigning region table121,,1261449041385 to 10.10.1.54,60853,1261448823301
> 2009-12-21 18:31:44,411 INFO [RegionServer:1] regionserver.HRegionServer(475): MSG_REGION_OPEN: table113,,1261449026166
> 2009-12-21 18:31:44,411 INFO [RegionServer:1] regionserver.HRegionServer(475): MSG_REGION_OPEN: table121,,1261449041385
> ...
> 2009-12-21 18:31:44,418 INFO [RegionServer:1.worker] regionserver.HRegion(343): region table113,,1261449026166/21044806 available; sequence id is 0
> ...
> 2009-12-21 18:31:44,445 DEBUG [IPC Server handler 4 on 60000] master.ChangeTableState(121): Adding region table113,,1261449026166 to setClosing list
> 2009-12-21 18:31:44,446 DEBUG [main] zookeeper.ZooKeeperWrapper(392): Read ZNode /hbase/root-region-server got 10.10.1.54:608532009-12-21 18:31:44,447 DEBUG [main] client.HConnectionManager$TableServers(990): Found ROOT at 10.10.1.54:60853
> 2009-12-21 18:31:44,447 DEBUG [main] client.HConnectionManager$TableServers(899): Cached location for .META.,,1 is 10.10.1.54:608552009-12-21 18:31:44,453 DEBUG [main] client.HConnectionManager$TableServers(554): Rowscanned=1, rowsOffline=0
> 2009-12-21 18:31:44,454 DEBUG [main] client.HBaseAdmin(397): Sleep. Waiting for all regions to be disabled from table1132009-12-21 18:31:44,554 DEBUG [main] client.HBaseAdmin(406): Wake. Waiting for all regions to be disabled from table113
> ...
> 2009-12-21 18:31:44,642 INFO [RegionServer:0] regionserver.HRegionServer(475): MSG_REGION_CLOSE: table113,,1261449026166
> ...
> 2009-12-21 18:31:44,642 INFO [RegionServer:0.worker] regionserver.HRegionServer$Worker(1332): Worker: MSG_REGION_CLOSE: table113,,1261449026166
> ...
> 2009-12-21 18:31:44,664 INFO [IPC Server handler 0 on 60000] master.ServerManager(421): Processing MSG_REPORT_PROCESS_OPEN: table113,,1261449026166 from 10.10.1.54,60853,1261448823301; 1 of 4
> ...
> 2009-12-21 18:31:44,664 INFO [IPC Server handler 0 on 60000] master.ServerManager(421): Processing MSG_REPORT_OPEN: table113,,1261449026166 from 10.10.1.54,60853,1261448823301; 3 of 4
> 2009-12-21 18:31:44,664 DEBUG [IPC Server handler 0 on 60000] master.ServerManager(562): region server 10.10.1.54:60853 should not have opened region table113,,1261449026166
> 2009-12-21 18:31:44,666 INFO [RegionServer:1] regionserver.HRegionServer(475): MSG_REGION_CLOSE_WITHOUT_REPORT: table113,,1261449026166: Duplicate assignment
> 2009-12-21 18:31:44,666 INFO [RegionServer:1.worker] regionserver.HRegionServer$Worker(1332): Worker: MSG_REGION_CLOSE_WITHOUT_REPORT: table113,,1261449026166: Duplicate assignment
> {code}
> Here the master reassigned table13 and told the old region server to close the region before the new one was able to report that it opened it. At the end the new region server (good one) is also told to close it After that my test times out, table13 is not disabled neither it is deployed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-2065) Cannot disable a table if any of its
region is opening at the same time
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jean-Daniel Cryans updated HBASE-2065:
--------------------------------------
Summary: Cannot disable a table if any of its region is opening at the same time (was: Cannot disable a table if any of its region is opening at the same)
> Cannot disable a table if any of its region is opening at the same time
> -----------------------------------------------------------------------
>
> Key: HBASE-2065
> URL: https://issues.apache.org/jira/browse/HBASE-2065
> Project: Hadoop HBase
> Issue Type: Sub-task
> Affects Versions: 0.20.2
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.20.3, 0.21.0
>
>
> Also found with the test in the parent jira:
> {code}
> 2009-12-21 18:31:44,411 INFO [IPC Server handler 0 on 60000] master.RegionManager(331): Assigning region table113,,1261449026166 to 10.10.1.54,60853,1261448823301
> 2009-12-21 18:31:44,411 INFO [IPC Server handler 0 on 60000] master.RegionManager(331): Assigning region table121,,1261449041385 to 10.10.1.54,60853,1261448823301
> 2009-12-21 18:31:44,411 INFO [RegionServer:1] regionserver.HRegionServer(475): MSG_REGION_OPEN: table113,,1261449026166
> 2009-12-21 18:31:44,411 INFO [RegionServer:1] regionserver.HRegionServer(475): MSG_REGION_OPEN: table121,,1261449041385
> ...
> 2009-12-21 18:31:44,418 INFO [RegionServer:1.worker] regionserver.HRegion(343): region table113,,1261449026166/21044806 available; sequence id is 0
> ...
> 2009-12-21 18:31:44,445 DEBUG [IPC Server handler 4 on 60000] master.ChangeTableState(121): Adding region table113,,1261449026166 to setClosing list
> 2009-12-21 18:31:44,446 DEBUG [main] zookeeper.ZooKeeperWrapper(392): Read ZNode /hbase/root-region-server got 10.10.1.54:608532009-12-21 18:31:44,447 DEBUG [main] client.HConnectionManager$TableServers(990): Found ROOT at 10.10.1.54:60853
> 2009-12-21 18:31:44,447 DEBUG [main] client.HConnectionManager$TableServers(899): Cached location for .META.,,1 is 10.10.1.54:608552009-12-21 18:31:44,453 DEBUG [main] client.HConnectionManager$TableServers(554): Rowscanned=1, rowsOffline=0
> 2009-12-21 18:31:44,454 DEBUG [main] client.HBaseAdmin(397): Sleep. Waiting for all regions to be disabled from table1132009-12-21 18:31:44,554 DEBUG [main] client.HBaseAdmin(406): Wake. Waiting for all regions to be disabled from table113
> ...
> 2009-12-21 18:31:44,642 INFO [RegionServer:0] regionserver.HRegionServer(475): MSG_REGION_CLOSE: table113,,1261449026166
> ...
> 2009-12-21 18:31:44,642 INFO [RegionServer:0.worker] regionserver.HRegionServer$Worker(1332): Worker: MSG_REGION_CLOSE: table113,,1261449026166
> ...
> 2009-12-21 18:31:44,664 INFO [IPC Server handler 0 on 60000] master.ServerManager(421): Processing MSG_REPORT_PROCESS_OPEN: table113,,1261449026166 from 10.10.1.54,60853,1261448823301; 1 of 4
> ...
> 2009-12-21 18:31:44,664 INFO [IPC Server handler 0 on 60000] master.ServerManager(421): Processing MSG_REPORT_OPEN: table113,,1261449026166 from 10.10.1.54,60853,1261448823301; 3 of 4
> 2009-12-21 18:31:44,664 DEBUG [IPC Server handler 0 on 60000] master.ServerManager(562): region server 10.10.1.54:60853 should not have opened region table113,,1261449026166
> 2009-12-21 18:31:44,666 INFO [RegionServer:1] regionserver.HRegionServer(475): MSG_REGION_CLOSE_WITHOUT_REPORT: table113,,1261449026166: Duplicate assignment
> 2009-12-21 18:31:44,666 INFO [RegionServer:1.worker] regionserver.HRegionServer$Worker(1332): Worker: MSG_REGION_CLOSE_WITHOUT_REPORT: table113,,1261449026166: Duplicate assignment
> {code}
> Here the master reassigned table13 and told the old region server to close the region before the new one was able to report that it opened it. At the end the new region server (good one) is also told to close it After that my test times out, table13 is not disabled neither it is deployed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-2065) Cannot disable a table if any of its
region is opening at the same time
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jean-Daniel Cryans updated HBASE-2065:
--------------------------------------
Attachment: HBASE-2065-branch.patch
Patch for branch.
> Cannot disable a table if any of its region is opening at the same time
> -----------------------------------------------------------------------
>
> Key: HBASE-2065
> URL: https://issues.apache.org/jira/browse/HBASE-2065
> Project: Hadoop HBase
> Issue Type: Sub-task
> Affects Versions: 0.20.2
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.20.3, 0.21.0
>
> Attachments: HBASE-2065-branch.patch, HBASE-2065.patch
>
>
> Also found with the test in the parent jira:
> {code}
> 2009-12-21 18:31:44,411 INFO [IPC Server handler 0 on 60000] master.RegionManager(331): Assigning region table113,,1261449026166 to 10.10.1.54,60853,1261448823301
> 2009-12-21 18:31:44,411 INFO [IPC Server handler 0 on 60000] master.RegionManager(331): Assigning region table121,,1261449041385 to 10.10.1.54,60853,1261448823301
> 2009-12-21 18:31:44,411 INFO [RegionServer:1] regionserver.HRegionServer(475): MSG_REGION_OPEN: table113,,1261449026166
> 2009-12-21 18:31:44,411 INFO [RegionServer:1] regionserver.HRegionServer(475): MSG_REGION_OPEN: table121,,1261449041385
> ...
> 2009-12-21 18:31:44,418 INFO [RegionServer:1.worker] regionserver.HRegion(343): region table113,,1261449026166/21044806 available; sequence id is 0
> ...
> 2009-12-21 18:31:44,445 DEBUG [IPC Server handler 4 on 60000] master.ChangeTableState(121): Adding region table113,,1261449026166 to setClosing list
> 2009-12-21 18:31:44,446 DEBUG [main] zookeeper.ZooKeeperWrapper(392): Read ZNode /hbase/root-region-server got 10.10.1.54:608532009-12-21 18:31:44,447 DEBUG [main] client.HConnectionManager$TableServers(990): Found ROOT at 10.10.1.54:60853
> 2009-12-21 18:31:44,447 DEBUG [main] client.HConnectionManager$TableServers(899): Cached location for .META.,,1 is 10.10.1.54:608552009-12-21 18:31:44,453 DEBUG [main] client.HConnectionManager$TableServers(554): Rowscanned=1, rowsOffline=0
> 2009-12-21 18:31:44,454 DEBUG [main] client.HBaseAdmin(397): Sleep. Waiting for all regions to be disabled from table1132009-12-21 18:31:44,554 DEBUG [main] client.HBaseAdmin(406): Wake. Waiting for all regions to be disabled from table113
> ...
> 2009-12-21 18:31:44,642 INFO [RegionServer:0] regionserver.HRegionServer(475): MSG_REGION_CLOSE: table113,,1261449026166
> ...
> 2009-12-21 18:31:44,642 INFO [RegionServer:0.worker] regionserver.HRegionServer$Worker(1332): Worker: MSG_REGION_CLOSE: table113,,1261449026166
> ...
> 2009-12-21 18:31:44,664 INFO [IPC Server handler 0 on 60000] master.ServerManager(421): Processing MSG_REPORT_PROCESS_OPEN: table113,,1261449026166 from 10.10.1.54,60853,1261448823301; 1 of 4
> ...
> 2009-12-21 18:31:44,664 INFO [IPC Server handler 0 on 60000] master.ServerManager(421): Processing MSG_REPORT_OPEN: table113,,1261449026166 from 10.10.1.54,60853,1261448823301; 3 of 4
> 2009-12-21 18:31:44,664 DEBUG [IPC Server handler 0 on 60000] master.ServerManager(562): region server 10.10.1.54:60853 should not have opened region table113,,1261449026166
> 2009-12-21 18:31:44,666 INFO [RegionServer:1] regionserver.HRegionServer(475): MSG_REGION_CLOSE_WITHOUT_REPORT: table113,,1261449026166: Duplicate assignment
> 2009-12-21 18:31:44,666 INFO [RegionServer:1.worker] regionserver.HRegionServer$Worker(1332): Worker: MSG_REGION_CLOSE_WITHOUT_REPORT: table113,,1261449026166: Duplicate assignment
> {code}
> Here the master reassigned table13 and told the old region server to close the region before the new one was able to report that it opened it. At the end the new region server (good one) is also told to close it After that my test times out, table13 is not disabled neither it is deployed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2065) Cannot disable a table if any of its
region is opening at the same time
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12793777#action_12793777 ]
stack commented on HBASE-2065:
------------------------------
+1 on patch. Would suggest running complete test suite before committing because jiggering the timings in src/test/hbase-site.xml can have unexpected repercussions out in other tests. Thanks for removing my noisy log of edit number.
> Cannot disable a table if any of its region is opening at the same time
> -----------------------------------------------------------------------
>
> Key: HBASE-2065
> URL: https://issues.apache.org/jira/browse/HBASE-2065
> Project: Hadoop HBase
> Issue Type: Sub-task
> Affects Versions: 0.20.2
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.20.3, 0.21.0
>
> Attachments: HBASE-2065.patch
>
>
> Also found with the test in the parent jira:
> {code}
> 2009-12-21 18:31:44,411 INFO [IPC Server handler 0 on 60000] master.RegionManager(331): Assigning region table113,,1261449026166 to 10.10.1.54,60853,1261448823301
> 2009-12-21 18:31:44,411 INFO [IPC Server handler 0 on 60000] master.RegionManager(331): Assigning region table121,,1261449041385 to 10.10.1.54,60853,1261448823301
> 2009-12-21 18:31:44,411 INFO [RegionServer:1] regionserver.HRegionServer(475): MSG_REGION_OPEN: table113,,1261449026166
> 2009-12-21 18:31:44,411 INFO [RegionServer:1] regionserver.HRegionServer(475): MSG_REGION_OPEN: table121,,1261449041385
> ...
> 2009-12-21 18:31:44,418 INFO [RegionServer:1.worker] regionserver.HRegion(343): region table113,,1261449026166/21044806 available; sequence id is 0
> ...
> 2009-12-21 18:31:44,445 DEBUG [IPC Server handler 4 on 60000] master.ChangeTableState(121): Adding region table113,,1261449026166 to setClosing list
> 2009-12-21 18:31:44,446 DEBUG [main] zookeeper.ZooKeeperWrapper(392): Read ZNode /hbase/root-region-server got 10.10.1.54:608532009-12-21 18:31:44,447 DEBUG [main] client.HConnectionManager$TableServers(990): Found ROOT at 10.10.1.54:60853
> 2009-12-21 18:31:44,447 DEBUG [main] client.HConnectionManager$TableServers(899): Cached location for .META.,,1 is 10.10.1.54:608552009-12-21 18:31:44,453 DEBUG [main] client.HConnectionManager$TableServers(554): Rowscanned=1, rowsOffline=0
> 2009-12-21 18:31:44,454 DEBUG [main] client.HBaseAdmin(397): Sleep. Waiting for all regions to be disabled from table1132009-12-21 18:31:44,554 DEBUG [main] client.HBaseAdmin(406): Wake. Waiting for all regions to be disabled from table113
> ...
> 2009-12-21 18:31:44,642 INFO [RegionServer:0] regionserver.HRegionServer(475): MSG_REGION_CLOSE: table113,,1261449026166
> ...
> 2009-12-21 18:31:44,642 INFO [RegionServer:0.worker] regionserver.HRegionServer$Worker(1332): Worker: MSG_REGION_CLOSE: table113,,1261449026166
> ...
> 2009-12-21 18:31:44,664 INFO [IPC Server handler 0 on 60000] master.ServerManager(421): Processing MSG_REPORT_PROCESS_OPEN: table113,,1261449026166 from 10.10.1.54,60853,1261448823301; 1 of 4
> ...
> 2009-12-21 18:31:44,664 INFO [IPC Server handler 0 on 60000] master.ServerManager(421): Processing MSG_REPORT_OPEN: table113,,1261449026166 from 10.10.1.54,60853,1261448823301; 3 of 4
> 2009-12-21 18:31:44,664 DEBUG [IPC Server handler 0 on 60000] master.ServerManager(562): region server 10.10.1.54:60853 should not have opened region table113,,1261449026166
> 2009-12-21 18:31:44,666 INFO [RegionServer:1] regionserver.HRegionServer(475): MSG_REGION_CLOSE_WITHOUT_REPORT: table113,,1261449026166: Duplicate assignment
> 2009-12-21 18:31:44,666 INFO [RegionServer:1.worker] regionserver.HRegionServer$Worker(1332): Worker: MSG_REGION_CLOSE_WITHOUT_REPORT: table113,,1261449026166: Duplicate assignment
> {code}
> Here the master reassigned table13 and told the old region server to close the region before the new one was able to report that it opened it. At the end the new region server (good one) is also told to close it After that my test times out, table13 is not disabled neither it is deployed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2065) Cannot disable a table if any of its
region is opening at the same time
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12794291#action_12794291 ]
Jean-Daniel Cryans commented on HBASE-2065:
-------------------------------------------
I committed the new patch to trunk and branch, let's see what Hudson says.
> Cannot disable a table if any of its region is opening at the same time
> -----------------------------------------------------------------------
>
> Key: HBASE-2065
> URL: https://issues.apache.org/jira/browse/HBASE-2065
> Project: Hadoop HBase
> Issue Type: Sub-task
> Affects Versions: 0.20.2
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.20.3, 0.21.0
>
> Attachments: HBASE-2065-2-branch.patch, HBASE-2065-2.patch, HBASE-2065-branch.patch, HBASE-2065.patch
>
>
> Also found with the test in the parent jira:
> {code}
> 2009-12-21 18:31:44,411 INFO [IPC Server handler 0 on 60000] master.RegionManager(331): Assigning region table113,,1261449026166 to 10.10.1.54,60853,1261448823301
> 2009-12-21 18:31:44,411 INFO [IPC Server handler 0 on 60000] master.RegionManager(331): Assigning region table121,,1261449041385 to 10.10.1.54,60853,1261448823301
> 2009-12-21 18:31:44,411 INFO [RegionServer:1] regionserver.HRegionServer(475): MSG_REGION_OPEN: table113,,1261449026166
> 2009-12-21 18:31:44,411 INFO [RegionServer:1] regionserver.HRegionServer(475): MSG_REGION_OPEN: table121,,1261449041385
> ...
> 2009-12-21 18:31:44,418 INFO [RegionServer:1.worker] regionserver.HRegion(343): region table113,,1261449026166/21044806 available; sequence id is 0
> ...
> 2009-12-21 18:31:44,445 DEBUG [IPC Server handler 4 on 60000] master.ChangeTableState(121): Adding region table113,,1261449026166 to setClosing list
> 2009-12-21 18:31:44,446 DEBUG [main] zookeeper.ZooKeeperWrapper(392): Read ZNode /hbase/root-region-server got 10.10.1.54:608532009-12-21 18:31:44,447 DEBUG [main] client.HConnectionManager$TableServers(990): Found ROOT at 10.10.1.54:60853
> 2009-12-21 18:31:44,447 DEBUG [main] client.HConnectionManager$TableServers(899): Cached location for .META.,,1 is 10.10.1.54:608552009-12-21 18:31:44,453 DEBUG [main] client.HConnectionManager$TableServers(554): Rowscanned=1, rowsOffline=0
> 2009-12-21 18:31:44,454 DEBUG [main] client.HBaseAdmin(397): Sleep. Waiting for all regions to be disabled from table1132009-12-21 18:31:44,554 DEBUG [main] client.HBaseAdmin(406): Wake. Waiting for all regions to be disabled from table113
> ...
> 2009-12-21 18:31:44,642 INFO [RegionServer:0] regionserver.HRegionServer(475): MSG_REGION_CLOSE: table113,,1261449026166
> ...
> 2009-12-21 18:31:44,642 INFO [RegionServer:0.worker] regionserver.HRegionServer$Worker(1332): Worker: MSG_REGION_CLOSE: table113,,1261449026166
> ...
> 2009-12-21 18:31:44,664 INFO [IPC Server handler 0 on 60000] master.ServerManager(421): Processing MSG_REPORT_PROCESS_OPEN: table113,,1261449026166 from 10.10.1.54,60853,1261448823301; 1 of 4
> ...
> 2009-12-21 18:31:44,664 INFO [IPC Server handler 0 on 60000] master.ServerManager(421): Processing MSG_REPORT_OPEN: table113,,1261449026166 from 10.10.1.54,60853,1261448823301; 3 of 4
> 2009-12-21 18:31:44,664 DEBUG [IPC Server handler 0 on 60000] master.ServerManager(562): region server 10.10.1.54:60853 should not have opened region table113,,1261449026166
> 2009-12-21 18:31:44,666 INFO [RegionServer:1] regionserver.HRegionServer(475): MSG_REGION_CLOSE_WITHOUT_REPORT: table113,,1261449026166: Duplicate assignment
> 2009-12-21 18:31:44,666 INFO [RegionServer:1.worker] regionserver.HRegionServer$Worker(1332): Worker: MSG_REGION_CLOSE_WITHOUT_REPORT: table113,,1261449026166: Duplicate assignment
> {code}
> Here the master reassigned table13 and told the old region server to close the region before the new one was able to report that it opened it. At the end the new region server (good one) is also told to close it After that my test times out, table13 is not disabled neither it is deployed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2065) Cannot disable a table if any of its
region is opening at the same time
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12793838#action_12793838 ]
Jean-Daniel Cryans commented on HBASE-2065:
-------------------------------------------
Testing on the cluster worked on a freshly loaded 100 regions table that I disabled/enabled a couple of times.
All tests are green.
> Cannot disable a table if any of its region is opening at the same time
> -----------------------------------------------------------------------
>
> Key: HBASE-2065
> URL: https://issues.apache.org/jira/browse/HBASE-2065
> Project: Hadoop HBase
> Issue Type: Sub-task
> Affects Versions: 0.20.2
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.20.3, 0.21.0
>
> Attachments: HBASE-2065.patch
>
>
> Also found with the test in the parent jira:
> {code}
> 2009-12-21 18:31:44,411 INFO [IPC Server handler 0 on 60000] master.RegionManager(331): Assigning region table113,,1261449026166 to 10.10.1.54,60853,1261448823301
> 2009-12-21 18:31:44,411 INFO [IPC Server handler 0 on 60000] master.RegionManager(331): Assigning region table121,,1261449041385 to 10.10.1.54,60853,1261448823301
> 2009-12-21 18:31:44,411 INFO [RegionServer:1] regionserver.HRegionServer(475): MSG_REGION_OPEN: table113,,1261449026166
> 2009-12-21 18:31:44,411 INFO [RegionServer:1] regionserver.HRegionServer(475): MSG_REGION_OPEN: table121,,1261449041385
> ...
> 2009-12-21 18:31:44,418 INFO [RegionServer:1.worker] regionserver.HRegion(343): region table113,,1261449026166/21044806 available; sequence id is 0
> ...
> 2009-12-21 18:31:44,445 DEBUG [IPC Server handler 4 on 60000] master.ChangeTableState(121): Adding region table113,,1261449026166 to setClosing list
> 2009-12-21 18:31:44,446 DEBUG [main] zookeeper.ZooKeeperWrapper(392): Read ZNode /hbase/root-region-server got 10.10.1.54:608532009-12-21 18:31:44,447 DEBUG [main] client.HConnectionManager$TableServers(990): Found ROOT at 10.10.1.54:60853
> 2009-12-21 18:31:44,447 DEBUG [main] client.HConnectionManager$TableServers(899): Cached location for .META.,,1 is 10.10.1.54:608552009-12-21 18:31:44,453 DEBUG [main] client.HConnectionManager$TableServers(554): Rowscanned=1, rowsOffline=0
> 2009-12-21 18:31:44,454 DEBUG [main] client.HBaseAdmin(397): Sleep. Waiting for all regions to be disabled from table1132009-12-21 18:31:44,554 DEBUG [main] client.HBaseAdmin(406): Wake. Waiting for all regions to be disabled from table113
> ...
> 2009-12-21 18:31:44,642 INFO [RegionServer:0] regionserver.HRegionServer(475): MSG_REGION_CLOSE: table113,,1261449026166
> ...
> 2009-12-21 18:31:44,642 INFO [RegionServer:0.worker] regionserver.HRegionServer$Worker(1332): Worker: MSG_REGION_CLOSE: table113,,1261449026166
> ...
> 2009-12-21 18:31:44,664 INFO [IPC Server handler 0 on 60000] master.ServerManager(421): Processing MSG_REPORT_PROCESS_OPEN: table113,,1261449026166 from 10.10.1.54,60853,1261448823301; 1 of 4
> ...
> 2009-12-21 18:31:44,664 INFO [IPC Server handler 0 on 60000] master.ServerManager(421): Processing MSG_REPORT_OPEN: table113,,1261449026166 from 10.10.1.54,60853,1261448823301; 3 of 4
> 2009-12-21 18:31:44,664 DEBUG [IPC Server handler 0 on 60000] master.ServerManager(562): region server 10.10.1.54:60853 should not have opened region table113,,1261449026166
> 2009-12-21 18:31:44,666 INFO [RegionServer:1] regionserver.HRegionServer(475): MSG_REGION_CLOSE_WITHOUT_REPORT: table113,,1261449026166: Duplicate assignment
> 2009-12-21 18:31:44,666 INFO [RegionServer:1.worker] regionserver.HRegionServer$Worker(1332): Worker: MSG_REGION_CLOSE_WITHOUT_REPORT: table113,,1261449026166: Duplicate assignment
> {code}
> Here the master reassigned table13 and told the old region server to close the region before the new one was able to report that it opened it. At the end the new region server (good one) is also told to close it After that my test times out, table13 is not disabled neither it is deployed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.