You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "ramkrishna.s.vasudevan (JIRA)" <ji...@apache.org> on 2011/08/16 11:22:27 UTC
[jira] [Created] (HBASE-4203) While master restarts and if the META
region's state is OPENING then master cannot assign META until timeout
monitor deducts
While master restarts and if the META region's state is OPENING then master cannot assign META until timeout monitor deducts
----------------------------------------------------------------------------------------------------------------------------
Key: HBASE-4203
URL: https://issues.apache.org/jira/browse/HBASE-4203
Project: HBase
Issue Type: Bug
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Priority: Minor
1. Start Master and 2 RS.
2. If any exception happens while opening the META region the state in znode will be OPENING.
3. If at this point the master restarts then the master will start processing the regions in RIT.
4. If the znode is found to be in OPENING then master waits for timeout monitor to deduct and then call opening.
5. If default timeout monitor is configured then it may take long time to open the META region.
Soln:
====
Better not to wait for the Timeout monitor period to open catalog tables on Master restart
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4203) While master restarts and if the
META region's state is OPENING then master cannot assign META until timeout
monitor deducts
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13085900#comment-13085900 ]
stack commented on HBASE-4203:
------------------------------
Your explanation above helps a lot. Now its clear what you are going on about. So, how do you distingush between the case where a master comes on line and the RIT is OPENING but NO exception has happened processing .META... its just a bit slow and will move to OPENED in a little while... how you distinguish this case form the pathological case you describe above where we have to wait 30 mins?
Otherwise, yes, this is important one to fix. Good on you Ram.
> While master restarts and if the META region's state is OPENING then master cannot assign META until timeout monitor deducts
> ----------------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-4203
> URL: https://issues.apache.org/jira/browse/HBASE-4203
> Project: HBase
> Issue Type: Bug
> Reporter: ramkrishna.s.vasudevan
> Assignee: ramkrishna.s.vasudevan
> Priority: Minor
>
> 1. Start Master and 2 RS.
> 2. If any exception happens while opening the META region the state in znode will be OPENING.
> 3. If at this point the master restarts then the master will start processing the regions in RIT.
> 4. If the znode is found to be in OPENING then master waits for timeout monitor to deduct and then call opening.
> 5. If default timeout monitor is configured(1800000 sec/30 min) then it will take 30 mins to open the META region itself.
> Soln:
> ====
> Better not to wait for the Timeout monitor period to open catalog tables on Master restart
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-4203) While master restarts and if the
META region's state is OPENING then master cannot assign META until timeout
monitor deducts
Posted by "Ted Yu (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ted Yu resolved HBASE-4203.
---------------------------
Resolution: Fixed
This was covered in HBASE-4015
> While master restarts and if the META region's state is OPENING then master cannot assign META until timeout monitor deducts
> ----------------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-4203
> URL: https://issues.apache.org/jira/browse/HBASE-4203
> Project: HBase
> Issue Type: Bug
> Reporter: ramkrishna.s.vasudevan
> Assignee: ramkrishna.s.vasudevan
> Priority: Minor
>
> 1. Start Master and 2 RS.
> 2. If any exception happens while opening the META region the state in znode will be OPENING.
> 3. If at this point the master restarts then the master will start processing the regions in RIT.
> 4. If the znode is found to be in OPENING then master waits for timeout monitor to deduct and then call opening.
> 5. If default timeout monitor is configured(1800000 sec/30 min) then it will take 30 mins to open the META region itself.
> Soln:
> ====
> Better not to wait for the Timeout monitor period to open catalog tables on Master restart
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4203) While master restarts and if the META
region's state is OPENING then master cannot assign META until timeout
monitor deducts
Posted by "ramkrishna.s.vasudevan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ramkrishna.s.vasudevan updated HBASE-4203:
------------------------------------------
Description:
1. Start Master and 2 RS.
2. If any exception happens while opening the META region the state in znode will be OPENING.
3. If at this point the master restarts then the master will start processing the regions in RIT.
4. If the znode is found to be in OPENING then master waits for timeout monitor to deduct and then call opening.
5. If default timeout monitor is configured(1800000 sec/30 min) then it will take 30 mins to open the META region itself.
Soln:
====
Better not to wait for the Timeout monitor period to open catalog tables on Master restart
was:
1. Start Master and 2 RS.
2. If any exception happens while opening the META region the state in znode will be OPENING.
3. If at this point the master restarts then the master will start processing the regions in RIT.
4. If the znode is found to be in OPENING then master waits for timeout monitor to deduct and then call opening.
5. If default timeout monitor is configured then it may take long time to open the META region.
Soln:
====
Better not to wait for the Timeout monitor period to open catalog tables on Master restart
> While master restarts and if the META region's state is OPENING then master cannot assign META until timeout monitor deducts
> ----------------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-4203
> URL: https://issues.apache.org/jira/browse/HBASE-4203
> Project: HBase
> Issue Type: Bug
> Reporter: ramkrishna.s.vasudevan
> Assignee: ramkrishna.s.vasudevan
> Priority: Minor
>
> 1. Start Master and 2 RS.
> 2. If any exception happens while opening the META region the state in znode will be OPENING.
> 3. If at this point the master restarts then the master will start processing the regions in RIT.
> 4. If the znode is found to be in OPENING then master waits for timeout monitor to deduct and then call opening.
> 5. If default timeout monitor is configured(1800000 sec/30 min) then it will take 30 mins to open the META region itself.
> Soln:
> ====
> Better not to wait for the Timeout monitor period to open catalog tables on Master restart
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4203) While master restarts and if the
META region's state is OPENING then master cannot assign META until timeout
monitor deducts
Posted by "ramkrishna.s.vasudevan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13085606#comment-13085606 ]
ramkrishna.s.vasudevan commented on HBASE-4203:
-----------------------------------------------
Attaching the logs to show the impact of how timetaking it is to wait for timeout monitor to deduct and then assign META
{color:red}2011-08-16 14:41:22,653 DEBUG org.apache.hadoop.hbase.master.handler.OpenedRegionHandler: Opened region -ROOT-,,0.70236052 on linux146,60020,1313485860002
2011-08-16 14:59:57,907 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: .META.,,1.1028785192 state=OPENING, ts=0, server=linux76,60020,1313485434045{color}
2011-08-16 14:59:57,907 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region has been OPENING for too long, reassigning region=.META.,,1.1028785192
2011-08-16 14:59:57,912 INFO org.apache.hadoop.hbase.master.AssignmentManager: Successfully transitioned region=.META.,,1.1028785192 into OFFLINE and forcing a new assignment
2011-08-16 14:59:57,912 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; was=.META.,,1.1028785192 state=OPENING, ts=0, server=linux76,60020,1313485434045
2011-08-16 14:59:57,912 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Handling transition=M_ZK_REGION_OFFLINE, server=linux76,60000,1313485532331, region=1028785192/.META.
2011-08-16 14:59:58,918 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: No previous transition plan was found (or we are ignoring an existing plan) for .META.,,1.1028785192 so generated a random one; hri=.META.,,1.1028785192, src=, dest=linux146,60020,1313485860002; 1 (online=1, exclude=null) available servers
2011-08-16 14:59:58,918 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Assigning region .META.,,1.1028785192 to linux146,60020,1313485860002
2011-08-16 14:59:58,918 DEBUG org.apache.hadoop.hbase.master.ServerManager: New connection to linux146,60020,1313485860002
2011-08-16 15:00:04,653 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Handling transition=RS_ZK_REGION_OPENING, server=linux146,60020,1313485860002, region=1028785192/.META.
2011-08-16 15:00:04,681 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Handling transition=RS_ZK_REGION_OPENING, server=linux146,60020,1313485860002, region=1028785192/.META.
2011-08-16 15:00:04,760 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Handling transition=RS_ZK_REGION_OPENED, server=linux146,60020,1313485860002, region=1028785192/.META.
2011-08-16 15:00:04,762 DEBUG org.apache.hadoop.hbase.master.handler.OpenedRegionHandler: Handling OPENED event for 1028785192; deleting unassigned node
> While master restarts and if the META region's state is OPENING then master cannot assign META until timeout monitor deducts
> ----------------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-4203
> URL: https://issues.apache.org/jira/browse/HBASE-4203
> Project: HBase
> Issue Type: Bug
> Reporter: ramkrishna.s.vasudevan
> Assignee: ramkrishna.s.vasudevan
> Priority: Minor
>
> 1. Start Master and 2 RS.
> 2. If any exception happens while opening the META region the state in znode will be OPENING.
> 3. If at this point the master restarts then the master will start processing the regions in RIT.
> 4. If the znode is found to be in OPENING then master waits for timeout monitor to deduct and then call opening.
> 5. If default timeout monitor is configured then it may take long time to open the META region.
> Soln:
> ====
> Better not to wait for the Timeout monitor period to open catalog tables on Master restart
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4203) While master restarts and if the
META region's state is OPENING then master cannot assign META until timeout
monitor deducts
Posted by "ramkrishna.s.vasudevan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13092707#comment-13092707 ]
ramkrishna.s.vasudevan commented on HBASE-4203:
-----------------------------------------------
Currently decided to give the patch for this JIRA along with HBASE-4015
> While master restarts and if the META region's state is OPENING then master cannot assign META until timeout monitor deducts
> ----------------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-4203
> URL: https://issues.apache.org/jira/browse/HBASE-4203
> Project: HBase
> Issue Type: Bug
> Reporter: ramkrishna.s.vasudevan
> Assignee: ramkrishna.s.vasudevan
> Priority: Minor
>
> 1. Start Master and 2 RS.
> 2. If any exception happens while opening the META region the state in znode will be OPENING.
> 3. If at this point the master restarts then the master will start processing the regions in RIT.
> 4. If the znode is found to be in OPENING then master waits for timeout monitor to deduct and then call opening.
> 5. If default timeout monitor is configured(1800000 sec/30 min) then it will take 30 mins to open the META region itself.
> Soln:
> ====
> Better not to wait for the Timeout monitor period to open catalog tables on Master restart
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4203) While master restarts and if the
META region's state is OPENING then master cannot assign META until timeout
monitor deducts
Posted by "ramkrishna.s.vasudevan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13086332#comment-13086332 ]
ramkrishna.s.vasudevan commented on HBASE-4203:
-----------------------------------------------
@Stack,
I am planning to implement the same logic that happens in timeoutmonitor when it finds a node in OPENING.
->The existing logic takes care of checking if the node had got changed to OPENED or not. If not forces the node to OFFLINE and again starts assignment. So we can also do the same here.
Also as per the current changes that am trying out in timeoutmonitor(HBASE-4015) this change can also be incorporated. Or do you want me to submit a seperate patch for this?
> While master restarts and if the META region's state is OPENING then master cannot assign META until timeout monitor deducts
> ----------------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-4203
> URL: https://issues.apache.org/jira/browse/HBASE-4203
> Project: HBase
> Issue Type: Bug
> Reporter: ramkrishna.s.vasudevan
> Assignee: ramkrishna.s.vasudevan
> Priority: Minor
>
> 1. Start Master and 2 RS.
> 2. If any exception happens while opening the META region the state in znode will be OPENING.
> 3. If at this point the master restarts then the master will start processing the regions in RIT.
> 4. If the znode is found to be in OPENING then master waits for timeout monitor to deduct and then call opening.
> 5. If default timeout monitor is configured(1800000 sec/30 min) then it will take 30 mins to open the META region itself.
> Soln:
> ====
> Better not to wait for the Timeout monitor period to open catalog tables on Master restart
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4203) While master restarts and if the
META region's state is OPENING then master cannot assign META until timeout
monitor deducts
Posted by "ramkrishna.s.vasudevan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13085610#comment-13085610 ]
ramkrishna.s.vasudevan commented on HBASE-4203:
-----------------------------------------------
In the above logs I had reduced the timeout monitor period to a lesser value.
Hence you can see that the META region assignment started in about 18mins
> While master restarts and if the META region's state is OPENING then master cannot assign META until timeout monitor deducts
> ----------------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-4203
> URL: https://issues.apache.org/jira/browse/HBASE-4203
> Project: HBase
> Issue Type: Bug
> Reporter: ramkrishna.s.vasudevan
> Assignee: ramkrishna.s.vasudevan
> Priority: Minor
>
> 1. Start Master and 2 RS.
> 2. If any exception happens while opening the META region the state in znode will be OPENING.
> 3. If at this point the master restarts then the master will start processing the regions in RIT.
> 4. If the znode is found to be in OPENING then master waits for timeout monitor to deduct and then call opening.
> 5. If default timeout monitor is configured(1800000 sec/30 min) then it will take 30 mins to open the META region itself.
> Soln:
> ====
> Better not to wait for the Timeout monitor period to open catalog tables on Master restart
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira