You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org> on 2010/02/07 09:30:27 UTC
[jira] Created: (HBASE-2190) HRS should report to master when HMsg
are available
HRS should report to master when HMsg are available
---------------------------------------------------
Key: HBASE-2190
URL: https://issues.apache.org/jira/browse/HBASE-2190
Project: Hadoop HBase
Issue Type: Improvement
Affects Versions: 0.20.3
Reporter: Jean-Daniel Cryans
Assignee: Jean-Daniel Cryans
Fix For: 0.20.4, 0.21.0
It still takes a lot of time for the client to see splits or just regions that move around, with default PE it takes around 4 seconds and creating a table takes a bit more than 2 seconds. I remember having the discussion with Stack that HRS.run was not suppose to sleep if any message to send.
Turns out it does sleep.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2190) HRS should report to master when
HMsg are available
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12833207#action_12833207 ]
stack commented on HBASE-2190:
------------------------------
@Kannan In the "commit message" above -- 11/Feb/10 12:15 AM -- I say this:
"Moved housekeeping AFTER pause since when before, its addition of
"processing open" would make it so we didn't sleep and we were
flooding the master with "processing open" messages (this new mechanism
is on a hair-trigger)."
What was happening was if opening a region, in housekeeping, we'd add a opening-region message to the BlockingQueue -- MSG_REPORT_OPENING_REGION (sor something like that) -- which in this new hair-trigger setup would immediately trip a send of the message to the master.... and keep sending messages to the master until the region successfully opened. We'd never sleep (block on the queue).
I moved the housekeeping till after we got something from the blockingqueue. If the region has opened in the meantime, we'll not bother sending MSG_REPORT_OPENING_REGION. If it hasn't opened, it'll be added and get sent on the next report to master.
I hope that helps.
> HRS should report to master when HMsg are available
> ---------------------------------------------------
>
> Key: HBASE-2190
> URL: https://issues.apache.org/jira/browse/HBASE-2190
> Project: Hadoop HBase
> Issue Type: Improvement
> Affects Versions: 0.20.3
> Reporter: Jean-Daniel Cryans
> Assignee: stack
> Fix For: 0.20.4, 0.21.0
>
> Attachments: 2190-v2.patch, 2190-v3.patch, 2190-v3.patch, HBASE-2190.patch
>
>
> It still takes a lot of time for the client to see splits or just regions that move around, with default PE it takes around 4 seconds and creating a table takes a bit more than 2 seconds. I remember having the discussion with Stack that HRS.run was not suppose to sleep if any message to send.
> Turns out it does sleep.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HBASE-2190) HRS should report to master when HMsg
are available
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack reassigned HBASE-2190:
----------------------------
Assignee: stack (was: Jean-Daniel Cryans)
> HRS should report to master when HMsg are available
> ---------------------------------------------------
>
> Key: HBASE-2190
> URL: https://issues.apache.org/jira/browse/HBASE-2190
> Project: Hadoop HBase
> Issue Type: Improvement
> Affects Versions: 0.20.3
> Reporter: Jean-Daniel Cryans
> Assignee: stack
> Fix For: 0.20.4, 0.21.0
>
> Attachments: 2190-v2.patch, 2190-v3.patch, HBASE-2190.patch
>
>
> It still takes a lot of time for the client to see splits or just regions that move around, with default PE it takes around 4 seconds and creating a table takes a bit more than 2 seconds. I remember having the discussion with Stack that HRS.run was not suppose to sleep if any message to send.
> Turns out it does sleep.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-2190) HRS should report to master when HMsg
are available
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-2190:
-------------------------
Release Note: This requires that making a rolling update, you upgrade the master first (so it knows what to do w/ the new split signal this patch introduces).
> HRS should report to master when HMsg are available
> ---------------------------------------------------
>
> Key: HBASE-2190
> URL: https://issues.apache.org/jira/browse/HBASE-2190
> Project: Hadoop HBase
> Issue Type: Improvement
> Affects Versions: 0.20.3
> Reporter: Jean-Daniel Cryans
> Assignee: stack
> Fix For: 0.20.4, 0.21.0
>
> Attachments: 2190-v2.patch, 2190-v3.patch, 2190-v3.patch, HBASE-2190.patch
>
>
> It still takes a lot of time for the client to see splits or just regions that move around, with default PE it takes around 4 seconds and creating a table takes a bit more than 2 seconds. I remember having the discussion with Stack that HRS.run was not suppose to sleep if any message to send.
> Turns out it does sleep.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-2190) HRS should report to master when HMsg
are available
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jean-Daniel Cryans updated HBASE-2190:
--------------------------------------
Attachment: HBASE-2190.patch
Very ugly patch that adds an ArrayBlockingQueue that HRS.run uses to poll without a timeout instead of sleeping. Every time a HMsg is added, an object is also put in that queue (I reuse the same one) so that we never sleep if something is in there. We could transform outboundMsgs into a concurrent structure but it seems problematic with splits since the three of them must be shipped together.
With that patch splits now take around 2 seconds in PE and creating a table takes only 1 second.
> HRS should report to master when HMsg are available
> ---------------------------------------------------
>
> Key: HBASE-2190
> URL: https://issues.apache.org/jira/browse/HBASE-2190
> Project: Hadoop HBase
> Issue Type: Improvement
> Affects Versions: 0.20.3
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.20.4, 0.21.0
>
> Attachments: HBASE-2190.patch
>
>
> It still takes a lot of time for the client to see splits or just regions that move around, with default PE it takes around 4 seconds and creating a table takes a bit more than 2 seconds. I remember having the discussion with Stack that HRS.run was not suppose to sleep if any message to send.
> Turns out it does sleep.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-2190) HRS should report to master when HMsg
are available
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-2190:
-------------------------
Attachment: 2190-v3.patch
THE patch.
> HRS should report to master when HMsg are available
> ---------------------------------------------------
>
> Key: HBASE-2190
> URL: https://issues.apache.org/jira/browse/HBASE-2190
> Project: Hadoop HBase
> Issue Type: Improvement
> Affects Versions: 0.20.3
> Reporter: Jean-Daniel Cryans
> Assignee: stack
> Fix For: 0.20.4, 0.21.0
>
> Attachments: 2190-v2.patch, 2190-v3.patch, 2190-v3.patch, HBASE-2190.patch
>
>
> It still takes a lot of time for the client to see splits or just regions that move around, with default PE it takes around 4 seconds and creating a table takes a bit more than 2 seconds. I remember having the discussion with Stack that HRS.run was not suppose to sleep if any message to send.
> Turns out it does sleep.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2190) HRS should report to master when
HMsg are available
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12832323#action_12832323 ]
stack commented on HBASE-2190:
------------------------------
It passes all tests. Patch is for branch.
> HRS should report to master when HMsg are available
> ---------------------------------------------------
>
> Key: HBASE-2190
> URL: https://issues.apache.org/jira/browse/HBASE-2190
> Project: Hadoop HBase
> Issue Type: Improvement
> Affects Versions: 0.20.3
> Reporter: Jean-Daniel Cryans
> Assignee: stack
> Fix For: 0.20.4, 0.21.0
>
> Attachments: 2190-v2.patch, 2190-v3.patch, HBASE-2190.patch
>
>
> It still takes a lot of time for the client to see splits or just regions that move around, with default PE it takes around 4 seconds and creating a table takes a bit more than 2 seconds. I remember having the discussion with Stack that HRS.run was not suppose to sleep if any message to send.
> Turns out it does sleep.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HBASE-2190) HRS should report to master when HMsg
are available
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack resolved HBASE-2190.
--------------------------
Resolution: Fixed
Hadoop Flags: [Reviewed]
Committed to branch and trunk after making j-d suggested change (thanks for review j-d).
> HRS should report to master when HMsg are available
> ---------------------------------------------------
>
> Key: HBASE-2190
> URL: https://issues.apache.org/jira/browse/HBASE-2190
> Project: Hadoop HBase
> Issue Type: Improvement
> Affects Versions: 0.20.3
> Reporter: Jean-Daniel Cryans
> Assignee: stack
> Fix For: 0.20.4, 0.21.0
>
> Attachments: 2190-v2.patch, 2190-v3.patch, 2190-v3.patch, HBASE-2190.patch
>
>
> It still takes a lot of time for the client to see splits or just regions that move around, with default PE it takes around 4 seconds and creating a table takes a bit more than 2 seconds. I remember having the discussion with Stack that HRS.run was not suppose to sleep if any message to send.
> Turns out it does sleep.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2190) HRS should report to master when
HMsg are available
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12832326#action_12832326 ]
stack commented on HBASE-2190:
------------------------------
I think this patch is good to go. Running tests it seems to do what other stuff used do. Review please.
> HRS should report to master when HMsg are available
> ---------------------------------------------------
>
> Key: HBASE-2190
> URL: https://issues.apache.org/jira/browse/HBASE-2190
> Project: Hadoop HBase
> Issue Type: Improvement
> Affects Versions: 0.20.3
> Reporter: Jean-Daniel Cryans
> Assignee: stack
> Fix For: 0.20.4, 0.21.0
>
> Attachments: 2190-v2.patch, 2190-v3.patch, 2190-v3.patch, HBASE-2190.patch
>
>
> It still takes a lot of time for the client to see splits or just regions that move around, with default PE it takes around 4 seconds and creating a table takes a bit more than 2 seconds. I remember having the discussion with Stack that HRS.run was not suppose to sleep if any message to send.
> Turns out it does sleep.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-2190) HRS should report to master when HMsg
are available
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-2190:
-------------------------
Attachment: 2190-v3.patch
Messages on RS are now on a hair-trigger; as soon as one available, its passed the master. All seems snappier -- splitting and table creation. I think this will help with table enable/disable type stuff too. We'll see. Still testing it up on cluster to make sure no message duplication, etc.
Here are some commit notes for attached patch:
{code}
M src/test/org/apache/hadoop/hbase/TestHMsg.java
Add serialization tests.
M src/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
Changed outboundMsgs from array to LinkedBlockingQueue.
Moved housekeeping AFTER pause since when before, its addition of
"processing open" would make it so we didn't sleep and we were
flooding the master with "processing open" messages (this new mechanism
is on a hair-trigger).
Use new MSG_REPORT_SPLIT_WITH_DAUGHTERS HMsg which bundles up three
HMsgs as one.
Removed all synchronization on this.outboundMsgs. Seems unnecessary.
M src/java/org/apache/hadoop/hbase/master/ServerManager.java
Added handling of new MSG_REPORT_SPLIT_WITH_DAUGHTERS message.
M src/java/org/apache/hadoop/hbase/HMsg.java
Added new MSG_REPORT_SPLIT_WITH_DAUGHTERS message. Carries split
daughters as payload rather than send them as new individual messages.
Should be self-migrating. Will need to restart master first though
when doing a rolling-restart.
{code}
> HRS should report to master when HMsg are available
> ---------------------------------------------------
>
> Key: HBASE-2190
> URL: https://issues.apache.org/jira/browse/HBASE-2190
> Project: Hadoop HBase
> Issue Type: Improvement
> Affects Versions: 0.20.3
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.20.4, 0.21.0
>
> Attachments: 2190-v2.patch, 2190-v3.patch, HBASE-2190.patch
>
>
> It still takes a lot of time for the client to see splits or just regions that move around, with default PE it takes around 4 seconds and creating a table takes a bit more than 2 seconds. I remember having the discussion with Stack that HRS.run was not suppose to sleep if any message to send.
> Turns out it does sleep.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2190) HRS should report to master when
HMsg are available
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12833208#action_12833208 ]
stack commented on HBASE-2190:
------------------------------
Oh, @Kannan, thanks for reviewing change.
> HRS should report to master when HMsg are available
> ---------------------------------------------------
>
> Key: HBASE-2190
> URL: https://issues.apache.org/jira/browse/HBASE-2190
> Project: Hadoop HBase
> Issue Type: Improvement
> Affects Versions: 0.20.3
> Reporter: Jean-Daniel Cryans
> Assignee: stack
> Fix For: 0.20.4, 0.21.0
>
> Attachments: 2190-v2.patch, 2190-v3.patch, 2190-v3.patch, HBASE-2190.patch
>
>
> It still takes a lot of time for the client to see splits or just regions that move around, with default PE it takes around 4 seconds and creating a table takes a bit more than 2 seconds. I remember having the discussion with Stack that HRS.run was not suppose to sleep if any message to send.
> Turns out it does sleep.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2190) HRS should report to master when
HMsg are available
Posted by "Kannan Muthukkaruppan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12833183#action_12833183 ]
Kannan Muthukkaruppan commented on HBASE-2190:
----------------------------------------------
One question about the fix.
Prior to the fix, the housekeeping() call (to report back to the master that it is processing an open region request) used to happen before it went to sleep at the bottom of the HRS run loop. Now, it happens after the sleep (i.e. poll on the LBQ). Is this a deliberate change? Just trying to understand. Wouldn't the former be a little better?
> HRS should report to master when HMsg are available
> ---------------------------------------------------
>
> Key: HBASE-2190
> URL: https://issues.apache.org/jira/browse/HBASE-2190
> Project: Hadoop HBase
> Issue Type: Improvement
> Affects Versions: 0.20.3
> Reporter: Jean-Daniel Cryans
> Assignee: stack
> Fix For: 0.20.4, 0.21.0
>
> Attachments: 2190-v2.patch, 2190-v3.patch, 2190-v3.patch, HBASE-2190.patch
>
>
> It still takes a lot of time for the client to see splits or just regions that move around, with default PE it takes around 4 seconds and creating a table takes a bit more than 2 seconds. I remember having the discussion with Stack that HRS.run was not suppose to sleep if any message to send.
> Turns out it does sleep.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2190) HRS should report to master when
HMsg are available
Posted by "Kannan Muthukkaruppan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12833244#action_12833244 ]
Kannan Muthukkaruppan commented on HBASE-2190:
----------------------------------------------
aah.. missed your commit comment earlier.
Also, I hadn't realized that we <<< keep sending messages to the master until the region successfully opened>>>. But upon reading the code further, it looks like this happens via the "Progressable" reporter mechanism (in HRegionServer:instantiateRegion).
Makes sense now.
> HRS should report to master when HMsg are available
> ---------------------------------------------------
>
> Key: HBASE-2190
> URL: https://issues.apache.org/jira/browse/HBASE-2190
> Project: Hadoop HBase
> Issue Type: Improvement
> Affects Versions: 0.20.3
> Reporter: Jean-Daniel Cryans
> Assignee: stack
> Fix For: 0.20.4, 0.21.0
>
> Attachments: 2190-v2.patch, 2190-v3.patch, 2190-v3.patch, HBASE-2190.patch
>
>
> It still takes a lot of time for the client to see splits or just regions that move around, with default PE it takes around 4 seconds and creating a table takes a bit more than 2 seconds. I remember having the discussion with Stack that HRS.run was not suppose to sleep if any message to send.
> Turns out it does sleep.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2190) HRS should report to master when
HMsg are available
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12832356#action_12832356 ]
Jean-Daniel Cryans commented on HBASE-2190:
-------------------------------------------
+1
I adapted and ran the patch on trunk with 1 machine, created 75 regions, did some enabling/disabling. It's working and it's faster.
For the patch itself, I think HRS. getOutboundMsgs should be renamed now that it doesn't get you anything. That can be fixed on commit.
> HRS should report to master when HMsg are available
> ---------------------------------------------------
>
> Key: HBASE-2190
> URL: https://issues.apache.org/jira/browse/HBASE-2190
> Project: Hadoop HBase
> Issue Type: Improvement
> Affects Versions: 0.20.3
> Reporter: Jean-Daniel Cryans
> Assignee: stack
> Fix For: 0.20.4, 0.21.0
>
> Attachments: 2190-v2.patch, 2190-v3.patch, 2190-v3.patch, HBASE-2190.patch
>
>
> It still takes a lot of time for the client to see splits or just regions that move around, with default PE it takes around 4 seconds and creating a table takes a bit more than 2 seconds. I remember having the discussion with Stack that HRS.run was not suppose to sleep if any message to send.
> Turns out it does sleep.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-2190) HRS should report to master when HMsg
are available
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-2190:
-------------------------
Attachment: 2190-v2.patch
Use a LinkedBlockingQueue. Removed the old array of HMsgs and use this LBQ instead. Making new HMsg type, a split that carries the daughters in it rather than send as new messages (goes w/ the general tendency of this patch making all operations more lively). Patch is not tested yet. Please don't review yet.
> HRS should report to master when HMsg are available
> ---------------------------------------------------
>
> Key: HBASE-2190
> URL: https://issues.apache.org/jira/browse/HBASE-2190
> Project: Hadoop HBase
> Issue Type: Improvement
> Affects Versions: 0.20.3
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.20.4, 0.21.0
>
> Attachments: 2190-v2.patch, HBASE-2190.patch
>
>
> It still takes a lot of time for the client to see splits or just regions that move around, with default PE it takes around 4 seconds and creating a table takes a bit more than 2 seconds. I remember having the discussion with Stack that HRS.run was not suppose to sleep if any message to send.
> Turns out it does sleep.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.