You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "nkeywal (JIRA)" <ji...@apache.org> on 2012/12/04 16:06:58 UTC
[jira] [Created] (HBASE-7271) Have a single executor for all
zkWorkers in the assignment manager
nkeywal created HBASE-7271:
------------------------------
Summary: Have a single executor for all zkWorkers in the assignment manager
Key: HBASE-7271
URL: https://issues.apache.org/jira/browse/HBASE-7271
Project: HBase
Issue Type: Bug
Components: master, Region Assignment
Affects Versions: 0.96.0
Reporter: nkeywal
Assignee: nkeywal
Fix For: 0.96.0
The current strategy is to have an array of monothreaded executor, and hash the zk path to ensure that there are no two events on the same region executed in parallel
I think a single executor, as presented in the attached patch, is better because:
- we're guaranteed to use all threads at any time
- if managing one of the event takes longer that expected, the slowness is limited to this region, and not to all regions that have the same hashed/moduloed code
- For the nodeChildrenChanged, there is no need to choose randomly one of the worker (or, once again, the risk to get stuck if one of the event takes time to be managed).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7271) Have a single executor for all
zkWorkers in the assignment manager
Posted by "nkeywal (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-7271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
nkeywal updated HBASE-7271:
---------------------------
Attachment: 7271.v1.patch
> Have a single executor for all zkWorkers in the assignment manager
> ------------------------------------------------------------------
>
> Key: HBASE-7271
> URL: https://issues.apache.org/jira/browse/HBASE-7271
> Project: HBase
> Issue Type: Bug
> Components: master, Region Assignment
> Affects Versions: 0.96.0
> Reporter: nkeywal
> Assignee: nkeywal
> Fix For: 0.96.0
>
> Attachments: 7271.v1.patch
>
>
> The current strategy is to have an array of monothreaded executor, and hash the zk path to ensure that there are no two events on the same region executed in parallel
> I think a single executor, as presented in the attached patch, is better because:
> - we're guaranteed to use all threads at any time
> - if managing one of the event takes longer that expected, the slowness is limited to this region, and not to all regions that have the same hashed/moduloed code
> - For the nodeChildrenChanged, there is no need to choose randomly one of the worker (or, once again, the risk to get stuck if one of the event takes time to be managed).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7271) Have a single executor for all
zkWorkers in the assignment manager
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-7271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13510692#comment-13510692 ]
Ted Yu commented on HBASE-7271:
-------------------------------
{code}
+ static interface RegionRunnable extends Runnable{
{code}
The above interface is only used within AssignmentManager - it can be made private.
> Have a single executor for all zkWorkers in the assignment manager
> ------------------------------------------------------------------
>
> Key: HBASE-7271
> URL: https://issues.apache.org/jira/browse/HBASE-7271
> Project: HBase
> Issue Type: Bug
> Components: master, Region Assignment
> Affects Versions: 0.96.0
> Reporter: nkeywal
> Assignee: nkeywal
> Fix For: 0.96.0
>
> Attachments: 7271.v1.patch
>
>
> The current strategy is to have an array of monothreaded executor, and hash the zk path to ensure that there are no two events on the same region executed in parallel
> I think a single executor, as presented in the attached patch, is better because:
> - we're guaranteed to use all threads at any time
> - if managing one of the event takes longer that expected, the slowness is limited to this region, and not to all regions that have the same hashed/moduloed code
> - For the nodeChildrenChanged, there is no need to choose randomly one of the worker (or, once again, the risk to get stuck if one of the event takes time to be managed).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-7271) Have a single executor for all
zkWorkers in the assignment manager
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-7271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack reassigned HBASE-7271:
----------------------------
Assignee: Jimmy Xiang (was: nkeywal)
Assigning Jimmy for his opinion on nkeyway's proposed change.
> Have a single executor for all zkWorkers in the assignment manager
> ------------------------------------------------------------------
>
> Key: HBASE-7271
> URL: https://issues.apache.org/jira/browse/HBASE-7271
> Project: HBase
> Issue Type: Bug
> Components: master, Region Assignment
> Affects Versions: 0.96.0
> Reporter: nkeywal
> Assignee: Jimmy Xiang
> Fix For: 0.96.0
>
> Attachments: 7271.v1.patch
>
>
> The current strategy is to have an array of monothreaded executor, and hash the zk path to ensure that there are no two events on the same region executed in parallel
> I think a single executor, as presented in the attached patch, is better because:
> - we're guaranteed to use all threads at any time
> - if managing one of the event takes longer that expected, the slowness is limited to this region, and not to all regions that have the same hashed/moduloed code
> - For the nodeChildrenChanged, there is no need to choose randomly one of the worker (or, once again, the risk to get stuck if one of the event takes time to be managed).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7271) Have a single executor for all
zkWorkers in the assignment manager
Posted by "Jimmy Xiang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-7271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13510574#comment-13510574 ]
Jimmy Xiang commented on HBASE-7271:
------------------------------------
Good idea, nkeyway! Back to you.
> Have a single executor for all zkWorkers in the assignment manager
> ------------------------------------------------------------------
>
> Key: HBASE-7271
> URL: https://issues.apache.org/jira/browse/HBASE-7271
> Project: HBase
> Issue Type: Bug
> Components: master, Region Assignment
> Affects Versions: 0.96.0
> Reporter: nkeywal
> Assignee: nkeywal
> Fix For: 0.96.0
>
> Attachments: 7271.v1.patch
>
>
> The current strategy is to have an array of monothreaded executor, and hash the zk path to ensure that there are no two events on the same region executed in parallel
> I think a single executor, as presented in the attached patch, is better because:
> - we're guaranteed to use all threads at any time
> - if managing one of the event takes longer that expected, the slowness is limited to this region, and not to all regions that have the same hashed/moduloed code
> - For the nodeChildrenChanged, there is no need to choose randomly one of the worker (or, once again, the risk to get stuck if one of the event takes time to be managed).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7271) Have a single executor for all
zkWorkers in the assignment manager
Posted by "nkeywal (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-7271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13510703#comment-13510703 ]
nkeywal commented on HBASE-7271:
--------------------------------
I'm ok will all the comments, I will provide an updated version tomorrow, my time.
> Have a single executor for all zkWorkers in the assignment manager
> ------------------------------------------------------------------
>
> Key: HBASE-7271
> URL: https://issues.apache.org/jira/browse/HBASE-7271
> Project: HBase
> Issue Type: Bug
> Components: master, Region Assignment
> Affects Versions: 0.96.0
> Reporter: nkeywal
> Assignee: nkeywal
> Fix For: 0.96.0
>
> Attachments: 7271.v1.patch
>
>
> The current strategy is to have an array of monothreaded executor, and hash the zk path to ensure that there are no two events on the same region executed in parallel
> I think a single executor, as presented in the attached patch, is better because:
> - we're guaranteed to use all threads at any time
> - if managing one of the event takes longer that expected, the slowness is limited to this region, and not to all regions that have the same hashed/moduloed code
> - For the nodeChildrenChanged, there is no need to choose randomly one of the worker (or, once again, the risk to get stuck if one of the event takes time to be managed).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7271) Have a single executor for all
zkWorkers in the assignment manager
Posted by "Jimmy Xiang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-7271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13510688#comment-13510688 ]
Jimmy Xiang commented on HBASE-7271:
------------------------------------
+1 for LinkedHashMultimap
> Have a single executor for all zkWorkers in the assignment manager
> ------------------------------------------------------------------
>
> Key: HBASE-7271
> URL: https://issues.apache.org/jira/browse/HBASE-7271
> Project: HBase
> Issue Type: Bug
> Components: master, Region Assignment
> Affects Versions: 0.96.0
> Reporter: nkeywal
> Assignee: nkeywal
> Fix For: 0.96.0
>
> Attachments: 7271.v1.patch
>
>
> The current strategy is to have an array of monothreaded executor, and hash the zk path to ensure that there are no two events on the same region executed in parallel
> I think a single executor, as presented in the attached patch, is better because:
> - we're guaranteed to use all threads at any time
> - if managing one of the event takes longer that expected, the slowness is limited to this region, and not to all regions that have the same hashed/moduloed code
> - For the nodeChildrenChanged, there is no need to choose randomly one of the worker (or, once again, the risk to get stuck if one of the event takes time to be managed).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7271) Have a single executor for all
zkWorkers in the assignment manager
Posted by "Jimmy Xiang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-7271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13510573#comment-13510573 ]
Jimmy Xiang commented on HBASE-7271:
------------------------------------
Looked at the patch. The idea is great.
{code}
+ // In a multimap, the put order is kept when we retrieve the collection back. We need this
+ // as we want the events to be managed in the same order as we received them.
{code}
Are we sure the order is kept? It seems to me HashMultiMap uses just a regular HashSet to store the values which don't keep the order.
If so, we can just use a hash map from region name to a linked list of RegionRunnable instead.
> Have a single executor for all zkWorkers in the assignment manager
> ------------------------------------------------------------------
>
> Key: HBASE-7271
> URL: https://issues.apache.org/jira/browse/HBASE-7271
> Project: HBase
> Issue Type: Bug
> Components: master, Region Assignment
> Affects Versions: 0.96.0
> Reporter: nkeywal
> Assignee: Jimmy Xiang
> Fix For: 0.96.0
>
> Attachments: 7271.v1.patch
>
>
> The current strategy is to have an array of monothreaded executor, and hash the zk path to ensure that there are no two events on the same region executed in parallel
> I think a single executor, as presented in the attached patch, is better because:
> - we're guaranteed to use all threads at any time
> - if managing one of the event takes longer that expected, the slowness is limited to this region, and not to all regions that have the same hashed/moduloed code
> - For the nodeChildrenChanged, there is no need to choose randomly one of the worker (or, once again, the risk to get stuck if one of the event takes time to be managed).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7271) Have a single executor for all
zkWorkers in the assignment manager
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-7271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13510674#comment-13510674 ]
Ted Yu commented on HBASE-7271:
-------------------------------
bq. If so, we can just use a hash map from region name to a linked list of RegionRunnable instead.
LinkedHashMultimap would give us the ordering guarantee. See its javadoc:
{code}
* Implementation of {@code Multimap} that does not allow duplicate key-value
* entries and that returns collections whose iterators follow the ordering in
* which the data was added to the multimap.
{code}
> Have a single executor for all zkWorkers in the assignment manager
> ------------------------------------------------------------------
>
> Key: HBASE-7271
> URL: https://issues.apache.org/jira/browse/HBASE-7271
> Project: HBase
> Issue Type: Bug
> Components: master, Region Assignment
> Affects Versions: 0.96.0
> Reporter: nkeywal
> Assignee: nkeywal
> Fix For: 0.96.0
>
> Attachments: 7271.v1.patch
>
>
> The current strategy is to have an array of monothreaded executor, and hash the zk path to ensure that there are no two events on the same region executed in parallel
> I think a single executor, as presented in the attached patch, is better because:
> - we're guaranteed to use all threads at any time
> - if managing one of the event takes longer that expected, the slowness is limited to this region, and not to all regions that have the same hashed/moduloed code
> - For the nodeChildrenChanged, there is no need to choose randomly one of the worker (or, once again, the risk to get stuck if one of the event takes time to be managed).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7271) Have a single executor for all
zkWorkers in the assignment manager
Posted by "Jimmy Xiang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-7271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jimmy Xiang updated HBASE-7271:
-------------------------------
Assignee: nkeywal (was: Jimmy Xiang)
> Have a single executor for all zkWorkers in the assignment manager
> ------------------------------------------------------------------
>
> Key: HBASE-7271
> URL: https://issues.apache.org/jira/browse/HBASE-7271
> Project: HBase
> Issue Type: Bug
> Components: master, Region Assignment
> Affects Versions: 0.96.0
> Reporter: nkeywal
> Assignee: nkeywal
> Fix For: 0.96.0
>
> Attachments: 7271.v1.patch
>
>
> The current strategy is to have an array of monothreaded executor, and hash the zk path to ensure that there are no two events on the same region executed in parallel
> I think a single executor, as presented in the attached patch, is better because:
> - we're guaranteed to use all threads at any time
> - if managing one of the event takes longer that expected, the slowness is limited to this region, and not to all regions that have the same hashed/moduloed code
> - For the nodeChildrenChanged, there is no need to choose randomly one of the worker (or, once again, the risk to get stuck if one of the event takes time to be managed).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7271) Have a single executor for all
zkWorkers in the assignment manager
Posted by "nkeywal (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-7271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
nkeywal updated HBASE-7271:
---------------------------
Status: Patch Available (was: Open)
> Have a single executor for all zkWorkers in the assignment manager
> ------------------------------------------------------------------
>
> Key: HBASE-7271
> URL: https://issues.apache.org/jira/browse/HBASE-7271
> Project: HBase
> Issue Type: Bug
> Components: master, Region Assignment
> Affects Versions: 0.96.0
> Reporter: nkeywal
> Assignee: nkeywal
> Fix For: 0.96.0
>
> Attachments: 7271.v1.patch
>
>
> The current strategy is to have an array of monothreaded executor, and hash the zk path to ensure that there are no two events on the same region executed in parallel
> I think a single executor, as presented in the attached patch, is better because:
> - we're guaranteed to use all threads at any time
> - if managing one of the event takes longer that expected, the slowness is limited to this region, and not to all regions that have the same hashed/moduloed code
> - For the nodeChildrenChanged, there is no need to choose randomly one of the worker (or, once again, the risk to get stuck if one of the event takes time to be managed).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7271) Have a single executor for all
zkWorkers in the assignment manager
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-7271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13510616#comment-13510616 ]
Ted Yu commented on HBASE-7271:
-------------------------------
{code}
+ // We want don't want to have two events on the same region managed simultaneously.
{code}
Remove the first 'want' above.
{code}
+ private final Set<String> inProgress = new HashSet<String>();
{code}
There're many actions in progress. How about naming the above variable regionsInProgress ?
{code}
+ private final Multimap<String, RegionRunnable> todo = HashMultimap.create()
{code}
Would zkEventWorkerWaitingList be a better name for todo ?
nit: remove one space before HashMultimap.
{code}
+ protected void zkEventWorkersSubmit(final RegionRunnable regRunnable) {
+ synchronized (inProgress) {
+ if (inProgress.contains(regRunnable.getRegionName())) {
+ synchronized (todo){
+ todo.put(regRunnable.getRegionName(), regRunnable);
+ }
+ } else {
{code}
nit: if you return from if block, you can save one indent for else block where else keyword can be omitted.
> Have a single executor for all zkWorkers in the assignment manager
> ------------------------------------------------------------------
>
> Key: HBASE-7271
> URL: https://issues.apache.org/jira/browse/HBASE-7271
> Project: HBase
> Issue Type: Bug
> Components: master, Region Assignment
> Affects Versions: 0.96.0
> Reporter: nkeywal
> Assignee: nkeywal
> Fix For: 0.96.0
>
> Attachments: 7271.v1.patch
>
>
> The current strategy is to have an array of monothreaded executor, and hash the zk path to ensure that there are no two events on the same region executed in parallel
> I think a single executor, as presented in the attached patch, is better because:
> - we're guaranteed to use all threads at any time
> - if managing one of the event takes longer that expected, the slowness is limited to this region, and not to all regions that have the same hashed/moduloed code
> - For the nodeChildrenChanged, there is no need to choose randomly one of the worker (or, once again, the risk to get stuck if one of the event takes time to be managed).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira