You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@helix.apache.org by lei-xia <gi...@git.apache.org> on 2017/02/06 04:15:50 UTC
[GitHub] helix pull request #67: Merge all new features, bugs and improvements develo...
GitHub user lei-xia opened a pull request:
https://github.com/apache/helix/pull/67
Merge all new features, bugs and improvements developed from LinkedIn's internal Helix branch.
A list of commits:
1. Lei Xia: Avoid moving partitions unnecessarily when auto-rebalancing using default AutoRebalanceStrategy.
2. Lei Xia: Fix bug in AutoRebalanceStrategy to try to assign orphan replicas to its preferred nodes instead of random nodes.`
3. Lei Xia: Fix a bug in BestPossibleExternalViewVerifier.
4. Lei Xia: Move all options from IdealState to ResourceConfig, add Bulder for building ResourceConfig, and a new RebalanceConfig to hold all rebalance options for a resource.
5. Lei Xia: Persist controller leader change history with timestamp for each leader controller.
6. Lei Xia: Persist the timestamp along with controller leader change history.
7. Lei Xia: Refactor/renames the instances releated methods in ClusterDataCache.
8. Lei Xia: Add cluster-level and resource-level config option to allow disable delayed rebalance of entire cluster or individual resource.
9. Boyan Li: Add more messaging metrics to participant.
10. Lei Xia: Log improvement: add resource name to logs in different places.
11. Lei Xia: New DelayedAutoRebalancer featured with delayed partition movements during rebalancing.
12. Lei Xia: Persist assignment map using specific format for MasterSlave resources (This is a short-term solution, we should get rid of this asap).
13. Boyan Li: Add a messaging monitoring mbean to ParticipantStatusMonitor object.
14. Lei Xia: More refactors on rebalancer releated pipelines. 1) Move get rebalancer and mappingCalculator logics out to separate methods to make the main flow clearer. 2) Move definition of ANY_LIVEINSTANCE from HelixConstants to IdealState.
15. Boyan Li: rename ParticipantMonitor class
16. Boyan Li: Revert "add a mbean for participant and emit received msgs"
17. Lei Xia: Persist participant's offline timestamp in ParticipantHistory.
18. Lei Xia: Refactor: add AbstractRebalancer as an abstract class, which contains the default implementations as well as utility functions that will be used by all specific rebalancers.
19. Lei Xia: Add reset() method to MockParticipantManager to allow reuse of the participant.
20. Lei Xia: Refactor: rename ParticipantManagerHelper to ParticipantManager and move all logic to handle new participant session into the ParticipangManager.
21. Lei Xia: Persist session change history with timestamp for each participant.
22. Boyan Li: Add a mbean for participant and emit received msgs
23. Boyan Li: Add logs for session sync messages
24. Boyan Li: participant syncs session id to controller
25. Junkai Xue: Fix BestPossibleExternalViewVerifier toString NPE
26. Lei Xia: Add support for flexible hirerachy representation of a cluster topology.
27. Lei Xia: Add StrictMatchExternalViewVerifier that verifies whether the ExternalViews of given resources (or all resources in the cluster) match exactly as its ideal mapping (in idealstate).
28. Lei Xia: Restructure ClusterVerifiers. Add HelixClusterVerifier interface, add abstract class ZkHelixClusterVerifier, and a BestPossibleExternViewVerifier implementation.
29. Lei Xia: Refactor: put all cluster verifiers into a sub-module of tools.
30. Lei Xia: Add Multi-round CRUSH rebalance strategy.
31. Lei Xia: Add option to allow persisting best possible partition assignment in IdealState for semi-auto and full-auto modes.
32. Lei Xia: Deprecated AutoModeISBuilder and AutoRebalanceModeIsBuilder and created SemiAutoISBuilder and FullAutoISBuilder instead.
33. Lei Xia: Support of client's customized threadpool for state-transition message handling.
34. Lei Xia: Minor fix: Do not set MaxPartitionPerNode in IdealState if it is not greater than 0.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/lei-xia/helix github.helix-0.6.x
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/helix/pull/67.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #67
----
commit 68e2fdca5c0fdf98d93ca59bee05a0ff30a3d54d
Author: Lei Xia <lx...@linkedin.com>
Date: 2017-02-06T00:37:29Z
Minor fix: Do not set MaxPartitionPerNode in IdealState if it is not greater than 0.
commit eb33370289e2fee521a12f6bae901560bd599f57
Author: Lei Xia <lx...@linkedin.com>
Date: 2016-04-15T22:38:15Z
Support of client's customized threadpool for state-transition message handling.
commit f1e9188733ffa72cdc68643061ac367e21bdf460
Author: Lei Xia <lx...@linkedin.com>
Date: 2016-06-07T23:25:36Z
Deprecated AutoModeISBuilder and AutoRebalanceModeIsBuilder and created SemiAutoISBuilder and FullAutoISBuilder instead.
commit 7d0885c78dbb94c2224490f53a161a752ee83de5
Author: Lei Xia <lx...@linkedin.com>
Date: 2016-07-01T23:27:53Z
Add option to allow persisting best possible partition assignment in IdealState for semi-auto and full-auto modes.
commit 0f7c3e42080ba8e2b17e36ca1c5c51c6209b0f03
Author: Lei Xia <lx...@linkedin.com>
Date: 2016-06-27T22:46:13Z
Add Multi-round CRUSH rebalance strategy.
commit 695228e0ec6fd5fc48d5408633205f8ab935838d
Author: Lei Xia <lx...@linkedin.com>
Date: 2016-07-18T23:06:22Z
Refactor: put all cluster verifiers into a sub-module of tools.
commit 04495e707a3a1a9067fd487cac96a169dc45c6dd
Author: Lei Xia <lx...@linkedin.com>
Date: 2016-07-20T01:17:19Z
Restructure ClusterVerifiers. Add HelixClusterVerifier interface, add abstract class ZkHelixClusterVerifier, and a BestPossibleExternViewVerifier implementation.
commit 981d0e295e01cd0839c8d5d0e54350f794ad52f7
Author: Lei Xia <lx...@linkedin.com>
Date: 2016-07-21T18:29:02Z
Add StrictMatchExternalViewVerifier that verifies whether the ExternalViews of given resources (or all resources in the cluster) match exactly as its ideal mapping (in idealstate).
commit 215039b374ba1457d892d5f0c7ad758b393ed23a
Author: Lei Xia <lx...@linkedin.com>
Date: 2016-07-01T17:20:50Z
Add support for flexible hirerachy representation of a cluster topology.
commit 9bf4437aa3ffe64d4cb983fea0a6b81162edeed9
Author: Junkai Xue <jx...@linkedin.com>
Date: 2016-07-29T23:25:01Z
Fix BestPossibleExternalViewVerifier toString NPE
commit 2981bbd11dc869f5235e85e71f1b30368ef7596e
Author: Boyan Li <bo...@linkedin.com>
Date: 2016-08-05T18:07:15Z
participant syncs session id to controller
commit a1278e199b54f29fb37abda20c23262ebb1b9d28
Author: Boyan Li <bo...@linkedin.com>
Date: 2016-08-23T04:24:43Z
Add logs for session sync messages
commit 821fd04dfb1dc66a85d0dfb76e5837a4c8107448
Author: Boyan Li <bo...@linkedin.com>
Date: 2016-08-29T16:47:27Z
Add a mbean for participant and emit received msgs
commit 8b320514da58e38652f79f625576bb9895f6312c
Author: Lei Xia <lx...@linkedin.com>
Date: 2016-08-30T20:29:08Z
Persist session change history with timestamp for each participant.
commit abecf2fe332a37de96ea5679654c38636772a2ed
Author: Lei Xia <lx...@linkedin.com>
Date: 2016-09-09T18:07:37Z
Refactor: rename ParticipantManagerHelper to ParticipantManager and move all logic to handle new participant session into the ParticipangManager.
commit 6f471606f18f6de416c3148bead43577c3a24167
Author: Lei Xia <lx...@linkedin.com>
Date: 2016-09-09T22:58:46Z
Add reset() method to MockParticipantManager to allow reuse of the participant.
RB=811078
G=nuage-reviewers
R=boli,jxue
A=boli
commit 752fd696ed1b8d098a63196a9fd02342d60c2e6d
Author: Lei Xia <lx...@linkedin.com>
Date: 2016-09-12T17:59:40Z
Refactor: add AbstractRebalancer as an abstract class, which contains the default implementations as well as utility functions that will be used by all specific rebalancers.
commit ebfe2ea3ce174b1431423402be84e064ad47bc6e
Author: Lei Xia <lx...@linkedin.com>
Date: 2016-09-02T01:21:36Z
Persist participant's offline timestamp in ParticipantHistory.
This is to persist the timestamp when a participant is going offline.
1) If a participant goes offlien gracefully (by calling disconnect()), participant will write a timestamp to its history record.
2) If a participant goes offline without calling disconnect() (e.g, GC, machine crashes), controller will try to set the timestamp in its pipeline triggered by liveInstanceChanges.
commit 543b3207e669710996a643f541ecc29ba2b2a8f8
Author: Boyan Li <bo...@linkedin.com>
Date: 2016-09-08T23:36:25Z
Revert "add a mbean for participant and emit received msgs"
This reverts commit ccff605eb311072644f355686853351a10ff5b95.
commit 3688630f3cb853658dc1aaefe54d70d29dab5243
Author: Boyan Li <bo...@linkedin.com>
Date: 2016-09-14T17:48:23Z
rename ParticipantMonitor class
RB=813995
G=nuage-reviewers
R=lxia,jxue
A=lxia
commit d9a47b43c3c16d18168247d1dabde1640932815d
Author: Lei Xia <lx...@linkedin.com>
Date: 2016-09-15T18:56:10Z
More refactors on rebalancer releated pipelines. 1) Move get rebalancer and mappingCalculator logics out to separate methods to make the main flow clearer. 2) Move definition of ANY_LIVEINSTANCE from HelixConstants to IdealState.
RB=815358
G=nuage-reviewers
R=boli,jxue
A=jxue
commit db6ab9fe9ea6ebca8c95c14cbc4635ac98935080
Author: Boyan Li <bo...@linkedin.com>
Date: 2016-09-14T22:29:54Z
Add a messaging monitoring mbean to ParticipantStatusMonitor object.
commit 99e6c4a659ce4616b4338ca14f196fbff3c75f4b
Author: Lei Xia <lx...@linkedin.com>
Date: 2016-09-09T20:31:14Z
Persist assignment map using specific format for MasterSlave resources (This is a short-term solution, we should get rid of this asap).
commit d7c035ca41de0c1d01589a3b1b232c5cef21f95c
Author: Lei Xia <lx...@linkedin.com>
Date: 2016-09-12T23:42:17Z
New DelayedAutoRebalancer featured with delayed partition movements during rebalancing.
commit e1f0678a0c918f74e57dc6ee2559ca813ecfd7d7
Author: Lei Xia <lx...@linkedin.com>
Date: 2016-09-19T17:20:01Z
Log improvement: add resource name to logs in different places.
commit 5d9f76e6674458e624fc9142511e7ffcbb7fdc0d
Author: Boyan Li <bo...@linkedin.com>
Date: 2016-09-19T18:42:11Z
Add more messaging metrics to participant.
commit ad4625e4c431903b29827259672446814e2fd0a0
Author: Lei Xia <lx...@linkedin.com>
Date: 2016-09-21T17:53:31Z
Add cluster-level and resource-level config option to allow disable delayed rebalance of entire cluster or individual resource.
commit 7bb3efd15b03ba698c7dc4a8a5277de7870f4014
Author: Lei Xia <lx...@linkedin.com>
Date: 2016-09-23T15:26:50Z
Refactor/renames the instances releated methods in ClusterDataCache.
commit b562026cb3e98b97b6b5359068de8ca925e89b0c
Author: Lei Xia <lx...@linkedin.com>
Date: 2016-08-30T20:26:52Z
Persist the timestamp along with controller leader change history.
commit c622d63c4b777e673edf2d26bfd2a3289f92bc9e
Author: Lei Xia <lx...@linkedin.com>
Date: 2016-10-13T01:15:10Z
Persist controller leader change history with timestamp for each leader controller.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] helix pull request #67: Merge all new features, bugs and improvements develo...
Posted by lei-xia <gi...@git.apache.org>.
Github user lei-xia closed the pull request at:
https://github.com/apache/helix/pull/67
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---