You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@brooklyn.apache.org by aledsage <gi...@git.apache.org> on 2016/03/25 16:25:45 UTC
[GitHub] brooklyn-server pull request: Fix DynamicMultiGroup rebind
GitHub user aledsage opened a pull request:
https://github.com/apache/brooklyn-server/pull/82
Fix DynamicMultiGroup rebind
The `isRebindComplete()` is really hacky, but not sure how better to do this - @ahgittin any advice?
I'd like to merge this PR asap to fix the problem, and then improve on the impl in subsequent PRs. Unless you think the code changes are dangerous!
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/aledsage/brooklyn-server fix/DynamicMultiGroup-rebind
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/brooklyn-server/pull/82.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #82
----
commit 7d7c1a836a5d426b47f82d64202d5859896e140b
Author: Aled Sage <al...@gmail.com>
Date: 2016-03-25T15:18:55Z
AbstractEntity: improve error message on previouslyOwned
commit 67a724e655f33188172c7a5784e98cde809a48ee
Author: Aled Sage <al...@gmail.com>
Date: 2016-03-25T15:19:32Z
DynamicMultiGroup: fix/test rebind
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] brooklyn-server pull request: Fix DynamicMultiGroup rebind
Posted by johnmccabe <gi...@git.apache.org>.
Github user johnmccabe commented on the pull request:
https://github.com/apache/brooklyn-server/pull/82#issuecomment-201363986
LGTM - checked with a clocker running Mesos with multiple managed tasks and an unmanaged tasks, all rebound successfully
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] brooklyn-server pull request: Fix DynamicMultiGroup rebind
Posted by neykov <gi...@git.apache.org>.
Github user neykov commented on a diff in the pull request:
https://github.com/apache/brooklyn-server/pull/82#discussion_r57455466
--- Diff: core/src/main/java/org/apache/brooklyn/entity/group/DynamicMultiGroupImpl.java ---
@@ -119,7 +125,36 @@ public void rebind() {
super.rebind();
if (rescan == null) {
- connectScanner();
+ // The rescan can (in a different thread) cause us to remove the empty groups - i.e. remove
+ // it as a child, and unmanage it. That is dangerous during rebind, because the rebind-thread
+ // may concurrently (or subsequently) be initialising that child entity. It has caused
+ // rebind errors where the child's entity-rebind complains in setParent() that it was
+ // "previouslyOwned". Therefore we defer registering/executing our scanner until rebind is
+ // complete, so all entities are reconstituted.
+ // We don't worry about other managementNodeStates, such as standby: if we were told to
+ // rebind then we are free to fully initialise ourselves. But we do double-check that we
+ // are still managed before trying to execute.
+
+ getExecutionContext().execute(new Runnable() {
+ @Override public void run() {
+ LOG.debug("Deferring scanner for {} until management context initialisation complete", DynamicMultiGroupImpl.this);
+ while (!isRebindComplete()) {
+ Time.sleep(100); // avoid thrashing
+ }
+ LOG.debug("Connecting scanner for {}", DynamicMultiGroupImpl.this);
+ connectScanner();
+ }
+ private boolean isRebindComplete() {
+ // TODO Want to determine if finished rebinding (either success or fail is fine).
+ // But not a clean way to do this that works for both unit tests and live server?!
+ // * In RebindTestFixtureWithApp tests, mgmt.getHighAvailabilityManager().getNodeState()
+ // always returns INITIALIZING.
+ // * The rebind metrics is a hack, and feels very risky for HOT_STANDBY nodes that
+ // may have executed the rebind code multiple times.
--- End diff --
`!mgmt().getRebindManager().isAwaitingInitialRebind()`should do the job.
Tests might need an additional `((LocalManagementContext)getManagementContext()).noteStartupComplete();`
See `HaHotCheckTest` for details.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] brooklyn-server pull request: Fix DynamicMultiGroup rebind
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/brooklyn-server/pull/82
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] brooklyn-server pull request: Fix DynamicMultiGroup rebind
Posted by aledsage <gi...@git.apache.org>.
Github user aledsage commented on a diff in the pull request:
https://github.com/apache/brooklyn-server/pull/82#discussion_r57459946
--- Diff: core/src/main/java/org/apache/brooklyn/entity/group/DynamicMultiGroupImpl.java ---
@@ -119,7 +125,36 @@ public void rebind() {
super.rebind();
if (rescan == null) {
- connectScanner();
+ // The rescan can (in a different thread) cause us to remove the empty groups - i.e. remove
+ // it as a child, and unmanage it. That is dangerous during rebind, because the rebind-thread
+ // may concurrently (or subsequently) be initialising that child entity. It has caused
+ // rebind errors where the child's entity-rebind complains in setParent() that it was
+ // "previouslyOwned". Therefore we defer registering/executing our scanner until rebind is
+ // complete, so all entities are reconstituted.
+ // We don't worry about other managementNodeStates, such as standby: if we were told to
+ // rebind then we are free to fully initialise ourselves. But we do double-check that we
+ // are still managed before trying to execute.
+
+ getExecutionContext().execute(new Runnable() {
+ @Override public void run() {
+ LOG.debug("Deferring scanner for {} until management context initialisation complete", DynamicMultiGroupImpl.this);
+ while (!isRebindComplete()) {
+ Time.sleep(100); // avoid thrashing
+ }
+ LOG.debug("Connecting scanner for {}", DynamicMultiGroupImpl.this);
+ connectScanner();
+ }
+ private boolean isRebindComplete() {
+ // TODO Want to determine if finished rebinding (either success or fail is fine).
+ // But not a clean way to do this that works for both unit tests and live server?!
+ // * In RebindTestFixtureWithApp tests, mgmt.getHighAvailabilityManager().getNodeState()
+ // always returns INITIALIZING.
+ // * The rebind metrics is a hack, and feels very risky for HOT_STANDBY nodes that
+ // may have executed the rebind code multiple times.
--- End diff --
Ah - the `noteStartupComplete()` is what is missing I think. All the sensible looking methods returned "initialising" in the unit tests. Longer term, it would be good to ensure the test code is more like the production code path!
I'll merge as-is. We can look at this again next week. Thanks for the fast review @neykov !
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] brooklyn-server pull request: Fix DynamicMultiGroup rebind
Posted by neykov <gi...@git.apache.org>.
Github user neykov commented on the pull request:
https://github.com/apache/brooklyn-server/pull/82#issuecomment-201340760
LGMT
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---