You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@brooklyn.apache.org by ahgittin <gi...@git.apache.org> on 2014/11/18 12:30:08 UTC
[GitHub] incubator-brooklyn pull request: Fix rebinding as master
GitHub user ahgittin opened a pull request:
https://github.com/apache/incubator-brooklyn/pull/341
Fix rebinding as master
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ahgittin/incubator-brooklyn fix-rebinding-as-master
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-brooklyn/pull/341.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #341
----
commit 2fdc3f9fe8f4984fb58ad1d7f43de18bab655691
Author: Alex Heneveld <al...@cloudsoftcorp.com>
Date: 2014-11-18T11:21:59Z
better handling for the case where an already-managed active item is rebinded active
do not clear the old location's record, and better logging
commit 5ebede9e7224e6457121cd62bd41003727fa95cc
Author: Alex Heneveld <al...@cloudsoftcorp.com>
Date: 2014-11-18T11:21:47Z
Make HA checker not promote again if node is stealing back master
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: Fix rebinding as master
Posted by ahgittin <gi...@git.apache.org>.
Github user ahgittin commented on the pull request:
https://github.com/apache/incubator-brooklyn/pull/341#issuecomment-63457250
the problems here were that:
* the promoteToMaster() call triggered a rebind, and that done when the node was already master would cause locations data to be cleared; where
* the promoteToMaster() call got triggered if N1 was master, but doesn't write or check for a while, then N2 steals master, then next time N1 checks he sees he is master but N2 stole it, however N1 would win an election and so he steals it back from N2
both are now fixed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: Fix rebinding as master
Posted by neykov <gi...@git.apache.org>.
Github user neykov commented on the pull request:
https://github.com/apache/incubator-brooklyn/pull/341#issuecomment-63471942
Apart from the comment above looks good.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: Fix rebinding as master
Posted by ahgittin <gi...@git.apache.org>.
Github user ahgittin commented on the pull request:
https://github.com/apache/incubator-brooklyn/pull/341#issuecomment-63681625
cool, merging
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: Fix rebinding as master
Posted by neykov <gi...@git.apache.org>.
Github user neykov commented on the pull request:
https://github.com/apache/incubator-brooklyn/pull/341#issuecomment-63625117
When `brooklyn.experimental.feature.defaultStandbyIsHot=true` logs report:
```
INFO Management node mYdxmvku running as HA HOT_STANDBY autodetected, master is TBXUmVmD at http://127.0.0.1:8081/
```
but behaves as `STANDBY` - only sync record is read:
```
Checkpointed delta of manager-memento in 3ms: brooklyn.management.ha.ManagementPlaneSyncRecordDeltaImpl[nodes: [BasicManagementNodeSyncRecord{nodeId=mYdxmvku, status=HOT_STANDBY}]]
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: Fix rebinding as master
Posted by ahgittin <gi...@git.apache.org>.
Github user ahgittin commented on the pull request:
https://github.com/apache/incubator-brooklyn/pull/341#issuecomment-63474223
@neykov i don't think so -- the first block checks if the registered (written his record) master is legitimate and if so the original (stolen-from) master defers; original only claims back if he detects the new one fails. (i hadn't accurately described this originally.)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: Fix rebinding as master
Posted by ahgittin <gi...@git.apache.org>.
Github user ahgittin commented on the pull request:
https://github.com/apache/incubator-brooklyn/pull/341#issuecomment-63641396
the failure if you start an explicit master when there is a master is deliberate; if you want to force something to be master then start with auto and then e.g.
curl -v -X POST -d mode=MASTER -H "Brooklyn-Allow-Non-Master-Access: true" http://localhost:8082/v1/server/ha/state
i think that is the right behaviour, a common use of `--highAvailaibility master` is to specify that a node should not run if there is already an existing master. we could introduce a new CLI option for this but sticking with `auto` and then using `curl` to set is what i'd recommend. (note this has always been the behaviour; a change in the PR however is to allow an explicit `curl` to cause something to steal master.)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: Fix rebinding as master
Posted by neykov <gi...@git.apache.org>.
Github user neykov commented on the pull request:
https://github.com/apache/incubator-brooklyn/pull/341#issuecomment-63655873
Can't repeat the above, let's attribute it to my error for the moment.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: Fix rebinding as master
Posted by aledsage <gi...@git.apache.org>.
Github user aledsage commented on a diff in the pull request:
https://github.com/apache/incubator-brooklyn/pull/341#discussion_r20500108
--- Diff: core/src/main/java/brooklyn/entity/rebind/persister/BrooklynMementoPersisterToObjectStore.java ---
@@ -557,7 +557,7 @@ public void delta(Delta delta, PersistenceExceptionHandler exceptionHandler) {
Stopwatch stopwatch = deltaImpl(delta, exceptionHandler);
- if (LOG.isDebugEnabled()) LOG.debug("Checkpointed delta of memento in {}: "
+ if (LOG.isDebugEnabled()) LOG.debug("7 in {}: "
--- End diff --
Why "7 in"?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: Fix rebinding as master
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-brooklyn/pull/341
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: Fix rebinding as master
Posted by ahgittin <gi...@git.apache.org>.
Github user ahgittin commented on the pull request:
https://github.com/apache/incubator-brooklyn/pull/341#issuecomment-63513945
added fixes to a couple related issues noticed when testing the original
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: Fix rebinding as master
Posted by neykov <gi...@git.apache.org>.
Github user neykov commented on the pull request:
https://github.com/apache/incubator-brooklyn/pull/341#issuecomment-63656031
The changes are looking good, tests show correct behaviour. HA feels solid.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: Fix rebinding as master
Posted by ahgittin <gi...@git.apache.org>.
Github user ahgittin commented on a diff in the pull request:
https://github.com/apache/incubator-brooklyn/pull/341#discussion_r20500143
--- Diff: core/src/main/java/brooklyn/entity/rebind/persister/BrooklynMementoPersisterToObjectStore.java ---
@@ -557,7 +557,7 @@ public void delta(Delta delta, PersistenceExceptionHandler exceptionHandler) {
Stopwatch stopwatch = deltaImpl(delta, exceptionHandler);
- if (LOG.isDebugEnabled()) LOG.debug("Checkpointed delta of memento in {}: "
+ if (LOG.isDebugEnabled()) LOG.debug("7 in {}: "
--- End diff --
oops eclipse cut and paste error, will fix
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: Fix rebinding as master
Posted by neykov <gi...@git.apache.org>.
Github user neykov commented on the pull request:
https://github.com/apache/incubator-brooklyn/pull/341#issuecomment-63650760
If HOT_BACKUP is forced into MASTER, then on demotion it reverts to HOT_BACKUP again, while expected would be HOT_STANDBY.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: Fix rebinding as master
Posted by ahgittin <gi...@git.apache.org>.
Github user ahgittin commented on the pull request:
https://github.com/apache/incubator-brooklyn/pull/341#issuecomment-63556803
i think i've fixed the start mode shenanigans, and a few other logging tidies
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: Fix rebinding as master
Posted by ahgittin <gi...@git.apache.org>.
Github user ahgittin commented on the pull request:
https://github.com/apache/incubator-brooklyn/pull/341#issuecomment-63642462
rebased on master, but bloody apache-git updates are being slow so the list has several things already merged. the commits in this PR are:
```
pick 911fb37 tidy think-we-are-master-when-we-are-not discrepancies
pick aae521e force catalog to load on access of getCatalogItems()
pick 9c6b602 log number of threads as part of GC info
pick 07048fc make BrooklynClassLoadingContext instances serializable
pick 1964bc0 fix hot proxying seeing old entities (below the app level)
pick 6127a75 misc tomcat blueprint fix
pick b0bb61a clean up logic for HA start+change mode
pick 288ffd7 allow explicit setting master
pick b9e8e6b more HA change mode fixes
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: Fix rebinding as master
Posted by ahgittin <gi...@git.apache.org>.
Github user ahgittin commented on the pull request:
https://github.com/apache/incubator-brooklyn/pull/341#issuecomment-63459633
slightly changed description. test failure needs a bit more work though :( .
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: Fix rebinding as master
Posted by neykov <gi...@git.apache.org>.
Github user neykov commented on the pull request:
https://github.com/apache/incubator-brooklyn/pull/341#issuecomment-63541176
Looking good, apart from the JSGUI comment.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: Fix rebinding as master
Posted by neykov <gi...@git.apache.org>.
Github user neykov commented on the pull request:
https://github.com/apache/incubator-brooklyn/pull/341#issuecomment-63628490
But `brooklyn.experimental.feature.defaultStandbyIsHot=true` works on demotion.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: Fix rebinding as master
Posted by neykov <gi...@git.apache.org>.
Github user neykov commented on the pull request:
https://github.com/apache/incubator-brooklyn/pull/341#issuecomment-63622823
Starting a second master explicitly causes:
```
java.lang.IllegalStateException: Master already exists; cannot run as master (master BasicManagementNodeSyncRecord{brooklynVersion=0.7.0-SNAPSHOT, nodeId=HwufKTUm, uri=http://127.0.0.1:8081/, status=MASTER, priority=0, localTimestamp=1416394091779=2014-11-19 12:48:11.779, remoteTimestamp=1416394091782=2014-11-19 12:48:11.782}); to trigger a promotion, set a priority and demote the current master
at brooklyn.management.ha.HighAvailabilityManagerImpl.changeMode(HighAvailabilityManagerImpl.java:345) ~[classes/:na]
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: Fix rebinding as master
Posted by neykov <gi...@git.apache.org>.
Github user neykov commented on the pull request:
https://github.com/apache/incubator-brooklyn/pull/341#issuecomment-63629886
Could be due high IO load on the system and
```
Unable to perform atomic file update (D:\Junk\persistence\rebind-test\nodes\LCc7nUAd.LCc7nUAd.tmp to D:\Junk\persistence\rebind-test\nodes\LCc7nUAd); file system not recommended for production HA/DR
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: Fix rebinding as master
Posted by ahgittin <gi...@git.apache.org>.
Github user ahgittin commented on the pull request:
https://github.com/apache/incubator-brooklyn/pull/341#issuecomment-63556691
it's going to be a natural thing to do to take catalogItemId in JS and pass it to a REST call. pointless to parse versions then pass back again.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: Fix rebinding as master
Posted by neykov <gi...@git.apache.org>.
Github user neykov commented on the pull request:
https://github.com/apache/incubator-brooklyn/pull/341#issuecomment-63629356
I get frequent ping pong failovers with local file system, can't figure out what's causing them.
```
Detected master heartbeat timeout. Initiating a new master election. Master was null
```
```
ERROR Management node LCc7nUAd detected master change, stolen from us, deferring to SeFyI0db
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: Fix rebinding as master
Posted by ahgittin <gi...@git.apache.org>.
Github user ahgittin commented on the pull request:
https://github.com/apache/incubator-brooklyn/pull/341#issuecomment-63641452
the failure of `auto` to trigger `hot_standby` is now fixed, in the final commit
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---