You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@brooklyn.apache.org by ahgittin <gi...@git.apache.org> on 2014/11/11 02:41:52 UTC
[GitHub] incubator-brooklyn pull request: Persistence can export, and more,...
GitHub user ahgittin opened a pull request:
https://github.com/apache/incubator-brooklyn/pull/314
Persistence can export, and more, including OSGi multi-versions
First part of a bunch of changes to make persistence and osgi more reliable
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ahgittin/incubator-brooklyn persistence-can-export-and-more
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-brooklyn/pull/314.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #314
----
commit a9dacaf80d696371e6ac58532cedf67d7b2b2fbd
Author: Alex Heneveld <al...@cloudsoftcorp.com>
Date: 2014-11-06T11:06:10Z
make heartbeat timeout (to trigger failover) and poll period configurable from brooklyn.properties
commit 6e59b05d690ae79fd3fc705b1011e93fdf92c640
Author: Alex Heneveld <al...@cloudsoftcorp.com>
Date: 2014-11-06T17:41:33Z
support multiple instances of the same osgi symbolic-name:version
commit 9e3e734243652057ef6f965d5c7d5a26947104a7
Author: Alex Heneveld <al...@cloudsoftcorp.com>
Date: 2014-11-07T08:38:24Z
clean up copyPersistedState code
move shared code to new `BrooklynPersistenceUtils`,
make a new `LocationWithPersistenceStore.newPersistenceObjectStore` interface,
and add documentation for `copy-state` command
commit 4f3c2b609553c9372c65128aed1fe8c7159075de
Author: Alex Heneveld <al...@cloudsoftcorp.com>
Date: 2014-11-10T14:18:47Z
support for exporting persistent state
commit fac445af597c7462d6b66447d31ac46ca06f38c3
Author: Alex Heneveld <al...@cloudsoftcorp.com>
Date: 2014-11-10T21:26:42Z
more persistence, mainly tidying code to treat all BrooklynObject instances the same
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: Persistence hardening - creates c...
Posted by neykov <gi...@git.apache.org>.
Github user neykov commented on the pull request:
https://github.com/apache/incubator-brooklyn/pull/314#issuecomment-63025527
Starting as HOT_BACKUP with existing master behaves as STANDBY (not rebinding).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: Persistence hardening - creates c...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-brooklyn/pull/314
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: Persistence hardening - creates c...
Posted by ahgittin <gi...@git.apache.org>.
Github user ahgittin commented on the pull request:
https://github.com/apache/incubator-brooklyn/pull/314#issuecomment-62934614
interim update, collating the issues:
> Can't use object store locations (unmanage problem)
Fixed
> When using object store for backups I get a NPE
TODO - what is `BrooklynPersistenceUtils.createBackup(BrooklynPersistenceUtils.java:248)` for you? i've looked in a few versions and i can't tell waht's going on here.
> NPE when changing standby state
I think this is user-error somehow. :) The only way `null` can get in there (it looks like `startMode`) is if it comes from the REST argument. I've tried your steps and it worked fine.
> /persist/export returns an empty file with missing extension
Added filename. Otherwise I could not reproduce the empty example. Make sure it is not a STANDBY node.
> Always defaults to local file system.
TODO
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: Persistence hardening - creates c...
Posted by ahgittin <gi...@git.apache.org>.
Github user ahgittin commented on the pull request:
https://github.com/apache/incubator-brooklyn/pull/314#issuecomment-62971864
latest commits fix all the issues, apart from the weirdness with state transitions. i tested that quite a lot -- but will play around some more.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: Persistence hardening - creates c...
Posted by neykov <gi...@git.apache.org>.
Github user neykov commented on the pull request:
https://github.com/apache/incubator-brooklyn/pull/314#issuecomment-62967070
> NPE when changing standby state
Confirming it was my error (wrong parameter name). Could you add a checkNotNull in the rest method to prevent future confusion.
Some weird messages when changing states
* Starting as master and switching to HOT_STANDBY results in
```
2014-11-13 22:45:54,255 WARN Management node pqvmrTbg detected master change, from (?) to <none>
2014-11-13 22:45:54,592 INFO Rebind complete (HOT_STANDBY, iteration 0) in 462ms: 0 apps, 0 entities, 0 locations, 0 policies, 0 enrichers, 0 feeds, 9 catalog items
2014-11-13 22:45:55,987 INFO Rebinding from JcloudsLocation[swift:https://ams01.objectstorage.softlayer.net/auth/v1.0:IBMOS321366-2:cloudsoft/swift:https://ams01.objectstorage.softlayer.net/auth/v1.0]:svet_test_201411...
2014-11-13 22:45:55,994 INFO Rebind complete (MASTER, iteration 1) in 491ms: 0 apps, 0 entities, 0 locations, 0 policies, 0 enrichers, 0 feeds, 9 catalog items
2014-11-13 22:45:56,744 INFO Back-up of persisted state created on promotion, in C:\Users\sam\.brooklyn\brooklyn-persisted-state\backups\2014-11-13-2245-pqvmrTbg-promotion-kU7a
2014-11-13 22:45:57,173 WARN This management node (pqvmrTbg) supposed to be master but reportedly unhealthy? no-op as expect other node to fix: self=BasicManagementNodeSyncRecord{brooklynVersion=0.7.0-SNAPSHOT, nodeId=pqvmrTbg, uri=http://127.0.0.1:8081/, status=STANDBY, priority=0, localTimestamp=1415911556744=2014-11-13 22:45:56.744, remoteTimestamp=1415911559000=2014-11-13 22:45:59.000}
2014-11-13 22:45:57,173 INFO Management node pqvmrTbg running as HA STANDBY (caller requested HOT_STANDBY, will attempt rebind for HOT_STANDBY next); master pqvmrTbg
2014-11-13 22:45:57,366 WARN Unable to promote pqvmrTbg to HOT_STANDBY, switching to FAILED: java.lang.IllegalStateException: Cannot start read-only when already running with persistence
java.lang.IllegalStateException: Cannot start read-only when already running with persistence
at brooklyn.entity.rebind.RebindManagerImpl.startReadOnly(RebindManagerImpl.java:312) ~[brooklyn-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
2014-11-13 22:45:58,270 WARN Management node pqvmrTbg unable to promote to HOT_STANDBY (currently FAILED); (see log for further details)
```
* MASTER -> STANDBY
```
2014-11-13 23:11:31,490 INFO Back-up of persisted state created on demotion, in C:\Users\sam\.brooklyn\brooklyn-persisted-state\backups\2014-11-13-2311-hOUE8a2g-demotion-foea
2014-11-13 23:11:33,639 INFO Rebinding from JcloudsLocation[swift:https://ams01.objectstorage.softlayer.net/auth/v1.0:IBMOS321366-2:cloudsoft/swift:https://ams01.objectstorage.softlayer.net/auth/v1.0]:svet_test_201411...
2014-11-13 23:11:33,649 INFO Rebind complete (MASTER) in 522ms: 0 apps, 0 entities, 0 locations, 0 policies, 0 enrichers, 0 feeds, 9 catalog items
2014-11-13 23:11:34,332 INFO Back-up of persisted state created on promotion, in C:\Users\sam\.brooklyn\brooklyn-persisted-state\backups\2014-11-13-2311-hOUE8a2g-promotion-DG1P
2014-11-13 23:11:34,752 WARN This management node (hOUE8a2g) supposed to be master but reportedly unhealthy? no-op as expect other node to fix: self=BasicManagementNodeSyncRecord{brooklynVersion=0.7.0-SNAPSHOT, nodeId=hOUE8a2g, uri=http://127.0.0.1:8081/, status=STANDBY, priority=0, localTimestamp=1415913094333=2014-11-13 23:11:34.333, remoteTimestamp=1415913096000=2014-11-13 23:11:36.000}
2014-11-13 23:11:34,753 INFO Management node hOUE8a2g running as HA STANDBY (explicitly requested); master hOUE8a2g
2014-11-13 23:11:35,169 WARN This management node (hOUE8a2g) supposed to be master but reportedly unhealthy? no-op as expect other node to fix: self=BasicManagementNodeSyncRecord{brooklynVersion=0.7.0-SNAPSHOT, nodeId=hOUE8a2g, uri=http://127.0.0.1:8081/, status=STANDBY, priority=0, localTimestamp=1415913094754=2014-11-13 23:11:34.754, remoteTimestamp=1415913097000=2014-11-13 23:11:37.000}
2014-11-13 23:11:36,576 WARN This management node (hOUE8a2g) supposed to be master but reportedly unhealthy? no-op as expect other node to fix: self=BasicManagementNodeSyncRecord{brooklynVersion=0.7.0-SNAPSHOT, nodeId=hOUE8a2g, uri=http://127.0.0.1:8081/, status=STANDBY, priority=0, localTimestamp=1415913096171=2014-11-13 23:11:36.171, remoteTimestamp=1415913098000=2014-11-13 23:11:38.000}
(repeating indefinitely)
```
* MASTER -> HOT_BACKUP
```
...
2014-11-13 22:49:11,004 WARN Cannot request read-only mode for brooklyn.entity.rebind.RebindManagerImpl@e64686[mgmt=ec8oxm5d] when already running - Task[Periodic read-only rebind:TZ3dbgWD]; ignoring
...
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: Persistence hardening - creates c...
Posted by neykov <gi...@git.apache.org>.
Github user neykov commented on a diff in the pull request:
https://github.com/apache/incubator-brooklyn/pull/314#discussion_r20349081
--- Diff: usage/rest-server/src/main/java/brooklyn/rest/resources/ServerResource.java ---
@@ -268,4 +294,31 @@ public String getUser() {
}
}
+ @Override
+ public Response exportPersistenceData(String preferredOrigin) {
+ return exportPersistenceData(TypeCoercions.coerce(preferredOrigin, MementoCopyMode.class));
+ }
+
+ protected Response exportPersistenceData(MementoCopyMode preferredOrigin) {
+ if (!Entitlements.isEntitled(mgmt().getEntitlementManager(), Entitlements.SEE_ALL_SERVER_INFO, null))
+ throw WebResourceUtils.unauthorized("User '%s' is not authorized for this operation", Entitlements.getEntitlementContext().user());
+
+ try {
+ String label = mgmt().getManagementNodeId()+"-"+Time.makeDateSimpleStampString();
+ PersistenceObjectStore targetStore = BrooklynPersistenceUtils.newPersistenceObjectStore(mgmt(), null,
+ "web-persistence-"+label+"-"+Identifiers.makeRandomId(4));
+ BrooklynPersistenceUtils.writeMemento(mgmt(), targetStore, preferredOrigin);
+
+ ByteArrayOutputStream baos = new ByteArrayOutputStream();
+ ArchiveBuilder.zip().addDirContentsAt( ((FileBasedObjectStore)targetStore).getBaseDir(), "/" ).stream(baos);
--- End diff --
* Still getting an empty archive. If I open it with another unzip app then I get a subfolder with no name which contains the files along with another empty subfolder.
* When replacing "/" with "" then the default unzip app shows the files at top level, but when opening with the alternative app (7Zip) I still get the empty subfolder.
* Replacing "/" with getBaseDir().getName() seems to work the best - no unexpected phantom folders, works with the default Windows unzip app.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: Persistence hardening - creates c...
Posted by ahgittin <gi...@git.apache.org>.
Github user ahgittin commented on the pull request:
https://github.com/apache/incubator-brooklyn/pull/314#issuecomment-62971999
i already added that rest check btw :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: Persistence hardening - creates c...
Posted by ahgittin <gi...@git.apache.org>.
Github user ahgittin commented on the pull request:
https://github.com/apache/incubator-brooklyn/pull/314#issuecomment-62886519
btw there seems to be some sort of leak which isn't cleaning up deleted entities properly from the persistence store, which can cause a lot of dangling references. i'm not sure if this is tests or where this is coming from. the workaround is to set danglingMinRequiredHealthy... to "alwaysHealthy" (from QuorumChecks), in most contexts, or too investigate if there is concern about data store integrity. (or `rm -rf ~/.brooklyn/brooklyn-persisted-state`!) would be good to find a long term fix finding out where the dangles are coming from.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: Persistence hardening - creates c...
Posted by ahgittin <gi...@git.apache.org>.
Github user ahgittin commented on the pull request:
https://github.com/apache/incubator-brooklyn/pull/314#issuecomment-62931180
jenkins failure is unrelated:
brooklyn.util.net.NetworkingUtilsTest.testIsPortAvailableReportsFalseWhenPortIsInUse
brooklyn.util.net.NetworkingUtilsTest.testIsPortAvailableReportsPromptly
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: Persistence hardening - creates c...
Posted by ahgittin <gi...@git.apache.org>.
Github user ahgittin commented on the pull request:
https://github.com/apache/incubator-brooklyn/pull/314#issuecomment-62938304
the backups in object store is due to:
```
org.jclouds.blobstore.KeyNotFoundException: backups/2014-11-13-1809-KltC3rit-promotion-JshU not found in container brooklyn-persisted-state: <html><h1>Not Found</h1><p>The resource could not be found.</p></html>
at org.jclouds.openstack.swift.handlers.ParseSwiftErrorFromHttpResponse.handleError(ParseSwiftErrorFromHttpResponse.java:82) ~[swift-1.8.1.jar:1.8.1]
Caused by: org.jclouds.http.HttpResponseException: command: PUT https://ams01.objectstorage.softlayer.net/v1/AUTH_46412265-236a-4d9a-9292-5f9a1a62d3e9/brooklyn-persisted-state/backups/2014-11-13-1809-KltC3rit-promotion-JshU HTTP/1.1 failed with response: HTTP/1.1 404 Not Found; content: [<html><h1>Not Found</h1><p>The resource could not be found.</p></html>]
at org.jclouds.openstack.swift.handlers.ParseSwiftErrorFromHttpResponse.handleError(ParseSwiftErrorFromHttpResponse.java:52) ~[swift-1.8.1.jar:1.8.1]
```
need to stop now but will fix this later
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: Persistence hardening - creates c...
Posted by ahgittin <gi...@git.apache.org>.
Github user ahgittin commented on the pull request:
https://github.com/apache/incubator-brooklyn/pull/314#issuecomment-63148440
good fixes @neykov -- there are a couple of niggles (summarised below) but working well now so will tidy these subsequently. merging.
niggles left:
* duplicate rebind log line about iteration 0
* persistence-test-Xxxx containers leftover
* empty folder created with the same name as its parent folder
can you confirm @neykov ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: Persistence hardening - creates c...
Posted by neykov <gi...@git.apache.org>.
Github user neykov commented on the pull request:
https://github.com/apache/incubator-brooklyn/pull/314#issuecomment-63021229
Lots of brooklyn-persistence-test-XXX files in object store left. Is the test cleaning up correctly?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: Persistence hardening - creates c...
Posted by ahgittin <gi...@git.apache.org>.
Github user ahgittin commented on the pull request:
https://github.com/apache/incubator-brooklyn/pull/314#issuecomment-62987072
rebased, and also in the last one fixed the state change problems you found @neykov -- i fiddled with something and it broke it all :( but now it's even better than it was
ready for merging i think. any last comments @neykov ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: Persistence hardening - creates c...
Posted by neykov <gi...@git.apache.org>.
Github user neykov commented on a diff in the pull request:
https://github.com/apache/incubator-brooklyn/pull/314#discussion_r20302497
--- Diff: core/src/main/java/brooklyn/config/BrooklynServerConfig.java ---
@@ -55,31 +54,50 @@
public static final ConfigKey<String> BROOKLYN_DATA_DIR = newStringConfigKey(
"brooklyn.datadir", "Directory for writing all brooklyn data");
- public static final String DEFAULT_PERSISTENCE_CONTAINER_NAME = "brooklyn-persisted-state";
- /** on file system, the 'data' subdir is used so that there is an obvious place to put backup dirs */
- public static final String DEFAULT_PERSISTENCE_DIR_FOR_FILESYSTEM = Os.mergePaths(DEFAULT_PERSISTENCE_CONTAINER_NAME, "data");
-
/**
* Provided for setting; consumers should query the management context persistence subsystem
- * for the actual target, or use {@link #resolvePersistencePath(String, StringConfigMap, String)}
+ * for the actual target, or use {@link BrooklynServerPaths#newMainPersistencePathResolver(ManagementContext)}
* if trying to resolve the value
*/
public static final ConfigKey<String> PERSISTENCE_DIR = newStringConfigKey(
"brooklyn.persistence.dir",
- "Directory or container name for writing brooklyn persisted state");
+ "Directory or container name for writing persisted state");
public static final ConfigKey<String> PERSISTENCE_LOCATION_SPEC = newStringConfigKey(
"brooklyn.persistence.location.spec",
- "Optional location spec string for an object store (e.g. jclouds:swift:URL) where persisted state should be kept;"
+ "Optional location spec string for an object store (e.g. jclouds:swift:URL) where persisted state should be kept; "
+ "if blank or not supplied, the file system is used");
+ public static final ConfigKey<String> PERSISTENCE_BACKUPS_DIR = newStringConfigKey(
+ "brooklyn.persistence.backups.dir",
+ "Directory or container name for writing backups of persisted state; "
+ + "defaults to 'backups' inside the default persistence directory");
+
+ public static final ConfigKey<String> PERSISTENCE_BACKUPS_LOCATION_SPEC = newStringConfigKey(
+ "brooklyn.persistence.backups.location.spec",
+ "Location spec string for an object store (e.g. jclouds:swift:URL) where backups of persisted state should be kept; "
+ + "defaults to the same location spec as regular persisted state, failing back to local file system");
--- End diff --
Always defaults to local file system.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: Persistence hardening - creates c...
Posted by ahgittin <gi...@git.apache.org>.
Github user ahgittin commented on the pull request:
https://github.com/apache/incubator-brooklyn/pull/314#issuecomment-62839318
Here is a sample of metrics which you can get from `/v1/server/ha/metrics/` btw:
```
{
"state": "HOT_BACKUP",
"uptime": "3m 1s",
"currentTimeUtc": 1415849932119,
"startTimeUtc": 1415849750199,
"highAvailability": {
"priority": 0,
"pollPeriod": 1000,
"heartbeatTimeout": 30000,
"history": [
{ "state": "HOT_BACKUP",
"timestamp": 1415849753591
},
{ "state": "HOT_BACKUP",
"timestamp": 1415849753551
}
]
},
"rebind": {
"count": 58,
"lastSuccessTimeUtc": 1415849932045,
"lastSuccessTimeMillisSince": 74,
"lastDuration": 75,
"failureCount": 110,
"lastFailureTimeUtc": 1415849925572,
"lastFailureTimeMillisSince": 6547,
"errorMessages": [{
"error": [
"brooklyn.util.exceptions.PropagatedRuntimeException: Failure rebinding: Too many dangling references: 6 of 10",
"No enricher found with id NBJrXNOH; returning null",
...
"Dangling items (6 of 10) found on rebind exceeds quorum, assuming failed: [ZSf8D8qH, QxhnTlzP, NBJrXNOH, QWZ1gHjQ, l8CgpwR4, A5WN9d0j]"
],
"timestamp": 1415849925572
},{
"error": [ ... ],
"timestamp": 1415849924521
}
]
},
"persist": {
"count": 0,
"lastSuccessTimeUtc": null,
"lastSuccessTimeMillisSince": null,
"lastDuration": null,
"failureCount": 0,
"lastFailureTimeUtc": null,
"lastFailureTimeMillisSince": null,
"errorMessages": []
},
"firstRebindCounts": {
"applications": 1,
"entities": 2,
"allItems": 10
},
"managementStatePersistence": {
"read": {
"count": 180,
"failureCount": 0,
"lastSuccessTime": 1415849931970,
"lastDuration": 2,
"lastFailureTime": null,
"errorMessages": []
},
"write": {
"count": 179,
"failureCount": 0,
"lastSuccessTime": 1415849931968,
"lastDuration": 0,
"lastFailureTime": null,
"errorMessages": []
}
}
}
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: Persistence hardening - creates c...
Posted by ahgittin <gi...@git.apache.org>.
Github user ahgittin commented on the pull request:
https://github.com/apache/incubator-brooklyn/pull/314#issuecomment-62909653
fixed those
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: Persistence hardening - creates c...
Posted by neykov <gi...@git.apache.org>.
Github user neykov commented on the pull request:
https://github.com/apache/incubator-brooklyn/pull/314#issuecomment-63030989
Apart from the above comments HA feels stable.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: Persistence hardening - creates c...
Posted by ahgittin <gi...@git.apache.org>.
Github user ahgittin commented on the pull request:
https://github.com/apache/incubator-brooklyn/pull/314#issuecomment-63040252
Comments:
> empty folder created with the same name as its parent folder
JcloudsBlobStoreBasedObjectStore has new logic to be smart about subpaths since obj store treats them specially; but given `seg1/seg2/seg3/name` it splits it into `seg1` and `seg2/seg3` so the obvious bugs there wouldn't explain why `seg3` is repeated twice. OTOH if the code is outwith that class then i'd expect the problem to happen on localhost/filesystem. similarly `BrooklynServerPaths.resolveWithSubpathFor(subpath)` seems correctly to only include the subpath once, and then forget it. so i'm confused. but those are the places to look.
> duplicate rebind log line about iteration 0
i thought i removed the duplicate "start rebinding" but apparently not
> persistence-test-Xxxx containers leftover
i put logic to delete the containers but i didn't examine the result
> zip issues, empty archive, etc
surprising; likely to do with our zip encoding routines. not sure how much they've been tested. putting everything into a subfolder is probably actually a good idea
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: Persistence hardening - creates c...
Posted by neykov <gi...@git.apache.org>.
Github user neykov commented on the pull request:
https://github.com/apache/incubator-brooklyn/pull/314#issuecomment-62926245
Found the following things while testing, noting them here so I can get back and re-test:
* Can't use object store locations (unmanage problem)
* NPE when changing standby state
* /persist/export returns an empty file with missing extension
* When using object store for backups I get a NPE
---
Failover error:
```
java.lang.NullPointerException: null
at brooklyn.management.ha.HighAvailabilityManagerImpl.changeMode(HighAvailabilityManagerImpl.java:300) ~[brooklyn-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
at brooklyn.management.ha.HighAvailabilityManagerImpl.changeMode(HighAvailabilityManagerImpl.java:254) ~[brooklyn-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
```
---
Backup error:
```
java.lang.NullPointerException: null
at brooklyn.entity.rebind.persister.BrooklynPersistenceUtils.createBackup(BrooklynPersistenceUtils.java:248) ~[
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: Persistence hardening - creates c...
Posted by neykov <gi...@git.apache.org>.
Github user neykov commented on the pull request:
https://github.com/apache/incubator-brooklyn/pull/314#issuecomment-63021176
When creating a backup on state change I get an empty folder at brooklyn-persisted-state/backups/2014-11-14-0948-X5CE0njT-demotion-ztmK/2014-11-14-0948-X5CE0njT-demotion-ztmK. Notice the duplicate folder name. The parent contains the correct files.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: Persistence hardening - creates c...
Posted by neykov <gi...@git.apache.org>.
Github user neykov commented on a diff in the pull request:
https://github.com/apache/incubator-brooklyn/pull/314#discussion_r20296070
--- Diff: core/src/main/java/brooklyn/entity/rebind/persister/BrooklynPersistenceUtils.java ---
@@ -0,0 +1,258 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package brooklyn.entity.rebind.persister;
+
+import java.util.List;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import brooklyn.basic.BrooklynObject;
+import brooklyn.catalog.CatalogItem;
+import brooklyn.config.BrooklynServerConfig;
+import brooklyn.config.BrooklynServerPaths;
+import brooklyn.entity.Entity;
+import brooklyn.entity.Feed;
+import brooklyn.entity.basic.Entities;
+import brooklyn.entity.basic.EntityInternal;
+import brooklyn.entity.rebind.BrooklynObjectType;
+import brooklyn.entity.rebind.PersistenceExceptionHandler;
+import brooklyn.entity.rebind.PersistenceExceptionHandlerImpl;
+import brooklyn.entity.rebind.dto.MementosGenerators;
+import brooklyn.entity.rebind.transformer.CompoundTransformer;
+import brooklyn.entity.rebind.transformer.CompoundTransformerLoader;
+import brooklyn.location.Location;
+import brooklyn.location.LocationSpec;
+import brooklyn.location.basic.LocalhostMachineProvisioningLocation;
+import brooklyn.management.ManagementContext;
+import brooklyn.management.ha.HighAvailabilityMode;
+import brooklyn.management.ha.ManagementNodeState;
+import brooklyn.management.ha.ManagementPlaneSyncRecord;
+import brooklyn.management.ha.ManagementPlaneSyncRecordPersisterToObjectStore;
+import brooklyn.management.ha.MementoCopyMode;
+import brooklyn.management.internal.ManagementContextInternal;
+import brooklyn.mementos.BrooklynMementoRawData;
+import brooklyn.mementos.Memento;
+import brooklyn.policy.Enricher;
+import brooklyn.policy.Policy;
+import brooklyn.util.ResourceUtils;
+import brooklyn.util.exceptions.Exceptions;
+import brooklyn.util.text.Strings;
+import brooklyn.util.time.Duration;
+import brooklyn.util.time.Time;
+
+import com.google.common.annotations.Beta;
+import com.google.common.base.Stopwatch;
+import com.google.common.collect.ImmutableList;
+
+public class BrooklynPersistenceUtils {
+
+ private static final Logger log = LoggerFactory.getLogger(BrooklynPersistenceUtils.class);
+
+ @Beta
+ public static final List<BrooklynObjectType> STANDARD_BROOKLYN_OBJECT_TYPE_PERSISTENCE_ORDER = ImmutableList.of(
+ BrooklynObjectType.ENTITY, BrooklynObjectType.LOCATION, BrooklynObjectType.POLICY,
+ BrooklynObjectType.ENRICHER, BrooklynObjectType.FEED, BrooklynObjectType.CATALOG_ITEM);
+
+ /** Creates a {@link PersistenceObjectStore} for general-purpose use. */
+ public static PersistenceObjectStore newPersistenceObjectStore(ManagementContext managementContext,
+ String locationSpec, String locationContainer) {
+
+ return newPersistenceObjectStore(managementContext, locationSpec, locationContainer,
+ PersistMode.AUTO, HighAvailabilityMode.STANDBY);
+ }
+
+ /** Creates a {@link PersistenceObjectStore} for use with a specified set of modes. */
+ public static PersistenceObjectStore newPersistenceObjectStore(ManagementContext managementContext,
+ String locationSpec, String locationContainer, PersistMode persistMode, HighAvailabilityMode highAvailabilityMode) {
+ PersistenceObjectStore destinationObjectStore;
+ locationContainer = BrooklynServerPaths.newMainPersistencePathResolver(managementContext).location(locationSpec).dir(locationContainer).resolve();
+
+ Location location = null;
+ try {
+ if (Strings.isBlank(locationSpec)) {
+ location = managementContext.getLocationManager().createLocation(LocationSpec.create(LocalhostMachineProvisioningLocation.class));
+ } else {
+ location = managementContext.getLocationRegistry().resolve(locationSpec);
+ if (!(location instanceof LocationWithObjectStore)) {
+ throw new IllegalArgumentException("Destination location "+location+" does not offer a persistent store");
+ }
+ }
+ } finally {
+ if (location!=null) managementContext.getLocationManager().unmanage(location);
--- End diff --
unmanage clears the config and subsequent calls to getIdentity & co fail.
Should request unmanaged location explicitly instead.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: Persistence hardening - creates c...
Posted by neykov <gi...@git.apache.org>.
Github user neykov commented on the pull request:
https://github.com/apache/incubator-brooklyn/pull/314#issuecomment-63022026
Switching to HOT_BACKUP generates a duplicate rebind log line (iteration 0):
```
2014-11-14 09:48:55,996 INFO Rebind complete (HOT_BACKUP, iteration 0) in 606ms: 1 app, 2 entities, 0 locations, 0 policies, 5 enrichers, 0 feeds, 10 catalog items
2014-11-14 09:48:56,842 INFO Rebind complete (HOT_BACKUP, iteration 0) in 844ms: 1 app, 2 entities, 0 locations, 0 policies, 5 enrichers, 0 feeds, 10 catalog items
2014-11-14 09:48:57,134 INFO Management node X5CE0njT running as HA HOT_BACKUP (explicitly requested); no master currently
2014-11-14 09:48:58,460 INFO Rebind complete (HOT_BACKUP, iteration 1) in 615ms: 1 app, 2 entities, 0 locations, 0 policies, 5 enrichers, 0 feeds, 10 catalog items
2014-11-14 09:49:00,078 INFO Rebind complete (HOT_BACKUP, iteration 2) in 613ms: 1 app, 2 entities, 0 locations, 0 policies, 5 enrichers, 0 feeds, 10 catalog items
2014-11-14 09:49:01,736 INFO Rebind complete (HOT_BACKUP, iteration 3) in 655ms: 1 app, 2 entities, 0 locations, 0 policies, 5 enrichers, 0 feeds, 10 catalog items
2014-11-14 09:49:03,387 INFO Rebind complete (HOT_BACKUP, iteration 4) in 648ms: 1 app, 2 entities, 0 locations, 0 policies, 5 enrichers, 0 feeds, 10 catalog items
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---