You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@brooklyn.apache.org by ahgittin <gi...@git.apache.org> on 2015/04/30 03:04:49 UTC
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
GitHub user ahgittin opened a pull request:
https://github.com/apache/incubator-brooklyn/pull/617
CLI commands for manipulating catalog, and cleaner catalog loading
approx as discussed on mailing list, except annotations now also supported (easier to maintain compatibility, in tests)
for review only at this point; i want to do more testing
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ahgittin/incubator-brooklyn catalog-cli
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-brooklyn/pull/617.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #617
----
commit 20810ac9e275eadbf5c00c8b2ba07534841e56f5
Author: Alex Heneveld <al...@cloudsoftcorp.com>
Date: 2015-04-23T06:46:32Z
adjust cli start sequence so web/rest is avail always, with isUp check on rest api
first step towards being able to interactively track the startup sequence and notify on errors
commit ddce439f17143e838291c8f60ab6b69610ed9249
Author: Alex Heneveld <al...@cloudsoftcorp.com>
Date: 2015-04-23T10:52:53Z
add ability to collect startup errors and check healthy through rest api
changes defaults to be to continue on error, and extends (and optimizes) the BrooklynLauncherTest which checks this
commit e629602b41613f1068c9fa4b319e99dafc5eeeeb
Author: Alex Heneveld <al...@cloudsoftcorp.com>
Date: 2015-04-23T15:31:30Z
javascript checks up on start, shows caution
popup now appears while server is starting, or if there are errors, and comes back if server changes to standby.
changed to plug in *before* the routes are evaluated, so we don't load the wrong data.
expands rest api to return several pieces of information to simplify gui checks.
simplifies code in rest filters to block requests not compatible with server state, including checking for upness there.
commit 65d7ee3c15afda7c5ed4507889fb632f1e2d5c8f
Author: Alex Heneveld <al...@cloudsoftcorp.com>
Date: 2015-04-27T11:53:09Z
actually force catalog load by default, and report errors better
and remove two incompatible items from the catalog, with comments in them - FollowTheSunPolicy and LoadBalancingPolicy
commit 7d0f1a0efeffd0e16bdca35ec73f0d7ebbd956ed
Author: Alex Heneveld <al...@cloudsoftcorp.com>
Date: 2015-04-27T13:30:26Z
show caution dialog if shutting down or if server is unresponsive
and include shutting down message in rest api.
tested that i can stop and restart the server and it nicely cycles through sequence of:
"shutting down", "server unreachable", then "starting up", then restores page.
commit df36d992ee081f231f3a38b922f0a25d4476307b
Author: Alex Heneveld <al...@cloudsoftcorp.com>
Date: 2015-04-27T13:38:05Z
introduce CatalogInitialization to cleanly init catalog at the right times
invoked during persistence cycles, and at startup, holding the new CLI catalog options.
this does not yet properly initialize things for real-world use, and tests fixed in next commit.
commit 9f42c513743f531e4722d5132a6b43b3e0a89141
Author: Alex Heneveld <al...@cloudsoftcorp.com>
Date: 2015-04-29T12:27:09Z
yaml catalog supports scanning, experimental, and default
this restores catalog scanning as the default, based on brooklyn/default.catalog.bom in the cli project;
there are some limitations on what can be scanned, described in the doc.
some of the tests configure other catalogs (with core's brooklyn/empty.catalog.bom used in many)
commit 9868d116b13e3b36b44edec47891f82a60d5436d
Author: Alex Heneveld <al...@cloudsoftcorp.com>
Date: 2015-04-29T12:32:34Z
item lister resources put in an appropriate subdir; other related tidies
code was very ad hoc, but also polluting the root of the all jar; now resources at least are in a clean subdir
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by ahgittin <gi...@git.apache.org>.
Github user ahgittin commented on a diff in the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#discussion_r30973616
--- Diff: core/src/main/java/brooklyn/entity/rebind/RebindIteration.java ---
@@ -341,38 +340,64 @@ protected void rebuildCatalog() {
}
}
}
+
+ // See notes in CatalogInitialization
- // Register catalogue items with the management context. Loads the bundles in the OSGi framework.
- CatalogLoadMode catalogLoadMode = managementContext.getConfig().getConfig(BrooklynServerConfig.CATALOG_LOAD_MODE);
+ Collection<CatalogItem<?, ?>> catalogItems = rebindContext.getCatalogItems();
+ CatalogInitialization catInit = ((ManagementContextInternal)managementContext).getCatalogInitialization();
+ catInit.applyCatalogLoadMode();
+ Collection<CatalogItem<?,?>> itemsForResettingCatalog = null;
+ boolean needsInitialCatalog;
if (rebindManager.persistCatalogItemsEnabled) {
- boolean shouldResetCatalog = catalogLoadMode == CatalogLoadMode.LOAD_PERSISTED_STATE
- || (!isEmpty && catalogLoadMode == CatalogLoadMode.LOAD_BROOKLYN_CATALOG_URL_IF_NO_PERSISTED_STATE);
- boolean shouldLoadDefaultCatalog = catalogLoadMode == CatalogLoadMode.LOAD_BROOKLYN_CATALOG_URL
- || (isEmpty && catalogLoadMode == CatalogLoadMode.LOAD_BROOKLYN_CATALOG_URL_IF_NO_PERSISTED_STATE);
- if (shouldResetCatalog) {
- // Reset catalog with previously persisted state
- logRebindingDebug("RebindManager resetting management context catalog to previously persisted state");
- managementContext.getCatalog().reset(rebindContext.getCatalogItems());
- } else if (shouldLoadDefaultCatalog) {
- // Load catalogue as normal
- // TODO in read-only mode, should do this less frequently than entities etc
- logRebindingDebug("RebindManager loading default catalog");
- ((BasicBrooklynCatalog) managementContext.getCatalog()).resetCatalogToContentsAtConfiguredUrl();
+ if (!catInit.hasRunOfficial() && catInit.isInitialResetRequested()) {
+ String message = "RebindManager resetting catalog on first run (catalog persistence enabled, but reset explicitly specified). ";
+ if (catalogItems.isEmpty()) {
+ message += "Catalog was empty anyway.";
+ } else {
+ message += "Deleting "+catalogItems.size()+" persisted catalog item"+Strings.s(catalogItems)+": "+catalogItems;
+ if (shouldLogRebinding()) {
+ LOG.info(message);
+ }
+ }
+ logRebindingDebug(message);
+
+ itemsForResettingCatalog = MutableList.<CatalogItem<?,?>>of();
--- End diff --
i see -- we could get failures while loading previously persisted catalog items, even if we are resetting. and yes this is an argument that the right thing to do, if we are going to reset the catalog anyway, is to clear things out earlier or skip the catalog item load. let me look at this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by ahgittin <gi...@git.apache.org>.
Github user ahgittin commented on a diff in the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#discussion_r30968430
--- Diff: core/src/main/java/brooklyn/entity/rebind/PeriodicDeltaChangeListener.java ---
@@ -348,13 +336,40 @@ private void addReferencedObjects(DeltaCollector deltaCollector) {
}
@VisibleForTesting
- public void persistNow() {
- if (!isActive()) {
+ public boolean persistNowSafely(boolean alreadyHasMutex) {
--- End diff --
agree - have done
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by neykov <gi...@git.apache.org>.
Github user neykov commented on a diff in the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#discussion_r30905989
--- Diff: usage/cli/src/main/java/brooklyn/cli/Main.java ---
@@ -756,9 +781,10 @@ public ToStringHelper string() {
.add("bindAddress", bindAddress)
.add("noConsole", noConsole)
.add("noConsoleSecurity", noConsoleSecurity)
- .add("ignorePersistenceErrors", ignorePersistenceErrors)
- .add("ignoreWebErrors", ignoreWebErrors)
- .add("ignoreAppErrors", ignoreAppErrors)
+ .add("startupFailOnPersistenceErrors", startupFailOnPersistenceErrors)
+ .add("startupFailsOnCatalogErrors", startupFailOnCatalogErrors)
+ .add("startupContinueOnWebErrors", startupContinueOnWebErrors)
+ .add("startupFailOnManagedAppsErrors", startupFailOnManagedAppsErrors)
--- End diff --
Add new catalog commands.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by ahgittin <gi...@git.apache.org>.
Github user ahgittin commented on a diff in the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#discussion_r30659071
--- Diff: usage/cli/src/main/java/brooklyn/cli/lister/ItemDescriptors.java ---
@@ -58,8 +65,17 @@
List<Map<String, Object>> itemDescriptors = Lists.newArrayList();
for (Class<? extends BrooklynObject> type : types) {
- Map<String, Object> itemDescriptor = toItemDescriptor(type, headingsOnly);
- itemDescriptors.add(itemDescriptor);
+ try {
+ Map<String, Object> itemDescriptor = toItemDescriptor(type, headingsOnly);
+ itemDescriptors.add(itemDescriptor);
+ } catch (Throwable throwable) {
+ if (throwable instanceof InterruptedException)
--- End diff --
you can get various `*Class*Error` instances here which we don't want to treat as fatal.
probably we should revisit `propagateIfFatal` not to crash and burn on arbitrary `Error` instances.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by neykov <gi...@git.apache.org>.
Github user neykov commented on a diff in the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#discussion_r30785837
--- Diff: core/src/main/java/brooklyn/catalog/internal/CatalogInitialization.java ---
@@ -0,0 +1,380 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package brooklyn.catalog.internal;
+
+import java.io.File;
+import java.util.Collection;
+import java.util.List;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import brooklyn.catalog.CatalogItem;
+import brooklyn.config.BrooklynServerConfig;
+import brooklyn.management.ManagementContext;
+import brooklyn.management.ManagementContextInjectable;
+import brooklyn.management.internal.ManagementContextInternal;
+import brooklyn.util.ResourceUtils;
+import brooklyn.util.collections.MutableList;
+import brooklyn.util.exceptions.Exceptions;
+import brooklyn.util.exceptions.FatalRuntimeException;
+import brooklyn.util.exceptions.RuntimeInterruptedException;
+import brooklyn.util.flags.TypeCoercions;
+import brooklyn.util.guava.Maybe;
+import brooklyn.util.os.Os;
+import brooklyn.util.text.Strings;
+
+import com.google.common.annotations.Beta;
+import com.google.common.base.Function;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.Iterables;
+
+@Beta
+public class CatalogInitialization implements ManagementContextInjectable {
+
+ /*
+
+ A1) if not persisting, go to B1
+ A2) if --catalog-reset, delete persisted catalog items
+ A3) if there is a persisted catalog (and it wasn't not deleted by A2), read it and go to C1
+ A4) go to B1
+
+ B1) look for --catalog-initial, if so read it, then go to C1
+ B2) look for BrooklynServerConfig.BROOKLYN_CATALOG_URL, if so, read it, supporting YAML or XML (warning if XML), then go to C1
+ B3) look for ~/.brooklyn/catalog.bom, if exists, read it then go to C1
+ B4) look for ~/.brooklyn/brooklyn.xml, if exists, warn, read it then go to C1
+ B5) read all classpath://brooklyn/default.catalog.bom items, if they exist (and for now they will)
+ B6) go to C1
+
+ C1) if --catalog-add, read and add those items
+
+ D1) if persisting, read the rest of the persisted items (entities etc)
+
+ */
+
+ private static final Logger log = LoggerFactory.getLogger(CatalogInitialization.class);
+
+ private String initialUri;
+ private boolean reset;
+ private String additionsUri;
+ private boolean force;
+
+ private boolean disallowLocal = false;
+ private List<Function<CatalogInitialization, Void>> callbacks = MutableList.of();
+ private boolean hasRunBestEffort = false, hasRunOfficial = false, isPopulating = false;
+
+ private ManagementContext managementContext;
+ private boolean isStartingUp = false;
+ private boolean failOnStartupErrors = false;
+
+ private Object populatingCatalogMutex = new Object();
+
+ public CatalogInitialization(String initialUri, boolean reset, String additionUri, boolean force) {
+ this.initialUri = initialUri;
+ this.reset = reset;
+ this.additionsUri = additionUri;
+ this.force = force;
+ }
+
+ public CatalogInitialization() {
+ this(null, false, null, false);
+ }
+
+ public void injectManagementContext(ManagementContext managementContext) {
+ Preconditions.checkNotNull(managementContext, "management context");
+ if (this.managementContext!=null && managementContext!=this.managementContext)
+ throw new IllegalStateException("Cannot switch management context, from "+this.managementContext+" to "+managementContext);
+ this.managementContext = managementContext;
+ }
+
+ /** Called by the framework to set true while starting up, and false afterwards,
+ * in order to assist in appropriate logging and error handling. */
+ public void setStartingUp(boolean isStartingUp) {
+ this.isStartingUp = isStartingUp;
+ }
+
+ public void setFailOnStartupErrors(boolean startupFailOnCatalogErrors) {
+ this.failOnStartupErrors = startupFailOnCatalogErrors;
+ }
+
+ public CatalogInitialization addPopulationCallback(Function<CatalogInitialization, Void> callback) {
+ callbacks.add(callback);
+ return this;
+ }
+
+ public ManagementContext getManagementContext() {
+ return Preconditions.checkNotNull(managementContext, "management context has not been injected into "+this);
+ }
+
+ public boolean isInitialResetRequested() {
+ return reset;
+ }
+
+ public boolean hasRunOfficial() { return hasRunOfficial; }
+ public boolean hasRunIncludingBestEffort() { return hasRunOfficial || hasRunBestEffort; }
+
+ /** makes or updates the mgmt catalog, based on the settings in this class */
+ public void populateCatalog(boolean needsInitial, Collection<CatalogItem<?, ?>> optionalItemsForResettingCatalog) {
+ synchronized (populatingCatalogMutex) {
+ try {
+ isPopulating = true;
+ BasicBrooklynCatalog catalog = (BasicBrooklynCatalog) managementContext.getCatalog();
+ if (!catalog.getCatalog().isLoaded()) {
+ catalog.load();
+ } else {
+ if (needsInitial && (hasRunOfficial || hasRunBestEffort)) {
+ // an indication that something caused it to load early; not severe, but unusual
+ log.warn("Catalog initialization has not properly run but management context has a catalog; re-populating, possibly overwriting items installed during earlier access (it may have been an early web request)");
+ catalog.reset(ImmutableList.<CatalogItem<?,?>>of());
+ }
+ }
+ hasRunOfficial = true;
+
+ populateCatalogImpl(catalog, needsInitial, optionalItemsForResettingCatalog);
+ } finally {
+ hasRunOfficial = true;
+ isPopulating = false;
+ }
+ }
+ }
+
+ private void populateCatalogImpl(BasicBrooklynCatalog catalog, boolean needsInitial, Collection<CatalogItem<?, ?>> optionalItemsForResettingCatalog) {
+ applyCatalogLoadMode();
+
+ if (optionalItemsForResettingCatalog!=null) {
+ catalog.reset(optionalItemsForResettingCatalog);
+ }
+
+ if (needsInitial) {
+ populateInitial(catalog);
+ }
+
+ populateAdditions(catalog);
+
+ populateViaCallbacks(catalog);
+ }
+
+ private enum PopulateMode { YAML, XML, AUTODETECT }
+
+ protected void populateInitial(BasicBrooklynCatalog catalog) {
+ if (disallowLocal) {
+ if (!hasRunOfficial()) {
+ log.debug("CLI initial catalog not being read with disallow-local mode set.");
+ }
+ return;
+ }
+
+// B1) look for --catalog-initial, if so read it, then go to C1
+// B2) look for BrooklynServerConfig.BROOKLYN_CATALOG_URL, if so, read it, supporting YAML or XML (warning if XML), then go to C1
+// B3) look for ~/.brooklyn/catalog.bom, if exists, read it then go to C1
+// B4) look for ~/.brooklyn/brooklyn.xml, if exists, warn, read it then go to C1
+// B5) read all classpath://brooklyn/default.catalog.bom items, if they exist (and for now they will)
+// B6) go to C1
+
+ if (initialUri!=null) {
+ populateInitialFromUri(catalog, initialUri, PopulateMode.AUTODETECT);
+ return;
+ }
+
+ String catalogUrl = managementContext.getConfig().getConfig(BrooklynServerConfig.BROOKLYN_CATALOG_URL);
+ if (Strings.isNonBlank(catalogUrl)) {
+ populateInitialFromUri(catalog, catalogUrl, PopulateMode.AUTODETECT);
+ return;
+ }
+
+ catalogUrl = Os.mergePaths(BrooklynServerConfig.getMgmtBaseDir( managementContext.getConfig() ), "catalog.bom");
+ if (new File(catalogUrl).exists()) {
+ populateInitialFromUri(catalog, "file:"+catalogUrl, PopulateMode.YAML);
--- End diff --
better use `new File(catalogUrl).toURI()`, same below.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by aledsage <gi...@git.apache.org>.
Github user aledsage commented on a diff in the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#discussion_r30546036
--- Diff: core/src/main/java/brooklyn/catalog/internal/CatalogInitialization.java ---
@@ -0,0 +1,377 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package brooklyn.catalog.internal;
+
+import java.io.File;
+import java.util.Collection;
+import java.util.List;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import brooklyn.catalog.CatalogItem;
+import brooklyn.config.BrooklynServerConfig;
+import brooklyn.management.ManagementContext;
+import brooklyn.management.ManagementContextInjectable;
+import brooklyn.management.internal.ManagementContextInternal;
+import brooklyn.util.ResourceUtils;
+import brooklyn.util.collections.MutableList;
+import brooklyn.util.exceptions.Exceptions;
+import brooklyn.util.exceptions.FatalRuntimeException;
+import brooklyn.util.exceptions.RuntimeInterruptedException;
+import brooklyn.util.flags.TypeCoercions;
+import brooklyn.util.guava.Maybe;
+import brooklyn.util.net.Urls;
+import brooklyn.util.text.Strings;
+
+import com.google.common.annotations.Beta;
+import com.google.common.base.Function;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.Iterables;
+
+@Beta
+public class CatalogInitialization implements ManagementContextInjectable {
+
+ /*
+
+ A1) if not persisting, go to B1
+ A2) if --catalog-reset, delete persisted catalog items
+ A3) read persisted catalog items (possibly deleted in A2), go to C1
+ A4) go to B1
+
+ B1) look for --catalog-initial, if so read it, then go to C1
+ B2) look for BrooklynServerConfig.BROOKLYN_CATALOG_URL, if so, read it, supporting YAML or XML (warning if XML), then go to C1
+ B3) look for ~/.brooklyn/catalog.bom, if exists, read it then go to C1
+ B4) look for ~/.brooklyn/brooklyn.xml, if exists, warn, read it then go to C1
+ B5) read all classpath://brooklyn/default.catalog.bom items, if they exist (and for now they will)
+ B6) go to C1
+
+ C1) if --catalog-add, read and add those items
+
+ D1) if persisting, read the rest of the persisted items (entities etc)
+
+ */
+
+ private static final Logger log = LoggerFactory.getLogger(CatalogInitialization.class);
+
+ String initialUri;
+ boolean reset;
+ String additionsUri;
+ boolean force;
+
+ boolean disallowLocal = false;
+ List<Function<CatalogInitialization, Void>> callbacks = MutableList.of();
+ boolean hasRunBestEffort = false, hasRunOfficial = false, isPopulating = false;
+
+ ManagementContext managementContext;
+ boolean isStartingUp = false;
+ boolean failOnStartupErrors = false;
+
+ Object mutex = new Object();
+
+ public CatalogInitialization(String initialUri, boolean reset, String additionUri, boolean force) {
+ this.initialUri = initialUri;
+ this.reset = reset;
+ this.additionsUri = additionUri;
+ this.force = force;
+ }
+
+ public CatalogInitialization() {
+ this(null, false, null, false);
+ }
+
+ public void injectManagementContext(ManagementContext managementContext) {
+ if (this.managementContext!=null && managementContext!=null && !this.managementContext.equals(managementContext))
+ throw new IllegalStateException("Cannot switch management context of "+this+"; from "+this.managementContext+" to "+managementContext);
+ this.managementContext = managementContext;
+ }
+
+ public ManagementContext getManagementContext() {
+ return Preconditions.checkNotNull(managementContext, "management context has not been injected into "+this);
+ }
+
+ public CatalogInitialization addPopulationCallback(Function<CatalogInitialization, Void> callback) {
+ callbacks.add(callback);
+ return this;
+ }
+
+ public boolean isInitialResetRequested() {
+ return reset;
+ }
+
+ public boolean hasRunOfficial() { return hasRunOfficial; }
+ public boolean hasRunIncludingBestEffort() { return hasRunOfficial || hasRunBestEffort; }
+
+ /** makes or updates the mgmt catalog, based on the settings in this class */
+ public void populateCatalog(boolean needsInitial, Collection<CatalogItem<?, ?>> optionalItemsForResettingCatalog) {
+ try {
+ isPopulating = true;
+ synchronized (mutex) {
+ BasicBrooklynCatalog catalog = (BasicBrooklynCatalog) managementContext.getCatalog();
+ if (!catalog.getCatalog().isLoaded()) {
+ catalog.load();
+ } else {
+ if (needsInitial && (hasRunOfficial || hasRunBestEffort)) {
+ // an indication that something caused it to load early; not severe, but unusual
+ log.warn("Catalog initialization has not properly run but management context has a catalog; re-populating, possibly overwriting items installed during earlier access (it may have been an early web request)");
+ catalog.reset(ImmutableList.<CatalogItem<?,?>>of());
+ }
+ }
+ hasRunOfficial = true;
+
+ populateCatalog(catalog, needsInitial, true, optionalItemsForResettingCatalog);
+ }
+ } finally {
+ hasRunOfficial = true;
+ isPopulating = false;
+ }
+ }
+
+ private void populateCatalog(BasicBrooklynCatalog catalog, boolean needsInitial, boolean runCallbacks, Collection<CatalogItem<?, ?>> optionalItemsForResettingCatalog) {
+ applyCatalogLoadMode();
+
+ if (optionalItemsForResettingCatalog!=null) {
+ catalog.reset(optionalItemsForResettingCatalog);
+ }
+
+ if (needsInitial) {
+ populateInitial(catalog);
+ }
+
+ populateAdditions(catalog);
+
+ if (runCallbacks) {
+ populateViaCallbacks(catalog);
+ }
+ }
+
+ private enum PopulateMode { YAML, XML, AUTODETECT }
+
+ protected void populateInitial(BasicBrooklynCatalog catalog) {
+ if (disallowLocal) {
+ if (!hasRunOfficial()) {
+ log.debug("CLI initial catalog not being read with disallow-local mode set.");
+ }
+ return;
+ }
+
+// B1) look for --catalog-initial, if so read it, then go to C1
+// B2) look for BrooklynServerConfig.BROOKLYN_CATALOG_URL, if so, read it, supporting YAML or XML (warning if XML), then go to C1
+// B3) look for ~/.brooklyn/catalog.bom, if exists, read it then go to C1
+// B4) look for ~/.brooklyn/brooklyn.xml, if exists, warn, read it then go to C1
+// B5) read all classpath://brooklyn/default.catalog.bom items, if they exist (and for now they will)
+// B6) go to C1
+
+ if (initialUri!=null) {
+ populateInitialFromUri(catalog, initialUri, PopulateMode.AUTODETECT);
+ return;
+ }
+
+ String catalogUrl = managementContext.getConfig().getConfig(BrooklynServerConfig.BROOKLYN_CATALOG_URL);
+ if (Strings.isNonBlank(catalogUrl)) {
+ populateInitialFromUri(catalog, catalogUrl, PopulateMode.AUTODETECT);
+ return;
+ }
+
+ catalogUrl = Urls.mergePaths(BrooklynServerConfig.getMgmtBaseDir( managementContext.getConfig() ), "catalog.bom");
+ if (new File(catalogUrl).exists()) {
+ populateInitialFromUri(catalog, "file:"+catalogUrl, PopulateMode.YAML);
+ return;
+ }
+
+ catalogUrl = Urls.mergePaths(BrooklynServerConfig.getMgmtBaseDir( managementContext.getConfig() ), "catalog.xml");
+ if (new File(catalogUrl).exists()) {
+ populateInitialFromUri(catalog, "file:"+catalogUrl, PopulateMode.XML);
+ return;
+ }
+
+ // otherwise look for for classpath:/brooklyn/default.catalog.bom --
+ // there is one on the classpath which says to scan, and provides a few templates;
+ // if one is supplied by user in the conf/ dir that will override the item from the classpath
+ // (TBD - we might want to scan for all such bom's?)
+
+ catalogUrl = "classpath:/brooklyn/default.catalog.bom";
+ if (new ResourceUtils(this).doesUrlExist(catalogUrl)) {
+ populateInitialFromUri(catalog, catalogUrl, PopulateMode.YAML);
+ return;
+ }
+
+ log.info("No catalog found on classpath or specified; catalog will not be initialized.");
+ return;
+ }
+
+ private void populateInitialFromUri(BasicBrooklynCatalog catalog, String catalogUrl, PopulateMode mode) {
+ log.debug("Loading initial catalog from {}", catalogUrl);
+
+ Exception problem = null;
+ Object result = null;
+
+ String contents = null;
+ try {
+ contents = new ResourceUtils(this).getResourceAsString(catalogUrl);
+ } catch (Exception e) {
+ Exceptions.propagateIfFatal(e);
+ if (problem==null) problem = e;
+ }
+
+ if (contents!=null && (mode==PopulateMode.YAML || mode==PopulateMode.AUTODETECT)) {
+ // try YAML first
+ try {
+ catalog.reset(MutableList.<CatalogItem<?,?>>of());
+ result = catalog.addItems(contents);
+ } catch (Exception e) {
+ Exceptions.propagateIfFatal(e);
+ if (problem==null) problem = e;
+ }
+ }
+
+ if (result==null && contents!=null && (mode==PopulateMode.XML || mode==PopulateMode.AUTODETECT)) {
+ // then try XML
+ problem = populateInitialFromUriXml(catalog, catalogUrl, problem, contents);
+ }
+
+ if (result!=null) {
+ log.debug("Loaded initial catalog from {}: {}", catalogUrl, result);
+ }
+ if (problem!=null) {
+ log.warn("Error importing catalog from " + catalogUrl + ": " + problem, problem);
+ // TODO inform mgmt of error
+ }
+
+ }
+
+ // deprecated XML format
+ @SuppressWarnings("deprecation")
+ private Exception populateInitialFromUriXml(BasicBrooklynCatalog catalog, String catalogUrl, Exception problem, String contents) {
+ CatalogDto dto = null;
+ try {
+ dto = CatalogDto.newDtoFromXmlContents(contents, catalogUrl);
+ problem = null;
+ } catch (Exception e) {
+ Exceptions.propagateIfFatal(e);
+ if (problem==null) problem = e;
+ }
+ if (dto!=null) {
+ catalog.reset(dto);
+ }
+ return problem;
+ }
+
+ boolean hasRunAdditions = false;
+ protected void populateAdditions(BasicBrooklynCatalog catalog) {
+ if (Strings.isNonBlank(additionsUri)) {
+ if (disallowLocal) {
+ if (!hasRunAdditions) {
+ log.warn("CLI additions supplied but not supported in disallow-local mode; ignoring.");
+ }
+ return;
+ }
+ if (!hasRunAdditions) {
+ log.debug("Adding to catalog from CLI: "+additionsUri+" (force: "+force+")");
+ }
+ Iterable<? extends CatalogItem<?, ?>> items = catalog.addItems(
+ new ResourceUtils(this).getResourceAsString(additionsUri), force);
+
+ if (!hasRunAdditions)
+ log.debug("Added to catalog from CLI: "+items);
+ else
+ log.debug("Added to catalog from CLI: count "+Iterables.size(items));
+
+ hasRunAdditions = true;
+ }
+ }
+
+ protected void populateViaCallbacks(BasicBrooklynCatalog catalog) {
+ for (Function<CatalogInitialization, Void> callback: callbacks)
--- End diff --
Would these be better called "actions" rather than "callbacks"? When I first saw the callbacks being passed in, I wondered if they were listeners of some description (being called on population).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by aledsage <gi...@git.apache.org>.
Github user aledsage commented on a diff in the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#discussion_r30589991
--- Diff: usage/rest-server/src/main/java/brooklyn/rest/resources/ServerResource.java ---
@@ -126,45 +126,62 @@ public void shutdown(final boolean stopAppsFirst, final boolean forceShutdownOnE
new Thread("shutdown") {
public void run() {
- if (stopAppsFirst) {
- CountdownTimer shutdownTimeoutTimer = null;
- if (!shutdownTimeout.equals(Duration.ZERO)) {
- shutdownTimeoutTimer = shutdownTimeout.countdownTimer();
- }
+ boolean terminateTried = false;
+ try {
+ if (stopAppsFirst) {
+ CountdownTimer shutdownTimeoutTimer = null;
+ if (!shutdownTimeout.equals(Duration.ZERO)) {
+ shutdownTimeoutTimer = shutdownTimeout.countdownTimer();
+ }
- List<Task<?>> stoppers = new ArrayList<Task<?>>();
- for (Application app: mgmt().getApplications()) {
- if (app instanceof StartableApplication)
- stoppers.add(Entities.invokeEffector((EntityLocal)app, app, StartableApplication.STOP));
- }
+ List<Task<?>> stoppers = new ArrayList<Task<?>>();
+ for (Application app: mgmt().getApplications()) {
+ if (app instanceof StartableApplication)
+ stoppers.add(Entities.invokeEffector((EntityLocal)app, app, StartableApplication.STOP));
+ }
- try {
for (Task<?> t: stoppers) {
if (!waitAppShutdown(shutdownTimeoutTimer, t)) {
//app stop error
hasAppErrorsOrTimeout.set(true);
}
}
- } catch (TimeoutException e) {
+ }
+
+ terminateTried = true;
+ ((ManagementContextInternal)mgmt()).terminate();
+
+ } catch (Throwable e) {
+ Throwable interesting = Exceptions.getFirstInteresting(e);
+ if (interesting instanceof TimeoutException) {
//timeout while waiting for apps to stop
+ log.warn("Timeout shutting down: "+Exceptions.collapseText(e));
+ log.debug("Timeout shutting down: "+e, e);
hasAppErrorsOrTimeout.set(true);
+
+ } else {
+ // swallow fatal, so we notify the outer loop to continue with shutdown
+ log.error("Unexpected error shutting down: "+Exceptions.collapseText(e), e);
+
}
-
- if (hasAppErrorsOrTimeout.get() && !forceShutdownOnError) {
- complete();
- //There are app errors, don't exit the process.
- return;
+ hasAppErrorsOrTimeout.set(true);
+
+ if (!terminateTried) {
+ ((ManagementContextInternal)mgmt()).terminate();
}
+ } finally {
+
+ complete();
+
+ if (!hasAppErrorsOrTimeout.get() || forceShutdownOnError) {
+ //give the http request a chance to complete gracefully
+ Time.sleep(delayForHttpReturn);
+ System.exit(0);
+ }
+
+ // There are app errors, don't exit the process, allowing any exception to continue throwing
+ log.warn("Abandoning shutdown because there were errors and shutdown was not forced.");
--- End diff --
personal preference to put this kind of thing in an else block (even though the above block will have done `System.exit`, so can only reach here if we didn't go into the if).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by neykov <gi...@git.apache.org>.
Github user neykov commented on a diff in the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#discussion_r30789091
--- Diff: core/src/main/java/brooklyn/catalog/internal/CatalogInitialization.java ---
@@ -0,0 +1,380 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package brooklyn.catalog.internal;
+
+import java.io.File;
+import java.util.Collection;
+import java.util.List;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import brooklyn.catalog.CatalogItem;
+import brooklyn.config.BrooklynServerConfig;
+import brooklyn.management.ManagementContext;
+import brooklyn.management.ManagementContextInjectable;
+import brooklyn.management.internal.ManagementContextInternal;
+import brooklyn.util.ResourceUtils;
+import brooklyn.util.collections.MutableList;
+import brooklyn.util.exceptions.Exceptions;
+import brooklyn.util.exceptions.FatalRuntimeException;
+import brooklyn.util.exceptions.RuntimeInterruptedException;
+import brooklyn.util.flags.TypeCoercions;
+import brooklyn.util.guava.Maybe;
+import brooklyn.util.os.Os;
+import brooklyn.util.text.Strings;
+
+import com.google.common.annotations.Beta;
+import com.google.common.base.Function;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.Iterables;
+
+@Beta
+public class CatalogInitialization implements ManagementContextInjectable {
+
+ /*
+
+ A1) if not persisting, go to B1
+ A2) if --catalog-reset, delete persisted catalog items
+ A3) if there is a persisted catalog (and it wasn't not deleted by A2), read it and go to C1
+ A4) go to B1
+
+ B1) look for --catalog-initial, if so read it, then go to C1
+ B2) look for BrooklynServerConfig.BROOKLYN_CATALOG_URL, if so, read it, supporting YAML or XML (warning if XML), then go to C1
+ B3) look for ~/.brooklyn/catalog.bom, if exists, read it then go to C1
+ B4) look for ~/.brooklyn/brooklyn.xml, if exists, warn, read it then go to C1
+ B5) read all classpath://brooklyn/default.catalog.bom items, if they exist (and for now they will)
+ B6) go to C1
+
+ C1) if --catalog-add, read and add those items
+
+ D1) if persisting, read the rest of the persisted items (entities etc)
+
+ */
+
+ private static final Logger log = LoggerFactory.getLogger(CatalogInitialization.class);
+
+ private String initialUri;
+ private boolean reset;
+ private String additionsUri;
+ private boolean force;
+
+ private boolean disallowLocal = false;
+ private List<Function<CatalogInitialization, Void>> callbacks = MutableList.of();
+ private boolean hasRunBestEffort = false, hasRunOfficial = false, isPopulating = false;
+
+ private ManagementContext managementContext;
+ private boolean isStartingUp = false;
+ private boolean failOnStartupErrors = false;
+
+ private Object populatingCatalogMutex = new Object();
+
+ public CatalogInitialization(String initialUri, boolean reset, String additionUri, boolean force) {
+ this.initialUri = initialUri;
+ this.reset = reset;
+ this.additionsUri = additionUri;
+ this.force = force;
+ }
+
+ public CatalogInitialization() {
+ this(null, false, null, false);
+ }
+
+ public void injectManagementContext(ManagementContext managementContext) {
--- End diff --
Missing `@Override`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by aledsage <gi...@git.apache.org>.
Github user aledsage commented on a diff in the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#discussion_r30545594
--- Diff: core/src/main/java/brooklyn/catalog/internal/CatalogInitialization.java ---
@@ -0,0 +1,377 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package brooklyn.catalog.internal;
+
+import java.io.File;
+import java.util.Collection;
+import java.util.List;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import brooklyn.catalog.CatalogItem;
+import brooklyn.config.BrooklynServerConfig;
+import brooklyn.management.ManagementContext;
+import brooklyn.management.ManagementContextInjectable;
+import brooklyn.management.internal.ManagementContextInternal;
+import brooklyn.util.ResourceUtils;
+import brooklyn.util.collections.MutableList;
+import brooklyn.util.exceptions.Exceptions;
+import brooklyn.util.exceptions.FatalRuntimeException;
+import brooklyn.util.exceptions.RuntimeInterruptedException;
+import brooklyn.util.flags.TypeCoercions;
+import brooklyn.util.guava.Maybe;
+import brooklyn.util.net.Urls;
+import brooklyn.util.text.Strings;
+
+import com.google.common.annotations.Beta;
+import com.google.common.base.Function;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.Iterables;
+
+@Beta
+public class CatalogInitialization implements ManagementContextInjectable {
+
+ /*
+
+ A1) if not persisting, go to B1
+ A2) if --catalog-reset, delete persisted catalog items
+ A3) read persisted catalog items (possibly deleted in A2), go to C1
+ A4) go to B1
+
+ B1) look for --catalog-initial, if so read it, then go to C1
+ B2) look for BrooklynServerConfig.BROOKLYN_CATALOG_URL, if so, read it, supporting YAML or XML (warning if XML), then go to C1
+ B3) look for ~/.brooklyn/catalog.bom, if exists, read it then go to C1
+ B4) look for ~/.brooklyn/brooklyn.xml, if exists, warn, read it then go to C1
+ B5) read all classpath://brooklyn/default.catalog.bom items, if they exist (and for now they will)
+ B6) go to C1
+
+ C1) if --catalog-add, read and add those items
+
+ D1) if persisting, read the rest of the persisted items (entities etc)
+
+ */
+
+ private static final Logger log = LoggerFactory.getLogger(CatalogInitialization.class);
+
+ String initialUri;
+ boolean reset;
+ String additionsUri;
+ boolean force;
+
+ boolean disallowLocal = false;
+ List<Function<CatalogInitialization, Void>> callbacks = MutableList.of();
+ boolean hasRunBestEffort = false, hasRunOfficial = false, isPopulating = false;
+
+ ManagementContext managementContext;
+ boolean isStartingUp = false;
+ boolean failOnStartupErrors = false;
+
+ Object mutex = new Object();
+
+ public CatalogInitialization(String initialUri, boolean reset, String additionUri, boolean force) {
+ this.initialUri = initialUri;
+ this.reset = reset;
+ this.additionsUri = additionUri;
+ this.force = force;
+ }
+
+ public CatalogInitialization() {
+ this(null, false, null, false);
+ }
+
+ public void injectManagementContext(ManagementContext managementContext) {
+ if (this.managementContext!=null && managementContext!=null && !this.managementContext.equals(managementContext))
+ throw new IllegalStateException("Cannot switch management context of "+this+"; from "+this.managementContext+" to "+managementContext);
+ this.managementContext = managementContext;
+ }
+
+ public ManagementContext getManagementContext() {
+ return Preconditions.checkNotNull(managementContext, "management context has not been injected into "+this);
+ }
+
+ public CatalogInitialization addPopulationCallback(Function<CatalogInitialization, Void> callback) {
+ callbacks.add(callback);
+ return this;
+ }
+
+ public boolean isInitialResetRequested() {
+ return reset;
+ }
+
+ public boolean hasRunOfficial() { return hasRunOfficial; }
+ public boolean hasRunIncludingBestEffort() { return hasRunOfficial || hasRunBestEffort; }
+
+ /** makes or updates the mgmt catalog, based on the settings in this class */
+ public void populateCatalog(boolean needsInitial, Collection<CatalogItem<?, ?>> optionalItemsForResettingCatalog) {
+ try {
+ isPopulating = true;
+ synchronized (mutex) {
+ BasicBrooklynCatalog catalog = (BasicBrooklynCatalog) managementContext.getCatalog();
+ if (!catalog.getCatalog().isLoaded()) {
+ catalog.load();
+ } else {
+ if (needsInitial && (hasRunOfficial || hasRunBestEffort)) {
+ // an indication that something caused it to load early; not severe, but unusual
+ log.warn("Catalog initialization has not properly run but management context has a catalog; re-populating, possibly overwriting items installed during earlier access (it may have been an early web request)");
+ catalog.reset(ImmutableList.<CatalogItem<?,?>>of());
+ }
+ }
+ hasRunOfficial = true;
+
+ populateCatalog(catalog, needsInitial, true, optionalItemsForResettingCatalog);
+ }
+ } finally {
+ hasRunOfficial = true;
+ isPopulating = false;
+ }
+ }
+
+ private void populateCatalog(BasicBrooklynCatalog catalog, boolean needsInitial, boolean runCallbacks, Collection<CatalogItem<?, ?>> optionalItemsForResettingCatalog) {
+ applyCatalogLoadMode();
+
+ if (optionalItemsForResettingCatalog!=null) {
+ catalog.reset(optionalItemsForResettingCatalog);
+ }
+
+ if (needsInitial) {
+ populateInitial(catalog);
+ }
+
+ populateAdditions(catalog);
+
+ if (runCallbacks) {
+ populateViaCallbacks(catalog);
+ }
+ }
+
+ private enum PopulateMode { YAML, XML, AUTODETECT }
+
+ protected void populateInitial(BasicBrooklynCatalog catalog) {
+ if (disallowLocal) {
+ if (!hasRunOfficial()) {
+ log.debug("CLI initial catalog not being read with disallow-local mode set.");
+ }
+ return;
+ }
+
+// B1) look for --catalog-initial, if so read it, then go to C1
+// B2) look for BrooklynServerConfig.BROOKLYN_CATALOG_URL, if so, read it, supporting YAML or XML (warning if XML), then go to C1
+// B3) look for ~/.brooklyn/catalog.bom, if exists, read it then go to C1
+// B4) look for ~/.brooklyn/brooklyn.xml, if exists, warn, read it then go to C1
+// B5) read all classpath://brooklyn/default.catalog.bom items, if they exist (and for now they will)
+// B6) go to C1
+
+ if (initialUri!=null) {
+ populateInitialFromUri(catalog, initialUri, PopulateMode.AUTODETECT);
+ return;
+ }
+
+ String catalogUrl = managementContext.getConfig().getConfig(BrooklynServerConfig.BROOKLYN_CATALOG_URL);
+ if (Strings.isNonBlank(catalogUrl)) {
+ populateInitialFromUri(catalog, catalogUrl, PopulateMode.AUTODETECT);
+ return;
+ }
+
+ catalogUrl = Urls.mergePaths(BrooklynServerConfig.getMgmtBaseDir( managementContext.getConfig() ), "catalog.bom");
+ if (new File(catalogUrl).exists()) {
+ populateInitialFromUri(catalog, "file:"+catalogUrl, PopulateMode.YAML);
+ return;
+ }
+
+ catalogUrl = Urls.mergePaths(BrooklynServerConfig.getMgmtBaseDir( managementContext.getConfig() ), "catalog.xml");
+ if (new File(catalogUrl).exists()) {
+ populateInitialFromUri(catalog, "file:"+catalogUrl, PopulateMode.XML);
+ return;
+ }
+
+ // otherwise look for for classpath:/brooklyn/default.catalog.bom --
+ // there is one on the classpath which says to scan, and provides a few templates;
+ // if one is supplied by user in the conf/ dir that will override the item from the classpath
+ // (TBD - we might want to scan for all such bom's?)
+
+ catalogUrl = "classpath:/brooklyn/default.catalog.bom";
+ if (new ResourceUtils(this).doesUrlExist(catalogUrl)) {
+ populateInitialFromUri(catalog, catalogUrl, PopulateMode.YAML);
+ return;
+ }
+
+ log.info("No catalog found on classpath or specified; catalog will not be initialized.");
+ return;
+ }
+
+ private void populateInitialFromUri(BasicBrooklynCatalog catalog, String catalogUrl, PopulateMode mode) {
+ log.debug("Loading initial catalog from {}", catalogUrl);
+
+ Exception problem = null;
+ Object result = null;
+
+ String contents = null;
+ try {
+ contents = new ResourceUtils(this).getResourceAsString(catalogUrl);
+ } catch (Exception e) {
+ Exceptions.propagateIfFatal(e);
+ if (problem==null) problem = e;
+ }
+
+ if (contents!=null && (mode==PopulateMode.YAML || mode==PopulateMode.AUTODETECT)) {
+ // try YAML first
+ try {
+ catalog.reset(MutableList.<CatalogItem<?,?>>of());
+ result = catalog.addItems(contents);
+ } catch (Exception e) {
+ Exceptions.propagateIfFatal(e);
+ if (problem==null) problem = e;
+ }
+ }
+
+ if (result==null && contents!=null && (mode==PopulateMode.XML || mode==PopulateMode.AUTODETECT)) {
+ // then try XML
+ problem = populateInitialFromUriXml(catalog, catalogUrl, problem, contents);
--- End diff --
Minor preference: I think the code would be clearer if the try-catch was in this method (similar to the YAML code above), rather than passing the `problem` in as a parameter in the method call here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by neykov <gi...@git.apache.org>.
Github user neykov commented on a diff in the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#discussion_r30787284
--- Diff: core/src/main/java/brooklyn/catalog/internal/CatalogInitialization.java ---
@@ -0,0 +1,380 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package brooklyn.catalog.internal;
+
+import java.io.File;
+import java.util.Collection;
+import java.util.List;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import brooklyn.catalog.CatalogItem;
+import brooklyn.config.BrooklynServerConfig;
+import brooklyn.management.ManagementContext;
+import brooklyn.management.ManagementContextInjectable;
+import brooklyn.management.internal.ManagementContextInternal;
+import brooklyn.util.ResourceUtils;
+import brooklyn.util.collections.MutableList;
+import brooklyn.util.exceptions.Exceptions;
+import brooklyn.util.exceptions.FatalRuntimeException;
+import brooklyn.util.exceptions.RuntimeInterruptedException;
+import brooklyn.util.flags.TypeCoercions;
+import brooklyn.util.guava.Maybe;
+import brooklyn.util.os.Os;
+import brooklyn.util.text.Strings;
+
+import com.google.common.annotations.Beta;
+import com.google.common.base.Function;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.Iterables;
+
+@Beta
+public class CatalogInitialization implements ManagementContextInjectable {
+
+ /*
+
+ A1) if not persisting, go to B1
+ A2) if --catalog-reset, delete persisted catalog items
+ A3) if there is a persisted catalog (and it wasn't not deleted by A2), read it and go to C1
+ A4) go to B1
+
+ B1) look for --catalog-initial, if so read it, then go to C1
+ B2) look for BrooklynServerConfig.BROOKLYN_CATALOG_URL, if so, read it, supporting YAML or XML (warning if XML), then go to C1
+ B3) look for ~/.brooklyn/catalog.bom, if exists, read it then go to C1
+ B4) look for ~/.brooklyn/brooklyn.xml, if exists, warn, read it then go to C1
+ B5) read all classpath://brooklyn/default.catalog.bom items, if they exist (and for now they will)
+ B6) go to C1
+
+ C1) if --catalog-add, read and add those items
+
+ D1) if persisting, read the rest of the persisted items (entities etc)
+
+ */
+
+ private static final Logger log = LoggerFactory.getLogger(CatalogInitialization.class);
+
+ private String initialUri;
+ private boolean reset;
+ private String additionsUri;
+ private boolean force;
+
+ private boolean disallowLocal = false;
+ private List<Function<CatalogInitialization, Void>> callbacks = MutableList.of();
+ private boolean hasRunBestEffort = false, hasRunOfficial = false, isPopulating = false;
+
+ private ManagementContext managementContext;
+ private boolean isStartingUp = false;
+ private boolean failOnStartupErrors = false;
+
+ private Object populatingCatalogMutex = new Object();
+
+ public CatalogInitialization(String initialUri, boolean reset, String additionUri, boolean force) {
+ this.initialUri = initialUri;
+ this.reset = reset;
+ this.additionsUri = additionUri;
+ this.force = force;
+ }
+
+ public CatalogInitialization() {
+ this(null, false, null, false);
+ }
+
+ public void injectManagementContext(ManagementContext managementContext) {
+ Preconditions.checkNotNull(managementContext, "management context");
+ if (this.managementContext!=null && managementContext!=this.managementContext)
+ throw new IllegalStateException("Cannot switch management context, from "+this.managementContext+" to "+managementContext);
+ this.managementContext = managementContext;
+ }
+
+ /** Called by the framework to set true while starting up, and false afterwards,
+ * in order to assist in appropriate logging and error handling. */
+ public void setStartingUp(boolean isStartingUp) {
+ this.isStartingUp = isStartingUp;
+ }
+
+ public void setFailOnStartupErrors(boolean startupFailOnCatalogErrors) {
+ this.failOnStartupErrors = startupFailOnCatalogErrors;
+ }
+
+ public CatalogInitialization addPopulationCallback(Function<CatalogInitialization, Void> callback) {
+ callbacks.add(callback);
+ return this;
+ }
+
+ public ManagementContext getManagementContext() {
+ return Preconditions.checkNotNull(managementContext, "management context has not been injected into "+this);
+ }
+
+ public boolean isInitialResetRequested() {
+ return reset;
+ }
+
+ public boolean hasRunOfficial() { return hasRunOfficial; }
+ public boolean hasRunIncludingBestEffort() { return hasRunOfficial || hasRunBestEffort; }
+
+ /** makes or updates the mgmt catalog, based on the settings in this class */
+ public void populateCatalog(boolean needsInitial, Collection<CatalogItem<?, ?>> optionalItemsForResettingCatalog) {
+ synchronized (populatingCatalogMutex) {
+ try {
+ isPopulating = true;
+ BasicBrooklynCatalog catalog = (BasicBrooklynCatalog) managementContext.getCatalog();
+ if (!catalog.getCatalog().isLoaded()) {
+ catalog.load();
+ } else {
+ if (needsInitial && (hasRunOfficial || hasRunBestEffort)) {
+ // an indication that something caused it to load early; not severe, but unusual
+ log.warn("Catalog initialization has not properly run but management context has a catalog; re-populating, possibly overwriting items installed during earlier access (it may have been an early web request)");
+ catalog.reset(ImmutableList.<CatalogItem<?,?>>of());
+ }
+ }
+ hasRunOfficial = true;
+
+ populateCatalogImpl(catalog, needsInitial, optionalItemsForResettingCatalog);
+ } finally {
+ hasRunOfficial = true;
+ isPopulating = false;
+ }
+ }
+ }
+
+ private void populateCatalogImpl(BasicBrooklynCatalog catalog, boolean needsInitial, Collection<CatalogItem<?, ?>> optionalItemsForResettingCatalog) {
+ applyCatalogLoadMode();
+
+ if (optionalItemsForResettingCatalog!=null) {
+ catalog.reset(optionalItemsForResettingCatalog);
+ }
+
+ if (needsInitial) {
+ populateInitial(catalog);
+ }
+
+ populateAdditions(catalog);
+
+ populateViaCallbacks(catalog);
+ }
+
+ private enum PopulateMode { YAML, XML, AUTODETECT }
+
+ protected void populateInitial(BasicBrooklynCatalog catalog) {
+ if (disallowLocal) {
+ if (!hasRunOfficial()) {
+ log.debug("CLI initial catalog not being read with disallow-local mode set.");
--- End diff --
disallow-local has no meaning to users, better use "brooklyn.catalog.mode=LOAD_PERSISTED_STATE"
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by aledsage <gi...@git.apache.org>.
Github user aledsage commented on a diff in the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#discussion_r30548244
--- Diff: core/src/main/java/brooklyn/entity/rebind/PeriodicDeltaChangeListener.java ---
@@ -247,68 +221,73 @@ void stop() {
stop(Duration.TEN_SECONDS, Duration.ONE_SECOND);
}
void stop(Duration timeout, Duration graceTimeoutForSubsequentOperations) {
- stopped = true;
- running = false;
-
- if (scheduledTask != null) {
- CountdownTimer expiry = timeout.countdownTimer();
- scheduledTask.cancel(false);
+ synchronized (startStopMutex) {
+ running = false;
try {
- waitForPendingComplete(expiry.getDurationRemaining().lowerBound(Duration.ZERO).add(graceTimeoutForSubsequentOperations));
- } catch (Exception e) {
- throw Exceptions.propagate(e);
- }
- scheduledTask.blockUntilEnded(expiry.getDurationRemaining().lowerBound(Duration.ZERO).add(graceTimeoutForSubsequentOperations));
- scheduledTask.cancel(true);
- boolean reallyEnded = Tasks.blockUntilInternalTasksEnded(scheduledTask, expiry.getDurationRemaining().lowerBound(Duration.ZERO).add(graceTimeoutForSubsequentOperations));
- if (!reallyEnded) {
- LOG.warn("Persistence tasks took too long to complete when stopping persistence (ignoring): "+scheduledTask);
- }
- scheduledTask = null;
- }
+ stopping = true;
+
+ if (scheduledTask != null) {
+ CountdownTimer expiry = timeout.countdownTimer();
+ try {
+ scheduledTask.cancel(false);
+ waitForPendingComplete(expiry.getDurationRemaining().lowerBound(Duration.ZERO).add(graceTimeoutForSubsequentOperations));
+ } catch (Exception e) {
+ throw Exceptions.propagate(e);
+ }
+ scheduledTask.blockUntilEnded(expiry.getDurationRemaining().lowerBound(Duration.ZERO).add(graceTimeoutForSubsequentOperations));
+ scheduledTask.cancel(true);
+ boolean reallyEnded = Tasks.blockUntilInternalTasksEnded(scheduledTask, expiry.getDurationRemaining().lowerBound(Duration.ZERO).add(graceTimeoutForSubsequentOperations));
+ if (!reallyEnded) {
+ LOG.warn("Persistence tasks took too long to terminate, when stopping persistence, although pending changes were persisted (ignoring): "+scheduledTask);
+ }
+ scheduledTask = null;
+ }
- // Discard all state that was waiting to be persisted
- synchronized (this) {
- deltaCollector = new DeltaCollector();
+ // Discard all state that was waiting to be persisted
+ synchronized (this) {
+ deltaCollector = new DeltaCollector();
+ }
+ } finally {
+ stopCompleted = true;
+ stopping = false;
+ }
}
}
/**
- * This method must only be used for testing. If required in production, then revisit implementation!
* @deprecated since 0.7.0, use {@link #waitForPendingComplete(Duration)}
*/
@VisibleForTesting
public void waitForPendingComplete(long timeout, TimeUnit unit) throws InterruptedException, TimeoutException {
waitForPendingComplete(Duration.of(timeout, unit));
}
+ /** Waits for any in-progress writes to be completed then for or any unwritten data to be written. */
@VisibleForTesting
public void waitForPendingComplete(Duration timeout) throws InterruptedException, TimeoutException {
- // Every time we finish writing, we increment a counter. We note the current val, and then
- // wait until we can guarantee that a complete additional write has been done. Not sufficient
- // to wait for `writeCount > origWriteCount` because we might have read the value when almost
- // finished a write.
+ if (!isActive() && !stopping) return;
- long startTime = System.currentTimeMillis();
- long maxEndtime = timeout.isPositive() ? startTime + timeout.toMillisecondsRoundingUp() : Long.MAX_VALUE;
- long origWriteCount = writeCount.get();
- while (true) {
- if (!isActive()) {
- return; // no pending activity;
- } else if (writeCount.get() > (origWriteCount+1)) {
- return;
- }
-
- if (System.currentTimeMillis() > maxEndtime) {
- throw new TimeoutException("Timeout waiting for pending complete of rebind-periodic-delta, after "+Time.makeTimeStringRounded(timeout));
+ CountdownTimer timer = timeout.isPositive() ? CountdownTimer.newInstanceStarted(timeout) : CountdownTimer.newInstancePaused(Duration.PRACTICALLY_FOREVER);
+ // wait for mutex, so we aren't tricked by an in-progress who has already recycled the collector
+ if (persistingMutex.tryAcquire(timer.getDurationRemaining().toMilliseconds(), TimeUnit.MILLISECONDS)) {
+ try {
+ // now no one else is writing
+ if (!deltaCollector.isEmpty()) {
+ // but there is data that needs to be written
+ persistNowSafely(true);
--- End diff --
I can see this is simpler than the previously `writeCount`. However, it means that test code (calling `waitForPendingComplete`) will change the behaviour of the persistence. It no longer just waits, but actually does the persist. Therefore it means a class of bug will no longer be detected where the persisting was not happening for whatever reason.
That's probably not an issue (because such an errors are unlikely, thought they could arise from some deadlock scenarios for example).
No particularly strong feelings.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by neykov <gi...@git.apache.org>.
Github user neykov commented on a diff in the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#discussion_r30885147
--- Diff: utils/common/src/main/java/brooklyn/util/javalang/AggregateClassLoader.java ---
@@ -57,24 +61,65 @@ public static AggregateClassLoader newInstanceWithNoLoaders() {
/** Add a loader to the first position in the search path. */
public void addFirst(ClassLoader classLoader) {
- if (classLoader != null) classLoaders.add(0, classLoader);
+ if (classLoader != null) {
+ synchronized (classLoaders) {
+ classLoaders.add(0, classLoader);
+ }
+ }
}
/** Add a loader to the last position in the search path. */
public void addLast(ClassLoader classLoader) {
- if (classLoader != null) classLoaders.add(classLoader);
+ if (classLoader != null) {
+ synchronized (classLoaders) {
+ classLoaders.add(classLoader);
+ }
+ }
}
/** Add a loader to the specific position in the search path.
* (It is callers responsibility to ensure that position is valid.) */
public void add(int index, ClassLoader classLoader) {
- if (classLoader != null) classLoaders.add(index, classLoader);
+ if (classLoader != null) {
+ synchronized (classLoaders) {
+ classLoaders.add(index, classLoader);
+ }
+ }
+ }
+
+ /** Resets the classloader shown here to be the given set */
+ public void reset(Collection<? extends ClassLoader> newClassLoaders) {
+ synchronized (classLoaders) {
+ // synchronize:
+ // * to prevent concurrent invocations
+ // * so add(0, cl) doesn't interfere
+ // * and for good measure we add before removing so that iterator always contains everything
+ // although since iterator access is synchronized that shouldn't be necessary
+ int count = classLoaders.size();
+ classLoaders.addAll(newClassLoaders);
+ for (int i=0; i<count; i++) {
+ classLoaders.remove(0);
+ }
+ }
+ }
+
+ /** True if nothing is in the list here */
+ public boolean isEmpty() {
+ return classLoaders.isEmpty();
}
- /** Returns the _live_ (and modifiable) list of classloaders
- * @return */
+ /** Returns the _live_ (and modifiable) list of classloaders; dangerous and discouraged.
+ * @deprecated since 0.7.0 */
+ @Deprecated
public List<ClassLoader> getList() {
return classLoaders;
}
+ public Iterator<ClassLoader> iterator() {
+ synchronized (classLoaders) {
+ // provides iterator of snapshot
+ return classLoaders.iterator();
--- End diff --
Should it be unmodifiable iterator? Why give access to the internals at all?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by neykov <gi...@git.apache.org>.
Github user neykov commented on a diff in the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#discussion_r30788475
--- Diff: core/src/main/java/brooklyn/catalog/internal/CatalogInitialization.java ---
@@ -0,0 +1,380 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package brooklyn.catalog.internal;
+
+import java.io.File;
+import java.util.Collection;
+import java.util.List;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import brooklyn.catalog.CatalogItem;
+import brooklyn.config.BrooklynServerConfig;
+import brooklyn.management.ManagementContext;
+import brooklyn.management.ManagementContextInjectable;
+import brooklyn.management.internal.ManagementContextInternal;
+import brooklyn.util.ResourceUtils;
+import brooklyn.util.collections.MutableList;
+import brooklyn.util.exceptions.Exceptions;
+import brooklyn.util.exceptions.FatalRuntimeException;
+import brooklyn.util.exceptions.RuntimeInterruptedException;
+import brooklyn.util.flags.TypeCoercions;
+import brooklyn.util.guava.Maybe;
+import brooklyn.util.os.Os;
+import brooklyn.util.text.Strings;
+
+import com.google.common.annotations.Beta;
+import com.google.common.base.Function;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.Iterables;
+
+@Beta
+public class CatalogInitialization implements ManagementContextInjectable {
+
+ /*
+
+ A1) if not persisting, go to B1
+ A2) if --catalog-reset, delete persisted catalog items
+ A3) if there is a persisted catalog (and it wasn't not deleted by A2), read it and go to C1
+ A4) go to B1
+
+ B1) look for --catalog-initial, if so read it, then go to C1
+ B2) look for BrooklynServerConfig.BROOKLYN_CATALOG_URL, if so, read it, supporting YAML or XML (warning if XML), then go to C1
+ B3) look for ~/.brooklyn/catalog.bom, if exists, read it then go to C1
+ B4) look for ~/.brooklyn/brooklyn.xml, if exists, warn, read it then go to C1
+ B5) read all classpath://brooklyn/default.catalog.bom items, if they exist (and for now they will)
+ B6) go to C1
+
+ C1) if --catalog-add, read and add those items
+
+ D1) if persisting, read the rest of the persisted items (entities etc)
+
+ */
+
+ private static final Logger log = LoggerFactory.getLogger(CatalogInitialization.class);
+
+ private String initialUri;
+ private boolean reset;
+ private String additionsUri;
+ private boolean force;
+
+ private boolean disallowLocal = false;
+ private List<Function<CatalogInitialization, Void>> callbacks = MutableList.of();
+ private boolean hasRunBestEffort = false, hasRunOfficial = false, isPopulating = false;
+
+ private ManagementContext managementContext;
+ private boolean isStartingUp = false;
+ private boolean failOnStartupErrors = false;
+
+ private Object populatingCatalogMutex = new Object();
+
+ public CatalogInitialization(String initialUri, boolean reset, String additionUri, boolean force) {
+ this.initialUri = initialUri;
+ this.reset = reset;
+ this.additionsUri = additionUri;
+ this.force = force;
+ }
+
+ public CatalogInitialization() {
+ this(null, false, null, false);
+ }
+
+ public void injectManagementContext(ManagementContext managementContext) {
+ Preconditions.checkNotNull(managementContext, "management context");
+ if (this.managementContext!=null && managementContext!=this.managementContext)
+ throw new IllegalStateException("Cannot switch management context, from "+this.managementContext+" to "+managementContext);
+ this.managementContext = managementContext;
+ }
+
+ /** Called by the framework to set true while starting up, and false afterwards,
+ * in order to assist in appropriate logging and error handling. */
+ public void setStartingUp(boolean isStartingUp) {
+ this.isStartingUp = isStartingUp;
+ }
+
+ public void setFailOnStartupErrors(boolean startupFailOnCatalogErrors) {
+ this.failOnStartupErrors = startupFailOnCatalogErrors;
+ }
+
+ public CatalogInitialization addPopulationCallback(Function<CatalogInitialization, Void> callback) {
+ callbacks.add(callback);
+ return this;
+ }
+
+ public ManagementContext getManagementContext() {
+ return Preconditions.checkNotNull(managementContext, "management context has not been injected into "+this);
+ }
+
+ public boolean isInitialResetRequested() {
+ return reset;
+ }
+
+ public boolean hasRunOfficial() { return hasRunOfficial; }
+ public boolean hasRunIncludingBestEffort() { return hasRunOfficial || hasRunBestEffort; }
+
+ /** makes or updates the mgmt catalog, based on the settings in this class */
+ public void populateCatalog(boolean needsInitial, Collection<CatalogItem<?, ?>> optionalItemsForResettingCatalog) {
+ synchronized (populatingCatalogMutex) {
+ try {
+ isPopulating = true;
+ BasicBrooklynCatalog catalog = (BasicBrooklynCatalog) managementContext.getCatalog();
+ if (!catalog.getCatalog().isLoaded()) {
+ catalog.load();
+ } else {
+ if (needsInitial && (hasRunOfficial || hasRunBestEffort)) {
+ // an indication that something caused it to load early; not severe, but unusual
+ log.warn("Catalog initialization has not properly run but management context has a catalog; re-populating, possibly overwriting items installed during earlier access (it may have been an early web request)");
+ catalog.reset(ImmutableList.<CatalogItem<?,?>>of());
+ }
+ }
+ hasRunOfficial = true;
+
+ populateCatalogImpl(catalog, needsInitial, optionalItemsForResettingCatalog);
+ } finally {
+ hasRunOfficial = true;
+ isPopulating = false;
+ }
+ }
+ }
+
+ private void populateCatalogImpl(BasicBrooklynCatalog catalog, boolean needsInitial, Collection<CatalogItem<?, ?>> optionalItemsForResettingCatalog) {
+ applyCatalogLoadMode();
+
+ if (optionalItemsForResettingCatalog!=null) {
+ catalog.reset(optionalItemsForResettingCatalog);
+ }
+
+ if (needsInitial) {
+ populateInitial(catalog);
+ }
+
+ populateAdditions(catalog);
+
+ populateViaCallbacks(catalog);
+ }
+
+ private enum PopulateMode { YAML, XML, AUTODETECT }
+
+ protected void populateInitial(BasicBrooklynCatalog catalog) {
+ if (disallowLocal) {
+ if (!hasRunOfficial()) {
+ log.debug("CLI initial catalog not being read with disallow-local mode set.");
+ }
+ return;
+ }
+
+// B1) look for --catalog-initial, if so read it, then go to C1
+// B2) look for BrooklynServerConfig.BROOKLYN_CATALOG_URL, if so, read it, supporting YAML or XML (warning if XML), then go to C1
+// B3) look for ~/.brooklyn/catalog.bom, if exists, read it then go to C1
+// B4) look for ~/.brooklyn/brooklyn.xml, if exists, warn, read it then go to C1
+// B5) read all classpath://brooklyn/default.catalog.bom items, if they exist (and for now they will)
+// B6) go to C1
+
+ if (initialUri!=null) {
+ populateInitialFromUri(catalog, initialUri, PopulateMode.AUTODETECT);
+ return;
+ }
+
+ String catalogUrl = managementContext.getConfig().getConfig(BrooklynServerConfig.BROOKLYN_CATALOG_URL);
+ if (Strings.isNonBlank(catalogUrl)) {
+ populateInitialFromUri(catalog, catalogUrl, PopulateMode.AUTODETECT);
+ return;
+ }
+
+ catalogUrl = Os.mergePaths(BrooklynServerConfig.getMgmtBaseDir( managementContext.getConfig() ), "catalog.bom");
+ if (new File(catalogUrl).exists()) {
+ populateInitialFromUri(catalog, "file:"+catalogUrl, PopulateMode.YAML);
+ return;
+ }
+
+ catalogUrl = Os.mergePaths(BrooklynServerConfig.getMgmtBaseDir( managementContext.getConfig() ), "catalog.xml");
+ if (new File(catalogUrl).exists()) {
+ populateInitialFromUri(catalog, "file:"+catalogUrl, PopulateMode.XML);
+ return;
+ }
+
+ // otherwise look for for classpath:/brooklyn/default.catalog.bom --
+ // there is one on the classpath which says to scan, and provides a few templates;
+ // if one is supplied by user in the conf/ dir that will override the item from the classpath
+ // (TBD - we might want to scan for all such bom's?)
+
+ catalogUrl = "classpath:/brooklyn/default.catalog.bom";
+ if (new ResourceUtils(this).doesUrlExist(catalogUrl)) {
+ populateInitialFromUri(catalog, catalogUrl, PopulateMode.YAML);
+ return;
+ }
+
+ log.info("No catalog found on classpath or specified; catalog will not be initialized.");
+ return;
+ }
+
+ private void populateInitialFromUri(BasicBrooklynCatalog catalog, String catalogUrl, PopulateMode mode) {
+ log.debug("Loading initial catalog from {}", catalogUrl);
+
+ Exception problem = null;
+ Object result = null;
+
+ String contents = null;
+ try {
+ contents = new ResourceUtils(this).getResourceAsString(catalogUrl);
+ } catch (Exception e) {
+ Exceptions.propagateIfFatal(e);
+ if (problem==null) problem = e;
+ }
+
+ if (contents!=null && (mode==PopulateMode.YAML || mode==PopulateMode.AUTODETECT)) {
+ // try YAML first
+ try {
+ catalog.reset(MutableList.<CatalogItem<?,?>>of());
+ result = catalog.addItems(contents);
+ } catch (Exception e) {
+ Exceptions.propagateIfFatal(e);
+ if (problem==null) problem = e;
+ }
+ }
+
+ if (result==null && contents!=null && (mode==PopulateMode.XML || mode==PopulateMode.AUTODETECT)) {
+ // then try XML
+ try {
+ populateInitialFromUriXml(catalog, catalogUrl, contents);
+ // clear YAML problem
+ problem = null;
+ } catch (Exception e) {
+ Exceptions.propagateIfFatal(e);
+ if (problem==null) problem = e;
+ }
+ }
+
+ if (result!=null) {
+ log.debug("Loaded initial catalog from {}: {}", catalogUrl, result);
+ }
+ if (problem!=null) {
+ log.warn("Error importing catalog from " + catalogUrl + ": " + problem, problem);
+ // TODO inform mgmt of error
+ }
+
+ }
+
+ // deprecated XML format
+ @SuppressWarnings("deprecation")
+ private void populateInitialFromUriXml(BasicBrooklynCatalog catalog, String catalogUrl, String contents) {
+ CatalogDto dto = CatalogDto.newDtoFromXmlContents(contents, catalogUrl);
+ if (dto!=null) {
+ catalog.reset(dto);
+ }
+ }
+
+ boolean hasRunAdditions = false;
+ protected void populateAdditions(BasicBrooklynCatalog catalog) {
+ if (Strings.isNonBlank(additionsUri)) {
+ if (disallowLocal) {
+ if (!hasRunAdditions) {
+ log.warn("CLI additions supplied but not supported in disallow-local mode; ignoring.");
+ }
+ return;
+ }
+ if (!hasRunAdditions) {
+ log.debug("Adding to catalog from CLI: "+additionsUri+" (force: "+force+")");
+ }
+ Iterable<? extends CatalogItem<?, ?>> items = catalog.addItems(
+ new ResourceUtils(this).getResourceAsString(additionsUri), force);
+
+ if (!hasRunAdditions)
+ log.debug("Added to catalog from CLI: "+items);
+ else
+ log.debug("Added to catalog from CLI: count "+Iterables.size(items));
+
+ hasRunAdditions = true;
+ }
+ }
+
+ protected void populateViaCallbacks(BasicBrooklynCatalog catalog) {
+ for (Function<CatalogInitialization, Void> callback: callbacks)
+ callback.apply(this);
+ }
+
+ private Object setFromCLMMutex = new Object();
+ private boolean setFromCatalogLoadMode = false;
+
+ /** @deprecated since introduced in 0.7.0, only for legacy compatibility with
+ * {@link CatalogLoadMode} {@link BrooklynServerConfig#CATALOG_LOAD_MODE},
+ * allowing control of catalog loading from a brooklyn property */
+ @Deprecated
+ public void applyCatalogLoadMode() {
--- End diff --
Can this be private to the class and called at `injectManagementContext` time instead of requiring the "user" to call it?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by aledsage <gi...@git.apache.org>.
Github user aledsage commented on a diff in the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#discussion_r30590730
--- Diff: core/src/main/java/brooklyn/catalog/internal/CatalogInitialization.java ---
@@ -0,0 +1,377 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package brooklyn.catalog.internal;
+
+import java.io.File;
+import java.util.Collection;
+import java.util.List;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import brooklyn.catalog.CatalogItem;
+import brooklyn.config.BrooklynServerConfig;
+import brooklyn.management.ManagementContext;
+import brooklyn.management.ManagementContextInjectable;
+import brooklyn.management.internal.ManagementContextInternal;
+import brooklyn.util.ResourceUtils;
+import brooklyn.util.collections.MutableList;
+import brooklyn.util.exceptions.Exceptions;
+import brooklyn.util.exceptions.FatalRuntimeException;
+import brooklyn.util.exceptions.RuntimeInterruptedException;
+import brooklyn.util.flags.TypeCoercions;
+import brooklyn.util.guava.Maybe;
+import brooklyn.util.net.Urls;
+import brooklyn.util.text.Strings;
+
+import com.google.common.annotations.Beta;
+import com.google.common.base.Function;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.Iterables;
+
+@Beta
+public class CatalogInitialization implements ManagementContextInjectable {
+
+ /*
+
+ A1) if not persisting, go to B1
+ A2) if --catalog-reset, delete persisted catalog items
+ A3) read persisted catalog items (possibly deleted in A2), go to C1
+ A4) go to B1
+
+ B1) look for --catalog-initial, if so read it, then go to C1
+ B2) look for BrooklynServerConfig.BROOKLYN_CATALOG_URL, if so, read it, supporting YAML or XML (warning if XML), then go to C1
+ B3) look for ~/.brooklyn/catalog.bom, if exists, read it then go to C1
+ B4) look for ~/.brooklyn/brooklyn.xml, if exists, warn, read it then go to C1
+ B5) read all classpath://brooklyn/default.catalog.bom items, if they exist (and for now they will)
+ B6) go to C1
+
+ C1) if --catalog-add, read and add those items
+
+ D1) if persisting, read the rest of the persisted items (entities etc)
+
+ */
+
+ private static final Logger log = LoggerFactory.getLogger(CatalogInitialization.class);
+
+ String initialUri;
+ boolean reset;
+ String additionsUri;
+ boolean force;
+
+ boolean disallowLocal = false;
+ List<Function<CatalogInitialization, Void>> callbacks = MutableList.of();
+ boolean hasRunBestEffort = false, hasRunOfficial = false, isPopulating = false;
+
+ ManagementContext managementContext;
+ boolean isStartingUp = false;
+ boolean failOnStartupErrors = false;
+
+ Object mutex = new Object();
+
+ public CatalogInitialization(String initialUri, boolean reset, String additionUri, boolean force) {
+ this.initialUri = initialUri;
+ this.reset = reset;
+ this.additionsUri = additionUri;
+ this.force = force;
+ }
+
+ public CatalogInitialization() {
+ this(null, false, null, false);
+ }
+
+ public void injectManagementContext(ManagementContext managementContext) {
+ if (this.managementContext!=null && managementContext!=null && !this.managementContext.equals(managementContext))
+ throw new IllegalStateException("Cannot switch management context of "+this+"; from "+this.managementContext+" to "+managementContext);
+ this.managementContext = managementContext;
+ }
+
+ public ManagementContext getManagementContext() {
+ return Preconditions.checkNotNull(managementContext, "management context has not been injected into "+this);
+ }
+
+ public CatalogInitialization addPopulationCallback(Function<CatalogInitialization, Void> callback) {
+ callbacks.add(callback);
+ return this;
+ }
+
+ public boolean isInitialResetRequested() {
+ return reset;
+ }
+
+ public boolean hasRunOfficial() { return hasRunOfficial; }
+ public boolean hasRunIncludingBestEffort() { return hasRunOfficial || hasRunBestEffort; }
+
+ /** makes or updates the mgmt catalog, based on the settings in this class */
+ public void populateCatalog(boolean needsInitial, Collection<CatalogItem<?, ?>> optionalItemsForResettingCatalog) {
+ try {
+ isPopulating = true;
+ synchronized (mutex) {
+ BasicBrooklynCatalog catalog = (BasicBrooklynCatalog) managementContext.getCatalog();
+ if (!catalog.getCatalog().isLoaded()) {
+ catalog.load();
+ } else {
+ if (needsInitial && (hasRunOfficial || hasRunBestEffort)) {
+ // an indication that something caused it to load early; not severe, but unusual
+ log.warn("Catalog initialization has not properly run but management context has a catalog; re-populating, possibly overwriting items installed during earlier access (it may have been an early web request)");
+ catalog.reset(ImmutableList.<CatalogItem<?,?>>of());
+ }
+ }
+ hasRunOfficial = true;
+
+ populateCatalog(catalog, needsInitial, true, optionalItemsForResettingCatalog);
+ }
+ } finally {
+ hasRunOfficial = true;
+ isPopulating = false;
+ }
+ }
+
+ private void populateCatalog(BasicBrooklynCatalog catalog, boolean needsInitial, boolean runCallbacks, Collection<CatalogItem<?, ?>> optionalItemsForResettingCatalog) {
+ applyCatalogLoadMode();
+
+ if (optionalItemsForResettingCatalog!=null) {
+ catalog.reset(optionalItemsForResettingCatalog);
+ }
+
+ if (needsInitial) {
+ populateInitial(catalog);
+ }
+
+ populateAdditions(catalog);
+
+ if (runCallbacks) {
+ populateViaCallbacks(catalog);
+ }
+ }
+
+ private enum PopulateMode { YAML, XML, AUTODETECT }
+
+ protected void populateInitial(BasicBrooklynCatalog catalog) {
+ if (disallowLocal) {
+ if (!hasRunOfficial()) {
+ log.debug("CLI initial catalog not being read with disallow-local mode set.");
+ }
+ return;
+ }
+
+// B1) look for --catalog-initial, if so read it, then go to C1
+// B2) look for BrooklynServerConfig.BROOKLYN_CATALOG_URL, if so, read it, supporting YAML or XML (warning if XML), then go to C1
+// B3) look for ~/.brooklyn/catalog.bom, if exists, read it then go to C1
+// B4) look for ~/.brooklyn/brooklyn.xml, if exists, warn, read it then go to C1
+// B5) read all classpath://brooklyn/default.catalog.bom items, if they exist (and for now they will)
+// B6) go to C1
+
+ if (initialUri!=null) {
+ populateInitialFromUri(catalog, initialUri, PopulateMode.AUTODETECT);
+ return;
+ }
+
+ String catalogUrl = managementContext.getConfig().getConfig(BrooklynServerConfig.BROOKLYN_CATALOG_URL);
+ if (Strings.isNonBlank(catalogUrl)) {
+ populateInitialFromUri(catalog, catalogUrl, PopulateMode.AUTODETECT);
+ return;
+ }
+
+ catalogUrl = Urls.mergePaths(BrooklynServerConfig.getMgmtBaseDir( managementContext.getConfig() ), "catalog.bom");
+ if (new File(catalogUrl).exists()) {
+ populateInitialFromUri(catalog, "file:"+catalogUrl, PopulateMode.YAML);
+ return;
+ }
+
+ catalogUrl = Urls.mergePaths(BrooklynServerConfig.getMgmtBaseDir( managementContext.getConfig() ), "catalog.xml");
+ if (new File(catalogUrl).exists()) {
+ populateInitialFromUri(catalog, "file:"+catalogUrl, PopulateMode.XML);
+ return;
+ }
+
+ // otherwise look for for classpath:/brooklyn/default.catalog.bom --
+ // there is one on the classpath which says to scan, and provides a few templates;
+ // if one is supplied by user in the conf/ dir that will override the item from the classpath
+ // (TBD - we might want to scan for all such bom's?)
+
+ catalogUrl = "classpath:/brooklyn/default.catalog.bom";
+ if (new ResourceUtils(this).doesUrlExist(catalogUrl)) {
+ populateInitialFromUri(catalog, catalogUrl, PopulateMode.YAML);
+ return;
+ }
+
+ log.info("No catalog found on classpath or specified; catalog will not be initialized.");
+ return;
+ }
+
+ private void populateInitialFromUri(BasicBrooklynCatalog catalog, String catalogUrl, PopulateMode mode) {
+ log.debug("Loading initial catalog from {}", catalogUrl);
+
+ Exception problem = null;
+ Object result = null;
+
+ String contents = null;
+ try {
+ contents = new ResourceUtils(this).getResourceAsString(catalogUrl);
+ } catch (Exception e) {
+ Exceptions.propagateIfFatal(e);
+ if (problem==null) problem = e;
+ }
+
+ if (contents!=null && (mode==PopulateMode.YAML || mode==PopulateMode.AUTODETECT)) {
+ // try YAML first
+ try {
+ catalog.reset(MutableList.<CatalogItem<?,?>>of());
+ result = catalog.addItems(contents);
+ } catch (Exception e) {
+ Exceptions.propagateIfFatal(e);
+ if (problem==null) problem = e;
+ }
+ }
+
+ if (result==null && contents!=null && (mode==PopulateMode.XML || mode==PopulateMode.AUTODETECT)) {
+ // then try XML
+ problem = populateInitialFromUriXml(catalog, catalogUrl, problem, contents);
+ }
+
+ if (result!=null) {
+ log.debug("Loaded initial catalog from {}: {}", catalogUrl, result);
+ }
+ if (problem!=null) {
+ log.warn("Error importing catalog from " + catalogUrl + ": " + problem, problem);
+ // TODO inform mgmt of error
+ }
+
+ }
+
+ // deprecated XML format
+ @SuppressWarnings("deprecation")
+ private Exception populateInitialFromUriXml(BasicBrooklynCatalog catalog, String catalogUrl, Exception problem, String contents) {
+ CatalogDto dto = null;
+ try {
+ dto = CatalogDto.newDtoFromXmlContents(contents, catalogUrl);
+ problem = null;
+ } catch (Exception e) {
+ Exceptions.propagateIfFatal(e);
+ if (problem==null) problem = e;
+ }
+ if (dto!=null) {
+ catalog.reset(dto);
+ }
+ return problem;
+ }
+
+ boolean hasRunAdditions = false;
+ protected void populateAdditions(BasicBrooklynCatalog catalog) {
+ if (Strings.isNonBlank(additionsUri)) {
+ if (disallowLocal) {
+ if (!hasRunAdditions) {
+ log.warn("CLI additions supplied but not supported in disallow-local mode; ignoring.");
+ }
+ return;
+ }
+ if (!hasRunAdditions) {
+ log.debug("Adding to catalog from CLI: "+additionsUri+" (force: "+force+")");
+ }
+ Iterable<? extends CatalogItem<?, ?>> items = catalog.addItems(
+ new ResourceUtils(this).getResourceAsString(additionsUri), force);
+
+ if (!hasRunAdditions)
+ log.debug("Added to catalog from CLI: "+items);
+ else
+ log.debug("Added to catalog from CLI: count "+Iterables.size(items));
+
+ hasRunAdditions = true;
+ }
+ }
+
+ protected void populateViaCallbacks(BasicBrooklynCatalog catalog) {
+ for (Function<CatalogInitialization, Void> callback: callbacks)
--- End diff --
Ok, no strong feelings. A line or two of javadoc would have removed the uncertainty in my mind.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by ahgittin <gi...@git.apache.org>.
Github user ahgittin commented on the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#issuecomment-103698264
A few I disagreed on, I've commented there, the rest should all be addresed in the last PR. (I've squashed what were three PR's.)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by ahgittin <gi...@git.apache.org>.
Github user ahgittin commented on the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#issuecomment-103962987
btw failure again is unrelated -- either there is something up with ports on this server or our logic (maybe around networks?) is not what we expect
```
java.lang.RuntimeException: unable to find a free port at or above 8080
at brooklyn.util.net.Networking.nextAvailablePort(Networking.java:156)
at io.brooklyn.camp.CampServer$CampServerUtils.startServer(CampServer.java:151)
at io.brooklyn.camp.CampServer.start(CampServer.java:104)
at io.brooklyn.camp.test.fixture.AbstractRestResourceTest.startServer(AbstractRestResourceTest.java:51)
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by neykov <gi...@git.apache.org>.
Github user neykov commented on a diff in the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#discussion_r30787212
--- Diff: core/src/main/java/brooklyn/catalog/internal/CatalogInitialization.java ---
@@ -0,0 +1,380 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package brooklyn.catalog.internal;
+
+import java.io.File;
+import java.util.Collection;
+import java.util.List;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import brooklyn.catalog.CatalogItem;
+import brooklyn.config.BrooklynServerConfig;
+import brooklyn.management.ManagementContext;
+import brooklyn.management.ManagementContextInjectable;
+import brooklyn.management.internal.ManagementContextInternal;
+import brooklyn.util.ResourceUtils;
+import brooklyn.util.collections.MutableList;
+import brooklyn.util.exceptions.Exceptions;
+import brooklyn.util.exceptions.FatalRuntimeException;
+import brooklyn.util.exceptions.RuntimeInterruptedException;
+import brooklyn.util.flags.TypeCoercions;
+import brooklyn.util.guava.Maybe;
+import brooklyn.util.os.Os;
+import brooklyn.util.text.Strings;
+
+import com.google.common.annotations.Beta;
+import com.google.common.base.Function;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.Iterables;
+
+@Beta
+public class CatalogInitialization implements ManagementContextInjectable {
+
+ /*
+
+ A1) if not persisting, go to B1
+ A2) if --catalog-reset, delete persisted catalog items
+ A3) if there is a persisted catalog (and it wasn't not deleted by A2), read it and go to C1
+ A4) go to B1
+
+ B1) look for --catalog-initial, if so read it, then go to C1
+ B2) look for BrooklynServerConfig.BROOKLYN_CATALOG_URL, if so, read it, supporting YAML or XML (warning if XML), then go to C1
+ B3) look for ~/.brooklyn/catalog.bom, if exists, read it then go to C1
+ B4) look for ~/.brooklyn/brooklyn.xml, if exists, warn, read it then go to C1
+ B5) read all classpath://brooklyn/default.catalog.bom items, if they exist (and for now they will)
+ B6) go to C1
+
+ C1) if --catalog-add, read and add those items
+
+ D1) if persisting, read the rest of the persisted items (entities etc)
+
+ */
+
+ private static final Logger log = LoggerFactory.getLogger(CatalogInitialization.class);
+
+ private String initialUri;
+ private boolean reset;
+ private String additionsUri;
+ private boolean force;
+
+ private boolean disallowLocal = false;
+ private List<Function<CatalogInitialization, Void>> callbacks = MutableList.of();
+ private boolean hasRunBestEffort = false, hasRunOfficial = false, isPopulating = false;
+
+ private ManagementContext managementContext;
+ private boolean isStartingUp = false;
+ private boolean failOnStartupErrors = false;
+
+ private Object populatingCatalogMutex = new Object();
+
+ public CatalogInitialization(String initialUri, boolean reset, String additionUri, boolean force) {
+ this.initialUri = initialUri;
+ this.reset = reset;
+ this.additionsUri = additionUri;
+ this.force = force;
+ }
+
+ public CatalogInitialization() {
+ this(null, false, null, false);
+ }
+
+ public void injectManagementContext(ManagementContext managementContext) {
+ Preconditions.checkNotNull(managementContext, "management context");
+ if (this.managementContext!=null && managementContext!=this.managementContext)
+ throw new IllegalStateException("Cannot switch management context, from "+this.managementContext+" to "+managementContext);
+ this.managementContext = managementContext;
+ }
+
+ /** Called by the framework to set true while starting up, and false afterwards,
+ * in order to assist in appropriate logging and error handling. */
+ public void setStartingUp(boolean isStartingUp) {
+ this.isStartingUp = isStartingUp;
+ }
+
+ public void setFailOnStartupErrors(boolean startupFailOnCatalogErrors) {
+ this.failOnStartupErrors = startupFailOnCatalogErrors;
+ }
+
+ public CatalogInitialization addPopulationCallback(Function<CatalogInitialization, Void> callback) {
+ callbacks.add(callback);
+ return this;
+ }
+
+ public ManagementContext getManagementContext() {
+ return Preconditions.checkNotNull(managementContext, "management context has not been injected into "+this);
+ }
+
+ public boolean isInitialResetRequested() {
+ return reset;
+ }
+
+ public boolean hasRunOfficial() { return hasRunOfficial; }
+ public boolean hasRunIncludingBestEffort() { return hasRunOfficial || hasRunBestEffort; }
+
+ /** makes or updates the mgmt catalog, based on the settings in this class */
+ public void populateCatalog(boolean needsInitial, Collection<CatalogItem<?, ?>> optionalItemsForResettingCatalog) {
+ synchronized (populatingCatalogMutex) {
+ try {
+ isPopulating = true;
+ BasicBrooklynCatalog catalog = (BasicBrooklynCatalog) managementContext.getCatalog();
+ if (!catalog.getCatalog().isLoaded()) {
+ catalog.load();
+ } else {
+ if (needsInitial && (hasRunOfficial || hasRunBestEffort)) {
+ // an indication that something caused it to load early; not severe, but unusual
+ log.warn("Catalog initialization has not properly run but management context has a catalog; re-populating, possibly overwriting items installed during earlier access (it may have been an early web request)");
+ catalog.reset(ImmutableList.<CatalogItem<?,?>>of());
+ }
+ }
+ hasRunOfficial = true;
+
+ populateCatalogImpl(catalog, needsInitial, optionalItemsForResettingCatalog);
+ } finally {
+ hasRunOfficial = true;
+ isPopulating = false;
+ }
+ }
+ }
+
+ private void populateCatalogImpl(BasicBrooklynCatalog catalog, boolean needsInitial, Collection<CatalogItem<?, ?>> optionalItemsForResettingCatalog) {
+ applyCatalogLoadMode();
+
+ if (optionalItemsForResettingCatalog!=null) {
+ catalog.reset(optionalItemsForResettingCatalog);
+ }
+
+ if (needsInitial) {
+ populateInitial(catalog);
+ }
+
+ populateAdditions(catalog);
+
+ populateViaCallbacks(catalog);
+ }
+
+ private enum PopulateMode { YAML, XML, AUTODETECT }
+
+ protected void populateInitial(BasicBrooklynCatalog catalog) {
+ if (disallowLocal) {
+ if (!hasRunOfficial()) {
+ log.debug("CLI initial catalog not being read with disallow-local mode set.");
+ }
+ return;
--- End diff --
If the user has set `initialUri`, but at the same time has `CATALOG_LOAD_MODE=LOAD_PERSISTED_STATE` configured, I'd expect the new style to take precedence. Or even fail early and refuse to start with a descriptive message.
Letting the old style win with only a debug message is confusing.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by aledsage <gi...@git.apache.org>.
Github user aledsage commented on a diff in the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#discussion_r30585960
--- Diff: docs/guide/ops/catalog/index.md ---
@@ -284,6 +291,19 @@ When referencing a blueprint, if a version number is not specified
the latest non-snapshot version will be loaded when an entity is instantiated.
+### CLI Options
+
+The `brooklyn` CLI includes several commands for working with the catalog.
+
+* `--catalogAdd <file.bom>` will add the catalog items in the `bom` file
--- End diff --
The term `bom` seems never to be defined.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by ahgittin <gi...@git.apache.org>.
Github user ahgittin commented on a diff in the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#discussion_r30967843
--- Diff: core/src/main/java/brooklyn/catalog/internal/CatalogInitialization.java ---
@@ -0,0 +1,380 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package brooklyn.catalog.internal;
+
+import java.io.File;
+import java.util.Collection;
+import java.util.List;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import brooklyn.catalog.CatalogItem;
+import brooklyn.config.BrooklynServerConfig;
+import brooklyn.management.ManagementContext;
+import brooklyn.management.ManagementContextInjectable;
+import brooklyn.management.internal.ManagementContextInternal;
+import brooklyn.util.ResourceUtils;
+import brooklyn.util.collections.MutableList;
+import brooklyn.util.exceptions.Exceptions;
+import brooklyn.util.exceptions.FatalRuntimeException;
+import brooklyn.util.exceptions.RuntimeInterruptedException;
+import brooklyn.util.flags.TypeCoercions;
+import brooklyn.util.guava.Maybe;
+import brooklyn.util.os.Os;
+import brooklyn.util.text.Strings;
+
+import com.google.common.annotations.Beta;
+import com.google.common.base.Function;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.Iterables;
+
+@Beta
+public class CatalogInitialization implements ManagementContextInjectable {
+
+ /*
+
+ A1) if not persisting, go to B1
+ A2) if --catalog-reset, delete persisted catalog items
+ A3) if there is a persisted catalog (and it wasn't not deleted by A2), read it and go to C1
+ A4) go to B1
+
+ B1) look for --catalog-initial, if so read it, then go to C1
+ B2) look for BrooklynServerConfig.BROOKLYN_CATALOG_URL, if so, read it, supporting YAML or XML (warning if XML), then go to C1
+ B3) look for ~/.brooklyn/catalog.bom, if exists, read it then go to C1
+ B4) look for ~/.brooklyn/brooklyn.xml, if exists, warn, read it then go to C1
+ B5) read all classpath://brooklyn/default.catalog.bom items, if they exist (and for now they will)
+ B6) go to C1
+
+ C1) if --catalog-add, read and add those items
+
+ D1) if persisting, read the rest of the persisted items (entities etc)
+
+ */
+
+ private static final Logger log = LoggerFactory.getLogger(CatalogInitialization.class);
+
+ private String initialUri;
+ private boolean reset;
+ private String additionsUri;
+ private boolean force;
+
+ private boolean disallowLocal = false;
+ private List<Function<CatalogInitialization, Void>> callbacks = MutableList.of();
+ private boolean hasRunBestEffort = false, hasRunOfficial = false, isPopulating = false;
+
+ private ManagementContext managementContext;
+ private boolean isStartingUp = false;
+ private boolean failOnStartupErrors = false;
+
+ private Object populatingCatalogMutex = new Object();
+
+ public CatalogInitialization(String initialUri, boolean reset, String additionUri, boolean force) {
+ this.initialUri = initialUri;
+ this.reset = reset;
+ this.additionsUri = additionUri;
+ this.force = force;
+ }
+
+ public CatalogInitialization() {
+ this(null, false, null, false);
+ }
+
+ public void injectManagementContext(ManagementContext managementContext) {
+ Preconditions.checkNotNull(managementContext, "management context");
+ if (this.managementContext!=null && managementContext!=this.managementContext)
+ throw new IllegalStateException("Cannot switch management context, from "+this.managementContext+" to "+managementContext);
+ this.managementContext = managementContext;
+ }
+
+ /** Called by the framework to set true while starting up, and false afterwards,
+ * in order to assist in appropriate logging and error handling. */
+ public void setStartingUp(boolean isStartingUp) {
+ this.isStartingUp = isStartingUp;
+ }
+
+ public void setFailOnStartupErrors(boolean startupFailOnCatalogErrors) {
+ this.failOnStartupErrors = startupFailOnCatalogErrors;
+ }
+
+ public CatalogInitialization addPopulationCallback(Function<CatalogInitialization, Void> callback) {
+ callbacks.add(callback);
+ return this;
+ }
+
+ public ManagementContext getManagementContext() {
+ return Preconditions.checkNotNull(managementContext, "management context has not been injected into "+this);
+ }
+
+ public boolean isInitialResetRequested() {
+ return reset;
+ }
+
+ public boolean hasRunOfficial() { return hasRunOfficial; }
+ public boolean hasRunIncludingBestEffort() { return hasRunOfficial || hasRunBestEffort; }
+
+ /** makes or updates the mgmt catalog, based on the settings in this class */
+ public void populateCatalog(boolean needsInitial, Collection<CatalogItem<?, ?>> optionalItemsForResettingCatalog) {
+ synchronized (populatingCatalogMutex) {
+ try {
+ isPopulating = true;
+ BasicBrooklynCatalog catalog = (BasicBrooklynCatalog) managementContext.getCatalog();
+ if (!catalog.getCatalog().isLoaded()) {
+ catalog.load();
+ } else {
+ if (needsInitial && (hasRunOfficial || hasRunBestEffort)) {
+ // an indication that something caused it to load early; not severe, but unusual
+ log.warn("Catalog initialization has not properly run but management context has a catalog; re-populating, possibly overwriting items installed during earlier access (it may have been an early web request)");
+ catalog.reset(ImmutableList.<CatalogItem<?,?>>of());
+ }
+ }
+ hasRunOfficial = true;
+
+ populateCatalogImpl(catalog, needsInitial, optionalItemsForResettingCatalog);
+ } finally {
+ hasRunOfficial = true;
+ isPopulating = false;
+ }
+ }
+ }
+
+ private void populateCatalogImpl(BasicBrooklynCatalog catalog, boolean needsInitial, Collection<CatalogItem<?, ?>> optionalItemsForResettingCatalog) {
+ applyCatalogLoadMode();
+
+ if (optionalItemsForResettingCatalog!=null) {
+ catalog.reset(optionalItemsForResettingCatalog);
+ }
+
+ if (needsInitial) {
+ populateInitial(catalog);
+ }
+
+ populateAdditions(catalog);
+
+ populateViaCallbacks(catalog);
+ }
+
+ private enum PopulateMode { YAML, XML, AUTODETECT }
+
+ protected void populateInitial(BasicBrooklynCatalog catalog) {
+ if (disallowLocal) {
+ if (!hasRunOfficial()) {
+ log.debug("CLI initial catalog not being read with disallow-local mode set.");
+ }
+ return;
+ }
+
+// B1) look for --catalog-initial, if so read it, then go to C1
+// B2) look for BrooklynServerConfig.BROOKLYN_CATALOG_URL, if so, read it, supporting YAML or XML (warning if XML), then go to C1
+// B3) look for ~/.brooklyn/catalog.bom, if exists, read it then go to C1
+// B4) look for ~/.brooklyn/brooklyn.xml, if exists, warn, read it then go to C1
+// B5) read all classpath://brooklyn/default.catalog.bom items, if they exist (and for now they will)
+// B6) go to C1
+
+ if (initialUri!=null) {
+ populateInitialFromUri(catalog, initialUri, PopulateMode.AUTODETECT);
+ return;
+ }
+
+ String catalogUrl = managementContext.getConfig().getConfig(BrooklynServerConfig.BROOKLYN_CATALOG_URL);
+ if (Strings.isNonBlank(catalogUrl)) {
+ populateInitialFromUri(catalog, catalogUrl, PopulateMode.AUTODETECT);
+ return;
+ }
+
+ catalogUrl = Os.mergePaths(BrooklynServerConfig.getMgmtBaseDir( managementContext.getConfig() ), "catalog.bom");
+ if (new File(catalogUrl).exists()) {
+ populateInitialFromUri(catalog, "file:"+catalogUrl, PopulateMode.YAML);
+ return;
+ }
+
+ catalogUrl = Os.mergePaths(BrooklynServerConfig.getMgmtBaseDir( managementContext.getConfig() ), "catalog.xml");
+ if (new File(catalogUrl).exists()) {
+ populateInitialFromUri(catalog, "file:"+catalogUrl, PopulateMode.XML);
+ return;
+ }
+
+ // otherwise look for for classpath:/brooklyn/default.catalog.bom --
+ // there is one on the classpath which says to scan, and provides a few templates;
+ // if one is supplied by user in the conf/ dir that will override the item from the classpath
+ // (TBD - we might want to scan for all such bom's?)
+
+ catalogUrl = "classpath:/brooklyn/default.catalog.bom";
+ if (new ResourceUtils(this).doesUrlExist(catalogUrl)) {
+ populateInitialFromUri(catalog, catalogUrl, PopulateMode.YAML);
+ return;
+ }
+
+ log.info("No catalog found on classpath or specified; catalog will not be initialized.");
+ return;
+ }
+
+ private void populateInitialFromUri(BasicBrooklynCatalog catalog, String catalogUrl, PopulateMode mode) {
+ log.debug("Loading initial catalog from {}", catalogUrl);
+
+ Exception problem = null;
+ Object result = null;
+
+ String contents = null;
+ try {
+ contents = new ResourceUtils(this).getResourceAsString(catalogUrl);
+ } catch (Exception e) {
+ Exceptions.propagateIfFatal(e);
+ if (problem==null) problem = e;
+ }
+
+ if (contents!=null && (mode==PopulateMode.YAML || mode==PopulateMode.AUTODETECT)) {
+ // try YAML first
+ try {
+ catalog.reset(MutableList.<CatalogItem<?,?>>of());
+ result = catalog.addItems(contents);
+ } catch (Exception e) {
+ Exceptions.propagateIfFatal(e);
+ if (problem==null) problem = e;
+ }
+ }
+
+ if (result==null && contents!=null && (mode==PopulateMode.XML || mode==PopulateMode.AUTODETECT)) {
+ // then try XML
+ try {
+ populateInitialFromUriXml(catalog, catalogUrl, contents);
+ // clear YAML problem
+ problem = null;
+ } catch (Exception e) {
+ Exceptions.propagateIfFatal(e);
+ if (problem==null) problem = e;
+ }
+ }
+
+ if (result!=null) {
+ log.debug("Loaded initial catalog from {}: {}", catalogUrl, result);
+ }
+ if (problem!=null) {
+ log.warn("Error importing catalog from " + catalogUrl + ": " + problem, problem);
+ // TODO inform mgmt of error
+ }
+
+ }
+
+ // deprecated XML format
+ @SuppressWarnings("deprecation")
+ private void populateInitialFromUriXml(BasicBrooklynCatalog catalog, String catalogUrl, String contents) {
+ CatalogDto dto = CatalogDto.newDtoFromXmlContents(contents, catalogUrl);
+ if (dto!=null) {
+ catalog.reset(dto);
+ }
+ }
+
+ boolean hasRunAdditions = false;
+ protected void populateAdditions(BasicBrooklynCatalog catalog) {
+ if (Strings.isNonBlank(additionsUri)) {
+ if (disallowLocal) {
+ if (!hasRunAdditions) {
+ log.warn("CLI additions supplied but not supported in disallow-local mode; ignoring.");
+ }
+ return;
+ }
+ if (!hasRunAdditions) {
+ log.debug("Adding to catalog from CLI: "+additionsUri+" (force: "+force+")");
+ }
+ Iterable<? extends CatalogItem<?, ?>> items = catalog.addItems(
+ new ResourceUtils(this).getResourceAsString(additionsUri), force);
+
+ if (!hasRunAdditions)
+ log.debug("Added to catalog from CLI: "+items);
+ else
+ log.debug("Added to catalog from CLI: count "+Iterables.size(items));
+
+ hasRunAdditions = true;
+ }
+ }
+
+ protected void populateViaCallbacks(BasicBrooklynCatalog catalog) {
+ for (Function<CatalogInitialization, Void> callback: callbacks)
+ callback.apply(this);
+ }
+
+ private Object setFromCLMMutex = new Object();
+ private boolean setFromCatalogLoadMode = false;
+
+ /** @deprecated since introduced in 0.7.0, only for legacy compatibility with
+ * {@link CatalogLoadMode} {@link BrooklynServerConfig#CATALOG_LOAD_MODE},
+ * allowing control of catalog loading from a brooklyn property */
+ @Deprecated
+ public void applyCatalogLoadMode() {
--- End diff --
probably it could, but just in case the properties are set later i wanted to reflect the original logic. this method should be removed soon (as soon as 0.7.0 is out) so not too concerned.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by neykov <gi...@git.apache.org>.
Github user neykov commented on the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#issuecomment-106340163
Tests passing now, let's merge this beast :).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by neykov <gi...@git.apache.org>.
Github user neykov commented on a diff in the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#discussion_r30694106
--- Diff: core/src/main/java/brooklyn/catalog/internal/BasicBrooklynCatalog.java ---
@@ -750,7 +780,37 @@ private String setFromItemIfUnset(String oldValue, Map<?,?> item, String fieldAt
return oldValue;
}
+ private Collection<CatalogItemDtoAbstract<?, ?>> scanAnnotationsFromLocal(ManagementContext mgmt) {
+ CatalogDto dto = CatalogDto.newNamedInstance("Local Scanned Catalog", "All annotated Brooklyn entities detected in the classpath", "scanning-local-classpath");
+ return scanAnnotationsInternal(mgmt, new CatalogDo(dto));
+ }
+ private Collection<CatalogItemDtoAbstract<?, ?>> scanAnnotationsFromBundles(ManagementContext mgmt, Collection<CatalogBundle> libraries) {
+ String[] urls = null;
+ CatalogDto dto = CatalogDto.newNamedInstance("Bundles Scanned Catalog", "All annotated Brooklyn entities detected in the classpath", "scanning-bundles-classpath-"+libraries.hashCode());
+ urls = new String[libraries.size()];
+ int i=0;
+ for (CatalogBundle b: libraries)
+ urls[i++] = b.getUrl();
+
+ CatalogDo subCatalog = new CatalogDo(dto);
+ subCatalog.addToClasspath(urls);
--- End diff --
* Will it work if annotated classes depend on a bundle not explicitly listed (no OSGi dependency mechanism in play here)?
* Can't figure out how the bundles in `urls` can see the brooklyn classpath. The classloader created for scanning is limited to only this list.
* Even if the brooklyn classpath is visible how are conflicting versions handled?
* The libraries list can contain name:version pairs only, leaving it to OSGi to figure out the bundle. `getUrl` will return `null` in this case.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by neykov <gi...@git.apache.org>.
Github user neykov commented on a diff in the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#discussion_r30796613
--- Diff: core/src/main/java/brooklyn/entity/rebind/RebindIteration.java ---
@@ -341,38 +340,64 @@ protected void rebuildCatalog() {
}
}
}
+
+ // See notes in CatalogInitialization
- // Register catalogue items with the management context. Loads the bundles in the OSGi framework.
- CatalogLoadMode catalogLoadMode = managementContext.getConfig().getConfig(BrooklynServerConfig.CATALOG_LOAD_MODE);
+ Collection<CatalogItem<?, ?>> catalogItems = rebindContext.getCatalogItems();
+ CatalogInitialization catInit = ((ManagementContextInternal)managementContext).getCatalogInitialization();
+ catInit.applyCatalogLoadMode();
+ Collection<CatalogItem<?,?>> itemsForResettingCatalog = null;
+ boolean needsInitialCatalog;
if (rebindManager.persistCatalogItemsEnabled) {
- boolean shouldResetCatalog = catalogLoadMode == CatalogLoadMode.LOAD_PERSISTED_STATE
- || (!isEmpty && catalogLoadMode == CatalogLoadMode.LOAD_BROOKLYN_CATALOG_URL_IF_NO_PERSISTED_STATE);
- boolean shouldLoadDefaultCatalog = catalogLoadMode == CatalogLoadMode.LOAD_BROOKLYN_CATALOG_URL
- || (isEmpty && catalogLoadMode == CatalogLoadMode.LOAD_BROOKLYN_CATALOG_URL_IF_NO_PERSISTED_STATE);
- if (shouldResetCatalog) {
- // Reset catalog with previously persisted state
- logRebindingDebug("RebindManager resetting management context catalog to previously persisted state");
- managementContext.getCatalog().reset(rebindContext.getCatalogItems());
- } else if (shouldLoadDefaultCatalog) {
- // Load catalogue as normal
- // TODO in read-only mode, should do this less frequently than entities etc
- logRebindingDebug("RebindManager loading default catalog");
- ((BasicBrooklynCatalog) managementContext.getCatalog()).resetCatalogToContentsAtConfiguredUrl();
+ if (!catInit.hasRunOfficial() && catInit.isInitialResetRequested()) {
+ String message = "RebindManager resetting catalog on first run (catalog persistence enabled, but reset explicitly specified). ";
+ if (catalogItems.isEmpty()) {
+ message += "Catalog was empty anyway.";
+ } else {
+ message += "Deleting "+catalogItems.size()+" persisted catalog item"+Strings.s(catalogItems)+": "+catalogItems;
+ if (shouldLogRebinding()) {
+ LOG.info(message);
+ }
+ }
+ logRebindingDebug(message);
+
+ itemsForResettingCatalog = MutableList.<CatalogItem<?,?>>of();
--- End diff --
If there are rebind errors [above](https://github.com/ahgittin/incubator-brooklyn/blob/d6ae0511c35ef4070c8a1637bad1ef3b113bda81/core/src/main/java/brooklyn/entity/rebind/RebindIteration.java#L308-L342) we would get a failed startup even though the catalog will be reset?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by aledsage <gi...@git.apache.org>.
Github user aledsage commented on a diff in the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#discussion_r30504909
--- Diff: core/src/main/java/brooklyn/catalog/internal/CatalogInitialization.java ---
@@ -0,0 +1,377 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package brooklyn.catalog.internal;
+
+import java.io.File;
+import java.util.Collection;
+import java.util.List;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import brooklyn.catalog.CatalogItem;
+import brooklyn.config.BrooklynServerConfig;
+import brooklyn.management.ManagementContext;
+import brooklyn.management.ManagementContextInjectable;
+import brooklyn.management.internal.ManagementContextInternal;
+import brooklyn.util.ResourceUtils;
+import brooklyn.util.collections.MutableList;
+import brooklyn.util.exceptions.Exceptions;
+import brooklyn.util.exceptions.FatalRuntimeException;
+import brooklyn.util.exceptions.RuntimeInterruptedException;
+import brooklyn.util.flags.TypeCoercions;
+import brooklyn.util.guava.Maybe;
+import brooklyn.util.net.Urls;
+import brooklyn.util.text.Strings;
+
+import com.google.common.annotations.Beta;
+import com.google.common.base.Function;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.Iterables;
+
+@Beta
+public class CatalogInitialization implements ManagementContextInjectable {
+
+ /*
+
+ A1) if not persisting, go to B1
+ A2) if --catalog-reset, delete persisted catalog items
+ A3) read persisted catalog items (possibly deleted in A2), go to C1
+ A4) go to B1
--- End diff --
To make this clearer, should say that A3 only goes to C1 if there was persisted state. I presume that is just non-blank state, rather than it having to have catalog items.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by ahgittin <gi...@git.apache.org>.
Github user ahgittin commented on a diff in the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#discussion_r30515186
--- Diff: core/src/main/java/brooklyn/catalog/internal/CatalogDo.java ---
@@ -254,8 +270,10 @@ CatalogDo addCatalog(CatalogDto child) {
public synchronized void addToClasspath(String ...urls) {
if (dto.classpath == null)
dto.classpath = new CatalogClasspathDto();
- for (String url: urls)
- dto.classpath.addEntry(url);
+ for (String url: urls) {
+ if (url!=null)
--- End diff --
the only non-test use is when we are registering declared osgi bundles. a pre-installed bundle might have a null url. we could filter this before calling but it is more work.
agree it's worth warning about nulls in situations where programming errors could crop up, but it seems unlikely here and the investment not worth it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by neykov <gi...@git.apache.org>.
Github user neykov commented on a diff in the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#discussion_r30711175
--- Diff: core/src/main/java/brooklyn/catalog/internal/BasicBrooklynCatalog.java ---
@@ -750,7 +780,37 @@ private String setFromItemIfUnset(String oldValue, Map<?,?> item, String fieldAt
return oldValue;
}
+ private Collection<CatalogItemDtoAbstract<?, ?>> scanAnnotationsFromLocal(ManagementContext mgmt) {
+ CatalogDto dto = CatalogDto.newNamedInstance("Local Scanned Catalog", "All annotated Brooklyn entities detected in the classpath", "scanning-local-classpath");
+ return scanAnnotationsInternal(mgmt, new CatalogDo(dto));
+ }
+ private Collection<CatalogItemDtoAbstract<?, ?>> scanAnnotationsFromBundles(ManagementContext mgmt, Collection<CatalogBundle> libraries) {
+ String[] urls = null;
+ CatalogDto dto = CatalogDto.newNamedInstance("Bundles Scanned Catalog", "All annotated Brooklyn entities detected in the classpath", "scanning-bundles-classpath-"+libraries.hashCode());
+ urls = new String[libraries.size()];
+ int i=0;
+ for (CatalogBundle b: libraries)
+ urls[i++] = b.getUrl();
+
+ CatalogDo subCatalog = new CatalogDo(dto);
+ subCatalog.addToClasspath(urls);
--- End diff --
Made a quick test - tried scanning `brooklyn-test-osgi-entities.jar` (after annotating `SimpleEntity`). Fails with
```
2015-05-20 17:46:39,482 ERROR Error loading catalog item 'brooklyn.catalog.internal.CatalogItemDo[CatalogEntityItemDto[brooklyn.osgi.tests.SimpleEntity:0.0.0.SNAPSHOT/Tomcat Server]]': java.lang.IllegalStateException: Error creating class brooklyn.entity.proxying.EntitySpec brooklyn.osgi.tests.SimpleEntity: java.lang.IllegalStateException: Unable to load brooklyn.osgi.tests.SimpleEntity from []: Invalid class: brooklyn.osgi.tests.SimpleEntity: java.lang.ClassNotFoundException: brooklyn.osgi.tests.SimpleEntity
```
with the yaml
```
brooklyn.catalog:
- libraries:
- file:///.../core/src/test/resources/brooklyn/osgi/brooklyn-test-osgi-entities.jar
scanJavaAnnotations: true
```
Worth adding it to the PR. But no classpath:// protocol supported by the catalog CL, as used in other tests.
If there's no explicit parent class loader (as in our case) only the bootstrap class loader will be consulted which doesn't have access to the app classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by neykov <gi...@git.apache.org>.
Github user neykov commented on a diff in the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#discussion_r30712561
--- Diff: core/src/main/java/brooklyn/catalog/internal/BasicBrooklynCatalog.java ---
@@ -750,7 +780,37 @@ private String setFromItemIfUnset(String oldValue, Map<?,?> item, String fieldAt
return oldValue;
}
+ private Collection<CatalogItemDtoAbstract<?, ?>> scanAnnotationsFromLocal(ManagementContext mgmt) {
+ CatalogDto dto = CatalogDto.newNamedInstance("Local Scanned Catalog", "All annotated Brooklyn entities detected in the classpath", "scanning-local-classpath");
+ return scanAnnotationsInternal(mgmt, new CatalogDo(dto));
+ }
+ private Collection<CatalogItemDtoAbstract<?, ?>> scanAnnotationsFromBundles(ManagementContext mgmt, Collection<CatalogBundle> libraries) {
+ String[] urls = null;
+ CatalogDto dto = CatalogDto.newNamedInstance("Bundles Scanned Catalog", "All annotated Brooklyn entities detected in the classpath", "scanning-bundles-classpath-"+libraries.hashCode());
+ urls = new String[libraries.size()];
+ int i=0;
+ for (CatalogBundle b: libraries)
+ urls[i++] = b.getUrl();
+
+ CatalogDo subCatalog = new CatalogDo(dto);
+ subCatalog.addToClasspath(urls);
--- End diff --
> that collection won't be empty if either is supplied, and so it then won't scanAnnotationsFromLocal
It will go through `scanAnnotationsFromBundles`, but in `CatalogClasspathDo.load` the list will be filtered out of the non-url bundles, making it go through [this code branch](https://github.com/apache/incubator-brooklyn/blob/09a64755e2c16c25f076af6a32f93ee0ff8c2599/core/src/main/java/brooklyn/catalog/internal/CatalogClasspathDo.java#L137)
I see you've already changed the code, just wanted to clear up what I meant.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by ahgittin <gi...@git.apache.org>.
Github user ahgittin commented on a diff in the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#discussion_r30659227
--- Diff: core/src/main/java/brooklyn/entity/rebind/persister/BrooklynMementoPersisterToObjectStore.java ---
@@ -560,9 +563,18 @@ public void checkpoint(BrooklynMemento newMemento, PersistenceExceptionHandler e
public void delta(Delta delta, PersistenceExceptionHandler exceptionHandler) {
checkWritesAllowed();
+ while (!queuedDeltas.isEmpty()) {
+ Delta extraDelta = queuedDeltas.remove(0);
+ doDelta(extraDelta, exceptionHandler, false);
--- End diff --
good spot
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by neykov <gi...@git.apache.org>.
Github user neykov commented on a diff in the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#discussion_r30780956
--- Diff: core/src/main/java/brooklyn/entity/rebind/RebindIteration.java ---
@@ -341,38 +340,64 @@ protected void rebuildCatalog() {
}
}
}
+
+ // See notes in CatalogInitialization
- // Register catalogue items with the management context. Loads the bundles in the OSGi framework.
- CatalogLoadMode catalogLoadMode = managementContext.getConfig().getConfig(BrooklynServerConfig.CATALOG_LOAD_MODE);
+ Collection<CatalogItem<?, ?>> catalogItems = rebindContext.getCatalogItems();
+ CatalogInitialization catInit = ((ManagementContextInternal)managementContext).getCatalogInitialization();
+ catInit.applyCatalogLoadMode();
+ Collection<CatalogItem<?,?>> itemsForResettingCatalog = null;
+ boolean needsInitialCatalog;
if (rebindManager.persistCatalogItemsEnabled) {
- boolean shouldResetCatalog = catalogLoadMode == CatalogLoadMode.LOAD_PERSISTED_STATE
- || (!isEmpty && catalogLoadMode == CatalogLoadMode.LOAD_BROOKLYN_CATALOG_URL_IF_NO_PERSISTED_STATE);
- boolean shouldLoadDefaultCatalog = catalogLoadMode == CatalogLoadMode.LOAD_BROOKLYN_CATALOG_URL
- || (isEmpty && catalogLoadMode == CatalogLoadMode.LOAD_BROOKLYN_CATALOG_URL_IF_NO_PERSISTED_STATE);
- if (shouldResetCatalog) {
- // Reset catalog with previously persisted state
- logRebindingDebug("RebindManager resetting management context catalog to previously persisted state");
- managementContext.getCatalog().reset(rebindContext.getCatalogItems());
- } else if (shouldLoadDefaultCatalog) {
- // Load catalogue as normal
- // TODO in read-only mode, should do this less frequently than entities etc
- logRebindingDebug("RebindManager loading default catalog");
- ((BasicBrooklynCatalog) managementContext.getCatalog()).resetCatalogToContentsAtConfiguredUrl();
+ if (!catInit.hasRunOfficial() && catInit.isInitialResetRequested()) {
+ String message = "RebindManager resetting catalog on first run (catalog persistence enabled, but reset explicitly specified). ";
+ if (catalogItems.isEmpty()) {
+ message += "Catalog was empty anyway.";
+ } else {
+ message += "Deleting "+catalogItems.size()+" persisted catalog item"+Strings.s(catalogItems)+": "+catalogItems;
+ if (shouldLogRebinding()) {
+ LOG.info(message);
+ }
+ }
+ logRebindingDebug(message);
+
+ itemsForResettingCatalog = MutableList.<CatalogItem<?,?>>of();
+
+ PersisterDeltaImpl delta = new PersisterDeltaImpl();
+ delta.removedCatalogItemIds.addAll(mementoRawData.getCatalogItems().keySet());
+ getPersister().queueDelta(delta);
+
+ mementoRawData.clearCatalogItems();
--- End diff --
This looks like too much manual housekeeping / wrong level of abstraction.
Better let the persister know how to clear the catalog instead of introducing two lower-level APIs (queueDelta & clearCatalogItems). Moreover `clearCatalogItems` breaks the immutable nature of `mementoRawData`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by ahgittin <gi...@git.apache.org>.
Github user ahgittin commented on the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#issuecomment-103898802
unrelated test failure:
2015-05-20 00:01:48,202 INFO TESTNG FAILED: "Surefire test" - brooklyn.rest.jsgui.BrooklynJavascriptGuiLauncherTest.testJavascriptWithoutRest() finished in 9455 ms
java.lang.RuntimeException: unable to find a free port at or above 8080
@aledsage @neykov anything else?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by ahgittin <gi...@git.apache.org>.
Github user ahgittin commented on the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#issuecomment-105235854
Thanks for keeping me straight @neykov .
I'll add your test, and try to fix access to the brooklyn classpath:
```
brooklyn.catalog:
- libraries:
- file:///.../core/src/test/resources/brooklyn/osgi/brooklyn-test-osgi-entities.jar
scanJavaAnnotations: true
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by ahgittin <gi...@git.apache.org>.
Github user ahgittin commented on a diff in the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#discussion_r30701852
--- Diff: core/src/main/java/brooklyn/catalog/internal/BasicBrooklynCatalog.java ---
@@ -597,7 +612,22 @@ private void collectCatalogItems(String sourceYaml, Map<?,?> itemMetadata, List<
Collection<CatalogBundle> libraryBundles = CatalogItemDtoAbstract.parseLibraries(librariesCombined);
// TODO as this may take a while if downloading, the REST call should be async
+ // (this load is required for the scan below and I think also for yaml resolution)
CatalogUtils.installLibraries(mgmt, libraryBundlesNew);
+
+ Boolean scanJavaAnnotations = getFirstAs(itemMetadata, Boolean.class, "scanJavaAnnotations", "scan_java_annotations").orNull();
+ if (scanJavaAnnotations==null || !scanJavaAnnotations) {
+ // don't scan
+ } else {
+ // scan for annotations: if libraries here, scan them; if inherited libraries error; else scan classpath
+ if (!libraryBundlesNew.isEmpty()) {
+ result.addAll(scanAnnotationsFromBundles(mgmt, libraryBundlesNew));
+ } else if (libraryBundles.isEmpty()) {
+ result.addAll(scanAnnotationsFromLocal(mgmt));
+ } else {
+ throw new IllegalStateException("Cannot scan catalog node no local bundles, and with inherited bundles we will not scan the classpath");
+ }
--- End diff --
personally i'd rather keep the yaml tight. and do away with local classpath autoscan altogether.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by ahgittin <gi...@git.apache.org>.
Github user ahgittin commented on a diff in the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#discussion_r30513608
--- Diff: core/src/main/java/brooklyn/catalog/internal/BasicBrooklynCatalog.java ---
@@ -186,12 +192,7 @@ public CatalogDo getCatalog() {
return null;
}
- String versionedId = CatalogUtils.getVersionedId(symbolicName, fixedVersionId);
- CatalogItemDo<?, ?> item = null;
- //TODO should remove "manual additions" bucket; just have one map a la osgi
- if (manualAdditionsCatalog!=null) item = manualAdditionsCatalog.getIdCache().get(versionedId);
- if (item == null) item = catalog.getIdCache().get(versionedId);
- return item;
+ return catalog.getIdCache().get( CatalogUtils.getVersionedId(symbolicName, fixedVersionId) );
--- End diff --
we can radically simplify everything -- perhaps even just rewrite it -- when we abandon the XML catalog
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by neykov <gi...@git.apache.org>.
Github user neykov commented on a diff in the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#discussion_r30796400
--- Diff: core/src/main/java/brooklyn/entity/rebind/RebindIteration.java ---
@@ -341,38 +340,64 @@ protected void rebuildCatalog() {
}
}
}
+
+ // See notes in CatalogInitialization
- // Register catalogue items with the management context. Loads the bundles in the OSGi framework.
- CatalogLoadMode catalogLoadMode = managementContext.getConfig().getConfig(BrooklynServerConfig.CATALOG_LOAD_MODE);
+ Collection<CatalogItem<?, ?>> catalogItems = rebindContext.getCatalogItems();
+ CatalogInitialization catInit = ((ManagementContextInternal)managementContext).getCatalogInitialization();
+ catInit.applyCatalogLoadMode();
+ Collection<CatalogItem<?,?>> itemsForResettingCatalog = null;
+ boolean needsInitialCatalog;
if (rebindManager.persistCatalogItemsEnabled) {
- boolean shouldResetCatalog = catalogLoadMode == CatalogLoadMode.LOAD_PERSISTED_STATE
- || (!isEmpty && catalogLoadMode == CatalogLoadMode.LOAD_BROOKLYN_CATALOG_URL_IF_NO_PERSISTED_STATE);
- boolean shouldLoadDefaultCatalog = catalogLoadMode == CatalogLoadMode.LOAD_BROOKLYN_CATALOG_URL
- || (isEmpty && catalogLoadMode == CatalogLoadMode.LOAD_BROOKLYN_CATALOG_URL_IF_NO_PERSISTED_STATE);
- if (shouldResetCatalog) {
- // Reset catalog with previously persisted state
- logRebindingDebug("RebindManager resetting management context catalog to previously persisted state");
- managementContext.getCatalog().reset(rebindContext.getCatalogItems());
- } else if (shouldLoadDefaultCatalog) {
- // Load catalogue as normal
- // TODO in read-only mode, should do this less frequently than entities etc
- logRebindingDebug("RebindManager loading default catalog");
- ((BasicBrooklynCatalog) managementContext.getCatalog()).resetCatalogToContentsAtConfiguredUrl();
+ if (!catInit.hasRunOfficial() && catInit.isInitialResetRequested()) {
+ String message = "RebindManager resetting catalog on first run (catalog persistence enabled, but reset explicitly specified). ";
+ if (catalogItems.isEmpty()) {
+ message += "Catalog was empty anyway.";
+ } else {
+ message += "Deleting "+catalogItems.size()+" persisted catalog item"+Strings.s(catalogItems)+": "+catalogItems;
+ if (shouldLogRebinding()) {
+ LOG.info(message);
+ }
+ }
+ logRebindingDebug(message);
+
+ itemsForResettingCatalog = MutableList.<CatalogItem<?,?>>of();
+
+ PersisterDeltaImpl delta = new PersisterDeltaImpl();
+ delta.removedCatalogItemIds.addAll(mementoRawData.getCatalogItems().keySet());
+ getPersister().queueDelta(delta);
+
+ mementoRawData.clearCatalogItems();
--- End diff --
Reset rebindContext catalog items as well?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by neykov <gi...@git.apache.org>.
Github user neykov commented on a diff in the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#discussion_r30794125
--- Diff: core/src/main/java/brooklyn/entity/rebind/RebindIteration.java ---
@@ -341,38 +340,64 @@ protected void rebuildCatalog() {
}
}
}
+
+ // See notes in CatalogInitialization
- // Register catalogue items with the management context. Loads the bundles in the OSGi framework.
- CatalogLoadMode catalogLoadMode = managementContext.getConfig().getConfig(BrooklynServerConfig.CATALOG_LOAD_MODE);
+ Collection<CatalogItem<?, ?>> catalogItems = rebindContext.getCatalogItems();
+ CatalogInitialization catInit = ((ManagementContextInternal)managementContext).getCatalogInitialization();
+ catInit.applyCatalogLoadMode();
+ Collection<CatalogItem<?,?>> itemsForResettingCatalog = null;
+ boolean needsInitialCatalog;
if (rebindManager.persistCatalogItemsEnabled) {
- boolean shouldResetCatalog = catalogLoadMode == CatalogLoadMode.LOAD_PERSISTED_STATE
- || (!isEmpty && catalogLoadMode == CatalogLoadMode.LOAD_BROOKLYN_CATALOG_URL_IF_NO_PERSISTED_STATE);
- boolean shouldLoadDefaultCatalog = catalogLoadMode == CatalogLoadMode.LOAD_BROOKLYN_CATALOG_URL
- || (isEmpty && catalogLoadMode == CatalogLoadMode.LOAD_BROOKLYN_CATALOG_URL_IF_NO_PERSISTED_STATE);
- if (shouldResetCatalog) {
- // Reset catalog with previously persisted state
- logRebindingDebug("RebindManager resetting management context catalog to previously persisted state");
- managementContext.getCatalog().reset(rebindContext.getCatalogItems());
- } else if (shouldLoadDefaultCatalog) {
- // Load catalogue as normal
- // TODO in read-only mode, should do this less frequently than entities etc
- logRebindingDebug("RebindManager loading default catalog");
- ((BasicBrooklynCatalog) managementContext.getCatalog()).resetCatalogToContentsAtConfiguredUrl();
+ if (!catInit.hasRunOfficial() && catInit.isInitialResetRequested()) {
--- End diff --
How does catalog initialization work with the different HA states? It doesn't matter because we get a brand new ManagementContext with each rebind, which will do the full catalog init procedure, is that right?
If so should we use the rebind logger logic during catalog initialization as well to prevent excessive logging?
What happens if we go master -> standby -> master, either due to persistence store issues or force manually through rest? Will the catalog be re-initialized on each promotion?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by aledsage <gi...@git.apache.org>.
Github user aledsage commented on a diff in the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#discussion_r30501884
--- Diff: core/src/main/java/brooklyn/catalog/internal/BasicBrooklynCatalog.java ---
@@ -1008,8 +1072,19 @@ private DeploymentPlan makePlanFromYaml(String yaml) {
return itemDto;
}
- private void checkItemNotExists(CatalogItem<?,?> itemDto, boolean forceUpdate) {
- if (!forceUpdate && getCatalogItemDo(itemDto.getSymbolicName(), itemDto.getVersion()) != null) {
+ /** returns item DTO if item is an allowed duplicate, null if it should be added, or false if the item is an allowed duplicate,
--- End diff --
Comment seems wrong - we don't return false.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by aledsage <gi...@git.apache.org>.
Github user aledsage commented on a diff in the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#discussion_r30501820
--- Diff: core/src/main/java/brooklyn/catalog/internal/BasicBrooklynCatalog.java ---
@@ -986,7 +1045,12 @@ private DeploymentPlan makePlanFromYaml(String yaml) {
}
private CatalogItem<?,?> addItemDto(CatalogItemDtoAbstract<?, ?> itemDto, boolean forceUpdate) {
- checkItemNotExists(itemDto, forceUpdate);
+ CatalogItem<?, ?> existingDto = checkItemIsDuplicateOrDisallowed(itemDto, true, forceUpdate);
+ if (existingDto!=null) {
+ // it's a duplicate, and not forced, just return it
+ log.trace("Using existing duplicate for catalog item {}", itemDto.getId());
--- End diff --
log.debug instead of trace: we're treating the caller's request as a no-op, so should log that we've done nothing on the caller's request. Or does this get called a lot by some automated mechanism?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by ahgittin <gi...@git.apache.org>.
Github user ahgittin commented on a diff in the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#discussion_r30704536
--- Diff: core/src/main/java/brooklyn/catalog/internal/BasicBrooklynCatalog.java ---
@@ -750,7 +780,37 @@ private String setFromItemIfUnset(String oldValue, Map<?,?> item, String fieldAt
return oldValue;
}
+ private Collection<CatalogItemDtoAbstract<?, ?>> scanAnnotationsFromLocal(ManagementContext mgmt) {
+ CatalogDto dto = CatalogDto.newNamedInstance("Local Scanned Catalog", "All annotated Brooklyn entities detected in the classpath", "scanning-local-classpath");
+ return scanAnnotationsInternal(mgmt, new CatalogDo(dto));
+ }
+ private Collection<CatalogItemDtoAbstract<?, ?>> scanAnnotationsFromBundles(ManagementContext mgmt, Collection<CatalogBundle> libraries) {
+ String[] urls = null;
+ CatalogDto dto = CatalogDto.newNamedInstance("Bundles Scanned Catalog", "All annotated Brooklyn entities detected in the classpath", "scanning-bundles-classpath-"+libraries.hashCode());
+ urls = new String[libraries.size()];
+ int i=0;
+ for (CatalogBundle b: libraries)
+ urls[i++] = b.getUrl();
+
+ CatalogDo subCatalog = new CatalogDo(dto);
+ subCatalog.addToClasspath(urls);
--- End diff --
i think the what-to-scan logic is correct. we do
Collection<CatalogBundle> libraryBundlesNew = CatalogItemDtoAbstract.parseLibraries(librariesNew);
which accepts both `name:version` pairs and URL's; that collection won't be empty if either is supplied, and so it then won't `scanAnnotationsFromLocal`.
the brooklyn classpath will always be available because it is the bootstrap class loader, won't it? it just won't be scanned when any bundles are defined.
i've added a log warning when only `name:version` is supplied, and comments in several places for the other issues (all of which i agree), including the docs for the `scanJavaAnnotations [experimental]`. i think this is the right balance until we know whether we want to support scanning of OSGi or instead have some other tool to extract a `bom` (or even a TOSCA definition) which we expect included in the bundle.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by ahgittin <gi...@git.apache.org>.
Github user ahgittin commented on a diff in the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#discussion_r30968278
--- Diff: core/src/main/java/brooklyn/entity/rebind/RebindIteration.java ---
@@ -341,38 +340,64 @@ protected void rebuildCatalog() {
}
}
}
+
+ // See notes in CatalogInitialization
- // Register catalogue items with the management context. Loads the bundles in the OSGi framework.
- CatalogLoadMode catalogLoadMode = managementContext.getConfig().getConfig(BrooklynServerConfig.CATALOG_LOAD_MODE);
+ Collection<CatalogItem<?, ?>> catalogItems = rebindContext.getCatalogItems();
+ CatalogInitialization catInit = ((ManagementContextInternal)managementContext).getCatalogInitialization();
+ catInit.applyCatalogLoadMode();
+ Collection<CatalogItem<?,?>> itemsForResettingCatalog = null;
+ boolean needsInitialCatalog;
if (rebindManager.persistCatalogItemsEnabled) {
- boolean shouldResetCatalog = catalogLoadMode == CatalogLoadMode.LOAD_PERSISTED_STATE
- || (!isEmpty && catalogLoadMode == CatalogLoadMode.LOAD_BROOKLYN_CATALOG_URL_IF_NO_PERSISTED_STATE);
- boolean shouldLoadDefaultCatalog = catalogLoadMode == CatalogLoadMode.LOAD_BROOKLYN_CATALOG_URL
- || (isEmpty && catalogLoadMode == CatalogLoadMode.LOAD_BROOKLYN_CATALOG_URL_IF_NO_PERSISTED_STATE);
- if (shouldResetCatalog) {
- // Reset catalog with previously persisted state
- logRebindingDebug("RebindManager resetting management context catalog to previously persisted state");
- managementContext.getCatalog().reset(rebindContext.getCatalogItems());
- } else if (shouldLoadDefaultCatalog) {
- // Load catalogue as normal
- // TODO in read-only mode, should do this less frequently than entities etc
- logRebindingDebug("RebindManager loading default catalog");
- ((BasicBrooklynCatalog) managementContext.getCatalog()).resetCatalogToContentsAtConfiguredUrl();
+ if (!catInit.hasRunOfficial() && catInit.isInitialResetRequested()) {
--- End diff --
will change the ambiguous `official` and check whether it has been in master state yet
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by neykov <gi...@git.apache.org>.
Github user neykov commented on the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#issuecomment-99854424
@ahgittin Could you rebase to remove changes from #614
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by neykov <gi...@git.apache.org>.
Github user neykov commented on a diff in the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#discussion_r30881057
--- Diff: usage/cli/src/main/java/brooklyn/cli/lister/ClassFinder.java ---
@@ -126,6 +129,8 @@ public boolean apply(Class<?> input) {
log.info("Cannot read "+file+"; not a file or directory");
}
}
+ } else {
+ result.add(new URL("file://"+tidiedFile));
--- End diff --
Use tidiedFile.toUri.toUrl
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by aledsage <gi...@git.apache.org>.
Github user aledsage commented on the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#issuecomment-103453887
Have finished going through this huge pull request! Lots of excellent stuff here; really like what it does.
Vast majority of my comments are quite low level.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by aledsage <gi...@git.apache.org>.
Github user aledsage commented on a diff in the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#discussion_r30502327
--- Diff: core/src/main/java/brooklyn/catalog/internal/CatalogDo.java ---
@@ -254,8 +270,10 @@ CatalogDo addCatalog(CatalogDto child) {
public synchronized void addToClasspath(String ...urls) {
if (dto.classpath == null)
dto.classpath = new CatalogClasspathDto();
- for (String url: urls)
- dto.classpath.addEntry(url);
+ for (String url: urls) {
+ if (url!=null)
--- End diff --
Let's hope we never hit this with `url == null`. I'd expect (at least some times) that will be caused by a programming error or missing resource, which we should tell the caller. This code will now ignore (without any indication to the user).
This relates to a more general discussion of how we should handle nulls. I think we should avoid nulls where feasible and accept them where required (e.g. getAttribute() and getConfig() can always return null). But I don't feel strongly enough to say more at this time!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by aledsage <gi...@git.apache.org>.
Github user aledsage commented on a diff in the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#discussion_r30585863
--- Diff: usage/cli/src/main/java/brooklyn/cli/ItemLister.java ---
@@ -66,6 +68,9 @@
public class ItemLister {
--- End diff --
Ah, yes I did know about it - it's for us generating documentation (config/attributes/effectors) about all our entities, policies, etc.
Anyway, the comment about docs doesn't need addressed in this PR.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by neykov <gi...@git.apache.org>.
Github user neykov commented on a diff in the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#discussion_r30885252
--- Diff: utils/common/src/main/java/brooklyn/util/javalang/Threads.java ---
@@ -44,7 +46,15 @@ public static boolean removeShutdownHook(Thread hook) {
return Runtime.getRuntime().removeShutdownHook(hook);
} catch (IllegalStateException e) {
// probably shutdown in progress
- log.debug("cannot remove shutdown hook "+hook+": "+e);
+ String text = Exceptions.collapseText(e);
+ if (text.contains("Shutdown in progress")) {
--- End diff --
This is a good candidate for a typed exception. Always wondered why the general dislike to typed exceptions in Brooklyn?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by neykov <gi...@git.apache.org>.
Github user neykov commented on a diff in the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#discussion_r30692077
--- Diff: core/src/main/java/brooklyn/catalog/internal/BasicBrooklynCatalog.java ---
@@ -597,7 +612,22 @@ private void collectCatalogItems(String sourceYaml, Map<?,?> itemMetadata, List<
Collection<CatalogBundle> libraryBundles = CatalogItemDtoAbstract.parseLibraries(librariesCombined);
// TODO as this may take a while if downloading, the REST call should be async
+ // (this load is required for the scan below and I think also for yaml resolution)
CatalogUtils.installLibraries(mgmt, libraryBundlesNew);
+
+ Boolean scanJavaAnnotations = getFirstAs(itemMetadata, Boolean.class, "scanJavaAnnotations", "scan_java_annotations").orNull();
+ if (scanJavaAnnotations==null || !scanJavaAnnotations) {
+ // don't scan
+ } else {
+ // scan for annotations: if libraries here, scan them; if inherited libraries error; else scan classpath
+ if (!libraryBundlesNew.isEmpty()) {
+ result.addAll(scanAnnotationsFromBundles(mgmt, libraryBundlesNew));
+ } else if (libraryBundles.isEmpty()) {
+ result.addAll(scanAnnotationsFromLocal(mgmt));
+ } else {
+ throw new IllegalStateException("Cannot scan catalog node no local bundles, and with inherited bundles we will not scan the classpath");
+ }
--- End diff --
Might be better to have two scan keywords - one for local bundles, the other for brooklyn classpath. This will avoid the context dependency, make intent explicit, easier for users to understand.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by ahgittin <gi...@git.apache.org>.
Github user ahgittin commented on a diff in the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#discussion_r30514140
--- Diff: core/src/main/java/brooklyn/catalog/internal/BasicBrooklynCatalog.java ---
@@ -750,7 +780,36 @@ private String setFromItemIfUnset(String oldValue, Map<?,?> item, String fieldAt
return oldValue;
}
-
+ /** scans the given libraries for annotated items, or if null scans the local classpath */
--- End diff --
for non-null is does *not* scan the local classpath. i've refactored to make 3 methods -- easier to read for sure, but at a cost of more private methods in the class.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by neykov <gi...@git.apache.org>.
Github user neykov commented on the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#issuecomment-105213647
I don't see changes related to `scanJavaAnnotations` and bundles. Is this working now?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by ahgittin <gi...@git.apache.org>.
Github user ahgittin commented on a diff in the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#discussion_r30659101
--- Diff: usage/cli/src/main/java/brooklyn/cli/Main.java ---
@@ -536,9 +570,10 @@ protected BrooklynLauncher createLauncher() {
BrooklynLauncher launcher;
launcher = BrooklynLauncher.newInstance();
launcher.localBrooklynPropertiesFile(localBrooklynProperties)
- .ignorePersistenceErrors(ignorePersistenceErrors)
- .ignoreWebErrors(ignoreWebErrors)
- .ignoreAppErrors(ignoreAppErrors)
+ .ignorePersistenceErrors(!startupFailOnPersistenceErrors)
--- End diff --
+1 -- let's wait and make sure these names (and logic) is good though
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by aledsage <gi...@git.apache.org>.
Github user aledsage commented on a diff in the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#discussion_r30583391
--- Diff: usage/launcher/src/main/java/brooklyn/launcher/BrooklynLauncher.java ---
@@ -682,6 +676,8 @@ private void initManagementContext() {
brooklynProperties.addFromMap(brooklynAdditionalProperties);
}
+ ((ManagementContextInternal)managementContext).setCatalogInitialization(catalogInitialization);
--- End diff --
guard call, to only do if `catalogInitialization` is non-null.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by ahgittin <gi...@git.apache.org>.
Github user ahgittin commented on a diff in the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#discussion_r30512982
--- Diff: core/src/main/java/brooklyn/catalog/internal/CatalogInitialization.java ---
@@ -0,0 +1,377 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package brooklyn.catalog.internal;
+
+import java.io.File;
+import java.util.Collection;
+import java.util.List;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import brooklyn.catalog.CatalogItem;
+import brooklyn.config.BrooklynServerConfig;
+import brooklyn.management.ManagementContext;
+import brooklyn.management.ManagementContextInjectable;
+import brooklyn.management.internal.ManagementContextInternal;
+import brooklyn.util.ResourceUtils;
+import brooklyn.util.collections.MutableList;
+import brooklyn.util.exceptions.Exceptions;
+import brooklyn.util.exceptions.FatalRuntimeException;
+import brooklyn.util.exceptions.RuntimeInterruptedException;
+import brooklyn.util.flags.TypeCoercions;
+import brooklyn.util.guava.Maybe;
+import brooklyn.util.net.Urls;
+import brooklyn.util.text.Strings;
+
+import com.google.common.annotations.Beta;
+import com.google.common.base.Function;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.Iterables;
+
+@Beta
+public class CatalogInitialization implements ManagementContextInjectable {
--- End diff --
i'll stick the `isPopulating` boolean in the mutex. i think it was fine before but it will be clearer with it inside. i'll also add comments to `mutex`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by robertgmoss <gi...@git.apache.org>.
Github user robertgmoss commented on a diff in the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#discussion_r30497792
--- Diff: core/src/main/java/brooklyn/catalog/internal/CatalogInitialization.java ---
@@ -0,0 +1,377 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package brooklyn.catalog.internal;
+
+import java.io.File;
+import java.util.Collection;
+import java.util.List;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import brooklyn.catalog.CatalogItem;
+import brooklyn.config.BrooklynServerConfig;
+import brooklyn.management.ManagementContext;
+import brooklyn.management.ManagementContextInjectable;
+import brooklyn.management.internal.ManagementContextInternal;
+import brooklyn.util.ResourceUtils;
+import brooklyn.util.collections.MutableList;
+import brooklyn.util.exceptions.Exceptions;
+import brooklyn.util.exceptions.FatalRuntimeException;
+import brooklyn.util.exceptions.RuntimeInterruptedException;
+import brooklyn.util.flags.TypeCoercions;
+import brooklyn.util.guava.Maybe;
+import brooklyn.util.net.Urls;
+import brooklyn.util.text.Strings;
+
+import com.google.common.annotations.Beta;
+import com.google.common.base.Function;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.Iterables;
+
+@Beta
+public class CatalogInitialization implements ManagementContextInjectable {
--- End diff --
It's the boolean variables that are concerning me. Sometimes they are updated/read under the lock, sometimes they are not.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by ahgittin <gi...@git.apache.org>.
Github user ahgittin commented on a diff in the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#discussion_r30968725
--- Diff: utils/common/src/main/java/brooklyn/util/javalang/Threads.java ---
@@ -44,7 +46,15 @@ public static boolean removeShutdownHook(Thread hook) {
return Runtime.getRuntime().removeShutdownHook(hook);
} catch (IllegalStateException e) {
// probably shutdown in progress
- log.debug("cannot remove shutdown hook "+hook+": "+e);
+ String text = Exceptions.collapseText(e);
+ if (text.contains("Shutdown in progress")) {
--- End diff --
i'd prefer a typed exception here, but the exception is thrown by java, not us, if you try to add a shutdown hook while a shutdown is in progress; this is just trying to clean up the log messages which are shown at the tail of the log on a shutdown
nothing against typed exceptions where useful but in general paths which might want typed exception handling i prefer the error-listener / callback pattern, avoiding throwing altogether -- for instance `Maybe.absentThrowing` and `ReferenceWithError`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by aledsage <gi...@git.apache.org>.
Github user aledsage commented on a diff in the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#discussion_r30543880
--- Diff: core/src/main/java/brooklyn/catalog/internal/CatalogInitialization.java ---
@@ -0,0 +1,377 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package brooklyn.catalog.internal;
+
+import java.io.File;
+import java.util.Collection;
+import java.util.List;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import brooklyn.catalog.CatalogItem;
+import brooklyn.config.BrooklynServerConfig;
+import brooklyn.management.ManagementContext;
+import brooklyn.management.ManagementContextInjectable;
+import brooklyn.management.internal.ManagementContextInternal;
+import brooklyn.util.ResourceUtils;
+import brooklyn.util.collections.MutableList;
+import brooklyn.util.exceptions.Exceptions;
+import brooklyn.util.exceptions.FatalRuntimeException;
+import brooklyn.util.exceptions.RuntimeInterruptedException;
+import brooklyn.util.flags.TypeCoercions;
+import brooklyn.util.guava.Maybe;
+import brooklyn.util.net.Urls;
+import brooklyn.util.text.Strings;
+
+import com.google.common.annotations.Beta;
+import com.google.common.base.Function;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.Iterables;
+
+@Beta
+public class CatalogInitialization implements ManagementContextInjectable {
+
+ /*
+
+ A1) if not persisting, go to B1
+ A2) if --catalog-reset, delete persisted catalog items
+ A3) read persisted catalog items (possibly deleted in A2), go to C1
+ A4) go to B1
+
+ B1) look for --catalog-initial, if so read it, then go to C1
+ B2) look for BrooklynServerConfig.BROOKLYN_CATALOG_URL, if so, read it, supporting YAML or XML (warning if XML), then go to C1
+ B3) look for ~/.brooklyn/catalog.bom, if exists, read it then go to C1
+ B4) look for ~/.brooklyn/brooklyn.xml, if exists, warn, read it then go to C1
+ B5) read all classpath://brooklyn/default.catalog.bom items, if they exist (and for now they will)
+ B6) go to C1
+
+ C1) if --catalog-add, read and add those items
+
+ D1) if persisting, read the rest of the persisted items (entities etc)
+
+ */
+
+ private static final Logger log = LoggerFactory.getLogger(CatalogInitialization.class);
+
+ String initialUri;
+ boolean reset;
+ String additionsUri;
+ boolean force;
+
+ boolean disallowLocal = false;
+ List<Function<CatalogInitialization, Void>> callbacks = MutableList.of();
+ boolean hasRunBestEffort = false, hasRunOfficial = false, isPopulating = false;
+
+ ManagementContext managementContext;
+ boolean isStartingUp = false;
+ boolean failOnStartupErrors = false;
+
+ Object mutex = new Object();
+
+ public CatalogInitialization(String initialUri, boolean reset, String additionUri, boolean force) {
--- End diff --
Worth javadoc to say what `initialUri` and `additionUri` are - not clear without digging through the code.
Rename to `additionsUri` to match the field.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by ahgittin <gi...@git.apache.org>.
Github user ahgittin commented on a diff in the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#discussion_r30510751
--- Diff: api/src/main/java/brooklyn/mementos/BrooklynMementoPersister.java ---
@@ -97,10 +97,13 @@
/** @deprecated since 0.7.0, use {@link #checkpoint(BrooklynMementoRawData, PersistenceExceptionHandler)}
* and javadoc on implementations of that */ @Deprecated // pretty sure this is not used outwith deprecated code
void checkpoint(BrooklynMemento memento, PersistenceExceptionHandler exceptionHandler);
-
- void checkpoint(BrooklynMementoRawData newMemento, PersistenceExceptionHandler exceptionHandler);
+ /** applies a full checkpoint (write) of all state */
+ void checkpoint(BrooklynMementoRawData newMemento, PersistenceExceptionHandler exceptionHandler);
+ /** applies a partial write of state delta */
void delta(Delta delta, PersistenceExceptionHandler exceptionHandler);
+ /** inserts an additional delta to be written on the next delta request */
+ void queueDelta(Delta delta);
--- End diff --
+1 `@Beta`. I couldn't see a way to add this batch-ahead-of-time functionality to the `PDCL`. Agree it's now suboptimal having two batches.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by ahgittin <gi...@git.apache.org>.
Github user ahgittin commented on the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#issuecomment-99796925
spurious failure - load balancing soak test
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by aledsage <gi...@git.apache.org>.
Github user aledsage commented on a diff in the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#discussion_r30578042
--- Diff: core/src/main/java/brooklyn/entity/rebind/RebindIteration.java ---
@@ -341,38 +340,67 @@ protected void rebuildCatalog() {
}
}
}
+
+ // See notes in CatalogInitialization
- // Register catalogue items with the management context. Loads the bundles in the OSGi framework.
- CatalogLoadMode catalogLoadMode = managementContext.getConfig().getConfig(BrooklynServerConfig.CATALOG_LOAD_MODE);
+ Collection<CatalogItem<?, ?>> catalogItems = rebindContext.getCatalogItems();
+ CatalogInitialization catInit = ((ManagementContextInternal)managementContext).getCatalogInitialization();
+ catInit.injectManagementContext(managementContext);
--- End diff --
Why would we have to inject the management context here, when we've obtained the instance by calling `managementContext.getCatalogInitialization()`? That should return an instance which already has the management context.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by ahgittin <gi...@git.apache.org>.
GitHub user ahgittin reopened a pull request:
https://github.com/apache/incubator-brooklyn/pull/617
CLI commands for manipulating catalog, and cleaner catalog loading
approx as discussed on mailing list, except annotations now also supported (easier to maintain compatibility, in tests)
for review only at this point; i want to do more testing
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ahgittin/incubator-brooklyn catalog-cli
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-brooklyn/pull/617.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #617
----
commit 20810ac9e275eadbf5c00c8b2ba07534841e56f5
Author: Alex Heneveld <al...@cloudsoftcorp.com>
Date: 2015-04-23T06:46:32Z
adjust cli start sequence so web/rest is avail always, with isUp check on rest api
first step towards being able to interactively track the startup sequence and notify on errors
commit ddce439f17143e838291c8f60ab6b69610ed9249
Author: Alex Heneveld <al...@cloudsoftcorp.com>
Date: 2015-04-23T10:52:53Z
add ability to collect startup errors and check healthy through rest api
changes defaults to be to continue on error, and extends (and optimizes) the BrooklynLauncherTest which checks this
commit e629602b41613f1068c9fa4b319e99dafc5eeeeb
Author: Alex Heneveld <al...@cloudsoftcorp.com>
Date: 2015-04-23T15:31:30Z
javascript checks up on start, shows caution
popup now appears while server is starting, or if there are errors, and comes back if server changes to standby.
changed to plug in *before* the routes are evaluated, so we don't load the wrong data.
expands rest api to return several pieces of information to simplify gui checks.
simplifies code in rest filters to block requests not compatible with server state, including checking for upness there.
commit 65d7ee3c15afda7c5ed4507889fb632f1e2d5c8f
Author: Alex Heneveld <al...@cloudsoftcorp.com>
Date: 2015-04-27T11:53:09Z
actually force catalog load by default, and report errors better
and remove two incompatible items from the catalog, with comments in them - FollowTheSunPolicy and LoadBalancingPolicy
commit 7d0f1a0efeffd0e16bdca35ec73f0d7ebbd956ed
Author: Alex Heneveld <al...@cloudsoftcorp.com>
Date: 2015-04-27T13:30:26Z
show caution dialog if shutting down or if server is unresponsive
and include shutting down message in rest api.
tested that i can stop and restart the server and it nicely cycles through sequence of:
"shutting down", "server unreachable", then "starting up", then restores page.
commit 7556c582ae253a655eb76abba92b75ac811a77b3
Author: Alex Heneveld <al...@cloudsoftcorp.com>
Date: 2015-05-06T09:48:16Z
address code review for startup info
JS masterUri fix, HA check cleanups, removing unneeded classes, and catch catalog error
commit 8bc70bc005cd3f7bc11e36536d31e0166c953603
Author: Alex Heneveld <al...@cloudsoftcorp.com>
Date: 2015-04-27T13:38:05Z
introduce CatalogInitialization to cleanly init catalog at the right times
invoked during persistence cycles, and at startup, holding the new CLI catalog options.
this does not yet properly initialize things for real-world use, and tests fixed in next commit.
commit 0701b7df68a1c8291cf157fc9fa7c05fa9bac0f8
Author: Alex Heneveld <al...@cloudsoftcorp.com>
Date: 2015-04-29T12:27:09Z
yaml catalog supports scanning, experimental, and default
this restores catalog scanning as the default, based on brooklyn/default.catalog.bom in the cli project;
there are some limitations on what can be scanned, described in the doc.
some of the tests configure other catalogs (with core's brooklyn/empty.catalog.bom used in many)
commit 4ac8385798adc9e9f11a7e4cd80d155d0dd2f361
Author: Alex Heneveld <al...@cloudsoftcorp.com>
Date: 2015-04-29T12:32:34Z
item lister resources put in an appropriate subdir; other related tidies
code was very ad hoc, but also polluting the root of the all jar; now resources at least are in a clean subdir
commit 23c31e0e97a5b5980e53e036344efb7773669fe9
Author: Alex Heneveld <al...@cloudsoftcorp.com>
Date: 2015-04-30T08:13:42Z
move riak.png to resources file so it ends up in the build
commit c778308d7362fa6ae719afb1aa6d0c7a7922b68c
Author: Alex Heneveld <al...@cloudsoftcorp.com>
Date: 2015-04-30T10:07:00Z
tidy CLI options, renaming startup-ignore-error flags; and other tidy
breaks backwards compatibility in CLI: previously we had `ignoreXxxOnStartup` fields, but now many of these default to true, and airlift offers no way to make them false, so they are called `--startupFailOn...` or `--startupContinueOn...`.
changes the nascent CatalogInitialization so that callbacks get it, and can query settings such as whether to throw on errors.
commit 718d5e208aef5c282e52a8638031b608233e8f17
Author: Alex Heneveld <al...@cloudsoftcorp.com>
Date: 2015-05-06T15:37:20Z
allow access to catalog classloader without populating catalog
required changes to how catalog is initialized, but i think it's a bit cleaner now (overwriting other changes in this PR).
(lots of files touched, unfortunately, but not a lot different here.)
commit 1f5e82d8f439572ab0c6ae52e07e010a61b8e483
Author: Alex Heneveld <al...@cloudsoftcorp.com>
Date: 2015-05-06T16:49:56Z
delete catalog items from persistence store on reset
commit 66d9bbc191877461f516698b88817d388c025d80
Author: Alex Heneveld <al...@cloudsoftcorp.com>
Date: 2015-05-06T22:02:03Z
allow "addition" of a catalog item which already exists if it's exactly the same
adds equals and hashCode to CatlogItemDto and CatalogBundleDto
commit 302dc98a5468ed29352fbeb8d1c325435095909c
Author: Alex Heneveld <al...@cloudsoftcorp.com>
Date: 2015-05-06T14:00:39Z
install a default.catalog.bom, and update docs
and tweak poms to exclude license since these are config files the user is meant to edit;
remove the old catalog.xml, including mentions in the docs
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by neykov <gi...@git.apache.org>.
Github user neykov commented on the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#issuecomment-104616332
Finished review. It's a great set of changes, making the yaml catalog a usable and useful feature. Really happy to see Brooklyn get to this point.
Two points stand out that need addressing:
* `scanJavaAnnotations` doesn't work with bundle items
* catalog initialization and HA states interplay
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by ahgittin <gi...@git.apache.org>.
Github user ahgittin commented on the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#issuecomment-105227109
@neykov `scanJavaAnnotations` addressed in https://github.com/ahgittin/incubator-brooklyn/commit/9cc7293eb45efd045edd81793db31fa8b6f64110 (mainly with comments and docs however as not sure how far we want to go supporting load-time scan for this, as opposed a dev/compile-time manifest -- it is still an experimental feature)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by aledsage <gi...@git.apache.org>.
Github user aledsage commented on a diff in the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#discussion_r30577981
--- Diff: core/src/main/java/brooklyn/management/internal/AbstractManagementContext.java ---
@@ -443,6 +441,22 @@ public void setManagementNodeUri(URI uri) {
return uri;
}
+ @Override
+ public CatalogInitialization getCatalogInitialization() {
+ if (catalogInitialization!=null) return catalogInitialization;
--- End diff --
Double-checked locking is broken in Java, unless you declare the variable volatile (http://www.cs.umd.edu/~pugh/java/memoryModel/DoubleCheckedLocking.html).
This code is not performance sensitive, so just delete this line to keep the code simpler.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by ahgittin <gi...@git.apache.org>.
Github user ahgittin commented on the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#issuecomment-103566754
Thanks @aledsage -- superb review. I've got most of them done, I should get the rest in tonight.
Sorry for the large volume of foolishness. :) It went through a few rewrites along the way.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by aledsage <gi...@git.apache.org>.
Github user aledsage commented on a diff in the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#discussion_r30583757
--- Diff: usage/launcher/src/main/java/brooklyn/launcher/BrooklynLauncher.java ---
@@ -682,6 +676,8 @@ private void initManagementContext() {
brooklynProperties.addFromMap(brooklynAdditionalProperties);
}
+ ((ManagementContextInternal)managementContext).setCatalogInitialization(catalogInitialization);
--- End diff --
Overall I'm ok with this, but I do have a gut feeling that I don't like us injecting the `catalogInitialization`.
It feels like we want to inject config to control the catalog initialization. That could be used to configure the actual catalogInitialization object, which would be purely internal.
Injecting a full-blown `CatalogInitialization` object is obviously more flexible, but the contract of what that object must do is not very well defined for it to be on an interface. I'd expect we'd only want to inject a different impl in tests, or in really extreme cases. If we really want to support that kind of thing, we should think seriously about a dependency injection framework.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by ahgittin <gi...@git.apache.org>.
Github user ahgittin commented on a diff in the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#discussion_r30514538
--- Diff: core/src/main/java/brooklyn/catalog/internal/BasicBrooklynCatalog.java ---
@@ -986,7 +1045,12 @@ private DeploymentPlan makePlanFromYaml(String yaml) {
}
private CatalogItem<?,?> addItemDto(CatalogItemDtoAbstract<?, ?> itemDto, boolean forceUpdate) {
- checkItemNotExists(itemDto, forceUpdate);
+ CatalogItem<?, ?> existingDto = checkItemIsDuplicateOrDisallowed(itemDto, true, forceUpdate);
+ if (existingDto!=null) {
+ // it's a duplicate, and not forced, just return it
+ log.trace("Using existing duplicate for catalog item {}", itemDto.getId());
--- End diff --
it could get called a lot in hot backup. and we verified that what we would do would be no-op before we did nothing so not a useful log message i think.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by ahgittin <gi...@git.apache.org>.
Github user ahgittin commented on a diff in the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#discussion_r30514374
--- Diff: core/src/main/java/brooklyn/catalog/internal/BasicBrooklynCatalog.java ---
@@ -986,7 +1045,12 @@ private DeploymentPlan makePlanFromYaml(String yaml) {
}
private CatalogItem<?,?> addItemDto(CatalogItemDtoAbstract<?, ?> itemDto, boolean forceUpdate) {
- checkItemNotExists(itemDto, forceUpdate);
+ CatalogItem<?, ?> existingDto = checkItemIsDuplicateOrDisallowed(itemDto, true, forceUpdate);
--- End diff --
have renamed to `checkItemAllowedAndIfSoReturnAnyDuplicate` -- it's private internal but this is clearer
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by ahgittin <gi...@git.apache.org>.
Github user ahgittin commented on a diff in the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#discussion_r30496433
--- Diff: core/src/main/java/brooklyn/catalog/internal/CatalogInitialization.java ---
@@ -0,0 +1,377 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package brooklyn.catalog.internal;
+
+import java.io.File;
+import java.util.Collection;
+import java.util.List;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import brooklyn.catalog.CatalogItem;
+import brooklyn.config.BrooklynServerConfig;
+import brooklyn.management.ManagementContext;
+import brooklyn.management.ManagementContextInjectable;
+import brooklyn.management.internal.ManagementContextInternal;
+import brooklyn.util.ResourceUtils;
+import brooklyn.util.collections.MutableList;
+import brooklyn.util.exceptions.Exceptions;
+import brooklyn.util.exceptions.FatalRuntimeException;
+import brooklyn.util.exceptions.RuntimeInterruptedException;
+import brooklyn.util.flags.TypeCoercions;
+import brooklyn.util.guava.Maybe;
+import brooklyn.util.net.Urls;
+import brooklyn.util.text.Strings;
+
+import com.google.common.annotations.Beta;
+import com.google.common.base.Function;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.Iterables;
+
+@Beta
+public class CatalogInitialization implements ManagementContextInjectable {
--- End diff --
the mutex governs the population of the catalog, to prevent two threads from running this at the same time. looks to me that this is functioning correctly. pure defensive programming could lock this down further and make everything private but i think that would be overkill -- the fields and methods are already protected or package private and the class marked `@Beta` which is to say use caution when extending. that's the right level of defensiveness for something which is evolving IMHO.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by aledsage <gi...@git.apache.org>.
Github user aledsage commented on a diff in the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#discussion_r30583128
--- Diff: core/src/main/java/brooklyn/entity/rebind/persister/AbstractBrooklynMementoPersister.java ---
@@ -110,6 +115,11 @@ public void delta(Delta delta, PersistenceExceptionHandler exceptionHanlder) {
memento.updateEnricherMementos(delta.enrichers());
memento.updateCatalogItemMementos(delta.catalogItems());
}
+
+ @Override
+ public void queueDelta(Delta delta) {
+ log.warn("Legacy persister ignoring queued delta: "+delta);
--- End diff --
Perhaps `throw new IllegalStateException("Not supported; use "+BrooklynMementoPersisterToObjectStore.class)`.
With this code, we'll log and ignore the requested delta, so the delta will never be applied. Better to fail than to do that, I think (unless the delta really is empty!)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by ahgittin <gi...@git.apache.org>.
Github user ahgittin commented on a diff in the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#discussion_r30589577
--- Diff: core/src/main/java/brooklyn/catalog/internal/CatalogInitialization.java ---
@@ -0,0 +1,377 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package brooklyn.catalog.internal;
+
+import java.io.File;
+import java.util.Collection;
+import java.util.List;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import brooklyn.catalog.CatalogItem;
+import brooklyn.config.BrooklynServerConfig;
+import brooklyn.management.ManagementContext;
+import brooklyn.management.ManagementContextInjectable;
+import brooklyn.management.internal.ManagementContextInternal;
+import brooklyn.util.ResourceUtils;
+import brooklyn.util.collections.MutableList;
+import brooklyn.util.exceptions.Exceptions;
+import brooklyn.util.exceptions.FatalRuntimeException;
+import brooklyn.util.exceptions.RuntimeInterruptedException;
+import brooklyn.util.flags.TypeCoercions;
+import brooklyn.util.guava.Maybe;
+import brooklyn.util.net.Urls;
+import brooklyn.util.text.Strings;
+
+import com.google.common.annotations.Beta;
+import com.google.common.base.Function;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.Iterables;
+
+@Beta
+public class CatalogInitialization implements ManagementContextInjectable {
+
+ /*
+
+ A1) if not persisting, go to B1
+ A2) if --catalog-reset, delete persisted catalog items
+ A3) read persisted catalog items (possibly deleted in A2), go to C1
+ A4) go to B1
+
+ B1) look for --catalog-initial, if so read it, then go to C1
+ B2) look for BrooklynServerConfig.BROOKLYN_CATALOG_URL, if so, read it, supporting YAML or XML (warning if XML), then go to C1
+ B3) look for ~/.brooklyn/catalog.bom, if exists, read it then go to C1
+ B4) look for ~/.brooklyn/brooklyn.xml, if exists, warn, read it then go to C1
+ B5) read all classpath://brooklyn/default.catalog.bom items, if they exist (and for now they will)
+ B6) go to C1
+
+ C1) if --catalog-add, read and add those items
+
+ D1) if persisting, read the rest of the persisted items (entities etc)
+
+ */
+
+ private static final Logger log = LoggerFactory.getLogger(CatalogInitialization.class);
+
+ String initialUri;
+ boolean reset;
+ String additionsUri;
+ boolean force;
+
+ boolean disallowLocal = false;
+ List<Function<CatalogInitialization, Void>> callbacks = MutableList.of();
+ boolean hasRunBestEffort = false, hasRunOfficial = false, isPopulating = false;
+
+ ManagementContext managementContext;
+ boolean isStartingUp = false;
+ boolean failOnStartupErrors = false;
+
+ Object mutex = new Object();
+
+ public CatalogInitialization(String initialUri, boolean reset, String additionUri, boolean force) {
+ this.initialUri = initialUri;
+ this.reset = reset;
+ this.additionsUri = additionUri;
+ this.force = force;
+ }
+
+ public CatalogInitialization() {
+ this(null, false, null, false);
+ }
+
+ public void injectManagementContext(ManagementContext managementContext) {
+ if (this.managementContext!=null && managementContext!=null && !this.managementContext.equals(managementContext))
+ throw new IllegalStateException("Cannot switch management context of "+this+"; from "+this.managementContext+" to "+managementContext);
+ this.managementContext = managementContext;
+ }
+
+ public ManagementContext getManagementContext() {
+ return Preconditions.checkNotNull(managementContext, "management context has not been injected into "+this);
+ }
+
+ public CatalogInitialization addPopulationCallback(Function<CatalogInitialization, Void> callback) {
+ callbacks.add(callback);
+ return this;
+ }
+
+ public boolean isInitialResetRequested() {
+ return reset;
+ }
+
+ public boolean hasRunOfficial() { return hasRunOfficial; }
+ public boolean hasRunIncludingBestEffort() { return hasRunOfficial || hasRunBestEffort; }
+
+ /** makes or updates the mgmt catalog, based on the settings in this class */
+ public void populateCatalog(boolean needsInitial, Collection<CatalogItem<?, ?>> optionalItemsForResettingCatalog) {
+ try {
+ isPopulating = true;
+ synchronized (mutex) {
+ BasicBrooklynCatalog catalog = (BasicBrooklynCatalog) managementContext.getCatalog();
+ if (!catalog.getCatalog().isLoaded()) {
+ catalog.load();
+ } else {
+ if (needsInitial && (hasRunOfficial || hasRunBestEffort)) {
+ // an indication that something caused it to load early; not severe, but unusual
+ log.warn("Catalog initialization has not properly run but management context has a catalog; re-populating, possibly overwriting items installed during earlier access (it may have been an early web request)");
+ catalog.reset(ImmutableList.<CatalogItem<?,?>>of());
+ }
+ }
+ hasRunOfficial = true;
+
+ populateCatalog(catalog, needsInitial, true, optionalItemsForResettingCatalog);
+ }
+ } finally {
+ hasRunOfficial = true;
+ isPopulating = false;
+ }
+ }
+
+ private void populateCatalog(BasicBrooklynCatalog catalog, boolean needsInitial, boolean runCallbacks, Collection<CatalogItem<?, ?>> optionalItemsForResettingCatalog) {
+ applyCatalogLoadMode();
+
+ if (optionalItemsForResettingCatalog!=null) {
+ catalog.reset(optionalItemsForResettingCatalog);
+ }
+
+ if (needsInitial) {
+ populateInitial(catalog);
+ }
+
+ populateAdditions(catalog);
+
+ if (runCallbacks) {
+ populateViaCallbacks(catalog);
+ }
+ }
+
+ private enum PopulateMode { YAML, XML, AUTODETECT }
+
+ protected void populateInitial(BasicBrooklynCatalog catalog) {
+ if (disallowLocal) {
+ if (!hasRunOfficial()) {
+ log.debug("CLI initial catalog not being read with disallow-local mode set.");
+ }
+ return;
+ }
+
+// B1) look for --catalog-initial, if so read it, then go to C1
+// B2) look for BrooklynServerConfig.BROOKLYN_CATALOG_URL, if so, read it, supporting YAML or XML (warning if XML), then go to C1
+// B3) look for ~/.brooklyn/catalog.bom, if exists, read it then go to C1
+// B4) look for ~/.brooklyn/brooklyn.xml, if exists, warn, read it then go to C1
+// B5) read all classpath://brooklyn/default.catalog.bom items, if they exist (and for now they will)
+// B6) go to C1
+
+ if (initialUri!=null) {
+ populateInitialFromUri(catalog, initialUri, PopulateMode.AUTODETECT);
+ return;
+ }
+
+ String catalogUrl = managementContext.getConfig().getConfig(BrooklynServerConfig.BROOKLYN_CATALOG_URL);
+ if (Strings.isNonBlank(catalogUrl)) {
+ populateInitialFromUri(catalog, catalogUrl, PopulateMode.AUTODETECT);
+ return;
+ }
+
+ catalogUrl = Urls.mergePaths(BrooklynServerConfig.getMgmtBaseDir( managementContext.getConfig() ), "catalog.bom");
+ if (new File(catalogUrl).exists()) {
+ populateInitialFromUri(catalog, "file:"+catalogUrl, PopulateMode.YAML);
+ return;
+ }
+
+ catalogUrl = Urls.mergePaths(BrooklynServerConfig.getMgmtBaseDir( managementContext.getConfig() ), "catalog.xml");
+ if (new File(catalogUrl).exists()) {
+ populateInitialFromUri(catalog, "file:"+catalogUrl, PopulateMode.XML);
+ return;
+ }
+
+ // otherwise look for for classpath:/brooklyn/default.catalog.bom --
+ // there is one on the classpath which says to scan, and provides a few templates;
+ // if one is supplied by user in the conf/ dir that will override the item from the classpath
+ // (TBD - we might want to scan for all such bom's?)
+
+ catalogUrl = "classpath:/brooklyn/default.catalog.bom";
+ if (new ResourceUtils(this).doesUrlExist(catalogUrl)) {
+ populateInitialFromUri(catalog, catalogUrl, PopulateMode.YAML);
+ return;
+ }
+
+ log.info("No catalog found on classpath or specified; catalog will not be initialized.");
+ return;
+ }
+
+ private void populateInitialFromUri(BasicBrooklynCatalog catalog, String catalogUrl, PopulateMode mode) {
+ log.debug("Loading initial catalog from {}", catalogUrl);
+
+ Exception problem = null;
+ Object result = null;
+
+ String contents = null;
+ try {
+ contents = new ResourceUtils(this).getResourceAsString(catalogUrl);
+ } catch (Exception e) {
+ Exceptions.propagateIfFatal(e);
+ if (problem==null) problem = e;
+ }
+
+ if (contents!=null && (mode==PopulateMode.YAML || mode==PopulateMode.AUTODETECT)) {
+ // try YAML first
+ try {
+ catalog.reset(MutableList.<CatalogItem<?,?>>of());
+ result = catalog.addItems(contents);
+ } catch (Exception e) {
+ Exceptions.propagateIfFatal(e);
+ if (problem==null) problem = e;
+ }
+ }
+
+ if (result==null && contents!=null && (mode==PopulateMode.XML || mode==PopulateMode.AUTODETECT)) {
+ // then try XML
+ problem = populateInitialFromUriXml(catalog, catalogUrl, problem, contents);
+ }
+
+ if (result!=null) {
+ log.debug("Loaded initial catalog from {}: {}", catalogUrl, result);
+ }
+ if (problem!=null) {
+ log.warn("Error importing catalog from " + catalogUrl + ": " + problem, problem);
+ // TODO inform mgmt of error
+ }
+
+ }
+
+ // deprecated XML format
+ @SuppressWarnings("deprecation")
+ private Exception populateInitialFromUriXml(BasicBrooklynCatalog catalog, String catalogUrl, Exception problem, String contents) {
+ CatalogDto dto = null;
+ try {
+ dto = CatalogDto.newDtoFromXmlContents(contents, catalogUrl);
+ problem = null;
+ } catch (Exception e) {
+ Exceptions.propagateIfFatal(e);
+ if (problem==null) problem = e;
+ }
+ if (dto!=null) {
+ catalog.reset(dto);
+ }
+ return problem;
+ }
+
+ boolean hasRunAdditions = false;
+ protected void populateAdditions(BasicBrooklynCatalog catalog) {
+ if (Strings.isNonBlank(additionsUri)) {
+ if (disallowLocal) {
+ if (!hasRunAdditions) {
+ log.warn("CLI additions supplied but not supported in disallow-local mode; ignoring.");
+ }
+ return;
+ }
+ if (!hasRunAdditions) {
+ log.debug("Adding to catalog from CLI: "+additionsUri+" (force: "+force+")");
+ }
+ Iterable<? extends CatalogItem<?, ?>> items = catalog.addItems(
+ new ResourceUtils(this).getResourceAsString(additionsUri), force);
+
+ if (!hasRunAdditions)
+ log.debug("Added to catalog from CLI: "+items);
+ else
+ log.debug("Added to catalog from CLI: count "+Iterables.size(items));
+
+ hasRunAdditions = true;
+ }
+ }
+
+ protected void populateViaCallbacks(BasicBrooklynCatalog catalog) {
+ for (Function<CatalogInitialization, Void> callback: callbacks)
--- End diff --
i think this is the only one i didn't fully agree with -- could go either way, but i think of them as callbacks because typically they are passed in by custom startup classes to have a callback to populate catalog in their space. (yes, they are actions, but that seems less clear to me.)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by neykov <gi...@git.apache.org>.
Github user neykov commented on a diff in the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#discussion_r30795389
--- Diff: core/src/main/java/brooklyn/entity/rebind/RebindIteration.java ---
@@ -341,38 +340,64 @@ protected void rebuildCatalog() {
}
}
}
+
+ // See notes in CatalogInitialization
- // Register catalogue items with the management context. Loads the bundles in the OSGi framework.
- CatalogLoadMode catalogLoadMode = managementContext.getConfig().getConfig(BrooklynServerConfig.CATALOG_LOAD_MODE);
+ Collection<CatalogItem<?, ?>> catalogItems = rebindContext.getCatalogItems();
+ CatalogInitialization catInit = ((ManagementContextInternal)managementContext).getCatalogInitialization();
+ catInit.applyCatalogLoadMode();
+ Collection<CatalogItem<?,?>> itemsForResettingCatalog = null;
+ boolean needsInitialCatalog;
if (rebindManager.persistCatalogItemsEnabled) {
- boolean shouldResetCatalog = catalogLoadMode == CatalogLoadMode.LOAD_PERSISTED_STATE
- || (!isEmpty && catalogLoadMode == CatalogLoadMode.LOAD_BROOKLYN_CATALOG_URL_IF_NO_PERSISTED_STATE);
- boolean shouldLoadDefaultCatalog = catalogLoadMode == CatalogLoadMode.LOAD_BROOKLYN_CATALOG_URL
- || (isEmpty && catalogLoadMode == CatalogLoadMode.LOAD_BROOKLYN_CATALOG_URL_IF_NO_PERSISTED_STATE);
- if (shouldResetCatalog) {
- // Reset catalog with previously persisted state
- logRebindingDebug("RebindManager resetting management context catalog to previously persisted state");
- managementContext.getCatalog().reset(rebindContext.getCatalogItems());
- } else if (shouldLoadDefaultCatalog) {
- // Load catalogue as normal
- // TODO in read-only mode, should do this less frequently than entities etc
- logRebindingDebug("RebindManager loading default catalog");
- ((BasicBrooklynCatalog) managementContext.getCatalog()).resetCatalogToContentsAtConfiguredUrl();
+ if (!catInit.hasRunOfficial() && catInit.isInitialResetRequested()) {
+ String message = "RebindManager resetting catalog on first run (catalog persistence enabled, but reset explicitly specified). ";
+ if (catalogItems.isEmpty()) {
+ message += "Catalog was empty anyway.";
+ } else {
+ message += "Deleting "+catalogItems.size()+" persisted catalog item"+Strings.s(catalogItems)+": "+catalogItems;
+ if (shouldLogRebinding()) {
+ LOG.info(message);
+ }
+ }
+ logRebindingDebug(message);
+
+ itemsForResettingCatalog = MutableList.<CatalogItem<?,?>>of();
+
+ PersisterDeltaImpl delta = new PersisterDeltaImpl();
+ delta.removedCatalogItemIds.addAll(mementoRawData.getCatalogItems().keySet());
+ getPersister().queueDelta(delta);
+
+ mementoRawData.clearCatalogItems();
+ needsInitialCatalog = true;
} else {
- // Management context should have taken care of loading the catalogue
- Collection<CatalogItem<?, ?>> catalogItems = rebindContext.getCatalogItems();
- String message = "RebindManager not resetting catalog to persisted state. Catalog load mode is {}.";
- if (!catalogItems.isEmpty() && shouldLogRebinding()) {
- LOG.info(message + " There {} {} item{} persisted.", new Object[]{
- catalogLoadMode, catalogItems.size() == 1 ? "was" : "were", catalogItems.size(), Strings.s(catalogItems)});
- } else if (LOG.isDebugEnabled()) {
- logRebindingDebug(message, catalogLoadMode);
+ if (!isEmpty) {
+ logRebindingDebug("RebindManager clearing local catalog and loading from persisted state");
+ itemsForResettingCatalog = rebindContext.getCatalogItems();
+ needsInitialCatalog = false;
+ } else {
+ if (catInit.hasRunOfficial()) {
--- End diff --
If we have already initialized the
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by neykov <gi...@git.apache.org>.
Github user neykov commented on a diff in the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#discussion_r30787423
--- Diff: core/src/main/java/brooklyn/catalog/internal/CatalogInitialization.java ---
@@ -0,0 +1,380 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package brooklyn.catalog.internal;
+
+import java.io.File;
+import java.util.Collection;
+import java.util.List;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import brooklyn.catalog.CatalogItem;
+import brooklyn.config.BrooklynServerConfig;
+import brooklyn.management.ManagementContext;
+import brooklyn.management.ManagementContextInjectable;
+import brooklyn.management.internal.ManagementContextInternal;
+import brooklyn.util.ResourceUtils;
+import brooklyn.util.collections.MutableList;
+import brooklyn.util.exceptions.Exceptions;
+import brooklyn.util.exceptions.FatalRuntimeException;
+import brooklyn.util.exceptions.RuntimeInterruptedException;
+import brooklyn.util.flags.TypeCoercions;
+import brooklyn.util.guava.Maybe;
+import brooklyn.util.os.Os;
+import brooklyn.util.text.Strings;
+
+import com.google.common.annotations.Beta;
+import com.google.common.base.Function;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.Iterables;
+
+@Beta
+public class CatalogInitialization implements ManagementContextInjectable {
+
+ /*
+
+ A1) if not persisting, go to B1
+ A2) if --catalog-reset, delete persisted catalog items
+ A3) if there is a persisted catalog (and it wasn't not deleted by A2), read it and go to C1
+ A4) go to B1
+
+ B1) look for --catalog-initial, if so read it, then go to C1
+ B2) look for BrooklynServerConfig.BROOKLYN_CATALOG_URL, if so, read it, supporting YAML or XML (warning if XML), then go to C1
+ B3) look for ~/.brooklyn/catalog.bom, if exists, read it then go to C1
+ B4) look for ~/.brooklyn/brooklyn.xml, if exists, warn, read it then go to C1
+ B5) read all classpath://brooklyn/default.catalog.bom items, if they exist (and for now they will)
+ B6) go to C1
+
+ C1) if --catalog-add, read and add those items
+
+ D1) if persisting, read the rest of the persisted items (entities etc)
+
+ */
+
+ private static final Logger log = LoggerFactory.getLogger(CatalogInitialization.class);
+
+ private String initialUri;
+ private boolean reset;
+ private String additionsUri;
+ private boolean force;
+
+ private boolean disallowLocal = false;
+ private List<Function<CatalogInitialization, Void>> callbacks = MutableList.of();
+ private boolean hasRunBestEffort = false, hasRunOfficial = false, isPopulating = false;
+
+ private ManagementContext managementContext;
+ private boolean isStartingUp = false;
+ private boolean failOnStartupErrors = false;
+
+ private Object populatingCatalogMutex = new Object();
+
+ public CatalogInitialization(String initialUri, boolean reset, String additionUri, boolean force) {
+ this.initialUri = initialUri;
+ this.reset = reset;
+ this.additionsUri = additionUri;
+ this.force = force;
+ }
+
+ public CatalogInitialization() {
+ this(null, false, null, false);
+ }
+
+ public void injectManagementContext(ManagementContext managementContext) {
+ Preconditions.checkNotNull(managementContext, "management context");
+ if (this.managementContext!=null && managementContext!=this.managementContext)
+ throw new IllegalStateException("Cannot switch management context, from "+this.managementContext+" to "+managementContext);
+ this.managementContext = managementContext;
+ }
+
+ /** Called by the framework to set true while starting up, and false afterwards,
+ * in order to assist in appropriate logging and error handling. */
+ public void setStartingUp(boolean isStartingUp) {
+ this.isStartingUp = isStartingUp;
+ }
+
+ public void setFailOnStartupErrors(boolean startupFailOnCatalogErrors) {
+ this.failOnStartupErrors = startupFailOnCatalogErrors;
+ }
+
+ public CatalogInitialization addPopulationCallback(Function<CatalogInitialization, Void> callback) {
+ callbacks.add(callback);
+ return this;
+ }
+
+ public ManagementContext getManagementContext() {
+ return Preconditions.checkNotNull(managementContext, "management context has not been injected into "+this);
+ }
+
+ public boolean isInitialResetRequested() {
+ return reset;
+ }
+
+ public boolean hasRunOfficial() { return hasRunOfficial; }
+ public boolean hasRunIncludingBestEffort() { return hasRunOfficial || hasRunBestEffort; }
+
+ /** makes or updates the mgmt catalog, based on the settings in this class */
+ public void populateCatalog(boolean needsInitial, Collection<CatalogItem<?, ?>> optionalItemsForResettingCatalog) {
+ synchronized (populatingCatalogMutex) {
+ try {
+ isPopulating = true;
+ BasicBrooklynCatalog catalog = (BasicBrooklynCatalog) managementContext.getCatalog();
+ if (!catalog.getCatalog().isLoaded()) {
+ catalog.load();
+ } else {
+ if (needsInitial && (hasRunOfficial || hasRunBestEffort)) {
+ // an indication that something caused it to load early; not severe, but unusual
+ log.warn("Catalog initialization has not properly run but management context has a catalog; re-populating, possibly overwriting items installed during earlier access (it may have been an early web request)");
+ catalog.reset(ImmutableList.<CatalogItem<?,?>>of());
+ }
+ }
+ hasRunOfficial = true;
+
+ populateCatalogImpl(catalog, needsInitial, optionalItemsForResettingCatalog);
+ } finally {
+ hasRunOfficial = true;
+ isPopulating = false;
+ }
+ }
+ }
+
+ private void populateCatalogImpl(BasicBrooklynCatalog catalog, boolean needsInitial, Collection<CatalogItem<?, ?>> optionalItemsForResettingCatalog) {
+ applyCatalogLoadMode();
+
+ if (optionalItemsForResettingCatalog!=null) {
+ catalog.reset(optionalItemsForResettingCatalog);
+ }
+
+ if (needsInitial) {
+ populateInitial(catalog);
+ }
+
+ populateAdditions(catalog);
+
+ populateViaCallbacks(catalog);
+ }
+
+ private enum PopulateMode { YAML, XML, AUTODETECT }
+
+ protected void populateInitial(BasicBrooklynCatalog catalog) {
+ if (disallowLocal) {
+ if (!hasRunOfficial()) {
+ log.debug("CLI initial catalog not being read with disallow-local mode set.");
+ }
+ return;
+ }
+
+// B1) look for --catalog-initial, if so read it, then go to C1
+// B2) look for BrooklynServerConfig.BROOKLYN_CATALOG_URL, if so, read it, supporting YAML or XML (warning if XML), then go to C1
+// B3) look for ~/.brooklyn/catalog.bom, if exists, read it then go to C1
+// B4) look for ~/.brooklyn/brooklyn.xml, if exists, warn, read it then go to C1
+// B5) read all classpath://brooklyn/default.catalog.bom items, if they exist (and for now they will)
+// B6) go to C1
+
+ if (initialUri!=null) {
+ populateInitialFromUri(catalog, initialUri, PopulateMode.AUTODETECT);
+ return;
+ }
+
+ String catalogUrl = managementContext.getConfig().getConfig(BrooklynServerConfig.BROOKLYN_CATALOG_URL);
+ if (Strings.isNonBlank(catalogUrl)) {
+ populateInitialFromUri(catalog, catalogUrl, PopulateMode.AUTODETECT);
+ return;
+ }
+
+ catalogUrl = Os.mergePaths(BrooklynServerConfig.getMgmtBaseDir( managementContext.getConfig() ), "catalog.bom");
+ if (new File(catalogUrl).exists()) {
+ populateInitialFromUri(catalog, "file:"+catalogUrl, PopulateMode.YAML);
+ return;
+ }
+
+ catalogUrl = Os.mergePaths(BrooklynServerConfig.getMgmtBaseDir( managementContext.getConfig() ), "catalog.xml");
+ if (new File(catalogUrl).exists()) {
+ populateInitialFromUri(catalog, "file:"+catalogUrl, PopulateMode.XML);
+ return;
+ }
+
+ // otherwise look for for classpath:/brooklyn/default.catalog.bom --
+ // there is one on the classpath which says to scan, and provides a few templates;
+ // if one is supplied by user in the conf/ dir that will override the item from the classpath
+ // (TBD - we might want to scan for all such bom's?)
+
+ catalogUrl = "classpath:/brooklyn/default.catalog.bom";
+ if (new ResourceUtils(this).doesUrlExist(catalogUrl)) {
+ populateInitialFromUri(catalog, catalogUrl, PopulateMode.YAML);
+ return;
+ }
+
+ log.info("No catalog found on classpath or specified; catalog will not be initialized.");
+ return;
+ }
+
+ private void populateInitialFromUri(BasicBrooklynCatalog catalog, String catalogUrl, PopulateMode mode) {
+ log.debug("Loading initial catalog from {}", catalogUrl);
+
+ Exception problem = null;
+ Object result = null;
+
+ String contents = null;
+ try {
+ contents = new ResourceUtils(this).getResourceAsString(catalogUrl);
+ } catch (Exception e) {
+ Exceptions.propagateIfFatal(e);
+ if (problem==null) problem = e;
+ }
+
+ if (contents!=null && (mode==PopulateMode.YAML || mode==PopulateMode.AUTODETECT)) {
+ // try YAML first
+ try {
+ catalog.reset(MutableList.<CatalogItem<?,?>>of());
+ result = catalog.addItems(contents);
+ } catch (Exception e) {
+ Exceptions.propagateIfFatal(e);
+ if (problem==null) problem = e;
+ }
+ }
+
+ if (result==null && contents!=null && (mode==PopulateMode.XML || mode==PopulateMode.AUTODETECT)) {
+ // then try XML
+ try {
+ populateInitialFromUriXml(catalog, catalogUrl, contents);
+ // clear YAML problem
+ problem = null;
+ } catch (Exception e) {
+ Exceptions.propagateIfFatal(e);
+ if (problem==null) problem = e;
+ }
+ }
+
+ if (result!=null) {
+ log.debug("Loaded initial catalog from {}: {}", catalogUrl, result);
+ }
+ if (problem!=null) {
+ log.warn("Error importing catalog from " + catalogUrl + ": " + problem, problem);
+ // TODO inform mgmt of error
+ }
+
+ }
+
+ // deprecated XML format
+ @SuppressWarnings("deprecation")
+ private void populateInitialFromUriXml(BasicBrooklynCatalog catalog, String catalogUrl, String contents) {
+ CatalogDto dto = CatalogDto.newDtoFromXmlContents(contents, catalogUrl);
+ if (dto!=null) {
+ catalog.reset(dto);
+ }
+ }
+
+ boolean hasRunAdditions = false;
+ protected void populateAdditions(BasicBrooklynCatalog catalog) {
+ if (Strings.isNonBlank(additionsUri)) {
+ if (disallowLocal) {
+ if (!hasRunAdditions) {
+ log.warn("CLI additions supplied but not supported in disallow-local mode; ignoring.");
+ }
+ return;
--- End diff --
Same as above - if the user explicitly uses `--catalogAdd` on the command line, better to take precedence over deprecated configuration or fail with a descriptive message.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-brooklyn/pull/617
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by ahgittin <gi...@git.apache.org>.
Github user ahgittin commented on a diff in the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#discussion_r30968350
--- Diff: core/src/main/java/brooklyn/entity/rebind/RebindIteration.java ---
@@ -341,38 +340,64 @@ protected void rebuildCatalog() {
}
}
}
+
+ // See notes in CatalogInitialization
- // Register catalogue items with the management context. Loads the bundles in the OSGi framework.
- CatalogLoadMode catalogLoadMode = managementContext.getConfig().getConfig(BrooklynServerConfig.CATALOG_LOAD_MODE);
+ Collection<CatalogItem<?, ?>> catalogItems = rebindContext.getCatalogItems();
+ CatalogInitialization catInit = ((ManagementContextInternal)managementContext).getCatalogInitialization();
+ catInit.applyCatalogLoadMode();
+ Collection<CatalogItem<?,?>> itemsForResettingCatalog = null;
+ boolean needsInitialCatalog;
if (rebindManager.persistCatalogItemsEnabled) {
- boolean shouldResetCatalog = catalogLoadMode == CatalogLoadMode.LOAD_PERSISTED_STATE
- || (!isEmpty && catalogLoadMode == CatalogLoadMode.LOAD_BROOKLYN_CATALOG_URL_IF_NO_PERSISTED_STATE);
- boolean shouldLoadDefaultCatalog = catalogLoadMode == CatalogLoadMode.LOAD_BROOKLYN_CATALOG_URL
- || (isEmpty && catalogLoadMode == CatalogLoadMode.LOAD_BROOKLYN_CATALOG_URL_IF_NO_PERSISTED_STATE);
- if (shouldResetCatalog) {
- // Reset catalog with previously persisted state
- logRebindingDebug("RebindManager resetting management context catalog to previously persisted state");
- managementContext.getCatalog().reset(rebindContext.getCatalogItems());
- } else if (shouldLoadDefaultCatalog) {
- // Load catalogue as normal
- // TODO in read-only mode, should do this less frequently than entities etc
- logRebindingDebug("RebindManager loading default catalog");
- ((BasicBrooklynCatalog) managementContext.getCatalog()).resetCatalogToContentsAtConfiguredUrl();
+ if (!catInit.hasRunOfficial() && catInit.isInitialResetRequested()) {
+ String message = "RebindManager resetting catalog on first run (catalog persistence enabled, but reset explicitly specified). ";
+ if (catalogItems.isEmpty()) {
+ message += "Catalog was empty anyway.";
+ } else {
+ message += "Deleting "+catalogItems.size()+" persisted catalog item"+Strings.s(catalogItems)+": "+catalogItems;
+ if (shouldLogRebinding()) {
+ LOG.info(message);
+ }
+ }
+ logRebindingDebug(message);
+
+ itemsForResettingCatalog = MutableList.<CatalogItem<?,?>>of();
--- End diff --
that's the right behavior isn't it? it won't persist the catalog clearance, so it's okay, and the instructions were invalid but were followed. what would you have it do instead?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by ahgittin <gi...@git.apache.org>.
Github user ahgittin commented on the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#issuecomment-103088238
@aledsage @robertgmoss these comments all addressed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by aledsage <gi...@git.apache.org>.
Github user aledsage commented on a diff in the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#discussion_r30586458
--- Diff: usage/cli/src/main/java/brooklyn/cli/Main.java ---
@@ -536,9 +570,10 @@ protected BrooklynLauncher createLauncher() {
BrooklynLauncher launcher;
launcher = BrooklynLauncher.newInstance();
launcher.localBrooklynPropertiesFile(localBrooklynProperties)
- .ignorePersistenceErrors(ignorePersistenceErrors)
- .ignoreWebErrors(ignoreWebErrors)
- .ignoreAppErrors(ignoreAppErrors)
+ .ignorePersistenceErrors(!startupFailOnPersistenceErrors)
--- End diff --
Should we (in separate PR?) align the method names in launcher with the cli names? e.g. `failOnPersistenceErrors(boolean)`, and deprecate the `ignorePersistenceErrors(boolean)`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by ahgittin <gi...@git.apache.org>.
Github user ahgittin commented on a diff in the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#discussion_r30973941
--- Diff: core/src/main/java/brooklyn/entity/rebind/RebindIteration.java ---
@@ -341,38 +340,64 @@ protected void rebuildCatalog() {
}
}
}
+
+ // See notes in CatalogInitialization
- // Register catalogue items with the management context. Loads the bundles in the OSGi framework.
- CatalogLoadMode catalogLoadMode = managementContext.getConfig().getConfig(BrooklynServerConfig.CATALOG_LOAD_MODE);
+ Collection<CatalogItem<?, ?>> catalogItems = rebindContext.getCatalogItems();
+ CatalogInitialization catInit = ((ManagementContextInternal)managementContext).getCatalogInitialization();
+ catInit.applyCatalogLoadMode();
+ Collection<CatalogItem<?,?>> itemsForResettingCatalog = null;
+ boolean needsInitialCatalog;
if (rebindManager.persistCatalogItemsEnabled) {
- boolean shouldResetCatalog = catalogLoadMode == CatalogLoadMode.LOAD_PERSISTED_STATE
- || (!isEmpty && catalogLoadMode == CatalogLoadMode.LOAD_BROOKLYN_CATALOG_URL_IF_NO_PERSISTED_STATE);
- boolean shouldLoadDefaultCatalog = catalogLoadMode == CatalogLoadMode.LOAD_BROOKLYN_CATALOG_URL
- || (isEmpty && catalogLoadMode == CatalogLoadMode.LOAD_BROOKLYN_CATALOG_URL_IF_NO_PERSISTED_STATE);
- if (shouldResetCatalog) {
- // Reset catalog with previously persisted state
- logRebindingDebug("RebindManager resetting management context catalog to previously persisted state");
- managementContext.getCatalog().reset(rebindContext.getCatalogItems());
- } else if (shouldLoadDefaultCatalog) {
- // Load catalogue as normal
- // TODO in read-only mode, should do this less frequently than entities etc
- logRebindingDebug("RebindManager loading default catalog");
- ((BasicBrooklynCatalog) managementContext.getCatalog()).resetCatalogToContentsAtConfiguredUrl();
+ if (!catInit.hasRunOfficial() && catInit.isInitialResetRequested()) {
+ String message = "RebindManager resetting catalog on first run (catalog persistence enabled, but reset explicitly specified). ";
+ if (catalogItems.isEmpty()) {
+ message += "Catalog was empty anyway.";
+ } else {
+ message += "Deleting "+catalogItems.size()+" persisted catalog item"+Strings.s(catalogItems)+": "+catalogItems;
+ if (shouldLogRebinding()) {
+ LOG.info(message);
+ }
+ }
+ logRebindingDebug(message);
+
+ itemsForResettingCatalog = MutableList.<CatalogItem<?,?>>of();
--- End diff --
agree with your synopsis. have added comments:
// we will have unnecessarily tried to load the catalog item manifests earlier in this iteration,
// and problems there could fail a rebind even when we are resetting;
// it might be cleaner to check earlier whether a reset is happening and not load those items at all,
// but that would be a significant new code path (to remove a directory in the persistent store, essentially),
// and as it stands we don't do much with those manifests (e.g. we won't register them or fail on missing types)
// so we think it's only really corrupted XML or CatalogItem schema changes which would cause such problems.
// in extremis someone might need to wipe their store but for most purposes i don't think there will be any issue
// with loading the catalog item manifests before wiping all those files.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by ahgittin <gi...@git.apache.org>.
Github user ahgittin commented on a diff in the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#discussion_r30658810
--- Diff: utils/common/src/main/java/brooklyn/util/javalang/AggregateClassLoader.java ---
@@ -69,8 +70,25 @@ public void add(int index, ClassLoader classLoader) {
if (classLoader != null) classLoaders.add(index, classLoader);
}
- /** Returns the _live_ (and modifiable) list of classloaders
- * @return */
+ /** Resets the classloader shown here to be the given set */
+ public void reset(Collection<? extends ClassLoader> newClassLoaders) {
+ synchronized (classLoaders) {
--- End diff --
have synched on all accesses, including access to `iterator()`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by ahgittin <gi...@git.apache.org>.
Github user ahgittin commented on a diff in the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#discussion_r30659264
--- Diff: core/src/main/java/brooklyn/entity/rebind/persister/AbstractBrooklynMementoPersister.java ---
@@ -110,6 +115,11 @@ public void delta(Delta delta, PersistenceExceptionHandler exceptionHanlder) {
memento.updateEnricherMementos(delta.enrichers());
memento.updateCatalogItemMementos(delta.catalogItems());
}
+
+ @Override
+ public void queueDelta(Delta delta) {
+ log.warn("Legacy persister ignoring queued delta: "+delta);
--- End diff --
this will be deleted as soon as `0.7.0` is cut. i'm sure we could delete it now, i'm just being super cautious. no need to make perfect log messages here!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by aledsage <gi...@git.apache.org>.
Github user aledsage commented on a diff in the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#discussion_r30499496
--- Diff: api/src/main/java/brooklyn/mementos/BrooklynMementoPersister.java ---
@@ -97,10 +97,13 @@
/** @deprecated since 0.7.0, use {@link #checkpoint(BrooklynMementoRawData, PersistenceExceptionHandler)}
* and javadoc on implementations of that */ @Deprecated // pretty sure this is not used outwith deprecated code
void checkpoint(BrooklynMemento memento, PersistenceExceptionHandler exceptionHandler);
-
- void checkpoint(BrooklynMementoRawData newMemento, PersistenceExceptionHandler exceptionHandler);
+ /** applies a full checkpoint (write) of all state */
+ void checkpoint(BrooklynMementoRawData newMemento, PersistenceExceptionHandler exceptionHandler);
+ /** applies a partial write of state delta */
void delta(Delta delta, PersistenceExceptionHandler exceptionHandler);
+ /** inserts an additional delta to be written on the next delta request */
+ void queueDelta(Delta delta);
--- End diff --
The semantics of `queueDelta` (to only be applied on the next delta request) is clear and good enough for current usage, but feels strange to have on an api. If means your queue won't be executed until either you call delta (in which case why did you queue it) or someone else calls delta (in which case you don't know when that will happen).
Worth at least having `@Beta` on this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by neykov <gi...@git.apache.org>.
Github user neykov commented on the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#issuecomment-104691624
Also would be nice if `--app` is treated analogously to `--catalogInit` with regard to rebind state, can be addressed in a separate PR.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by ahgittin <gi...@git.apache.org>.
Github user ahgittin commented on the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#issuecomment-97624025
BTW it's just the last 3 commits, the others are from #614
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by ahgittin <gi...@git.apache.org>.
Github user ahgittin commented on the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#issuecomment-105529608
@neykov test added. the issue wasn't in the scanner - it could see brooklyn types - but in transferring the metadata including libraries into to the scanner-generated dto. hackish in places but happy path now works, while we figure out how much to continue with this experimental feature. good catch @neykov!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by ahgittin <gi...@git.apache.org>.
Github user ahgittin commented on a diff in the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#discussion_r30968077
--- Diff: core/src/main/java/brooklyn/entity/rebind/PeriodicDeltaChangeListener.java ---
@@ -164,14 +163,12 @@ public void remove(BrooklynObject instance) {
private final PersistenceExceptionHandler exceptionHandler;
private final Duration period;
-
- private final AtomicLong writeCount = new AtomicLong();
-
+
private DeltaCollector deltaCollector = new DeltaCollector();
private volatile boolean running = false;
- private volatile boolean stopped = false;
+ private volatile boolean stopping = false, stopCompleted = false;
--- End diff --
+1 that's cleaner, have applied
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by aledsage <gi...@git.apache.org>.
Github user aledsage commented on a diff in the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#discussion_r30546417
--- Diff: core/src/main/java/brooklyn/catalog/internal/CatalogInitialization.java ---
@@ -0,0 +1,377 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package brooklyn.catalog.internal;
+
+import java.io.File;
+import java.util.Collection;
+import java.util.List;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import brooklyn.catalog.CatalogItem;
+import brooklyn.config.BrooklynServerConfig;
+import brooklyn.management.ManagementContext;
+import brooklyn.management.ManagementContextInjectable;
+import brooklyn.management.internal.ManagementContextInternal;
+import brooklyn.util.ResourceUtils;
+import brooklyn.util.collections.MutableList;
+import brooklyn.util.exceptions.Exceptions;
+import brooklyn.util.exceptions.FatalRuntimeException;
+import brooklyn.util.exceptions.RuntimeInterruptedException;
+import brooklyn.util.flags.TypeCoercions;
+import brooklyn.util.guava.Maybe;
+import brooklyn.util.net.Urls;
+import brooklyn.util.text.Strings;
+
+import com.google.common.annotations.Beta;
+import com.google.common.base.Function;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.Iterables;
+
+@Beta
+public class CatalogInitialization implements ManagementContextInjectable {
+
+ /*
+
+ A1) if not persisting, go to B1
+ A2) if --catalog-reset, delete persisted catalog items
+ A3) read persisted catalog items (possibly deleted in A2), go to C1
+ A4) go to B1
+
+ B1) look for --catalog-initial, if so read it, then go to C1
+ B2) look for BrooklynServerConfig.BROOKLYN_CATALOG_URL, if so, read it, supporting YAML or XML (warning if XML), then go to C1
+ B3) look for ~/.brooklyn/catalog.bom, if exists, read it then go to C1
+ B4) look for ~/.brooklyn/brooklyn.xml, if exists, warn, read it then go to C1
+ B5) read all classpath://brooklyn/default.catalog.bom items, if they exist (and for now they will)
+ B6) go to C1
+
+ C1) if --catalog-add, read and add those items
+
+ D1) if persisting, read the rest of the persisted items (entities etc)
+
+ */
+
+ private static final Logger log = LoggerFactory.getLogger(CatalogInitialization.class);
+
+ String initialUri;
+ boolean reset;
+ String additionsUri;
+ boolean force;
+
+ boolean disallowLocal = false;
+ List<Function<CatalogInitialization, Void>> callbacks = MutableList.of();
+ boolean hasRunBestEffort = false, hasRunOfficial = false, isPopulating = false;
+
+ ManagementContext managementContext;
+ boolean isStartingUp = false;
+ boolean failOnStartupErrors = false;
+
+ Object mutex = new Object();
+
+ public CatalogInitialization(String initialUri, boolean reset, String additionUri, boolean force) {
+ this.initialUri = initialUri;
+ this.reset = reset;
+ this.additionsUri = additionUri;
+ this.force = force;
+ }
+
+ public CatalogInitialization() {
+ this(null, false, null, false);
+ }
+
+ public void injectManagementContext(ManagementContext managementContext) {
+ if (this.managementContext!=null && managementContext!=null && !this.managementContext.equals(managementContext))
+ throw new IllegalStateException("Cannot switch management context of "+this+"; from "+this.managementContext+" to "+managementContext);
+ this.managementContext = managementContext;
+ }
+
+ public ManagementContext getManagementContext() {
+ return Preconditions.checkNotNull(managementContext, "management context has not been injected into "+this);
+ }
+
+ public CatalogInitialization addPopulationCallback(Function<CatalogInitialization, Void> callback) {
+ callbacks.add(callback);
+ return this;
+ }
+
+ public boolean isInitialResetRequested() {
+ return reset;
+ }
+
+ public boolean hasRunOfficial() { return hasRunOfficial; }
+ public boolean hasRunIncludingBestEffort() { return hasRunOfficial || hasRunBestEffort; }
+
+ /** makes or updates the mgmt catalog, based on the settings in this class */
+ public void populateCatalog(boolean needsInitial, Collection<CatalogItem<?, ?>> optionalItemsForResettingCatalog) {
+ try {
+ isPopulating = true;
+ synchronized (mutex) {
+ BasicBrooklynCatalog catalog = (BasicBrooklynCatalog) managementContext.getCatalog();
+ if (!catalog.getCatalog().isLoaded()) {
+ catalog.load();
+ } else {
+ if (needsInitial && (hasRunOfficial || hasRunBestEffort)) {
+ // an indication that something caused it to load early; not severe, but unusual
+ log.warn("Catalog initialization has not properly run but management context has a catalog; re-populating, possibly overwriting items installed during earlier access (it may have been an early web request)");
+ catalog.reset(ImmutableList.<CatalogItem<?,?>>of());
+ }
+ }
+ hasRunOfficial = true;
+
+ populateCatalog(catalog, needsInitial, true, optionalItemsForResettingCatalog);
+ }
+ } finally {
+ hasRunOfficial = true;
+ isPopulating = false;
+ }
+ }
+
+ private void populateCatalog(BasicBrooklynCatalog catalog, boolean needsInitial, boolean runCallbacks, Collection<CatalogItem<?, ?>> optionalItemsForResettingCatalog) {
+ applyCatalogLoadMode();
+
+ if (optionalItemsForResettingCatalog!=null) {
+ catalog.reset(optionalItemsForResettingCatalog);
+ }
+
+ if (needsInitial) {
+ populateInitial(catalog);
+ }
+
+ populateAdditions(catalog);
+
+ if (runCallbacks) {
+ populateViaCallbacks(catalog);
+ }
+ }
+
+ private enum PopulateMode { YAML, XML, AUTODETECT }
+
+ protected void populateInitial(BasicBrooklynCatalog catalog) {
+ if (disallowLocal) {
+ if (!hasRunOfficial()) {
+ log.debug("CLI initial catalog not being read with disallow-local mode set.");
+ }
+ return;
+ }
+
+// B1) look for --catalog-initial, if so read it, then go to C1
+// B2) look for BrooklynServerConfig.BROOKLYN_CATALOG_URL, if so, read it, supporting YAML or XML (warning if XML), then go to C1
+// B3) look for ~/.brooklyn/catalog.bom, if exists, read it then go to C1
+// B4) look for ~/.brooklyn/brooklyn.xml, if exists, warn, read it then go to C1
+// B5) read all classpath://brooklyn/default.catalog.bom items, if they exist (and for now they will)
+// B6) go to C1
+
+ if (initialUri!=null) {
+ populateInitialFromUri(catalog, initialUri, PopulateMode.AUTODETECT);
+ return;
+ }
+
+ String catalogUrl = managementContext.getConfig().getConfig(BrooklynServerConfig.BROOKLYN_CATALOG_URL);
+ if (Strings.isNonBlank(catalogUrl)) {
+ populateInitialFromUri(catalog, catalogUrl, PopulateMode.AUTODETECT);
+ return;
+ }
+
+ catalogUrl = Urls.mergePaths(BrooklynServerConfig.getMgmtBaseDir( managementContext.getConfig() ), "catalog.bom");
+ if (new File(catalogUrl).exists()) {
+ populateInitialFromUri(catalog, "file:"+catalogUrl, PopulateMode.YAML);
+ return;
+ }
+
+ catalogUrl = Urls.mergePaths(BrooklynServerConfig.getMgmtBaseDir( managementContext.getConfig() ), "catalog.xml");
+ if (new File(catalogUrl).exists()) {
+ populateInitialFromUri(catalog, "file:"+catalogUrl, PopulateMode.XML);
+ return;
+ }
+
+ // otherwise look for for classpath:/brooklyn/default.catalog.bom --
+ // there is one on the classpath which says to scan, and provides a few templates;
+ // if one is supplied by user in the conf/ dir that will override the item from the classpath
+ // (TBD - we might want to scan for all such bom's?)
+
+ catalogUrl = "classpath:/brooklyn/default.catalog.bom";
+ if (new ResourceUtils(this).doesUrlExist(catalogUrl)) {
+ populateInitialFromUri(catalog, catalogUrl, PopulateMode.YAML);
+ return;
+ }
+
+ log.info("No catalog found on classpath or specified; catalog will not be initialized.");
+ return;
+ }
+
+ private void populateInitialFromUri(BasicBrooklynCatalog catalog, String catalogUrl, PopulateMode mode) {
+ log.debug("Loading initial catalog from {}", catalogUrl);
+
+ Exception problem = null;
+ Object result = null;
+
+ String contents = null;
+ try {
+ contents = new ResourceUtils(this).getResourceAsString(catalogUrl);
+ } catch (Exception e) {
+ Exceptions.propagateIfFatal(e);
+ if (problem==null) problem = e;
+ }
+
+ if (contents!=null && (mode==PopulateMode.YAML || mode==PopulateMode.AUTODETECT)) {
+ // try YAML first
+ try {
+ catalog.reset(MutableList.<CatalogItem<?,?>>of());
+ result = catalog.addItems(contents);
+ } catch (Exception e) {
+ Exceptions.propagateIfFatal(e);
+ if (problem==null) problem = e;
+ }
+ }
+
+ if (result==null && contents!=null && (mode==PopulateMode.XML || mode==PopulateMode.AUTODETECT)) {
+ // then try XML
+ problem = populateInitialFromUriXml(catalog, catalogUrl, problem, contents);
+ }
+
+ if (result!=null) {
+ log.debug("Loaded initial catalog from {}: {}", catalogUrl, result);
+ }
+ if (problem!=null) {
+ log.warn("Error importing catalog from " + catalogUrl + ": " + problem, problem);
+ // TODO inform mgmt of error
+ }
+
+ }
+
+ // deprecated XML format
+ @SuppressWarnings("deprecation")
+ private Exception populateInitialFromUriXml(BasicBrooklynCatalog catalog, String catalogUrl, Exception problem, String contents) {
+ CatalogDto dto = null;
+ try {
+ dto = CatalogDto.newDtoFromXmlContents(contents, catalogUrl);
+ problem = null;
+ } catch (Exception e) {
+ Exceptions.propagateIfFatal(e);
+ if (problem==null) problem = e;
+ }
+ if (dto!=null) {
+ catalog.reset(dto);
+ }
+ return problem;
+ }
+
+ boolean hasRunAdditions = false;
+ protected void populateAdditions(BasicBrooklynCatalog catalog) {
+ if (Strings.isNonBlank(additionsUri)) {
+ if (disallowLocal) {
+ if (!hasRunAdditions) {
+ log.warn("CLI additions supplied but not supported in disallow-local mode; ignoring.");
+ }
+ return;
+ }
+ if (!hasRunAdditions) {
+ log.debug("Adding to catalog from CLI: "+additionsUri+" (force: "+force+")");
+ }
+ Iterable<? extends CatalogItem<?, ?>> items = catalog.addItems(
+ new ResourceUtils(this).getResourceAsString(additionsUri), force);
+
+ if (!hasRunAdditions)
+ log.debug("Added to catalog from CLI: "+items);
+ else
+ log.debug("Added to catalog from CLI: count "+Iterables.size(items));
+
+ hasRunAdditions = true;
+ }
+ }
+
+ protected void populateViaCallbacks(BasicBrooklynCatalog catalog) {
+ for (Function<CatalogInitialization, Void> callback: callbacks)
+ callback.apply(this);
+ }
+
+ private boolean setFromCatalogLoadMode = false;
+
+ /** @deprecated since introduced in 0.7.0, only for legacy compatibility with
+ * {@link CatalogLoadMode} {@link BrooklynServerConfig#CATALOG_LOAD_MODE},
+ * allowing control of catalog loading from a brooklyn property */
+ @Deprecated
+ public void applyCatalogLoadMode() {
+ if (setFromCatalogLoadMode) return;
+ setFromCatalogLoadMode = true;
+ Maybe<Object> clmm = ((ManagementContextInternal)managementContext).getConfig().getConfigRaw(BrooklynServerConfig.CATALOG_LOAD_MODE, false);
+ if (clmm.isAbsent()) return;
+ brooklyn.catalog.CatalogLoadMode clm = TypeCoercions.coerce(clmm.get(), brooklyn.catalog.CatalogLoadMode.class);
+ log.warn("Legacy CatalogLoadMode "+clm+" set: applying, but this should be changed to use new CLI --catalogXxx commands");
+ switch (clm) {
+ case LOAD_BROOKLYN_CATALOG_URL:
+ reset = true;
+ break;
+ case LOAD_BROOKLYN_CATALOG_URL_IF_NO_PERSISTED_STATE:
+ // now the default
+ break;
+ case LOAD_PERSISTED_STATE:
+ disallowLocal = true;
+ break;
+ }
+ }
+
+ /** makes the catalog, warning if persistence is on and hasn't run yet
+ * (as the catalog will be subsequently replaced) */
+ public void populateBestEffort(BasicBrooklynCatalog catalog) {
+ synchronized (mutex) {
+ if (hasRunOfficial || hasRunBestEffort || isPopulating) return;
+ // if a thread calls back in to this, ie calling to it from a getCatalog() call while populating,
+ // it will own the mutex and observe isRunningBestEffort, returning quickly
+ isPopulating = true;
+ try {
+ if (isStartingUp) {
+ log.warn("Catalog access requested when not yet initialized; populating best effort rather than through recommended pathway. Catalog data may be replaced subsequently.");
+ }
+ populateCatalog(catalog, true, true, null);
+ } finally {
+ hasRunBestEffort = true;
+ isPopulating = false;
+ }
+ }
+ }
+
+ public void setStartingUp(boolean isStartingUp) {
--- End diff --
The lifecycle of this class is quite hard to follow. I presume that the caller is supposed to call `setStartingUp(true)` when first initialising, and `setStartingUp(false)` when done (but to not call that on rebind?). However, I don't feel confident about understanding that context by just looking at this code.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by aledsage <gi...@git.apache.org>.
Github user aledsage commented on a diff in the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#discussion_r30544663
--- Diff: core/src/main/java/brooklyn/catalog/internal/CatalogInitialization.java ---
@@ -0,0 +1,377 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package brooklyn.catalog.internal;
+
+import java.io.File;
+import java.util.Collection;
+import java.util.List;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import brooklyn.catalog.CatalogItem;
+import brooklyn.config.BrooklynServerConfig;
+import brooklyn.management.ManagementContext;
+import brooklyn.management.ManagementContextInjectable;
+import brooklyn.management.internal.ManagementContextInternal;
+import brooklyn.util.ResourceUtils;
+import brooklyn.util.collections.MutableList;
+import brooklyn.util.exceptions.Exceptions;
+import brooklyn.util.exceptions.FatalRuntimeException;
+import brooklyn.util.exceptions.RuntimeInterruptedException;
+import brooklyn.util.flags.TypeCoercions;
+import brooklyn.util.guava.Maybe;
+import brooklyn.util.net.Urls;
+import brooklyn.util.text.Strings;
+
+import com.google.common.annotations.Beta;
+import com.google.common.base.Function;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.Iterables;
+
+@Beta
+public class CatalogInitialization implements ManagementContextInjectable {
+
+ /*
+
+ A1) if not persisting, go to B1
+ A2) if --catalog-reset, delete persisted catalog items
+ A3) read persisted catalog items (possibly deleted in A2), go to C1
+ A4) go to B1
+
+ B1) look for --catalog-initial, if so read it, then go to C1
+ B2) look for BrooklynServerConfig.BROOKLYN_CATALOG_URL, if so, read it, supporting YAML or XML (warning if XML), then go to C1
+ B3) look for ~/.brooklyn/catalog.bom, if exists, read it then go to C1
+ B4) look for ~/.brooklyn/brooklyn.xml, if exists, warn, read it then go to C1
+ B5) read all classpath://brooklyn/default.catalog.bom items, if they exist (and for now they will)
+ B6) go to C1
+
+ C1) if --catalog-add, read and add those items
+
+ D1) if persisting, read the rest of the persisted items (entities etc)
+
+ */
+
+ private static final Logger log = LoggerFactory.getLogger(CatalogInitialization.class);
+
+ String initialUri;
+ boolean reset;
+ String additionsUri;
+ boolean force;
+
+ boolean disallowLocal = false;
+ List<Function<CatalogInitialization, Void>> callbacks = MutableList.of();
+ boolean hasRunBestEffort = false, hasRunOfficial = false, isPopulating = false;
+
+ ManagementContext managementContext;
+ boolean isStartingUp = false;
+ boolean failOnStartupErrors = false;
+
+ Object mutex = new Object();
+
+ public CatalogInitialization(String initialUri, boolean reset, String additionUri, boolean force) {
+ this.initialUri = initialUri;
+ this.reset = reset;
+ this.additionsUri = additionUri;
+ this.force = force;
+ }
+
+ public CatalogInitialization() {
+ this(null, false, null, false);
+ }
+
+ public void injectManagementContext(ManagementContext managementContext) {
+ if (this.managementContext!=null && managementContext!=null && !this.managementContext.equals(managementContext))
+ throw new IllegalStateException("Cannot switch management context of "+this+"; from "+this.managementContext+" to "+managementContext);
+ this.managementContext = managementContext;
+ }
+
+ public ManagementContext getManagementContext() {
+ return Preconditions.checkNotNull(managementContext, "management context has not been injected into "+this);
+ }
+
+ public CatalogInitialization addPopulationCallback(Function<CatalogInitialization, Void> callback) {
+ callbacks.add(callback);
+ return this;
+ }
+
+ public boolean isInitialResetRequested() {
+ return reset;
+ }
+
+ public boolean hasRunOfficial() { return hasRunOfficial; }
+ public boolean hasRunIncludingBestEffort() { return hasRunOfficial || hasRunBestEffort; }
+
+ /** makes or updates the mgmt catalog, based on the settings in this class */
+ public void populateCatalog(boolean needsInitial, Collection<CatalogItem<?, ?>> optionalItemsForResettingCatalog) {
+ try {
+ isPopulating = true;
--- End diff --
To @robertgmoss's point, why deliberately do the `isPopulating` and the finally block outside of the mutex?
For example, the other method to be synchronized (`populateBestEffort `) grabs the mutex and the does:
`if (hasRunOfficial || hasRunBestEffort || isPopulating) return;`
Therefore `populateBestEffort ` return immediately even though this method hasn't actually populated the catalog yet.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by aledsage <gi...@git.apache.org>.
Github user aledsage commented on a diff in the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#discussion_r30586220
--- Diff: usage/cli/src/main/java/brooklyn/cli/Main.java ---
@@ -266,7 +290,7 @@ private String readLine(InputStream in) throws IOException {
// looks like: {@linktourl https://gist.github.com/47066f72d6f6f79b953e}
@Beta
@Option(name = { "-sk", "--stopOnKeyPress" },
- description = "After startup, shutdown on user text entry (default false)")
+ description = "Shutdown immediately on user text entry after startup (useful for debugging and demos)")
--- End diff --
Definite improvement; we should also remove mention of Whirr in comment above this (can be in a separate PR)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by neykov <gi...@git.apache.org>.
Github user neykov commented on the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#issuecomment-105228200
Saw the commit and the change are good, but my reading of the code and subsequent testing shows that `scanJavaAnnotations` doesn't work for bundles. Could you add a test to prove me wrong :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by aledsage <gi...@git.apache.org>.
Github user aledsage commented on a diff in the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#discussion_r30583877
--- Diff: core/src/main/java/brooklyn/management/internal/AbstractManagementContext.java ---
@@ -443,6 +441,22 @@ public void setManagementNodeUri(URI uri) {
return uri;
}
+ @Override
+ public CatalogInitialization getCatalogInitialization() {
+ if (catalogInitialization!=null) return catalogInitialization;
+ synchronized (this) {
+ if (catalogInitialization!=null) return catalogInitialization;
+ CatalogInitialization ci = new CatalogInitialization();
+ setCatalogInitialization(ci);
+ return ci;
+ }
+ }
+
+ @Override
+ public synchronized void setCatalogInitialization(CatalogInitialization catalogInitialization) {
+ if (catalogInitialization!=null) catalogInitialization.injectManagementContext(this);
--- End diff --
I'd much prefer us to fail if the user passes in null, rather than failing on an NPE in some unrelated code/thread later.
I'd do something like:
this.catalogInitialization = checkNotNull(catalogInitialization, "catalogInitialization");
catalogInitialization.injectManagementContext(this);
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by aledsage <gi...@git.apache.org>.
Github user aledsage commented on a diff in the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#discussion_r30545285
--- Diff: core/src/main/java/brooklyn/catalog/internal/CatalogInitialization.java ---
@@ -0,0 +1,377 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package brooklyn.catalog.internal;
+
+import java.io.File;
+import java.util.Collection;
+import java.util.List;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import brooklyn.catalog.CatalogItem;
+import brooklyn.config.BrooklynServerConfig;
+import brooklyn.management.ManagementContext;
+import brooklyn.management.ManagementContextInjectable;
+import brooklyn.management.internal.ManagementContextInternal;
+import brooklyn.util.ResourceUtils;
+import brooklyn.util.collections.MutableList;
+import brooklyn.util.exceptions.Exceptions;
+import brooklyn.util.exceptions.FatalRuntimeException;
+import brooklyn.util.exceptions.RuntimeInterruptedException;
+import brooklyn.util.flags.TypeCoercions;
+import brooklyn.util.guava.Maybe;
+import brooklyn.util.net.Urls;
+import brooklyn.util.text.Strings;
+
+import com.google.common.annotations.Beta;
+import com.google.common.base.Function;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.Iterables;
+
+@Beta
+public class CatalogInitialization implements ManagementContextInjectable {
+
+ /*
+
+ A1) if not persisting, go to B1
+ A2) if --catalog-reset, delete persisted catalog items
+ A3) read persisted catalog items (possibly deleted in A2), go to C1
+ A4) go to B1
+
+ B1) look for --catalog-initial, if so read it, then go to C1
+ B2) look for BrooklynServerConfig.BROOKLYN_CATALOG_URL, if so, read it, supporting YAML or XML (warning if XML), then go to C1
+ B3) look for ~/.brooklyn/catalog.bom, if exists, read it then go to C1
+ B4) look for ~/.brooklyn/brooklyn.xml, if exists, warn, read it then go to C1
+ B5) read all classpath://brooklyn/default.catalog.bom items, if they exist (and for now they will)
+ B6) go to C1
+
+ C1) if --catalog-add, read and add those items
+
+ D1) if persisting, read the rest of the persisted items (entities etc)
+
+ */
+
+ private static final Logger log = LoggerFactory.getLogger(CatalogInitialization.class);
+
+ String initialUri;
+ boolean reset;
+ String additionsUri;
+ boolean force;
+
+ boolean disallowLocal = false;
+ List<Function<CatalogInitialization, Void>> callbacks = MutableList.of();
+ boolean hasRunBestEffort = false, hasRunOfficial = false, isPopulating = false;
+
+ ManagementContext managementContext;
+ boolean isStartingUp = false;
+ boolean failOnStartupErrors = false;
+
+ Object mutex = new Object();
+
+ public CatalogInitialization(String initialUri, boolean reset, String additionUri, boolean force) {
+ this.initialUri = initialUri;
+ this.reset = reset;
+ this.additionsUri = additionUri;
+ this.force = force;
+ }
+
+ public CatalogInitialization() {
+ this(null, false, null, false);
+ }
+
+ public void injectManagementContext(ManagementContext managementContext) {
+ if (this.managementContext!=null && managementContext!=null && !this.managementContext.equals(managementContext))
+ throw new IllegalStateException("Cannot switch management context of "+this+"; from "+this.managementContext+" to "+managementContext);
+ this.managementContext = managementContext;
+ }
+
+ public ManagementContext getManagementContext() {
+ return Preconditions.checkNotNull(managementContext, "management context has not been injected into "+this);
+ }
+
+ public CatalogInitialization addPopulationCallback(Function<CatalogInitialization, Void> callback) {
+ callbacks.add(callback);
+ return this;
+ }
+
+ public boolean isInitialResetRequested() {
+ return reset;
+ }
+
+ public boolean hasRunOfficial() { return hasRunOfficial; }
+ public boolean hasRunIncludingBestEffort() { return hasRunOfficial || hasRunBestEffort; }
+
+ /** makes or updates the mgmt catalog, based on the settings in this class */
+ public void populateCatalog(boolean needsInitial, Collection<CatalogItem<?, ?>> optionalItemsForResettingCatalog) {
+ try {
+ isPopulating = true;
+ synchronized (mutex) {
+ BasicBrooklynCatalog catalog = (BasicBrooklynCatalog) managementContext.getCatalog();
+ if (!catalog.getCatalog().isLoaded()) {
+ catalog.load();
+ } else {
+ if (needsInitial && (hasRunOfficial || hasRunBestEffort)) {
+ // an indication that something caused it to load early; not severe, but unusual
+ log.warn("Catalog initialization has not properly run but management context has a catalog; re-populating, possibly overwriting items installed during earlier access (it may have been an early web request)");
+ catalog.reset(ImmutableList.<CatalogItem<?,?>>of());
+ }
+ }
+ hasRunOfficial = true;
+
+ populateCatalog(catalog, needsInitial, true, optionalItemsForResettingCatalog);
+ }
+ } finally {
+ hasRunOfficial = true;
+ isPopulating = false;
+ }
+ }
+
+ private void populateCatalog(BasicBrooklynCatalog catalog, boolean needsInitial, boolean runCallbacks, Collection<CatalogItem<?, ?>> optionalItemsForResettingCatalog) {
+ applyCatalogLoadMode();
+
+ if (optionalItemsForResettingCatalog!=null) {
+ catalog.reset(optionalItemsForResettingCatalog);
+ }
+
+ if (needsInitial) {
+ populateInitial(catalog);
+ }
+
+ populateAdditions(catalog);
+
+ if (runCallbacks) {
+ populateViaCallbacks(catalog);
+ }
+ }
+
+ private enum PopulateMode { YAML, XML, AUTODETECT }
+
+ protected void populateInitial(BasicBrooklynCatalog catalog) {
+ if (disallowLocal) {
+ if (!hasRunOfficial()) {
+ log.debug("CLI initial catalog not being read with disallow-local mode set.");
+ }
+ return;
+ }
+
+// B1) look for --catalog-initial, if so read it, then go to C1
+// B2) look for BrooklynServerConfig.BROOKLYN_CATALOG_URL, if so, read it, supporting YAML or XML (warning if XML), then go to C1
+// B3) look for ~/.brooklyn/catalog.bom, if exists, read it then go to C1
+// B4) look for ~/.brooklyn/brooklyn.xml, if exists, warn, read it then go to C1
+// B5) read all classpath://brooklyn/default.catalog.bom items, if they exist (and for now they will)
+// B6) go to C1
+
+ if (initialUri!=null) {
+ populateInitialFromUri(catalog, initialUri, PopulateMode.AUTODETECT);
+ return;
+ }
+
+ String catalogUrl = managementContext.getConfig().getConfig(BrooklynServerConfig.BROOKLYN_CATALOG_URL);
+ if (Strings.isNonBlank(catalogUrl)) {
+ populateInitialFromUri(catalog, catalogUrl, PopulateMode.AUTODETECT);
+ return;
+ }
+
+ catalogUrl = Urls.mergePaths(BrooklynServerConfig.getMgmtBaseDir( managementContext.getConfig() ), "catalog.bom");
--- End diff --
If getMgmtBaseDir returns a dir, then clearer to use `Os.mergePaths()`. If it returns a URL, then can't do `new File(catalogUrl).exists()` below, I presume.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by neykov <gi...@git.apache.org>.
Github user neykov commented on the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#issuecomment-104423104
Is `riak.png`renamed on purpose?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by robertgmoss <gi...@git.apache.org>.
Github user robertgmoss commented on a diff in the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#discussion_r30495366
--- Diff: core/src/main/java/brooklyn/catalog/internal/CatalogInitialization.java ---
@@ -0,0 +1,377 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package brooklyn.catalog.internal;
+
+import java.io.File;
+import java.util.Collection;
+import java.util.List;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import brooklyn.catalog.CatalogItem;
+import brooklyn.config.BrooklynServerConfig;
+import brooklyn.management.ManagementContext;
+import brooklyn.management.ManagementContextInjectable;
+import brooklyn.management.internal.ManagementContextInternal;
+import brooklyn.util.ResourceUtils;
+import brooklyn.util.collections.MutableList;
+import brooklyn.util.exceptions.Exceptions;
+import brooklyn.util.exceptions.FatalRuntimeException;
+import brooklyn.util.exceptions.RuntimeInterruptedException;
+import brooklyn.util.flags.TypeCoercions;
+import brooklyn.util.guava.Maybe;
+import brooklyn.util.net.Urls;
+import brooklyn.util.text.Strings;
+
+import com.google.common.annotations.Beta;
+import com.google.common.base.Function;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.Iterables;
+
+@Beta
+public class CatalogInitialization implements ManagementContextInjectable {
--- End diff --
I'm concerned about the Thread safety of this class. There is a mutex, but it is not clear what state it is guarding; all reads and writes of the state it is guarding should be guarded by this lock to ensure visibility - that is not happening, this much is clear. Additionally, the state should be made private to prevent read/write access without the lock.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by neykov <gi...@git.apache.org>.
Github user neykov commented on a diff in the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#discussion_r30801647
--- Diff: core/src/main/java/brooklyn/entity/rebind/RebindIteration.java ---
@@ -341,38 +340,64 @@ protected void rebuildCatalog() {
}
}
}
+
+ // See notes in CatalogInitialization
- // Register catalogue items with the management context. Loads the bundles in the OSGi framework.
- CatalogLoadMode catalogLoadMode = managementContext.getConfig().getConfig(BrooklynServerConfig.CATALOG_LOAD_MODE);
+ Collection<CatalogItem<?, ?>> catalogItems = rebindContext.getCatalogItems();
+ CatalogInitialization catInit = ((ManagementContextInternal)managementContext).getCatalogInitialization();
+ catInit.applyCatalogLoadMode();
+ Collection<CatalogItem<?,?>> itemsForResettingCatalog = null;
+ boolean needsInitialCatalog;
if (rebindManager.persistCatalogItemsEnabled) {
- boolean shouldResetCatalog = catalogLoadMode == CatalogLoadMode.LOAD_PERSISTED_STATE
- || (!isEmpty && catalogLoadMode == CatalogLoadMode.LOAD_BROOKLYN_CATALOG_URL_IF_NO_PERSISTED_STATE);
- boolean shouldLoadDefaultCatalog = catalogLoadMode == CatalogLoadMode.LOAD_BROOKLYN_CATALOG_URL
- || (isEmpty && catalogLoadMode == CatalogLoadMode.LOAD_BROOKLYN_CATALOG_URL_IF_NO_PERSISTED_STATE);
- if (shouldResetCatalog) {
- // Reset catalog with previously persisted state
- logRebindingDebug("RebindManager resetting management context catalog to previously persisted state");
- managementContext.getCatalog().reset(rebindContext.getCatalogItems());
- } else if (shouldLoadDefaultCatalog) {
- // Load catalogue as normal
- // TODO in read-only mode, should do this less frequently than entities etc
- logRebindingDebug("RebindManager loading default catalog");
- ((BasicBrooklynCatalog) managementContext.getCatalog()).resetCatalogToContentsAtConfiguredUrl();
+ if (!catInit.hasRunOfficial() && catInit.isInitialResetRequested()) {
--- End diff --
After discussion - clarify "official" as being first master rebind OR (in non-rebind mode) the load triggered by CLI
Then, feels like the cases should be
```
if (catalog persistence enabled) {
if (!hasBeenMaster) {
if (reset || isEmpty) {
don't use persistence catalog items
init initial catalog
}
init add'tl catalog
}
} else {//catalog persistence disabled
if (didn't initialize the catalog) {
init initial catalog
init add'tl catalog
}
}
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by aledsage <gi...@git.apache.org>.
Github user aledsage commented on a diff in the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#discussion_r30583177
--- Diff: core/src/main/java/brooklyn/entity/rebind/persister/BrooklynMementoPersisterToObjectStore.java ---
@@ -560,9 +563,18 @@ public void checkpoint(BrooklynMemento newMemento, PersistenceExceptionHandler e
public void delta(Delta delta, PersistenceExceptionHandler exceptionHandler) {
checkWritesAllowed();
+ while (!queuedDeltas.isEmpty()) {
+ Delta extraDelta = queuedDeltas.remove(0);
+ doDelta(extraDelta, exceptionHandler, false);
--- End diff --
why does this pass `false` for `previouslyQueued`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by aledsage <gi...@git.apache.org>.
Github user aledsage commented on a diff in the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#discussion_r30501739
--- Diff: core/src/main/java/brooklyn/catalog/internal/BasicBrooklynCatalog.java ---
@@ -986,7 +1045,12 @@ private DeploymentPlan makePlanFromYaml(String yaml) {
}
private CatalogItem<?,?> addItemDto(CatalogItemDtoAbstract<?, ?> itemDto, boolean forceUpdate) {
- checkItemNotExists(itemDto, forceUpdate);
+ CatalogItem<?, ?> existingDto = checkItemIsDuplicateOrDisallowed(itemDto, true, forceUpdate);
--- End diff --
Name `checkItemIsDuplicateOrDisallowed` is confusing. The `check...` prefix elsewhere follows the guava convention: it throws an exception if the check fails (e.g. `checkNotNull` throws if the condition "not null" fails). Reading this suggests we would throw if "is duplicate or disallowed" fails. But it's the opposite: we're checking that either it's not a duplicate or is allowed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by ahgittin <gi...@git.apache.org>.
Github user ahgittin commented on a diff in the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#discussion_r30967595
--- Diff: core/src/main/java/brooklyn/entity/rebind/RebindIteration.java ---
@@ -341,38 +340,64 @@ protected void rebuildCatalog() {
}
}
}
+
+ // See notes in CatalogInitialization
- // Register catalogue items with the management context. Loads the bundles in the OSGi framework.
- CatalogLoadMode catalogLoadMode = managementContext.getConfig().getConfig(BrooklynServerConfig.CATALOG_LOAD_MODE);
+ Collection<CatalogItem<?, ?>> catalogItems = rebindContext.getCatalogItems();
+ CatalogInitialization catInit = ((ManagementContextInternal)managementContext).getCatalogInitialization();
+ catInit.applyCatalogLoadMode();
+ Collection<CatalogItem<?,?>> itemsForResettingCatalog = null;
+ boolean needsInitialCatalog;
if (rebindManager.persistCatalogItemsEnabled) {
- boolean shouldResetCatalog = catalogLoadMode == CatalogLoadMode.LOAD_PERSISTED_STATE
- || (!isEmpty && catalogLoadMode == CatalogLoadMode.LOAD_BROOKLYN_CATALOG_URL_IF_NO_PERSISTED_STATE);
- boolean shouldLoadDefaultCatalog = catalogLoadMode == CatalogLoadMode.LOAD_BROOKLYN_CATALOG_URL
- || (isEmpty && catalogLoadMode == CatalogLoadMode.LOAD_BROOKLYN_CATALOG_URL_IF_NO_PERSISTED_STATE);
- if (shouldResetCatalog) {
- // Reset catalog with previously persisted state
- logRebindingDebug("RebindManager resetting management context catalog to previously persisted state");
- managementContext.getCatalog().reset(rebindContext.getCatalogItems());
- } else if (shouldLoadDefaultCatalog) {
- // Load catalogue as normal
- // TODO in read-only mode, should do this less frequently than entities etc
- logRebindingDebug("RebindManager loading default catalog");
- ((BasicBrooklynCatalog) managementContext.getCatalog()).resetCatalogToContentsAtConfiguredUrl();
+ if (!catInit.hasRunOfficial() && catInit.isInitialResetRequested()) {
+ String message = "RebindManager resetting catalog on first run (catalog persistence enabled, but reset explicitly specified). ";
+ if (catalogItems.isEmpty()) {
+ message += "Catalog was empty anyway.";
+ } else {
+ message += "Deleting "+catalogItems.size()+" persisted catalog item"+Strings.s(catalogItems)+": "+catalogItems;
+ if (shouldLogRebinding()) {
+ LOG.info(message);
+ }
+ }
+ logRebindingDebug(message);
+
+ itemsForResettingCatalog = MutableList.<CatalogItem<?,?>>of();
+
+ PersisterDeltaImpl delta = new PersisterDeltaImpl();
+ delta.removedCatalogItemIds.addAll(mementoRawData.getCatalogItems().keySet());
+ getPersister().queueDelta(delta);
+
+ mementoRawData.clearCatalogItems();
--- End diff --
yes to `rebindContext`, good spot. as for the level of abstraction it feels right as is to me, with the persister being relatively dumb apart from persisting things, all notions of what to be persisted is controlled from here. it is too bad to lose immutability on `mementoRawData` but the alternative i think would be either to replace it (which would be bad since the rawData might have been passed elsewhere) or analyse the catalog earlier (introducing new phases). i think it makes sense that the rawData can be mutable when needed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by aledsage <gi...@git.apache.org>.
Github user aledsage commented on a diff in the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#discussion_r30500187
--- Diff: core/src/main/java/brooklyn/catalog/internal/BasicBrooklynCatalog.java ---
@@ -186,12 +192,7 @@ public CatalogDo getCatalog() {
return null;
}
- String versionedId = CatalogUtils.getVersionedId(symbolicName, fixedVersionId);
- CatalogItemDo<?, ?> item = null;
- //TODO should remove "manual additions" bucket; just have one map a la osgi
- if (manualAdditionsCatalog!=null) item = manualAdditionsCatalog.getIdCache().get(versionedId);
- if (item == null) item = catalog.getIdCache().get(versionedId);
- return item;
+ return catalog.getIdCache().get( CatalogUtils.getVersionedId(symbolicName, fixedVersionId) );
--- End diff --
I don't follow the use of `manualAdditionsCatalog`. Can we safely ignore it in this method?
I see manualAdditionsCatalog is used in the deprecated `addItem(CatalogItem<?,?> item)`, but `catalog` is not.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by ahgittin <gi...@git.apache.org>.
Github user ahgittin commented on a diff in the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#discussion_r30589701
--- Diff: usage/dist/src/main/dist/conf/brooklyn/default.catalog.bom ---
@@ -0,0 +1,41 @@
+
+# this catalog bom is an illustration supplying a few useful sample items
+# and templates to get started using Brooklyn
+
+brooklyn.catalog:
+ version: 0.7.0-SNAPSHOT # BROOKLYN_VERSION
+ items:
+
+ # load everything in the classpath with a @Catalog annotation
+ - scanJavaAnnotations: true
+
+ - id: server
+ description: |
+ Provision a server, with customizable provisioning.properties and credentials installed,
+ but no other special software process or scripts executed.
+ item:
+ type: brooklyn.entity.basic.EmptySoftwareProcess
--- End diff --
templates *must* have a services block -- everything else generally should *not* -- as per current catalog description
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by ahgittin <gi...@git.apache.org>.
Github user ahgittin commented on the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#issuecomment-104440320
yes `riak.png` is deliberately moved, from `src/main/java` to `src/main/resources`. maven sometimes ignores it in the former location so the icon doesn't show up when we view it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by ahgittin <gi...@git.apache.org>.
Github user ahgittin commented on a diff in the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#discussion_r30659421
--- Diff: core/src/main/java/brooklyn/entity/rebind/RebindIteration.java ---
@@ -341,38 +340,67 @@ protected void rebuildCatalog() {
}
}
}
+
+ // See notes in CatalogInitialization
- // Register catalogue items with the management context. Loads the bundles in the OSGi framework.
- CatalogLoadMode catalogLoadMode = managementContext.getConfig().getConfig(BrooklynServerConfig.CATALOG_LOAD_MODE);
+ Collection<CatalogItem<?, ?>> catalogItems = rebindContext.getCatalogItems();
+ CatalogInitialization catInit = ((ManagementContextInternal)managementContext).getCatalogInitialization();
+ catInit.injectManagementContext(managementContext);
+ catInit.applyCatalogLoadMode();
+ Collection<CatalogItem<?,?>> itemsForResettingCatalog = null;
+ boolean needsInitialCatalog;
if (rebindManager.persistCatalogItemsEnabled) {
- boolean shouldResetCatalog = catalogLoadMode == CatalogLoadMode.LOAD_PERSISTED_STATE
- || (!isEmpty && catalogLoadMode == CatalogLoadMode.LOAD_BROOKLYN_CATALOG_URL_IF_NO_PERSISTED_STATE);
- boolean shouldLoadDefaultCatalog = catalogLoadMode == CatalogLoadMode.LOAD_BROOKLYN_CATALOG_URL
- || (isEmpty && catalogLoadMode == CatalogLoadMode.LOAD_BROOKLYN_CATALOG_URL_IF_NO_PERSISTED_STATE);
- if (shouldResetCatalog) {
- // Reset catalog with previously persisted state
- logRebindingDebug("RebindManager resetting management context catalog to previously persisted state");
- managementContext.getCatalog().reset(rebindContext.getCatalogItems());
- } else if (shouldLoadDefaultCatalog) {
- // Load catalogue as normal
- // TODO in read-only mode, should do this less frequently than entities etc
- logRebindingDebug("RebindManager loading default catalog");
- ((BasicBrooklynCatalog) managementContext.getCatalog()).resetCatalogToContentsAtConfiguredUrl();
+ if (!catInit.hasRunOfficial() && catInit.isInitialResetRequested()) {
+ String message = "RebindManager resetting catalog on first run (catalog persistence enabled, but reset explicitly specified). ";
+ if (catalogItems.isEmpty()) {
+ message += "Catalog was empty anyway.";
+ } else {
+ message += "Deleting "+catalogItems.size()+" persisted catalog item"+Strings.s(catalogItems)+": "+catalogItems;
+ if (shouldLogRebinding()) {
+ LOG.info(message);
+ }
+ }
+ logRebindingDebug(message);
+
+ itemsForResettingCatalog = MutableList.<CatalogItem<?,?>>of();
+
+ PersisterDeltaImpl delta = new PersisterDeltaImpl();
+ for (String catalogItemId: mementoRawData.getCatalogItems().keySet()) {
+ delta.removedCatalogItemIds.add(catalogItemId);
+ }
+ getPersister().queueDelta(delta);
+
+ mementoRawData.clearCatalogItems();
+ needsInitialCatalog = true;
} else {
- // Management context should have taken care of loading the catalogue
- Collection<CatalogItem<?, ?>> catalogItems = rebindContext.getCatalogItems();
- String message = "RebindManager not resetting catalog to persisted state. Catalog load mode is {}.";
- if (!catalogItems.isEmpty() && shouldLogRebinding()) {
- LOG.info(message + " There {} {} item{} persisted.", new Object[]{
- catalogLoadMode, catalogItems.size() == 1 ? "was" : "were", catalogItems.size(), Strings.s(catalogItems)});
- } else if (LOG.isDebugEnabled()) {
- logRebindingDebug(message, catalogLoadMode);
+ if (!isEmpty) {
+ logRebindingDebug("RebindManager clearing local catalog and loading from persisted state");
+ itemsForResettingCatalog = rebindContext.getCatalogItems();
+ needsInitialCatalog = false;
+ } else {
+ if (catInit.hasRunOfficial()) {
+ logRebindingDebug("RebindManager will re-add any new items (persisted state empty)");
+ needsInitialCatalog = false;
+ } else {
+ logRebindingDebug("RebindManager loading initial catalog locally because persisted state empty");
+ needsInitialCatalog = true;
+ }
}
}
- // TODO destroy old (as above)
} else {
- logRebindingDebug("RebindManager not resetting catalog because catalog persistence is disabled");
+ if (catInit.hasRunOfficial()) {
+ logRebindingDebug("RebindManager skipping catalog init because it has already run (catalog persistence disabled)");
+ needsInitialCatalog = false;
+ } else {
+ logRebindingDebug("RebindManager will initialize catalog locally because catalog persistence is disabled");
+ needsInitialCatalog = true;
+ }
}
+
+ // TODO in read-only mode, perhaps do this less frequently than entities etc ?
+ // both in RW and in RO mode, the first run reads the initialization data;
+ // maybe not desired for RO as it defers problems, although if it's standalone it is desired
+ catInit.populateCatalog(needsInitialCatalog, itemsForResettingCatalog);
--- End diff --
meh i dislike `Optional` except when null is ambiguous or you want to chain or maintain an error on access (and in that last case `Maybe` is better).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by aledsage <gi...@git.apache.org>.
Github user aledsage commented on a diff in the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#discussion_r30590383
--- Diff: utils/common/src/main/java/brooklyn/util/javalang/AggregateClassLoader.java ---
@@ -69,8 +70,25 @@ public void add(int index, ClassLoader classLoader) {
if (classLoader != null) classLoaders.add(index, classLoader);
}
- /** Returns the _live_ (and modifiable) list of classloaders
- * @return */
+ /** Resets the classloader shown here to be the given set */
+ public void reset(Collection<? extends ClassLoader> newClassLoaders) {
+ synchronized (classLoaders) {
--- End diff --
Clever adding all to end, and then removing `count` from start. However, it will break if there is a concurrent call to `add(int index, ClassLoader classLoader)`.
Personally I'd prefer a simpler synchronization model, that does not try to be clever.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by ahgittin <gi...@git.apache.org>.
Github user ahgittin commented on a diff in the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#discussion_r30659149
--- Diff: docs/guide/ops/catalog/index.md ---
@@ -284,6 +291,19 @@ When referencing a blueprint, if a version number is not specified
the latest non-snapshot version will be loaded when an entity is instantiated.
+### CLI Options
+
+The `brooklyn` CLI includes several commands for working with the catalog.
+
+* `--catalogAdd <file.bom>` will add the catalog items in the `bom` file
--- End diff --
i expect we'll be adding a lot more, explaining this, and converting `*.yaml` to `*.bom` in near future
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by aledsage <gi...@git.apache.org>.
Github user aledsage commented on a diff in the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#discussion_r30546607
--- Diff: core/src/main/java/brooklyn/catalog/internal/CatalogInitialization.java ---
@@ -0,0 +1,377 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package brooklyn.catalog.internal;
+
+import java.io.File;
+import java.util.Collection;
+import java.util.List;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import brooklyn.catalog.CatalogItem;
+import brooklyn.config.BrooklynServerConfig;
+import brooklyn.management.ManagementContext;
+import brooklyn.management.ManagementContextInjectable;
+import brooklyn.management.internal.ManagementContextInternal;
+import brooklyn.util.ResourceUtils;
+import brooklyn.util.collections.MutableList;
+import brooklyn.util.exceptions.Exceptions;
+import brooklyn.util.exceptions.FatalRuntimeException;
+import brooklyn.util.exceptions.RuntimeInterruptedException;
+import brooklyn.util.flags.TypeCoercions;
+import brooklyn.util.guava.Maybe;
+import brooklyn.util.net.Urls;
+import brooklyn.util.text.Strings;
+
+import com.google.common.annotations.Beta;
+import com.google.common.base.Function;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.Iterables;
+
+@Beta
+public class CatalogInitialization implements ManagementContextInjectable {
+
+ /*
+
+ A1) if not persisting, go to B1
+ A2) if --catalog-reset, delete persisted catalog items
+ A3) read persisted catalog items (possibly deleted in A2), go to C1
+ A4) go to B1
+
+ B1) look for --catalog-initial, if so read it, then go to C1
+ B2) look for BrooklynServerConfig.BROOKLYN_CATALOG_URL, if so, read it, supporting YAML or XML (warning if XML), then go to C1
+ B3) look for ~/.brooklyn/catalog.bom, if exists, read it then go to C1
+ B4) look for ~/.brooklyn/brooklyn.xml, if exists, warn, read it then go to C1
+ B5) read all classpath://brooklyn/default.catalog.bom items, if they exist (and for now they will)
+ B6) go to C1
+
+ C1) if --catalog-add, read and add those items
+
+ D1) if persisting, read the rest of the persisted items (entities etc)
+
+ */
+
+ private static final Logger log = LoggerFactory.getLogger(CatalogInitialization.class);
+
+ String initialUri;
+ boolean reset;
+ String additionsUri;
+ boolean force;
+
+ boolean disallowLocal = false;
+ List<Function<CatalogInitialization, Void>> callbacks = MutableList.of();
+ boolean hasRunBestEffort = false, hasRunOfficial = false, isPopulating = false;
+
+ ManagementContext managementContext;
+ boolean isStartingUp = false;
+ boolean failOnStartupErrors = false;
+
+ Object mutex = new Object();
+
+ public CatalogInitialization(String initialUri, boolean reset, String additionUri, boolean force) {
+ this.initialUri = initialUri;
+ this.reset = reset;
+ this.additionsUri = additionUri;
+ this.force = force;
+ }
+
+ public CatalogInitialization() {
+ this(null, false, null, false);
+ }
+
+ public void injectManagementContext(ManagementContext managementContext) {
+ if (this.managementContext!=null && managementContext!=null && !this.managementContext.equals(managementContext))
+ throw new IllegalStateException("Cannot switch management context of "+this+"; from "+this.managementContext+" to "+managementContext);
+ this.managementContext = managementContext;
+ }
+
+ public ManagementContext getManagementContext() {
+ return Preconditions.checkNotNull(managementContext, "management context has not been injected into "+this);
+ }
+
+ public CatalogInitialization addPopulationCallback(Function<CatalogInitialization, Void> callback) {
+ callbacks.add(callback);
+ return this;
+ }
+
+ public boolean isInitialResetRequested() {
+ return reset;
+ }
+
+ public boolean hasRunOfficial() { return hasRunOfficial; }
+ public boolean hasRunIncludingBestEffort() { return hasRunOfficial || hasRunBestEffort; }
+
+ /** makes or updates the mgmt catalog, based on the settings in this class */
+ public void populateCatalog(boolean needsInitial, Collection<CatalogItem<?, ?>> optionalItemsForResettingCatalog) {
+ try {
+ isPopulating = true;
+ synchronized (mutex) {
+ BasicBrooklynCatalog catalog = (BasicBrooklynCatalog) managementContext.getCatalog();
+ if (!catalog.getCatalog().isLoaded()) {
+ catalog.load();
+ } else {
+ if (needsInitial && (hasRunOfficial || hasRunBestEffort)) {
+ // an indication that something caused it to load early; not severe, but unusual
+ log.warn("Catalog initialization has not properly run but management context has a catalog; re-populating, possibly overwriting items installed during earlier access (it may have been an early web request)");
+ catalog.reset(ImmutableList.<CatalogItem<?,?>>of());
+ }
+ }
+ hasRunOfficial = true;
+
+ populateCatalog(catalog, needsInitial, true, optionalItemsForResettingCatalog);
+ }
+ } finally {
+ hasRunOfficial = true;
+ isPopulating = false;
+ }
+ }
+
+ private void populateCatalog(BasicBrooklynCatalog catalog, boolean needsInitial, boolean runCallbacks, Collection<CatalogItem<?, ?>> optionalItemsForResettingCatalog) {
+ applyCatalogLoadMode();
+
+ if (optionalItemsForResettingCatalog!=null) {
+ catalog.reset(optionalItemsForResettingCatalog);
+ }
+
+ if (needsInitial) {
+ populateInitial(catalog);
+ }
+
+ populateAdditions(catalog);
+
+ if (runCallbacks) {
+ populateViaCallbacks(catalog);
+ }
+ }
+
+ private enum PopulateMode { YAML, XML, AUTODETECT }
+
+ protected void populateInitial(BasicBrooklynCatalog catalog) {
+ if (disallowLocal) {
+ if (!hasRunOfficial()) {
+ log.debug("CLI initial catalog not being read with disallow-local mode set.");
+ }
+ return;
+ }
+
+// B1) look for --catalog-initial, if so read it, then go to C1
+// B2) look for BrooklynServerConfig.BROOKLYN_CATALOG_URL, if so, read it, supporting YAML or XML (warning if XML), then go to C1
+// B3) look for ~/.brooklyn/catalog.bom, if exists, read it then go to C1
+// B4) look for ~/.brooklyn/brooklyn.xml, if exists, warn, read it then go to C1
+// B5) read all classpath://brooklyn/default.catalog.bom items, if they exist (and for now they will)
+// B6) go to C1
+
+ if (initialUri!=null) {
+ populateInitialFromUri(catalog, initialUri, PopulateMode.AUTODETECT);
+ return;
+ }
+
+ String catalogUrl = managementContext.getConfig().getConfig(BrooklynServerConfig.BROOKLYN_CATALOG_URL);
+ if (Strings.isNonBlank(catalogUrl)) {
+ populateInitialFromUri(catalog, catalogUrl, PopulateMode.AUTODETECT);
+ return;
+ }
+
+ catalogUrl = Urls.mergePaths(BrooklynServerConfig.getMgmtBaseDir( managementContext.getConfig() ), "catalog.bom");
+ if (new File(catalogUrl).exists()) {
+ populateInitialFromUri(catalog, "file:"+catalogUrl, PopulateMode.YAML);
+ return;
+ }
+
+ catalogUrl = Urls.mergePaths(BrooklynServerConfig.getMgmtBaseDir( managementContext.getConfig() ), "catalog.xml");
+ if (new File(catalogUrl).exists()) {
+ populateInitialFromUri(catalog, "file:"+catalogUrl, PopulateMode.XML);
+ return;
+ }
+
+ // otherwise look for for classpath:/brooklyn/default.catalog.bom --
+ // there is one on the classpath which says to scan, and provides a few templates;
+ // if one is supplied by user in the conf/ dir that will override the item from the classpath
+ // (TBD - we might want to scan for all such bom's?)
+
+ catalogUrl = "classpath:/brooklyn/default.catalog.bom";
+ if (new ResourceUtils(this).doesUrlExist(catalogUrl)) {
+ populateInitialFromUri(catalog, catalogUrl, PopulateMode.YAML);
+ return;
+ }
+
+ log.info("No catalog found on classpath or specified; catalog will not be initialized.");
+ return;
+ }
+
+ private void populateInitialFromUri(BasicBrooklynCatalog catalog, String catalogUrl, PopulateMode mode) {
+ log.debug("Loading initial catalog from {}", catalogUrl);
+
+ Exception problem = null;
+ Object result = null;
+
+ String contents = null;
+ try {
+ contents = new ResourceUtils(this).getResourceAsString(catalogUrl);
+ } catch (Exception e) {
+ Exceptions.propagateIfFatal(e);
+ if (problem==null) problem = e;
+ }
+
+ if (contents!=null && (mode==PopulateMode.YAML || mode==PopulateMode.AUTODETECT)) {
+ // try YAML first
+ try {
+ catalog.reset(MutableList.<CatalogItem<?,?>>of());
+ result = catalog.addItems(contents);
+ } catch (Exception e) {
+ Exceptions.propagateIfFatal(e);
+ if (problem==null) problem = e;
+ }
+ }
+
+ if (result==null && contents!=null && (mode==PopulateMode.XML || mode==PopulateMode.AUTODETECT)) {
+ // then try XML
+ problem = populateInitialFromUriXml(catalog, catalogUrl, problem, contents);
+ }
+
+ if (result!=null) {
+ log.debug("Loaded initial catalog from {}: {}", catalogUrl, result);
+ }
+ if (problem!=null) {
+ log.warn("Error importing catalog from " + catalogUrl + ": " + problem, problem);
+ // TODO inform mgmt of error
+ }
+
+ }
+
+ // deprecated XML format
+ @SuppressWarnings("deprecation")
+ private Exception populateInitialFromUriXml(BasicBrooklynCatalog catalog, String catalogUrl, Exception problem, String contents) {
+ CatalogDto dto = null;
+ try {
+ dto = CatalogDto.newDtoFromXmlContents(contents, catalogUrl);
+ problem = null;
+ } catch (Exception e) {
+ Exceptions.propagateIfFatal(e);
+ if (problem==null) problem = e;
+ }
+ if (dto!=null) {
+ catalog.reset(dto);
+ }
+ return problem;
+ }
+
+ boolean hasRunAdditions = false;
+ protected void populateAdditions(BasicBrooklynCatalog catalog) {
+ if (Strings.isNonBlank(additionsUri)) {
+ if (disallowLocal) {
+ if (!hasRunAdditions) {
+ log.warn("CLI additions supplied but not supported in disallow-local mode; ignoring.");
+ }
+ return;
+ }
+ if (!hasRunAdditions) {
+ log.debug("Adding to catalog from CLI: "+additionsUri+" (force: "+force+")");
+ }
+ Iterable<? extends CatalogItem<?, ?>> items = catalog.addItems(
+ new ResourceUtils(this).getResourceAsString(additionsUri), force);
+
+ if (!hasRunAdditions)
+ log.debug("Added to catalog from CLI: "+items);
+ else
+ log.debug("Added to catalog from CLI: count "+Iterables.size(items));
+
+ hasRunAdditions = true;
+ }
+ }
+
+ protected void populateViaCallbacks(BasicBrooklynCatalog catalog) {
+ for (Function<CatalogInitialization, Void> callback: callbacks)
+ callback.apply(this);
+ }
+
+ private boolean setFromCatalogLoadMode = false;
+
+ /** @deprecated since introduced in 0.7.0, only for legacy compatibility with
+ * {@link CatalogLoadMode} {@link BrooklynServerConfig#CATALOG_LOAD_MODE},
+ * allowing control of catalog loading from a brooklyn property */
+ @Deprecated
+ public void applyCatalogLoadMode() {
+ if (setFromCatalogLoadMode) return;
+ setFromCatalogLoadMode = true;
+ Maybe<Object> clmm = ((ManagementContextInternal)managementContext).getConfig().getConfigRaw(BrooklynServerConfig.CATALOG_LOAD_MODE, false);
+ if (clmm.isAbsent()) return;
+ brooklyn.catalog.CatalogLoadMode clm = TypeCoercions.coerce(clmm.get(), brooklyn.catalog.CatalogLoadMode.class);
+ log.warn("Legacy CatalogLoadMode "+clm+" set: applying, but this should be changed to use new CLI --catalogXxx commands");
+ switch (clm) {
+ case LOAD_BROOKLYN_CATALOG_URL:
+ reset = true;
+ break;
+ case LOAD_BROOKLYN_CATALOG_URL_IF_NO_PERSISTED_STATE:
+ // now the default
+ break;
+ case LOAD_PERSISTED_STATE:
+ disallowLocal = true;
+ break;
+ }
+ }
+
+ /** makes the catalog, warning if persistence is on and hasn't run yet
+ * (as the catalog will be subsequently replaced) */
+ public void populateBestEffort(BasicBrooklynCatalog catalog) {
+ synchronized (mutex) {
+ if (hasRunOfficial || hasRunBestEffort || isPopulating) return;
+ // if a thread calls back in to this, ie calling to it from a getCatalog() call while populating,
+ // it will own the mutex and observe isRunningBestEffort, returning quickly
+ isPopulating = true;
+ try {
+ if (isStartingUp) {
+ log.warn("Catalog access requested when not yet initialized; populating best effort rather than through recommended pathway. Catalog data may be replaced subsequently.");
+ }
+ populateCatalog(catalog, true, true, null);
+ } finally {
+ hasRunBestEffort = true;
+ isPopulating = false;
+ }
+ }
+ }
+
+ public void setStartingUp(boolean isStartingUp) {
+ this.isStartingUp = isStartingUp;
+ }
+
+ public void setFailOnStartupErrors(boolean startupFailOnCatalogErrors) {
--- End diff --
Personal preference to have the setters at the top - they are presumably being used like a builder'y type of thing. They are conveniences, rather than having a huge number of args in the constructor. Having all of those together at the top make it easier to see how the behaviour of the class is set.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by ahgittin <gi...@git.apache.org>.
Github user ahgittin commented on a diff in the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#discussion_r30514565
--- Diff: core/src/main/java/brooklyn/catalog/internal/BasicBrooklynCatalog.java ---
@@ -1008,8 +1072,19 @@ private DeploymentPlan makePlanFromYaml(String yaml) {
return itemDto;
}
- private void checkItemNotExists(CatalogItem<?,?> itemDto, boolean forceUpdate) {
- if (!forceUpdate && getCatalogItemDo(itemDto.getSymbolicName(), itemDto.getVersion()) != null) {
+ /** returns item DTO if item is an allowed duplicate, null if it should be added, or false if the item is an allowed duplicate,
--- End diff --
yeah, semantics got updated, tidied as part of method rename
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by aledsage <gi...@git.apache.org>.
Github user aledsage commented on a diff in the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#discussion_r30590479
--- Diff: utils/common/src/main/java/brooklyn/util/javalang/Threads.java ---
@@ -44,7 +46,15 @@ public static boolean removeShutdownHook(Thread hook) {
return Runtime.getRuntime().removeShutdownHook(hook);
} catch (IllegalStateException e) {
// probably shutdown in progress
- log.debug("cannot remove shutdown hook "+hook+": "+e);
+ String text = Exceptions.collapseText(e);
+ if (text.contains("Shutdown in progress")) {
+ if (log.isTraceEnabled()) {
+ log.trace("Could not remove shutdown hook "+hook+": "+text);
--- End diff --
Add trace and debug level, we should just include the entire stacktrace rather than truncating. Otherwise we risk losing important information for debugging problems that customers send via log files.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by ahgittin <gi...@git.apache.org>.
Github user ahgittin commented on a diff in the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#discussion_r30659214
--- Diff: docs/guide/start/blueprints.md ---
@@ -45,17 +45,14 @@ application. Your application will be shown as "Starting" on the web console's f
Instead of pasting the YAML blueprint each time,
this blueprint can be [added to the catalog](../ops/catalog/).
-Or, even easier, you can download a sample [catalog.xml](catalog.xml).
-Install this to your `~/.brooklyn/` folder and relaunch Brooklyn
-(navigating to the "Help" tab in order to shutdown Brooklyn *and* the application you launched in the previous step).
-
-Now when the web console is re-opened, the catalog contains our blueprints.
-Select the "Demo Web Cluster with DB" and click "Next".
+With this YAML blueprint added, including the location, the Add Application dialog will offer
--- End diff --
i expect we'll tidy up the gui and this discussion of the gui in future
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by aledsage <gi...@git.apache.org>.
Github user aledsage commented on a diff in the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#discussion_r30545124
--- Diff: core/src/main/java/brooklyn/catalog/internal/CatalogInitialization.java ---
@@ -0,0 +1,377 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package brooklyn.catalog.internal;
+
+import java.io.File;
+import java.util.Collection;
+import java.util.List;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import brooklyn.catalog.CatalogItem;
+import brooklyn.config.BrooklynServerConfig;
+import brooklyn.management.ManagementContext;
+import brooklyn.management.ManagementContextInjectable;
+import brooklyn.management.internal.ManagementContextInternal;
+import brooklyn.util.ResourceUtils;
+import brooklyn.util.collections.MutableList;
+import brooklyn.util.exceptions.Exceptions;
+import brooklyn.util.exceptions.FatalRuntimeException;
+import brooklyn.util.exceptions.RuntimeInterruptedException;
+import brooklyn.util.flags.TypeCoercions;
+import brooklyn.util.guava.Maybe;
+import brooklyn.util.net.Urls;
+import brooklyn.util.text.Strings;
+
+import com.google.common.annotations.Beta;
+import com.google.common.base.Function;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.Iterables;
+
+@Beta
+public class CatalogInitialization implements ManagementContextInjectable {
+
+ /*
+
+ A1) if not persisting, go to B1
+ A2) if --catalog-reset, delete persisted catalog items
+ A3) read persisted catalog items (possibly deleted in A2), go to C1
+ A4) go to B1
+
+ B1) look for --catalog-initial, if so read it, then go to C1
+ B2) look for BrooklynServerConfig.BROOKLYN_CATALOG_URL, if so, read it, supporting YAML or XML (warning if XML), then go to C1
+ B3) look for ~/.brooklyn/catalog.bom, if exists, read it then go to C1
+ B4) look for ~/.brooklyn/brooklyn.xml, if exists, warn, read it then go to C1
+ B5) read all classpath://brooklyn/default.catalog.bom items, if they exist (and for now they will)
+ B6) go to C1
+
+ C1) if --catalog-add, read and add those items
+
+ D1) if persisting, read the rest of the persisted items (entities etc)
+
+ */
+
+ private static final Logger log = LoggerFactory.getLogger(CatalogInitialization.class);
+
+ String initialUri;
+ boolean reset;
+ String additionsUri;
+ boolean force;
+
+ boolean disallowLocal = false;
+ List<Function<CatalogInitialization, Void>> callbacks = MutableList.of();
+ boolean hasRunBestEffort = false, hasRunOfficial = false, isPopulating = false;
+
+ ManagementContext managementContext;
+ boolean isStartingUp = false;
+ boolean failOnStartupErrors = false;
+
+ Object mutex = new Object();
+
+ public CatalogInitialization(String initialUri, boolean reset, String additionUri, boolean force) {
+ this.initialUri = initialUri;
+ this.reset = reset;
+ this.additionsUri = additionUri;
+ this.force = force;
+ }
+
+ public CatalogInitialization() {
+ this(null, false, null, false);
+ }
+
+ public void injectManagementContext(ManagementContext managementContext) {
+ if (this.managementContext!=null && managementContext!=null && !this.managementContext.equals(managementContext))
+ throw new IllegalStateException("Cannot switch management context of "+this+"; from "+this.managementContext+" to "+managementContext);
+ this.managementContext = managementContext;
+ }
+
+ public ManagementContext getManagementContext() {
+ return Preconditions.checkNotNull(managementContext, "management context has not been injected into "+this);
+ }
+
+ public CatalogInitialization addPopulationCallback(Function<CatalogInitialization, Void> callback) {
+ callbacks.add(callback);
+ return this;
+ }
+
+ public boolean isInitialResetRequested() {
+ return reset;
+ }
+
+ public boolean hasRunOfficial() { return hasRunOfficial; }
+ public boolean hasRunIncludingBestEffort() { return hasRunOfficial || hasRunBestEffort; }
+
+ /** makes or updates the mgmt catalog, based on the settings in this class */
+ public void populateCatalog(boolean needsInitial, Collection<CatalogItem<?, ?>> optionalItemsForResettingCatalog) {
+ try {
+ isPopulating = true;
+ synchronized (mutex) {
+ BasicBrooklynCatalog catalog = (BasicBrooklynCatalog) managementContext.getCatalog();
+ if (!catalog.getCatalog().isLoaded()) {
+ catalog.load();
+ } else {
+ if (needsInitial && (hasRunOfficial || hasRunBestEffort)) {
+ // an indication that something caused it to load early; not severe, but unusual
+ log.warn("Catalog initialization has not properly run but management context has a catalog; re-populating, possibly overwriting items installed during earlier access (it may have been an early web request)");
+ catalog.reset(ImmutableList.<CatalogItem<?,?>>of());
+ }
+ }
+ hasRunOfficial = true;
+
+ populateCatalog(catalog, needsInitial, true, optionalItemsForResettingCatalog);
+ }
+ } finally {
+ hasRunOfficial = true;
+ isPopulating = false;
+ }
+ }
+
+ private void populateCatalog(BasicBrooklynCatalog catalog, boolean needsInitial, boolean runCallbacks, Collection<CatalogItem<?, ?>> optionalItemsForResettingCatalog) {
--- End diff --
I'd prefer to not bother passing in arguments that are always true. For example, `runCallbacks` is true in every caller, so we can miss it out.
I'd also go for a name like `populateCatalogImpl` to indicate that it's the internal implementation (but no strong feelings on that one).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by aledsage <gi...@git.apache.org>.
Github user aledsage commented on a diff in the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#discussion_r30501517
--- Diff: core/src/main/java/brooklyn/catalog/internal/BasicBrooklynCatalog.java ---
@@ -750,7 +780,36 @@ private String setFromItemIfUnset(String oldValue, Map<?,?> item, String fieldAt
return oldValue;
}
-
+ /** scans the given libraries for annotated items, or if null scans the local classpath */
--- End diff --
I'd prefer it if `null` does not mean "local scan". In (too) many other places we treat null the same as an empty list. Treating it as meaning "local" will be surprising.
If non-null, does it scan the local classpath as well? I'm finding it hard to tell from the code what triggers it to scan the local classpath.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by ahgittin <gi...@git.apache.org>.
Github user ahgittin commented on a diff in the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#discussion_r30967798
--- Diff: core/src/main/java/brooklyn/catalog/internal/CatalogInitialization.java ---
@@ -0,0 +1,380 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package brooklyn.catalog.internal;
+
+import java.io.File;
+import java.util.Collection;
+import java.util.List;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import brooklyn.catalog.CatalogItem;
+import brooklyn.config.BrooklynServerConfig;
+import brooklyn.management.ManagementContext;
+import brooklyn.management.ManagementContextInjectable;
+import brooklyn.management.internal.ManagementContextInternal;
+import brooklyn.util.ResourceUtils;
+import brooklyn.util.collections.MutableList;
+import brooklyn.util.exceptions.Exceptions;
+import brooklyn.util.exceptions.FatalRuntimeException;
+import brooklyn.util.exceptions.RuntimeInterruptedException;
+import brooklyn.util.flags.TypeCoercions;
+import brooklyn.util.guava.Maybe;
+import brooklyn.util.os.Os;
+import brooklyn.util.text.Strings;
+
+import com.google.common.annotations.Beta;
+import com.google.common.base.Function;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.Iterables;
+
+@Beta
+public class CatalogInitialization implements ManagementContextInjectable {
+
+ /*
+
+ A1) if not persisting, go to B1
+ A2) if --catalog-reset, delete persisted catalog items
+ A3) if there is a persisted catalog (and it wasn't not deleted by A2), read it and go to C1
+ A4) go to B1
+
+ B1) look for --catalog-initial, if so read it, then go to C1
+ B2) look for BrooklynServerConfig.BROOKLYN_CATALOG_URL, if so, read it, supporting YAML or XML (warning if XML), then go to C1
+ B3) look for ~/.brooklyn/catalog.bom, if exists, read it then go to C1
+ B4) look for ~/.brooklyn/brooklyn.xml, if exists, warn, read it then go to C1
+ B5) read all classpath://brooklyn/default.catalog.bom items, if they exist (and for now they will)
+ B6) go to C1
+
+ C1) if --catalog-add, read and add those items
+
+ D1) if persisting, read the rest of the persisted items (entities etc)
+
+ */
+
+ private static final Logger log = LoggerFactory.getLogger(CatalogInitialization.class);
+
+ private String initialUri;
+ private boolean reset;
+ private String additionsUri;
+ private boolean force;
+
+ private boolean disallowLocal = false;
+ private List<Function<CatalogInitialization, Void>> callbacks = MutableList.of();
+ private boolean hasRunBestEffort = false, hasRunOfficial = false, isPopulating = false;
+
+ private ManagementContext managementContext;
+ private boolean isStartingUp = false;
+ private boolean failOnStartupErrors = false;
+
+ private Object populatingCatalogMutex = new Object();
+
+ public CatalogInitialization(String initialUri, boolean reset, String additionUri, boolean force) {
+ this.initialUri = initialUri;
+ this.reset = reset;
+ this.additionsUri = additionUri;
+ this.force = force;
+ }
+
+ public CatalogInitialization() {
+ this(null, false, null, false);
+ }
+
+ public void injectManagementContext(ManagementContext managementContext) {
+ Preconditions.checkNotNull(managementContext, "management context");
+ if (this.managementContext!=null && managementContext!=this.managementContext)
+ throw new IllegalStateException("Cannot switch management context, from "+this.managementContext+" to "+managementContext);
+ this.managementContext = managementContext;
+ }
+
+ /** Called by the framework to set true while starting up, and false afterwards,
+ * in order to assist in appropriate logging and error handling. */
+ public void setStartingUp(boolean isStartingUp) {
+ this.isStartingUp = isStartingUp;
+ }
+
+ public void setFailOnStartupErrors(boolean startupFailOnCatalogErrors) {
+ this.failOnStartupErrors = startupFailOnCatalogErrors;
+ }
+
+ public CatalogInitialization addPopulationCallback(Function<CatalogInitialization, Void> callback) {
+ callbacks.add(callback);
+ return this;
+ }
+
+ public ManagementContext getManagementContext() {
+ return Preconditions.checkNotNull(managementContext, "management context has not been injected into "+this);
+ }
+
+ public boolean isInitialResetRequested() {
+ return reset;
+ }
+
+ public boolean hasRunOfficial() { return hasRunOfficial; }
+ public boolean hasRunIncludingBestEffort() { return hasRunOfficial || hasRunBestEffort; }
+
+ /** makes or updates the mgmt catalog, based on the settings in this class */
+ public void populateCatalog(boolean needsInitial, Collection<CatalogItem<?, ?>> optionalItemsForResettingCatalog) {
+ synchronized (populatingCatalogMutex) {
+ try {
+ isPopulating = true;
+ BasicBrooklynCatalog catalog = (BasicBrooklynCatalog) managementContext.getCatalog();
+ if (!catalog.getCatalog().isLoaded()) {
+ catalog.load();
+ } else {
+ if (needsInitial && (hasRunOfficial || hasRunBestEffort)) {
+ // an indication that something caused it to load early; not severe, but unusual
+ log.warn("Catalog initialization has not properly run but management context has a catalog; re-populating, possibly overwriting items installed during earlier access (it may have been an early web request)");
+ catalog.reset(ImmutableList.<CatalogItem<?,?>>of());
+ }
+ }
+ hasRunOfficial = true;
+
+ populateCatalogImpl(catalog, needsInitial, optionalItemsForResettingCatalog);
+ } finally {
+ hasRunOfficial = true;
+ isPopulating = false;
+ }
+ }
+ }
+
+ private void populateCatalogImpl(BasicBrooklynCatalog catalog, boolean needsInitial, Collection<CatalogItem<?, ?>> optionalItemsForResettingCatalog) {
+ applyCatalogLoadMode();
+
+ if (optionalItemsForResettingCatalog!=null) {
+ catalog.reset(optionalItemsForResettingCatalog);
+ }
+
+ if (needsInitial) {
+ populateInitial(catalog);
+ }
+
+ populateAdditions(catalog);
+
+ populateViaCallbacks(catalog);
+ }
+
+ private enum PopulateMode { YAML, XML, AUTODETECT }
+
+ protected void populateInitial(BasicBrooklynCatalog catalog) {
+ if (disallowLocal) {
+ if (!hasRunOfficial()) {
+ log.debug("CLI initial catalog not being read with disallow-local mode set.");
+ }
+ return;
+ }
+
+// B1) look for --catalog-initial, if so read it, then go to C1
+// B2) look for BrooklynServerConfig.BROOKLYN_CATALOG_URL, if so, read it, supporting YAML or XML (warning if XML), then go to C1
+// B3) look for ~/.brooklyn/catalog.bom, if exists, read it then go to C1
+// B4) look for ~/.brooklyn/brooklyn.xml, if exists, warn, read it then go to C1
+// B5) read all classpath://brooklyn/default.catalog.bom items, if they exist (and for now they will)
+// B6) go to C1
+
+ if (initialUri!=null) {
+ populateInitialFromUri(catalog, initialUri, PopulateMode.AUTODETECT);
+ return;
+ }
+
+ String catalogUrl = managementContext.getConfig().getConfig(BrooklynServerConfig.BROOKLYN_CATALOG_URL);
+ if (Strings.isNonBlank(catalogUrl)) {
+ populateInitialFromUri(catalog, catalogUrl, PopulateMode.AUTODETECT);
+ return;
+ }
+
+ catalogUrl = Os.mergePaths(BrooklynServerConfig.getMgmtBaseDir( managementContext.getConfig() ), "catalog.bom");
+ if (new File(catalogUrl).exists()) {
+ populateInitialFromUri(catalog, "file:"+catalogUrl, PopulateMode.YAML);
+ return;
+ }
+
+ catalogUrl = Os.mergePaths(BrooklynServerConfig.getMgmtBaseDir( managementContext.getConfig() ), "catalog.xml");
+ if (new File(catalogUrl).exists()) {
+ populateInitialFromUri(catalog, "file:"+catalogUrl, PopulateMode.XML);
+ return;
+ }
+
+ // otherwise look for for classpath:/brooklyn/default.catalog.bom --
+ // there is one on the classpath which says to scan, and provides a few templates;
+ // if one is supplied by user in the conf/ dir that will override the item from the classpath
+ // (TBD - we might want to scan for all such bom's?)
+
+ catalogUrl = "classpath:/brooklyn/default.catalog.bom";
+ if (new ResourceUtils(this).doesUrlExist(catalogUrl)) {
+ populateInitialFromUri(catalog, catalogUrl, PopulateMode.YAML);
+ return;
+ }
+
+ log.info("No catalog found on classpath or specified; catalog will not be initialized.");
+ return;
+ }
+
+ private void populateInitialFromUri(BasicBrooklynCatalog catalog, String catalogUrl, PopulateMode mode) {
+ log.debug("Loading initial catalog from {}", catalogUrl);
+
+ Exception problem = null;
+ Object result = null;
+
+ String contents = null;
+ try {
+ contents = new ResourceUtils(this).getResourceAsString(catalogUrl);
+ } catch (Exception e) {
+ Exceptions.propagateIfFatal(e);
+ if (problem==null) problem = e;
+ }
+
+ if (contents!=null && (mode==PopulateMode.YAML || mode==PopulateMode.AUTODETECT)) {
+ // try YAML first
+ try {
+ catalog.reset(MutableList.<CatalogItem<?,?>>of());
+ result = catalog.addItems(contents);
+ } catch (Exception e) {
+ Exceptions.propagateIfFatal(e);
+ if (problem==null) problem = e;
+ }
+ }
+
+ if (result==null && contents!=null && (mode==PopulateMode.XML || mode==PopulateMode.AUTODETECT)) {
+ // then try XML
+ try {
+ populateInitialFromUriXml(catalog, catalogUrl, contents);
+ // clear YAML problem
+ problem = null;
+ } catch (Exception e) {
+ Exceptions.propagateIfFatal(e);
+ if (problem==null) problem = e;
+ }
+ }
+
+ if (result!=null) {
+ log.debug("Loaded initial catalog from {}: {}", catalogUrl, result);
+ }
+ if (problem!=null) {
+ log.warn("Error importing catalog from " + catalogUrl + ": " + problem, problem);
+ // TODO inform mgmt of error
+ }
+
+ }
+
+ // deprecated XML format
+ @SuppressWarnings("deprecation")
+ private void populateInitialFromUriXml(BasicBrooklynCatalog catalog, String catalogUrl, String contents) {
+ CatalogDto dto = CatalogDto.newDtoFromXmlContents(contents, catalogUrl);
+ if (dto!=null) {
+ catalog.reset(dto);
+ }
+ }
+
+ boolean hasRunAdditions = false;
+ protected void populateAdditions(BasicBrooklynCatalog catalog) {
+ if (Strings.isNonBlank(additionsUri)) {
+ if (disallowLocal) {
+ if (!hasRunAdditions) {
+ log.warn("CLI additions supplied but not supported in disallow-local mode; ignoring.");
+ }
+ return;
--- End diff --
here we get a warning. i think that's fine for a deprecated option. have updated message.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by ahgittin <gi...@git.apache.org>.
Github user ahgittin commented on a diff in the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#discussion_r30658508
--- Diff: utils/common/src/main/java/brooklyn/util/javalang/Threads.java ---
@@ -44,7 +46,15 @@ public static boolean removeShutdownHook(Thread hook) {
return Runtime.getRuntime().removeShutdownHook(hook);
} catch (IllegalStateException e) {
// probably shutdown in progress
- log.debug("cannot remove shutdown hook "+hook+": "+e);
+ String text = Exceptions.collapseText(e);
+ if (text.contains("Shutdown in progress")) {
+ if (log.isTraceEnabled()) {
+ log.trace("Could not remove shutdown hook "+hook+": "+text);
--- End diff --
Normally I'd agree but this is a known exception which is noise in the logs, only arising from one place. And it appears at the very end of the log and so is very visible, giving the wrong impression if it looks like a real problem.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by aledsage <gi...@git.apache.org>.
Github user aledsage commented on a diff in the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#discussion_r30544000
--- Diff: core/src/main/java/brooklyn/catalog/internal/CatalogInitialization.java ---
@@ -0,0 +1,377 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package brooklyn.catalog.internal;
+
+import java.io.File;
+import java.util.Collection;
+import java.util.List;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import brooklyn.catalog.CatalogItem;
+import brooklyn.config.BrooklynServerConfig;
+import brooklyn.management.ManagementContext;
+import brooklyn.management.ManagementContextInjectable;
+import brooklyn.management.internal.ManagementContextInternal;
+import brooklyn.util.ResourceUtils;
+import brooklyn.util.collections.MutableList;
+import brooklyn.util.exceptions.Exceptions;
+import brooklyn.util.exceptions.FatalRuntimeException;
+import brooklyn.util.exceptions.RuntimeInterruptedException;
+import brooklyn.util.flags.TypeCoercions;
+import brooklyn.util.guava.Maybe;
+import brooklyn.util.net.Urls;
+import brooklyn.util.text.Strings;
+
+import com.google.common.annotations.Beta;
+import com.google.common.base.Function;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.Iterables;
+
+@Beta
+public class CatalogInitialization implements ManagementContextInjectable {
+
+ /*
+
+ A1) if not persisting, go to B1
+ A2) if --catalog-reset, delete persisted catalog items
+ A3) read persisted catalog items (possibly deleted in A2), go to C1
+ A4) go to B1
+
+ B1) look for --catalog-initial, if so read it, then go to C1
+ B2) look for BrooklynServerConfig.BROOKLYN_CATALOG_URL, if so, read it, supporting YAML or XML (warning if XML), then go to C1
+ B3) look for ~/.brooklyn/catalog.bom, if exists, read it then go to C1
+ B4) look for ~/.brooklyn/brooklyn.xml, if exists, warn, read it then go to C1
+ B5) read all classpath://brooklyn/default.catalog.bom items, if they exist (and for now they will)
+ B6) go to C1
+
+ C1) if --catalog-add, read and add those items
+
+ D1) if persisting, read the rest of the persisted items (entities etc)
+
+ */
+
+ private static final Logger log = LoggerFactory.getLogger(CatalogInitialization.class);
+
+ String initialUri;
--- End diff --
Why package private and not final?
It leaves the reader wondering if it is supposed to support the field being modified at any point by other code in the same package - which makes it much harder to reason about the code if anyone is going to modify it later or investigate any bugs that are reported.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by aledsage <gi...@git.apache.org>.
Github user aledsage commented on a diff in the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#discussion_r30590279
--- Diff: utils/common/src/main/java/brooklyn/util/javalang/AggregateClassLoader.java ---
@@ -69,8 +70,25 @@ public void add(int index, ClassLoader classLoader) {
if (classLoader != null) classLoaders.add(index, classLoader);
}
- /** Returns the _live_ (and modifiable) list of classloaders
- * @return */
+ /** Resets the classloader shown here to be the given set */
+ public void reset(Collection<? extends ClassLoader> newClassLoaders) {
+ synchronized (classLoaders) {
--- End diff --
What is the point of the synchronized? Is it just to guard against multiple concurrent calls to `reset()`? None of the other modifications or accesses to `classLoaders` is synchronized.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by ahgittin <gi...@git.apache.org>.
Github user ahgittin commented on a diff in the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#discussion_r30973465
--- Diff: core/src/main/java/brooklyn/entity/rebind/RebindIteration.java ---
@@ -341,38 +340,64 @@ protected void rebuildCatalog() {
}
}
}
+
+ // See notes in CatalogInitialization
- // Register catalogue items with the management context. Loads the bundles in the OSGi framework.
- CatalogLoadMode catalogLoadMode = managementContext.getConfig().getConfig(BrooklynServerConfig.CATALOG_LOAD_MODE);
+ Collection<CatalogItem<?, ?>> catalogItems = rebindContext.getCatalogItems();
+ CatalogInitialization catInit = ((ManagementContextInternal)managementContext).getCatalogInitialization();
+ catInit.applyCatalogLoadMode();
+ Collection<CatalogItem<?,?>> itemsForResettingCatalog = null;
+ boolean needsInitialCatalog;
if (rebindManager.persistCatalogItemsEnabled) {
- boolean shouldResetCatalog = catalogLoadMode == CatalogLoadMode.LOAD_PERSISTED_STATE
- || (!isEmpty && catalogLoadMode == CatalogLoadMode.LOAD_BROOKLYN_CATALOG_URL_IF_NO_PERSISTED_STATE);
- boolean shouldLoadDefaultCatalog = catalogLoadMode == CatalogLoadMode.LOAD_BROOKLYN_CATALOG_URL
- || (isEmpty && catalogLoadMode == CatalogLoadMode.LOAD_BROOKLYN_CATALOG_URL_IF_NO_PERSISTED_STATE);
- if (shouldResetCatalog) {
- // Reset catalog with previously persisted state
- logRebindingDebug("RebindManager resetting management context catalog to previously persisted state");
- managementContext.getCatalog().reset(rebindContext.getCatalogItems());
- } else if (shouldLoadDefaultCatalog) {
- // Load catalogue as normal
- // TODO in read-only mode, should do this less frequently than entities etc
- logRebindingDebug("RebindManager loading default catalog");
- ((BasicBrooklynCatalog) managementContext.getCatalog()).resetCatalogToContentsAtConfiguredUrl();
+ if (!catInit.hasRunOfficial() && catInit.isInitialResetRequested()) {
--- End diff --
i've set it up so we distinguish between `unofficial` initialization, `official final` initialization, and `official transient` initialization. that last is done if we run rebind but aren't master, and it allows subsequent runs also to apply initialization. if we are master (or are a non-persisting launch sequence) then we set `official final` and subsequent re-initialization does *not* apply of the parameters.
added quite a bit more doc around it as well. unfortunate it is so complex but i'm not sure of a way around it, unless we distinguished things you could do on first run from on subsequent run.
(the edge cases are very edge, as most CLI uses should be when there is no brooklyn running; but i think we do now handle them nicely.)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by ahgittin <gi...@git.apache.org>.
Github user ahgittin commented on the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#issuecomment-100333416
@neykov thanks for the comments so far - updated and rebased
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by ahgittin <gi...@git.apache.org>.
Github user ahgittin commented on the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#issuecomment-106319443
@neykov I think these are all fixed. The test case now does not depend on `~/.brooklyn/`. Can you confirm it works for you?
The `URLConnection` class is an ugly beast, that's why I backed out of the `URLStreamHandler` approach.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by aledsage <gi...@git.apache.org>.
Github user aledsage commented on a diff in the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#discussion_r30582714
--- Diff: core/src/main/java/brooklyn/entity/rebind/RebindIteration.java ---
@@ -341,38 +340,67 @@ protected void rebuildCatalog() {
}
}
}
+
+ // See notes in CatalogInitialization
- // Register catalogue items with the management context. Loads the bundles in the OSGi framework.
- CatalogLoadMode catalogLoadMode = managementContext.getConfig().getConfig(BrooklynServerConfig.CATALOG_LOAD_MODE);
+ Collection<CatalogItem<?, ?>> catalogItems = rebindContext.getCatalogItems();
+ CatalogInitialization catInit = ((ManagementContextInternal)managementContext).getCatalogInitialization();
+ catInit.injectManagementContext(managementContext);
+ catInit.applyCatalogLoadMode();
+ Collection<CatalogItem<?,?>> itemsForResettingCatalog = null;
+ boolean needsInitialCatalog;
if (rebindManager.persistCatalogItemsEnabled) {
- boolean shouldResetCatalog = catalogLoadMode == CatalogLoadMode.LOAD_PERSISTED_STATE
- || (!isEmpty && catalogLoadMode == CatalogLoadMode.LOAD_BROOKLYN_CATALOG_URL_IF_NO_PERSISTED_STATE);
- boolean shouldLoadDefaultCatalog = catalogLoadMode == CatalogLoadMode.LOAD_BROOKLYN_CATALOG_URL
- || (isEmpty && catalogLoadMode == CatalogLoadMode.LOAD_BROOKLYN_CATALOG_URL_IF_NO_PERSISTED_STATE);
- if (shouldResetCatalog) {
- // Reset catalog with previously persisted state
- logRebindingDebug("RebindManager resetting management context catalog to previously persisted state");
- managementContext.getCatalog().reset(rebindContext.getCatalogItems());
- } else if (shouldLoadDefaultCatalog) {
- // Load catalogue as normal
- // TODO in read-only mode, should do this less frequently than entities etc
- logRebindingDebug("RebindManager loading default catalog");
- ((BasicBrooklynCatalog) managementContext.getCatalog()).resetCatalogToContentsAtConfiguredUrl();
+ if (!catInit.hasRunOfficial() && catInit.isInitialResetRequested()) {
+ String message = "RebindManager resetting catalog on first run (catalog persistence enabled, but reset explicitly specified). ";
+ if (catalogItems.isEmpty()) {
+ message += "Catalog was empty anyway.";
+ } else {
+ message += "Deleting "+catalogItems.size()+" persisted catalog item"+Strings.s(catalogItems)+": "+catalogItems;
+ if (shouldLogRebinding()) {
+ LOG.info(message);
+ }
+ }
+ logRebindingDebug(message);
+
+ itemsForResettingCatalog = MutableList.<CatalogItem<?,?>>of();
+
+ PersisterDeltaImpl delta = new PersisterDeltaImpl();
+ for (String catalogItemId: mementoRawData.getCatalogItems().keySet()) {
+ delta.removedCatalogItemIds.add(catalogItemId);
+ }
+ getPersister().queueDelta(delta);
+
+ mementoRawData.clearCatalogItems();
+ needsInitialCatalog = true;
} else {
- // Management context should have taken care of loading the catalogue
- Collection<CatalogItem<?, ?>> catalogItems = rebindContext.getCatalogItems();
- String message = "RebindManager not resetting catalog to persisted state. Catalog load mode is {}.";
- if (!catalogItems.isEmpty() && shouldLogRebinding()) {
- LOG.info(message + " There {} {} item{} persisted.", new Object[]{
- catalogLoadMode, catalogItems.size() == 1 ? "was" : "were", catalogItems.size(), Strings.s(catalogItems)});
- } else if (LOG.isDebugEnabled()) {
- logRebindingDebug(message, catalogLoadMode);
+ if (!isEmpty) {
+ logRebindingDebug("RebindManager clearing local catalog and loading from persisted state");
+ itemsForResettingCatalog = rebindContext.getCatalogItems();
+ needsInitialCatalog = false;
+ } else {
+ if (catInit.hasRunOfficial()) {
+ logRebindingDebug("RebindManager will re-add any new items (persisted state empty)");
+ needsInitialCatalog = false;
+ } else {
+ logRebindingDebug("RebindManager loading initial catalog locally because persisted state empty");
+ needsInitialCatalog = true;
+ }
}
}
- // TODO destroy old (as above)
} else {
- logRebindingDebug("RebindManager not resetting catalog because catalog persistence is disabled");
+ if (catInit.hasRunOfficial()) {
+ logRebindingDebug("RebindManager skipping catalog init because it has already run (catalog persistence disabled)");
+ needsInitialCatalog = false;
+ } else {
+ logRebindingDebug("RebindManager will initialize catalog locally because catalog persistence is disabled");
+ needsInitialCatalog = true;
+ }
}
+
+ // TODO in read-only mode, perhaps do this less frequently than entities etc ?
+ // both in RW and in RO mode, the first run reads the initialization data;
+ // maybe not desired for RO as it defers problems, although if it's standalone it is desired
+ catInit.populateCatalog(needsInitialCatalog, itemsForResettingCatalog);
--- End diff --
Personal preference: I like `Optional<Collection<CatalogItem<?,?>>>` rather than using null to indicate absence. That makes the behaviour of the method more explicit.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by neykov <gi...@git.apache.org>.
Github user neykov commented on a diff in the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#discussion_r31217476
--- Diff: core/src/test/java/brooklyn/catalog/internal/CatalogScanTest.java ---
@@ -161,13 +162,14 @@ public void testMoreTypesThanAnnotationsForApps() {
@Test
public void testAnnotationIsDefault() {
--- End diff --
This test started failing for me after the merge of #634
Seems to be caused by me having a default catalog in ~/.brooklyn which replaces the `default.catalog.bom` expected here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by aledsage <gi...@git.apache.org>.
Github user aledsage commented on a diff in the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#discussion_r30499409
--- Diff: api/src/main/java/brooklyn/mementos/BrooklynMementoPersister.java ---
@@ -97,10 +97,13 @@
/** @deprecated since 0.7.0, use {@link #checkpoint(BrooklynMementoRawData, PersistenceExceptionHandler)}
* and javadoc on implementations of that */ @Deprecated // pretty sure this is not used outwith deprecated code
void checkpoint(BrooklynMemento memento, PersistenceExceptionHandler exceptionHandler);
-
- void checkpoint(BrooklynMementoRawData newMemento, PersistenceExceptionHandler exceptionHandler);
+ /** applies a full checkpoint (write) of all state */
+ void checkpoint(BrooklynMementoRawData newMemento, PersistenceExceptionHandler exceptionHandler);
+ /** applies a partial write of state delta */
void delta(Delta delta, PersistenceExceptionHandler exceptionHandler);
+ /** inserts an additional delta to be written on the next delta request */
+ void queueDelta(Delta delta);
--- End diff --
We already have the `PeriodicDeltaChangeListener`, which wraps calls to `persister.delta`. It handles queues, and does smart stuff like if you do a modify then a remove within the same time period, it will discard the modify and just do the remove.
Should we try to reuse `PeriodicDeltaChangeListener` somehow, rather than having `queueDelta` here?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by aledsage <gi...@git.apache.org>.
Github user aledsage commented on a diff in the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#discussion_r30584982
--- Diff: docs/guide/start/blueprints.md ---
@@ -45,17 +45,14 @@ application. Your application will be shown as "Starting" on the web console's f
Instead of pasting the YAML blueprint each time,
this blueprint can be [added to the catalog](../ops/catalog/).
-Or, even easier, you can download a sample [catalog.xml](catalog.xml).
-Install this to your `~/.brooklyn/` folder and relaunch Brooklyn
-(navigating to the "Help" tab in order to shutdown Brooklyn *and* the application you launched in the previous step).
-
-Now when the web console is re-opened, the catalog contains our blueprints.
-Select the "Demo Web Cluster with DB" and click "Next".
+With this YAML blueprint added, including the location, the Add Application dialog will offer
--- End diff --
Is this the only mention in the docs that "including the location" will treat the catalog item as a template? Deserves a section of its own in `docs/guide/ops/catalog/index.md`, including how to add the catalog item as a "tile" (i.e. as a clickable icon that will not show the yaml, but will instead allow it to be deployed directly - a customer was recently stressing the importance of that when they demo Brooklyn to their own customers).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by ahgittin <gi...@git.apache.org>.
Github user ahgittin commented on a diff in the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#discussion_r30968526
--- Diff: utils/common/src/main/java/brooklyn/util/javalang/AggregateClassLoader.java ---
@@ -57,24 +61,65 @@ public static AggregateClassLoader newInstanceWithNoLoaders() {
/** Add a loader to the first position in the search path. */
public void addFirst(ClassLoader classLoader) {
- if (classLoader != null) classLoaders.add(0, classLoader);
+ if (classLoader != null) {
+ synchronized (classLoaders) {
+ classLoaders.add(0, classLoader);
+ }
+ }
}
/** Add a loader to the last position in the search path. */
public void addLast(ClassLoader classLoader) {
- if (classLoader != null) classLoaders.add(classLoader);
+ if (classLoader != null) {
+ synchronized (classLoaders) {
+ classLoaders.add(classLoader);
+ }
+ }
}
/** Add a loader to the specific position in the search path.
* (It is callers responsibility to ensure that position is valid.) */
public void add(int index, ClassLoader classLoader) {
- if (classLoader != null) classLoaders.add(index, classLoader);
+ if (classLoader != null) {
+ synchronized (classLoaders) {
+ classLoaders.add(index, classLoader);
+ }
+ }
+ }
+
+ /** Resets the classloader shown here to be the given set */
+ public void reset(Collection<? extends ClassLoader> newClassLoaders) {
+ synchronized (classLoaders) {
+ // synchronize:
+ // * to prevent concurrent invocations
+ // * so add(0, cl) doesn't interfere
+ // * and for good measure we add before removing so that iterator always contains everything
+ // although since iterator access is synchronized that shouldn't be necessary
+ int count = classLoaders.size();
+ classLoaders.addAll(newClassLoaders);
+ for (int i=0; i<count; i++) {
+ classLoaders.remove(0);
+ }
+ }
+ }
+
+ /** True if nothing is in the list here */
+ public boolean isEmpty() {
+ return classLoaders.isEmpty();
}
- /** Returns the _live_ (and modifiable) list of classloaders
- * @return */
+ /** Returns the _live_ (and modifiable) list of classloaders; dangerous and discouraged.
+ * @deprecated since 0.7.0 */
+ @Deprecated
public List<ClassLoader> getList() {
return classLoaders;
}
+ public Iterator<ClassLoader> iterator() {
+ synchronized (classLoaders) {
+ // provides iterator of snapshot
+ return classLoaders.iterator();
--- End diff --
semantics of `CopyOnWriteList.iterator()` is to take a snapshot, with unmodifiable iterator; have expanded comment
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by aledsage <gi...@git.apache.org>.
Github user aledsage commented on a diff in the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#discussion_r30578361
--- Diff: core/src/main/java/brooklyn/entity/rebind/RebindIteration.java ---
@@ -341,38 +340,67 @@ protected void rebuildCatalog() {
}
}
}
+
+ // See notes in CatalogInitialization
- // Register catalogue items with the management context. Loads the bundles in the OSGi framework.
- CatalogLoadMode catalogLoadMode = managementContext.getConfig().getConfig(BrooklynServerConfig.CATALOG_LOAD_MODE);
+ Collection<CatalogItem<?, ?>> catalogItems = rebindContext.getCatalogItems();
+ CatalogInitialization catInit = ((ManagementContextInternal)managementContext).getCatalogInitialization();
+ catInit.injectManagementContext(managementContext);
+ catInit.applyCatalogLoadMode();
+ Collection<CatalogItem<?,?>> itemsForResettingCatalog = null;
+ boolean needsInitialCatalog;
if (rebindManager.persistCatalogItemsEnabled) {
- boolean shouldResetCatalog = catalogLoadMode == CatalogLoadMode.LOAD_PERSISTED_STATE
- || (!isEmpty && catalogLoadMode == CatalogLoadMode.LOAD_BROOKLYN_CATALOG_URL_IF_NO_PERSISTED_STATE);
- boolean shouldLoadDefaultCatalog = catalogLoadMode == CatalogLoadMode.LOAD_BROOKLYN_CATALOG_URL
- || (isEmpty && catalogLoadMode == CatalogLoadMode.LOAD_BROOKLYN_CATALOG_URL_IF_NO_PERSISTED_STATE);
- if (shouldResetCatalog) {
- // Reset catalog with previously persisted state
- logRebindingDebug("RebindManager resetting management context catalog to previously persisted state");
- managementContext.getCatalog().reset(rebindContext.getCatalogItems());
- } else if (shouldLoadDefaultCatalog) {
- // Load catalogue as normal
- // TODO in read-only mode, should do this less frequently than entities etc
- logRebindingDebug("RebindManager loading default catalog");
- ((BasicBrooklynCatalog) managementContext.getCatalog()).resetCatalogToContentsAtConfiguredUrl();
+ if (!catInit.hasRunOfficial() && catInit.isInitialResetRequested()) {
+ String message = "RebindManager resetting catalog on first run (catalog persistence enabled, but reset explicitly specified). ";
+ if (catalogItems.isEmpty()) {
+ message += "Catalog was empty anyway.";
+ } else {
+ message += "Deleting "+catalogItems.size()+" persisted catalog item"+Strings.s(catalogItems)+": "+catalogItems;
+ if (shouldLogRebinding()) {
+ LOG.info(message);
+ }
+ }
+ logRebindingDebug(message);
+
+ itemsForResettingCatalog = MutableList.<CatalogItem<?,?>>of();
+
+ PersisterDeltaImpl delta = new PersisterDeltaImpl();
+ for (String catalogItemId: mementoRawData.getCatalogItems().keySet()) {
--- End diff --
just do `delta.removedCatalogItemIds.addAll(mementoRawData.getCatalogItems().keySet())`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by neykov <gi...@git.apache.org>.
Github user neykov commented on a diff in the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#discussion_r30790013
--- Diff: core/src/main/java/brooklyn/entity/rebind/PeriodicDeltaChangeListener.java ---
@@ -164,14 +163,12 @@ public void remove(BrooklynObject instance) {
private final PersistenceExceptionHandler exceptionHandler;
private final Duration period;
-
- private final AtomicLong writeCount = new AtomicLong();
-
+
private DeltaCollector deltaCollector = new DeltaCollector();
private volatile boolean running = false;
- private volatile boolean stopped = false;
+ private volatile boolean stopping = false, stopCompleted = false;
--- End diff --
`stopping`, `stopCompleted` and `running` could be merged into a single enum to manage state easier, can be done in a future PR.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by ahgittin <gi...@git.apache.org>.
Github user ahgittin commented on the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#issuecomment-105203872
@neykov Thanks for excellent comments. The last commit addresses them, I think. Code is clearer now, even if still not as simple as I'd like. I have done a fair bit of testing with HA modes and catalog initialization and no surprises :) .
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by neykov <gi...@git.apache.org>.
Github user neykov commented on the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#issuecomment-106286291
Finished review of new commits. Couldn't understand why you didn't prefer the `URLStreamHandler` approach for fetching the classpath:// URLs. Seems cleaner.
Mostly minor comments, can be merged after `testAnnotationIsDefault` can handle user catalogs.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by aledsage <gi...@git.apache.org>.
Github user aledsage commented on a diff in the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#discussion_r30501548
--- Diff: core/src/main/java/brooklyn/catalog/internal/BasicBrooklynCatalog.java ---
@@ -750,7 +780,36 @@ private String setFromItemIfUnset(String oldValue, Map<?,?> item, String fieldAt
return oldValue;
}
-
+ /** scans the given libraries for annotated items, or if null scans the local classpath */
+ private Collection<CatalogItemDtoAbstract<?, ?>> scanAnnotations(ManagementContext mgmt, Collection<CatalogBundle> libraries) {
+// CatalogDto dto = CatalogDto.newDefaultLocalScanningDto(CatalogClasspathDo.CatalogScanningModes.ANNOTATIONS);
--- End diff --
Don't comment out code without a comment to say when and why it would be useful to uncomment it (or some such).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by ahgittin <gi...@git.apache.org>.
Github user ahgittin commented on the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#issuecomment-103448606
ah i caught up with your comments while on the plane just now but i see you've also been busy @aledsage . will look in to your latest batch of comments.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by ahgittin <gi...@git.apache.org>.
Github user ahgittin commented on a diff in the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#discussion_r30967764
--- Diff: core/src/main/java/brooklyn/catalog/internal/CatalogInitialization.java ---
@@ -0,0 +1,380 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package brooklyn.catalog.internal;
+
+import java.io.File;
+import java.util.Collection;
+import java.util.List;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import brooklyn.catalog.CatalogItem;
+import brooklyn.config.BrooklynServerConfig;
+import brooklyn.management.ManagementContext;
+import brooklyn.management.ManagementContextInjectable;
+import brooklyn.management.internal.ManagementContextInternal;
+import brooklyn.util.ResourceUtils;
+import brooklyn.util.collections.MutableList;
+import brooklyn.util.exceptions.Exceptions;
+import brooklyn.util.exceptions.FatalRuntimeException;
+import brooklyn.util.exceptions.RuntimeInterruptedException;
+import brooklyn.util.flags.TypeCoercions;
+import brooklyn.util.guava.Maybe;
+import brooklyn.util.os.Os;
+import brooklyn.util.text.Strings;
+
+import com.google.common.annotations.Beta;
+import com.google.common.base.Function;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.Iterables;
+
+@Beta
+public class CatalogInitialization implements ManagementContextInjectable {
+
+ /*
+
+ A1) if not persisting, go to B1
+ A2) if --catalog-reset, delete persisted catalog items
+ A3) if there is a persisted catalog (and it wasn't not deleted by A2), read it and go to C1
+ A4) go to B1
+
+ B1) look for --catalog-initial, if so read it, then go to C1
+ B2) look for BrooklynServerConfig.BROOKLYN_CATALOG_URL, if so, read it, supporting YAML or XML (warning if XML), then go to C1
+ B3) look for ~/.brooklyn/catalog.bom, if exists, read it then go to C1
+ B4) look for ~/.brooklyn/brooklyn.xml, if exists, warn, read it then go to C1
+ B5) read all classpath://brooklyn/default.catalog.bom items, if they exist (and for now they will)
+ B6) go to C1
+
+ C1) if --catalog-add, read and add those items
+
+ D1) if persisting, read the rest of the persisted items (entities etc)
+
+ */
+
+ private static final Logger log = LoggerFactory.getLogger(CatalogInitialization.class);
+
+ private String initialUri;
+ private boolean reset;
+ private String additionsUri;
+ private boolean force;
+
+ private boolean disallowLocal = false;
+ private List<Function<CatalogInitialization, Void>> callbacks = MutableList.of();
+ private boolean hasRunBestEffort = false, hasRunOfficial = false, isPopulating = false;
+
+ private ManagementContext managementContext;
+ private boolean isStartingUp = false;
+ private boolean failOnStartupErrors = false;
+
+ private Object populatingCatalogMutex = new Object();
+
+ public CatalogInitialization(String initialUri, boolean reset, String additionUri, boolean force) {
+ this.initialUri = initialUri;
+ this.reset = reset;
+ this.additionsUri = additionUri;
+ this.force = force;
+ }
+
+ public CatalogInitialization() {
+ this(null, false, null, false);
+ }
+
+ public void injectManagementContext(ManagementContext managementContext) {
+ Preconditions.checkNotNull(managementContext, "management context");
+ if (this.managementContext!=null && managementContext!=this.managementContext)
+ throw new IllegalStateException("Cannot switch management context, from "+this.managementContext+" to "+managementContext);
+ this.managementContext = managementContext;
+ }
+
+ /** Called by the framework to set true while starting up, and false afterwards,
+ * in order to assist in appropriate logging and error handling. */
+ public void setStartingUp(boolean isStartingUp) {
+ this.isStartingUp = isStartingUp;
+ }
+
+ public void setFailOnStartupErrors(boolean startupFailOnCatalogErrors) {
+ this.failOnStartupErrors = startupFailOnCatalogErrors;
+ }
+
+ public CatalogInitialization addPopulationCallback(Function<CatalogInitialization, Void> callback) {
+ callbacks.add(callback);
+ return this;
+ }
+
+ public ManagementContext getManagementContext() {
+ return Preconditions.checkNotNull(managementContext, "management context has not been injected into "+this);
+ }
+
+ public boolean isInitialResetRequested() {
+ return reset;
+ }
+
+ public boolean hasRunOfficial() { return hasRunOfficial; }
+ public boolean hasRunIncludingBestEffort() { return hasRunOfficial || hasRunBestEffort; }
+
+ /** makes or updates the mgmt catalog, based on the settings in this class */
+ public void populateCatalog(boolean needsInitial, Collection<CatalogItem<?, ?>> optionalItemsForResettingCatalog) {
+ synchronized (populatingCatalogMutex) {
+ try {
+ isPopulating = true;
+ BasicBrooklynCatalog catalog = (BasicBrooklynCatalog) managementContext.getCatalog();
+ if (!catalog.getCatalog().isLoaded()) {
+ catalog.load();
+ } else {
+ if (needsInitial && (hasRunOfficial || hasRunBestEffort)) {
+ // an indication that something caused it to load early; not severe, but unusual
+ log.warn("Catalog initialization has not properly run but management context has a catalog; re-populating, possibly overwriting items installed during earlier access (it may have been an early web request)");
+ catalog.reset(ImmutableList.<CatalogItem<?,?>>of());
+ }
+ }
+ hasRunOfficial = true;
+
+ populateCatalogImpl(catalog, needsInitial, optionalItemsForResettingCatalog);
+ } finally {
+ hasRunOfficial = true;
+ isPopulating = false;
+ }
+ }
+ }
+
+ private void populateCatalogImpl(BasicBrooklynCatalog catalog, boolean needsInitial, Collection<CatalogItem<?, ?>> optionalItemsForResettingCatalog) {
+ applyCatalogLoadMode();
+
+ if (optionalItemsForResettingCatalog!=null) {
+ catalog.reset(optionalItemsForResettingCatalog);
+ }
+
+ if (needsInitial) {
+ populateInitial(catalog);
+ }
+
+ populateAdditions(catalog);
+
+ populateViaCallbacks(catalog);
+ }
+
+ private enum PopulateMode { YAML, XML, AUTODETECT }
+
+ protected void populateInitial(BasicBrooklynCatalog catalog) {
+ if (disallowLocal) {
+ if (!hasRunOfficial()) {
+ log.debug("CLI initial catalog not being read with disallow-local mode set.");
--- End diff --
have amended
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by neykov <gi...@git.apache.org>.
Github user neykov commented on a diff in the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#discussion_r30697461
--- Diff: core/src/main/java/brooklyn/catalog/internal/BasicBrooklynCatalog.java ---
@@ -750,7 +780,37 @@ private String setFromItemIfUnset(String oldValue, Map<?,?> item, String fieldAt
return oldValue;
}
+ private Collection<CatalogItemDtoAbstract<?, ?>> scanAnnotationsFromLocal(ManagementContext mgmt) {
+ CatalogDto dto = CatalogDto.newNamedInstance("Local Scanned Catalog", "All annotated Brooklyn entities detected in the classpath", "scanning-local-classpath");
+ return scanAnnotationsInternal(mgmt, new CatalogDo(dto));
+ }
+ private Collection<CatalogItemDtoAbstract<?, ?>> scanAnnotationsFromBundles(ManagementContext mgmt, Collection<CatalogBundle> libraries) {
+ String[] urls = null;
+ CatalogDto dto = CatalogDto.newNamedInstance("Bundles Scanned Catalog", "All annotated Brooklyn entities detected in the classpath", "scanning-bundles-classpath-"+libraries.hashCode());
+ urls = new String[libraries.size()];
+ int i=0;
+ for (CatalogBundle b: libraries)
+ urls[i++] = b.getUrl();
+
+ CatalogDo subCatalog = new CatalogDo(dto);
+ subCatalog.addToClasspath(urls);
--- End diff --
I see that below you are handling the `null` case, but then if all libraries are name-value pairs, the code path will go in the "scanning catalog root classpath" case. Better move the null check here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by aledsage <gi...@git.apache.org>.
Github user aledsage commented on a diff in the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#discussion_r30543745
--- Diff: core/src/main/java/brooklyn/catalog/internal/CatalogInitialization.java ---
@@ -0,0 +1,377 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package brooklyn.catalog.internal;
+
+import java.io.File;
+import java.util.Collection;
+import java.util.List;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import brooklyn.catalog.CatalogItem;
+import brooklyn.config.BrooklynServerConfig;
+import brooklyn.management.ManagementContext;
+import brooklyn.management.ManagementContextInjectable;
+import brooklyn.management.internal.ManagementContextInternal;
+import brooklyn.util.ResourceUtils;
+import brooklyn.util.collections.MutableList;
+import brooklyn.util.exceptions.Exceptions;
+import brooklyn.util.exceptions.FatalRuntimeException;
+import brooklyn.util.exceptions.RuntimeInterruptedException;
+import brooklyn.util.flags.TypeCoercions;
+import brooklyn.util.guava.Maybe;
+import brooklyn.util.net.Urls;
+import brooklyn.util.text.Strings;
+
+import com.google.common.annotations.Beta;
+import com.google.common.base.Function;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.Iterables;
+
+@Beta
+public class CatalogInitialization implements ManagementContextInjectable {
+
+ /*
+
+ A1) if not persisting, go to B1
+ A2) if --catalog-reset, delete persisted catalog items
+ A3) read persisted catalog items (possibly deleted in A2), go to C1
+ A4) go to B1
+
+ B1) look for --catalog-initial, if so read it, then go to C1
+ B2) look for BrooklynServerConfig.BROOKLYN_CATALOG_URL, if so, read it, supporting YAML or XML (warning if XML), then go to C1
+ B3) look for ~/.brooklyn/catalog.bom, if exists, read it then go to C1
+ B4) look for ~/.brooklyn/brooklyn.xml, if exists, warn, read it then go to C1
+ B5) read all classpath://brooklyn/default.catalog.bom items, if they exist (and for now they will)
+ B6) go to C1
+
+ C1) if --catalog-add, read and add those items
+
+ D1) if persisting, read the rest of the persisted items (entities etc)
+
+ */
+
+ private static final Logger log = LoggerFactory.getLogger(CatalogInitialization.class);
+
+ String initialUri;
+ boolean reset;
+ String additionsUri;
+ boolean force;
+
+ boolean disallowLocal = false;
+ List<Function<CatalogInitialization, Void>> callbacks = MutableList.of();
+ boolean hasRunBestEffort = false, hasRunOfficial = false, isPopulating = false;
+
+ ManagementContext managementContext;
+ boolean isStartingUp = false;
+ boolean failOnStartupErrors = false;
+
+ Object mutex = new Object();
+
+ public CatalogInitialization(String initialUri, boolean reset, String additionUri, boolean force) {
+ this.initialUri = initialUri;
+ this.reset = reset;
+ this.additionsUri = additionUri;
+ this.force = force;
+ }
+
+ public CatalogInitialization() {
+ this(null, false, null, false);
+ }
+
+ public void injectManagementContext(ManagementContext managementContext) {
+ if (this.managementContext!=null && managementContext!=null && !this.managementContext.equals(managementContext))
--- End diff --
I'd prefer us to NPE if the injected managementContext is null. That would presumably be either a programming error or very lazy test code.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by aledsage <gi...@git.apache.org>.
Github user aledsage commented on a diff in the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#discussion_r30582994
--- Diff: core/src/main/java/brooklyn/entity/rebind/PeriodicDeltaChangeListener.java ---
@@ -348,13 +327,40 @@ private void addReferencedObjects(DeltaCollector deltaCollector) {
}
@VisibleForTesting
- public void persistNow() {
- if (!isActive()) {
+ public boolean persistNowSafely(boolean alreadyHasMutex) {
+ Stopwatch timer = Stopwatch.createStarted();
+ try {
+ persistNowInternal(alreadyHasMutex);
+ metrics.noteSuccess(Duration.of(timer));
+ return true;
+ } catch (RuntimeInterruptedException e) {
+ LOG.debug("Interrupted persisting change-delta (rethrowing)", e);
+ metrics.noteFailure(Duration.of(timer));
+ metrics.noteError(e.toString());
+ Thread.currentThread().interrupt();
+ return false;
+ } catch (Exception e) {
--- End diff --
I'd probably merge the `catch Exception` and the `catch Throwable`, with:
} catch (Throwable t) {
// Don't rethrow: the behaviour of executionManager is different from a scheduledExecutorService,
// if we throw an exception, then our task will never get executed again
LOG.error("Problem persisting change-delta", t);
metrics.noteFailure(Duration.of(timer));
metrics.noteError(t.toString());
Exceptions.propagateIfFatal(t);
return false;
Downside is that log message is always error, and doesn't include whether or not it is "(rethrowing)".
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by ahgittin <gi...@git.apache.org>.
Github user ahgittin commented on a diff in the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#discussion_r30967740
--- Diff: core/src/main/java/brooklyn/catalog/internal/CatalogInitialization.java ---
@@ -0,0 +1,380 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package brooklyn.catalog.internal;
+
+import java.io.File;
+import java.util.Collection;
+import java.util.List;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import brooklyn.catalog.CatalogItem;
+import brooklyn.config.BrooklynServerConfig;
+import brooklyn.management.ManagementContext;
+import brooklyn.management.ManagementContextInjectable;
+import brooklyn.management.internal.ManagementContextInternal;
+import brooklyn.util.ResourceUtils;
+import brooklyn.util.collections.MutableList;
+import brooklyn.util.exceptions.Exceptions;
+import brooklyn.util.exceptions.FatalRuntimeException;
+import brooklyn.util.exceptions.RuntimeInterruptedException;
+import brooklyn.util.flags.TypeCoercions;
+import brooklyn.util.guava.Maybe;
+import brooklyn.util.os.Os;
+import brooklyn.util.text.Strings;
+
+import com.google.common.annotations.Beta;
+import com.google.common.base.Function;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.Iterables;
+
+@Beta
+public class CatalogInitialization implements ManagementContextInjectable {
+
+ /*
+
+ A1) if not persisting, go to B1
+ A2) if --catalog-reset, delete persisted catalog items
+ A3) if there is a persisted catalog (and it wasn't not deleted by A2), read it and go to C1
+ A4) go to B1
+
+ B1) look for --catalog-initial, if so read it, then go to C1
+ B2) look for BrooklynServerConfig.BROOKLYN_CATALOG_URL, if so, read it, supporting YAML or XML (warning if XML), then go to C1
+ B3) look for ~/.brooklyn/catalog.bom, if exists, read it then go to C1
+ B4) look for ~/.brooklyn/brooklyn.xml, if exists, warn, read it then go to C1
+ B5) read all classpath://brooklyn/default.catalog.bom items, if they exist (and for now they will)
+ B6) go to C1
+
+ C1) if --catalog-add, read and add those items
+
+ D1) if persisting, read the rest of the persisted items (entities etc)
+
+ */
+
+ private static final Logger log = LoggerFactory.getLogger(CatalogInitialization.class);
+
+ private String initialUri;
+ private boolean reset;
+ private String additionsUri;
+ private boolean force;
+
+ private boolean disallowLocal = false;
+ private List<Function<CatalogInitialization, Void>> callbacks = MutableList.of();
+ private boolean hasRunBestEffort = false, hasRunOfficial = false, isPopulating = false;
+
+ private ManagementContext managementContext;
+ private boolean isStartingUp = false;
+ private boolean failOnStartupErrors = false;
+
+ private Object populatingCatalogMutex = new Object();
+
+ public CatalogInitialization(String initialUri, boolean reset, String additionUri, boolean force) {
+ this.initialUri = initialUri;
+ this.reset = reset;
+ this.additionsUri = additionUri;
+ this.force = force;
+ }
+
+ public CatalogInitialization() {
+ this(null, false, null, false);
+ }
+
+ public void injectManagementContext(ManagementContext managementContext) {
+ Preconditions.checkNotNull(managementContext, "management context");
+ if (this.managementContext!=null && managementContext!=this.managementContext)
+ throw new IllegalStateException("Cannot switch management context, from "+this.managementContext+" to "+managementContext);
+ this.managementContext = managementContext;
+ }
+
+ /** Called by the framework to set true while starting up, and false afterwards,
+ * in order to assist in appropriate logging and error handling. */
+ public void setStartingUp(boolean isStartingUp) {
+ this.isStartingUp = isStartingUp;
+ }
+
+ public void setFailOnStartupErrors(boolean startupFailOnCatalogErrors) {
+ this.failOnStartupErrors = startupFailOnCatalogErrors;
+ }
+
+ public CatalogInitialization addPopulationCallback(Function<CatalogInitialization, Void> callback) {
+ callbacks.add(callback);
+ return this;
+ }
+
+ public ManagementContext getManagementContext() {
+ return Preconditions.checkNotNull(managementContext, "management context has not been injected into "+this);
+ }
+
+ public boolean isInitialResetRequested() {
+ return reset;
+ }
+
+ public boolean hasRunOfficial() { return hasRunOfficial; }
+ public boolean hasRunIncludingBestEffort() { return hasRunOfficial || hasRunBestEffort; }
+
+ /** makes or updates the mgmt catalog, based on the settings in this class */
+ public void populateCatalog(boolean needsInitial, Collection<CatalogItem<?, ?>> optionalItemsForResettingCatalog) {
+ synchronized (populatingCatalogMutex) {
+ try {
+ isPopulating = true;
+ BasicBrooklynCatalog catalog = (BasicBrooklynCatalog) managementContext.getCatalog();
+ if (!catalog.getCatalog().isLoaded()) {
+ catalog.load();
+ } else {
+ if (needsInitial && (hasRunOfficial || hasRunBestEffort)) {
+ // an indication that something caused it to load early; not severe, but unusual
+ log.warn("Catalog initialization has not properly run but management context has a catalog; re-populating, possibly overwriting items installed during earlier access (it may have been an early web request)");
+ catalog.reset(ImmutableList.<CatalogItem<?,?>>of());
+ }
+ }
+ hasRunOfficial = true;
+
+ populateCatalogImpl(catalog, needsInitial, optionalItemsForResettingCatalog);
+ } finally {
+ hasRunOfficial = true;
+ isPopulating = false;
+ }
+ }
+ }
+
+ private void populateCatalogImpl(BasicBrooklynCatalog catalog, boolean needsInitial, Collection<CatalogItem<?, ?>> optionalItemsForResettingCatalog) {
+ applyCatalogLoadMode();
+
+ if (optionalItemsForResettingCatalog!=null) {
+ catalog.reset(optionalItemsForResettingCatalog);
+ }
+
+ if (needsInitial) {
+ populateInitial(catalog);
+ }
+
+ populateAdditions(catalog);
+
+ populateViaCallbacks(catalog);
+ }
+
+ private enum PopulateMode { YAML, XML, AUTODETECT }
+
+ protected void populateInitial(BasicBrooklynCatalog catalog) {
+ if (disallowLocal) {
+ if (!hasRunOfficial()) {
+ log.debug("CLI initial catalog not being read with disallow-local mode set.");
+ }
+ return;
--- End diff --
i see `initialUri` as often being embedded in a script, so that deliberately it only has effect on a new catalog-initializing run. so it is normal for options to be combined to mask it, hence to warning or failure. the log will say where catalog is being loaded from, and so it shouldn't be too hard to work out that the initial uri is not being applied, and the description is clear why it wouldn't be (it's not the initial run). also `CATALOG_LOAD_MODE` is now deprecated; it had a short shelf life so unlikely this code path will get executed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by aledsage <gi...@git.apache.org>.
Github user aledsage commented on a diff in the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#discussion_r30584097
--- Diff: core/src/test/java/brooklyn/catalog/internal/CatalogScanTest.java ---
@@ -161,13 +162,13 @@ public void testMoreTypesThanAnnotationsForApps() {
@Test
public void testAnnotationIsDefault() {
- loadAnnotationsOnlyCatalog();
loadTheDefaultCatalog();
+ int numInDefault = Iterables.size(defaultCatalog.getCatalogItems(Predicates.alwaysTrue()));
+ loadAnnotationsOnlyCatalog();
int numFromAnnots = Iterables.size(annotsCatalog.getCatalogItems(Predicates.alwaysTrue()));
- int numInDefault = Iterables.size(defaultCatalog.getCatalogItems(Predicates.alwaysTrue()));
- Assert.assertEquals(numFromAnnots, numInDefault);
+ Assert.assertEquals(numInDefault, numFromAnnots);
--- End diff --
Could also assert that `> 0`; otherwise could be a bug and a coincidence that both are zero?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by neykov <gi...@git.apache.org>.
Github user neykov commented on a diff in the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#discussion_r30813782
--- Diff: core/src/main/java/brooklyn/entity/rebind/PeriodicDeltaChangeListener.java ---
@@ -348,13 +336,40 @@ private void addReferencedObjects(DeltaCollector deltaCollector) {
}
@VisibleForTesting
- public void persistNow() {
- if (!isActive()) {
+ public boolean persistNowSafely(boolean alreadyHasMutex) {
--- End diff --
Since the mutex is private it makes sense to expose to the outside only with alreadyHasMutex=false (i.e. a separate public metod which passes false), but not important.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by ahgittin <gi...@git.apache.org>.
Github user ahgittin closed the pull request at:
https://github.com/apache/incubator-brooklyn/pull/617
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by ahgittin <gi...@git.apache.org>.
Github user ahgittin commented on a diff in the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#discussion_r30589477
--- Diff: core/src/main/java/brooklyn/catalog/internal/CatalogInitialization.java ---
@@ -0,0 +1,377 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package brooklyn.catalog.internal;
+
+import java.io.File;
+import java.util.Collection;
+import java.util.List;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import brooklyn.catalog.CatalogItem;
+import brooklyn.config.BrooklynServerConfig;
+import brooklyn.management.ManagementContext;
+import brooklyn.management.ManagementContextInjectable;
+import brooklyn.management.internal.ManagementContextInternal;
+import brooklyn.util.ResourceUtils;
+import brooklyn.util.collections.MutableList;
+import brooklyn.util.exceptions.Exceptions;
+import brooklyn.util.exceptions.FatalRuntimeException;
+import brooklyn.util.exceptions.RuntimeInterruptedException;
+import brooklyn.util.flags.TypeCoercions;
+import brooklyn.util.guava.Maybe;
+import brooklyn.util.net.Urls;
+import brooklyn.util.text.Strings;
+
+import com.google.common.annotations.Beta;
+import com.google.common.base.Function;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.Iterables;
+
+@Beta
+public class CatalogInitialization implements ManagementContextInjectable {
+
+ /*
+
+ A1) if not persisting, go to B1
+ A2) if --catalog-reset, delete persisted catalog items
+ A3) read persisted catalog items (possibly deleted in A2), go to C1
+ A4) go to B1
+
+ B1) look for --catalog-initial, if so read it, then go to C1
+ B2) look for BrooklynServerConfig.BROOKLYN_CATALOG_URL, if so, read it, supporting YAML or XML (warning if XML), then go to C1
+ B3) look for ~/.brooklyn/catalog.bom, if exists, read it then go to C1
+ B4) look for ~/.brooklyn/brooklyn.xml, if exists, warn, read it then go to C1
+ B5) read all classpath://brooklyn/default.catalog.bom items, if they exist (and for now they will)
+ B6) go to C1
+
+ C1) if --catalog-add, read and add those items
+
+ D1) if persisting, read the rest of the persisted items (entities etc)
+
+ */
+
+ private static final Logger log = LoggerFactory.getLogger(CatalogInitialization.class);
+
+ String initialUri;
+ boolean reset;
+ String additionsUri;
+ boolean force;
+
+ boolean disallowLocal = false;
+ List<Function<CatalogInitialization, Void>> callbacks = MutableList.of();
+ boolean hasRunBestEffort = false, hasRunOfficial = false, isPopulating = false;
+
+ ManagementContext managementContext;
+ boolean isStartingUp = false;
+ boolean failOnStartupErrors = false;
+
+ Object mutex = new Object();
+
+ public CatalogInitialization(String initialUri, boolean reset, String additionUri, boolean force) {
+ this.initialUri = initialUri;
+ this.reset = reset;
+ this.additionsUri = additionUri;
+ this.force = force;
+ }
+
+ public CatalogInitialization() {
+ this(null, false, null, false);
+ }
+
+ public void injectManagementContext(ManagementContext managementContext) {
+ if (this.managementContext!=null && managementContext!=null && !this.managementContext.equals(managementContext))
+ throw new IllegalStateException("Cannot switch management context of "+this+"; from "+this.managementContext+" to "+managementContext);
+ this.managementContext = managementContext;
+ }
+
+ public ManagementContext getManagementContext() {
+ return Preconditions.checkNotNull(managementContext, "management context has not been injected into "+this);
+ }
+
+ public CatalogInitialization addPopulationCallback(Function<CatalogInitialization, Void> callback) {
+ callbacks.add(callback);
+ return this;
+ }
+
+ public boolean isInitialResetRequested() {
+ return reset;
+ }
+
+ public boolean hasRunOfficial() { return hasRunOfficial; }
+ public boolean hasRunIncludingBestEffort() { return hasRunOfficial || hasRunBestEffort; }
+
+ /** makes or updates the mgmt catalog, based on the settings in this class */
+ public void populateCatalog(boolean needsInitial, Collection<CatalogItem<?, ?>> optionalItemsForResettingCatalog) {
+ try {
+ isPopulating = true;
+ synchronized (mutex) {
+ BasicBrooklynCatalog catalog = (BasicBrooklynCatalog) managementContext.getCatalog();
+ if (!catalog.getCatalog().isLoaded()) {
+ catalog.load();
+ } else {
+ if (needsInitial && (hasRunOfficial || hasRunBestEffort)) {
+ // an indication that something caused it to load early; not severe, but unusual
+ log.warn("Catalog initialization has not properly run but management context has a catalog; re-populating, possibly overwriting items installed during earlier access (it may have been an early web request)");
+ catalog.reset(ImmutableList.<CatalogItem<?,?>>of());
+ }
+ }
+ hasRunOfficial = true;
+
+ populateCatalog(catalog, needsInitial, true, optionalItemsForResettingCatalog);
+ }
+ } finally {
+ hasRunOfficial = true;
+ isPopulating = false;
+ }
+ }
+
+ private void populateCatalog(BasicBrooklynCatalog catalog, boolean needsInitial, boolean runCallbacks, Collection<CatalogItem<?, ?>> optionalItemsForResettingCatalog) {
+ applyCatalogLoadMode();
+
+ if (optionalItemsForResettingCatalog!=null) {
+ catalog.reset(optionalItemsForResettingCatalog);
+ }
+
+ if (needsInitial) {
+ populateInitial(catalog);
+ }
+
+ populateAdditions(catalog);
+
+ if (runCallbacks) {
+ populateViaCallbacks(catalog);
+ }
+ }
+
+ private enum PopulateMode { YAML, XML, AUTODETECT }
+
+ protected void populateInitial(BasicBrooklynCatalog catalog) {
+ if (disallowLocal) {
+ if (!hasRunOfficial()) {
+ log.debug("CLI initial catalog not being read with disallow-local mode set.");
+ }
+ return;
+ }
+
+// B1) look for --catalog-initial, if so read it, then go to C1
+// B2) look for BrooklynServerConfig.BROOKLYN_CATALOG_URL, if so, read it, supporting YAML or XML (warning if XML), then go to C1
+// B3) look for ~/.brooklyn/catalog.bom, if exists, read it then go to C1
+// B4) look for ~/.brooklyn/brooklyn.xml, if exists, warn, read it then go to C1
+// B5) read all classpath://brooklyn/default.catalog.bom items, if they exist (and for now they will)
+// B6) go to C1
+
+ if (initialUri!=null) {
+ populateInitialFromUri(catalog, initialUri, PopulateMode.AUTODETECT);
+ return;
+ }
+
+ String catalogUrl = managementContext.getConfig().getConfig(BrooklynServerConfig.BROOKLYN_CATALOG_URL);
+ if (Strings.isNonBlank(catalogUrl)) {
+ populateInitialFromUri(catalog, catalogUrl, PopulateMode.AUTODETECT);
+ return;
+ }
+
+ catalogUrl = Urls.mergePaths(BrooklynServerConfig.getMgmtBaseDir( managementContext.getConfig() ), "catalog.bom");
+ if (new File(catalogUrl).exists()) {
+ populateInitialFromUri(catalog, "file:"+catalogUrl, PopulateMode.YAML);
+ return;
+ }
+
+ catalogUrl = Urls.mergePaths(BrooklynServerConfig.getMgmtBaseDir( managementContext.getConfig() ), "catalog.xml");
+ if (new File(catalogUrl).exists()) {
+ populateInitialFromUri(catalog, "file:"+catalogUrl, PopulateMode.XML);
+ return;
+ }
+
+ // otherwise look for for classpath:/brooklyn/default.catalog.bom --
+ // there is one on the classpath which says to scan, and provides a few templates;
+ // if one is supplied by user in the conf/ dir that will override the item from the classpath
+ // (TBD - we might want to scan for all such bom's?)
+
+ catalogUrl = "classpath:/brooklyn/default.catalog.bom";
+ if (new ResourceUtils(this).doesUrlExist(catalogUrl)) {
+ populateInitialFromUri(catalog, catalogUrl, PopulateMode.YAML);
+ return;
+ }
+
+ log.info("No catalog found on classpath or specified; catalog will not be initialized.");
+ return;
+ }
+
+ private void populateInitialFromUri(BasicBrooklynCatalog catalog, String catalogUrl, PopulateMode mode) {
+ log.debug("Loading initial catalog from {}", catalogUrl);
+
+ Exception problem = null;
+ Object result = null;
+
+ String contents = null;
+ try {
+ contents = new ResourceUtils(this).getResourceAsString(catalogUrl);
+ } catch (Exception e) {
+ Exceptions.propagateIfFatal(e);
+ if (problem==null) problem = e;
+ }
+
+ if (contents!=null && (mode==PopulateMode.YAML || mode==PopulateMode.AUTODETECT)) {
+ // try YAML first
+ try {
+ catalog.reset(MutableList.<CatalogItem<?,?>>of());
+ result = catalog.addItems(contents);
+ } catch (Exception e) {
+ Exceptions.propagateIfFatal(e);
+ if (problem==null) problem = e;
+ }
+ }
+
+ if (result==null && contents!=null && (mode==PopulateMode.XML || mode==PopulateMode.AUTODETECT)) {
+ // then try XML
+ problem = populateInitialFromUriXml(catalog, catalogUrl, problem, contents);
+ }
+
+ if (result!=null) {
+ log.debug("Loaded initial catalog from {}: {}", catalogUrl, result);
+ }
+ if (problem!=null) {
+ log.warn("Error importing catalog from " + catalogUrl + ": " + problem, problem);
+ // TODO inform mgmt of error
+ }
+
+ }
+
+ // deprecated XML format
+ @SuppressWarnings("deprecation")
+ private Exception populateInitialFromUriXml(BasicBrooklynCatalog catalog, String catalogUrl, Exception problem, String contents) {
+ CatalogDto dto = null;
+ try {
+ dto = CatalogDto.newDtoFromXmlContents(contents, catalogUrl);
+ problem = null;
+ } catch (Exception e) {
+ Exceptions.propagateIfFatal(e);
+ if (problem==null) problem = e;
+ }
+ if (dto!=null) {
+ catalog.reset(dto);
+ }
+ return problem;
+ }
+
+ boolean hasRunAdditions = false;
+ protected void populateAdditions(BasicBrooklynCatalog catalog) {
+ if (Strings.isNonBlank(additionsUri)) {
+ if (disallowLocal) {
+ if (!hasRunAdditions) {
+ log.warn("CLI additions supplied but not supported in disallow-local mode; ignoring.");
+ }
+ return;
+ }
+ if (!hasRunAdditions) {
+ log.debug("Adding to catalog from CLI: "+additionsUri+" (force: "+force+")");
+ }
+ Iterable<? extends CatalogItem<?, ?>> items = catalog.addItems(
+ new ResourceUtils(this).getResourceAsString(additionsUri), force);
+
+ if (!hasRunAdditions)
+ log.debug("Added to catalog from CLI: "+items);
+ else
+ log.debug("Added to catalog from CLI: count "+Iterables.size(items));
+
+ hasRunAdditions = true;
+ }
+ }
+
+ protected void populateViaCallbacks(BasicBrooklynCatalog catalog) {
+ for (Function<CatalogInitialization, Void> callback: callbacks)
+ callback.apply(this);
+ }
+
+ private boolean setFromCatalogLoadMode = false;
+
+ /** @deprecated since introduced in 0.7.0, only for legacy compatibility with
+ * {@link CatalogLoadMode} {@link BrooklynServerConfig#CATALOG_LOAD_MODE},
+ * allowing control of catalog loading from a brooklyn property */
+ @Deprecated
+ public void applyCatalogLoadMode() {
+ if (setFromCatalogLoadMode) return;
+ setFromCatalogLoadMode = true;
+ Maybe<Object> clmm = ((ManagementContextInternal)managementContext).getConfig().getConfigRaw(BrooklynServerConfig.CATALOG_LOAD_MODE, false);
+ if (clmm.isAbsent()) return;
+ brooklyn.catalog.CatalogLoadMode clm = TypeCoercions.coerce(clmm.get(), brooklyn.catalog.CatalogLoadMode.class);
+ log.warn("Legacy CatalogLoadMode "+clm+" set: applying, but this should be changed to use new CLI --catalogXxx commands");
+ switch (clm) {
+ case LOAD_BROOKLYN_CATALOG_URL:
+ reset = true;
+ break;
+ case LOAD_BROOKLYN_CATALOG_URL_IF_NO_PERSISTED_STATE:
+ // now the default
+ break;
+ case LOAD_PERSISTED_STATE:
+ disallowLocal = true;
+ break;
+ }
+ }
+
+ /** makes the catalog, warning if persistence is on and hasn't run yet
+ * (as the catalog will be subsequently replaced) */
+ public void populateBestEffort(BasicBrooklynCatalog catalog) {
+ synchronized (mutex) {
+ if (hasRunOfficial || hasRunBestEffort || isPopulating) return;
+ // if a thread calls back in to this, ie calling to it from a getCatalog() call while populating,
+ // it will own the mutex and observe isRunningBestEffort, returning quickly
+ isPopulating = true;
+ try {
+ if (isStartingUp) {
+ log.warn("Catalog access requested when not yet initialized; populating best effort rather than through recommended pathway. Catalog data may be replaced subsequently.");
+ }
+ populateCatalog(catalog, true, true, null);
+ } finally {
+ hasRunBestEffort = true;
+ isPopulating = false;
+ }
+ }
+ }
+
+ public void setStartingUp(boolean isStartingUp) {
+ this.isStartingUp = isStartingUp;
+ }
+
+ public void setFailOnStartupErrors(boolean startupFailOnCatalogErrors) {
--- End diff --
agree, moved
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by neykov <gi...@git.apache.org>.
Github user neykov commented on a diff in the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#discussion_r30971988
--- Diff: core/src/main/java/brooklyn/entity/rebind/RebindIteration.java ---
@@ -341,38 +340,64 @@ protected void rebuildCatalog() {
}
}
}
+
+ // See notes in CatalogInitialization
- // Register catalogue items with the management context. Loads the bundles in the OSGi framework.
- CatalogLoadMode catalogLoadMode = managementContext.getConfig().getConfig(BrooklynServerConfig.CATALOG_LOAD_MODE);
+ Collection<CatalogItem<?, ?>> catalogItems = rebindContext.getCatalogItems();
+ CatalogInitialization catInit = ((ManagementContextInternal)managementContext).getCatalogInitialization();
+ catInit.applyCatalogLoadMode();
+ Collection<CatalogItem<?,?>> itemsForResettingCatalog = null;
+ boolean needsInitialCatalog;
if (rebindManager.persistCatalogItemsEnabled) {
- boolean shouldResetCatalog = catalogLoadMode == CatalogLoadMode.LOAD_PERSISTED_STATE
- || (!isEmpty && catalogLoadMode == CatalogLoadMode.LOAD_BROOKLYN_CATALOG_URL_IF_NO_PERSISTED_STATE);
- boolean shouldLoadDefaultCatalog = catalogLoadMode == CatalogLoadMode.LOAD_BROOKLYN_CATALOG_URL
- || (isEmpty && catalogLoadMode == CatalogLoadMode.LOAD_BROOKLYN_CATALOG_URL_IF_NO_PERSISTED_STATE);
- if (shouldResetCatalog) {
- // Reset catalog with previously persisted state
- logRebindingDebug("RebindManager resetting management context catalog to previously persisted state");
- managementContext.getCatalog().reset(rebindContext.getCatalogItems());
- } else if (shouldLoadDefaultCatalog) {
- // Load catalogue as normal
- // TODO in read-only mode, should do this less frequently than entities etc
- logRebindingDebug("RebindManager loading default catalog");
- ((BasicBrooklynCatalog) managementContext.getCatalog()).resetCatalogToContentsAtConfiguredUrl();
+ if (!catInit.hasRunOfficial() && catInit.isInitialResetRequested()) {
+ String message = "RebindManager resetting catalog on first run (catalog persistence enabled, but reset explicitly specified). ";
+ if (catalogItems.isEmpty()) {
+ message += "Catalog was empty anyway.";
+ } else {
+ message += "Deleting "+catalogItems.size()+" persisted catalog item"+Strings.s(catalogItems)+": "+catalogItems;
+ if (shouldLogRebinding()) {
+ LOG.info(message);
+ }
+ }
+ logRebindingDebug(message);
+
+ itemsForResettingCatalog = MutableList.<CatalogItem<?,?>>of();
--- End diff --
My first thought was that if we are resetting the catalog then there's no point in downloading all the bundles and dealing with missing bundles/catalog items, but these cases will result in soft errors (i.e. be logged only), won't abort the rebind.
Only schema changes will abort the rebind, which for catalog items is not so easy to do.
So there's no point in loading the old catalog if resetting, but won't hurt if we do so.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by aledsage <gi...@git.apache.org>.
Github user aledsage commented on a diff in the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#discussion_r30589120
--- Diff: usage/cli/src/main/java/brooklyn/cli/lister/ItemDescriptors.java ---
@@ -58,8 +65,17 @@
List<Map<String, Object>> itemDescriptors = Lists.newArrayList();
for (Class<? extends BrooklynObject> type : types) {
- Map<String, Object> itemDescriptor = toItemDescriptor(type, headingsOnly);
- itemDescriptors.add(itemDescriptor);
+ try {
+ Map<String, Object> itemDescriptor = toItemDescriptor(type, headingsOnly);
+ itemDescriptors.add(itemDescriptor);
+ } catch (Throwable throwable) {
+ if (throwable instanceof InterruptedException)
--- End diff --
Can just do `Exceptions.propagateIfFatal(throwable)`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by ahgittin <gi...@git.apache.org>.
Github user ahgittin commented on a diff in the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#discussion_r30659321
--- Diff: core/src/main/java/brooklyn/entity/rebind/PeriodicDeltaChangeListener.java ---
@@ -348,13 +327,40 @@ private void addReferencedObjects(DeltaCollector deltaCollector) {
}
@VisibleForTesting
- public void persistNow() {
- if (!isActive()) {
+ public boolean persistNowSafely(boolean alreadyHasMutex) {
+ Stopwatch timer = Stopwatch.createStarted();
+ try {
+ persistNowInternal(alreadyHasMutex);
+ metrics.noteSuccess(Duration.of(timer));
+ return true;
+ } catch (RuntimeInterruptedException e) {
+ LOG.debug("Interrupted persisting change-delta (rethrowing)", e);
+ metrics.noteFailure(Duration.of(timer));
+ metrics.noteError(e.toString());
+ Thread.currentThread().interrupt();
+ return false;
+ } catch (Exception e) {
--- End diff --
handling log messages correctly, with the right message, is important to prevent spurious errors on shutdown
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by aledsage <gi...@git.apache.org>.
Github user aledsage commented on a diff in the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#discussion_r30585763
--- Diff: usage/cli/src/main/java/brooklyn/cli/ItemLister.java ---
@@ -66,6 +68,9 @@
public class ItemLister {
--- End diff --
I didn't know about this before! Needs documented somewhere, perhaps in `docs/guide/ops/launch.md` where other CLI options are documented such as `brooklyn cloud-compute`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by neykov <gi...@git.apache.org>.
Github user neykov commented on a diff in the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#discussion_r30789390
--- Diff: core/src/main/java/brooklyn/catalog/internal/CatalogItemDtoAbstract.java ---
@@ -162,6 +163,37 @@ public String getPlanYaml() {
}
@Override
+ public int hashCode() {
+ return Objects.hashCode(symbolicName, planYaml, javaType, nullIfEmpty(libraries), version, getCatalogItemId());
+ }
+
+ @Override
+ public boolean equals(Object obj) {
+ if (this == obj) return true;
+ if (obj == null) return false;
+ if (getClass() != obj.getClass()) return false;
+ CatalogItemDtoAbstract<?,?> other = (CatalogItemDtoAbstract<?,?>) obj;
+ if (!Objects.equal(symbolicName, other.symbolicName)) return false;
+ if (!Objects.equal(planYaml, other.planYaml)) return false;
+ if (!Objects.equal(javaType, other.javaType)) return false;
+ if (!Objects.equal(nullIfEmpty(libraries), nullIfEmpty(other.libraries))) return false;
+ if (!Objects.equal(getCatalogItemId(), other.getCatalogItemId())) return false;
+ if (!Objects.equal(version, other.version)) return false;
+ if (!Objects.equal(deprecated, other.deprecated)) return false;
+ if (!Objects.equal(description, other.description)) return false;
+ if (!Objects.equal(displayName, other.displayName)) return false;
+ if (!Objects.equal(iconUrl, other.iconUrl)) return false;
+ if (!Objects.equal(tags, other.tags)) return false;
+ // 'type' not checked, because deprecated, we might want to allow removal in future
--- End diff --
`symbolicName` is set to `type.getName()`, so shouldn't matter.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-brooklyn pull request: CLI commands for manipulating cat...
Posted by aledsage <gi...@git.apache.org>.
Github user aledsage commented on a diff in the pull request:
https://github.com/apache/incubator-brooklyn/pull/617#discussion_r30589313
--- Diff: usage/dist/src/main/dist/conf/brooklyn/default.catalog.bom ---
@@ -0,0 +1,41 @@
+
+# this catalog bom is an illustration supplying a few useful sample items
+# and templates to get started using Brooklyn
+
+brooklyn.catalog:
+ version: 0.7.0-SNAPSHOT # BROOKLYN_VERSION
+ items:
+
+ # load everything in the classpath with a @Catalog annotation
+ - scanJavaAnnotations: true
+
+ - id: server
+ description: |
+ Provision a server, with customizable provisioning.properties and credentials installed,
+ but no other special software process or scripts executed.
+ item:
+ type: brooklyn.entity.basic.EmptySoftwareProcess
--- End diff --
I'd prefer if we have consistent structure, rather than demonstrating different possible structures within the same file (that will just confuse users, as to whether they mean the same thing).
Suggest we include `services: `, as is done in `server-template`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---