You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@brooklyn.apache.org by aledsage <gi...@git.apache.org> on 2016/05/03 18:12:04 UTC
[GitHub] brooklyn-server pull request: Fix JcloudsByonLocationResolver
GitHub user aledsage opened a pull request:
https://github.com/apache/brooklyn-server/pull/132
Fix JcloudsByonLocationResolver
- Previously, it created locations every time the resolver was called
(even if that was just to query about what it would produce), which
caused us to leak locations.
- Now it defers creating the locations (though doing this with
LocationSpec is not practical because of the relationship between
the JcloudsLocation and the machines)!
- Adds FixedListMachineProvisioningLocation.initialMachinesFactory,
for deferring the machine-lookup until the byon location is really
being created.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/aledsage/brooklyn-server fix/JcloudsByonLocationResolver
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/brooklyn-server/pull/132.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #132
----
commit bae0726d3a2ed984fa4a657168daf570b041625e
Author: Aled Sage <al...@gmail.com>
Date: 2016-05-03T15:58:32Z
Fix JcloudsByonLocationResolver
- Previously, it created locations every time the resolver was called
(even if that was just to query about what it would produce), which
caused us to leak locations.
- Now it defers creating the locations (though doing this with
LocationSpec is not practical because of the relationship between
the JcloudsLocation and the machines)!
- Adds FixedListMachineProvisioningLocation.initialMachinesFactory,
for deferring the machine-lookup until the byon location is really
being created.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] brooklyn-server pull request: BROOKLYN-259: Fix JcloudsByonLocationResolver
Posted by johnmccabe <gi...@git.apache.org>.
Github user johnmccabe commented on the pull request:
https://github.com/apache/brooklyn-server/pull/132
LGTM, thanks @aledsage
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] brooklyn-server pull request: BROOKLYN-259: Fix JcloudsByonLocatio...
Posted by ahgittin <gi...@git.apache.org>.
Github user ahgittin commented on the pull request:
https://github.com/apache/brooklyn-server/pull/132#issuecomment-218986193
@aledsage important comments from @johnmccabe need addressing before can merge
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] brooklyn-server pull request: BROOKLYN-259: Fix JcloudsByonLocatio...
Posted by sjcorbett <gi...@git.apache.org>.
Github user sjcorbett commented on the pull request:
https://github.com/apache/brooklyn-server/pull/132#issuecomment-217880451
Looks good, only a few minor comments. Believe @johnmccabe is testing this with a deployment.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] brooklyn-server pull request: BROOKLYN-259: Fix JcloudsByonLocatio...
Posted by johnmccabe <gi...@git.apache.org>.
Github user johnmccabe commented on the pull request:
https://github.com/apache/brooklyn-server/pull/132#issuecomment-217979935
Looks like we're not overriding the `null` credentials returned in the `NodeMetadata` in JcloudsLocation.findNodeOrThrow. These should be getting replaced with the user/password set in the `ConfigBag`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] brooklyn-server pull request: BROOKLYN-259: Fix JcloudsByonLocatio...
Posted by sjcorbett <gi...@git.apache.org>.
Github user sjcorbett commented on a diff in the pull request:
https://github.com/apache/brooklyn-server/pull/132#discussion_r62505553
--- Diff: locations/jclouds/src/main/java/org/apache/brooklyn/location/jclouds/JcloudsByonLocationResolver.java ---
@@ -61,129 +56,140 @@
* @author aled
*/
@SuppressWarnings({"unchecked","rawtypes"})
-public class JcloudsByonLocationResolver implements LocationResolver {
+public class JcloudsByonLocationResolver extends AbstractLocationResolver implements LocationResolver {
public static final Logger log = LoggerFactory.getLogger(JcloudsByonLocationResolver.class);
public static final String BYON = "jcloudsByon";
- private static final Pattern PATTERN = Pattern.compile("("+BYON+"|"+BYON.toUpperCase()+")" + ":" + "\\((.*)\\)$");
-
- private ManagementContext managementContext;
-
@Override
- public void init(ManagementContext managementContext) {
- this.managementContext = checkNotNull(managementContext, "managementContext");
+ public boolean isEnabled() {
+ return LocationConfigUtils.isResolverPrefixEnabled(managementContext, getPrefix());
}
@Override
- public boolean isEnabled() {
- return LocationConfigUtils.isResolverPrefixEnabled(managementContext, getPrefix());
+ public String getPrefix() {
+ return BYON;
}
- // TODO Remove some duplication from JcloudsResolver; needs more careful review
@Override
- public LocationSpec<? extends Location> newLocationSpecFromString(String spec, Map<?, ?> locationFlags, LocationRegistry registry) {
- Map globalProperties = registry.getProperties();
+ protected Class<? extends Location> getLocationType() {
+ return FixedListMachineProvisioningLocation.class;
+ }
- Matcher matcher = PATTERN.matcher(spec);
- if (!matcher.matches()) {
- throw new IllegalArgumentException("Invalid location '"+spec+"'; must specify something like jcloudsByon(provider=\"aws-ec2\",region=\"us-east-1\",hosts=\"i-f2014593,i-d1234567\")");
- }
-
- String argsPart = matcher.group(2);
- Map<String, String> argsMap = KeyValueParser.parseMap(argsPart);
-
- // prefer args map over location flags
-
- String namedLocation = (String) locationFlags.get(LocationInternal.NAMED_SPEC_NAME.getName());
+ @Override
+ protected SpecParser getSpecParser() {
+ return new AbstractLocationResolver.SpecParser(getPrefix()).setExampleUsage("\"jcloudsByon(provider='aws-ec2',region='us-east-1',hosts='i-12345678,i-90123456')\"");
+ }
+
+ @Override
+ protected ConfigBag extractConfig(Map<?,?> locationFlags, String spec, final LocationRegistry registry) {
+ ConfigBag config = super.extractConfig(locationFlags, spec, registry);
- String providerOrApi = argsMap.containsKey("provider") ? argsMap.get("provider") : (String)locationFlags.get("provider");
+ String providerOrApi = (String) config.getStringKey("provider");
+ String regionName = (String) config.getStringKey("region");
+ String endpoint = (String) config.getStringKey("endpoint");
+ String namedLocation = (String) config.get(LocationInternal.NAMED_SPEC_NAME);
+ config.remove(LocationInternal.NAMED_SPEC_NAME.getName());
- String regionName = argsMap.containsKey("region") ? argsMap.get("region") : (String)locationFlags.get("region");
-
- String endpoint = argsMap.containsKey("endpoint") ? argsMap.get("endpoint") : (String)locationFlags.get("endpoint");
-
- String name = argsMap.containsKey("name") ? argsMap.get("name") : (String)locationFlags.get("name");
+ Object hosts = config.getStringKey("hosts");
+ config.remove("hosts");
- String user = argsMap.containsKey("user") ? argsMap.get("user") : (String)locationFlags.get("user");
+ Map jcloudsProperties = new JcloudsPropertiesFromBrooklynProperties().getJcloudsProperties(providerOrApi, regionName, namedLocation, config.getAllConfig());
+ config.putIfAbsent(jcloudsProperties);
- String privateKeyFile = argsMap.containsKey("privateKeyFile") ? argsMap.get("privateKeyFile") : (String)locationFlags.get("privateKeyFile");
-
- String hosts = argsMap.get("hosts");
-
if (Strings.isEmpty(providerOrApi)) {
throw new IllegalArgumentException("Invalid location '"+spec+"'; provider must be defined");
}
- if (hosts == null || hosts.isEmpty()) {
+ if (hosts == null || (hosts instanceof String && Strings.isBlank((String)hosts))) {
throw new IllegalArgumentException("Invalid location '"+spec+"'; at least one host must be defined");
}
- if (argsMap.containsKey("name") && (Strings.isEmpty(name))) {
- throw new IllegalArgumentException("Invalid location '"+spec+"'; if name supplied then value must be non-empty");
+
+ final String jcloudsSpec = "jclouds:"+providerOrApi + (regionName != null ? ":"+regionName : "") + (endpoint != null ? ":"+endpoint : "");
+ final Maybe<LocationSpec<? extends Location>> jcloudsLocationSpec = registry.getLocationSpec(jcloudsSpec, config.getAllConfig());
+ if (jcloudsLocationSpec.isAbsent()) {
+ throw new IllegalArgumentException("Invalid location '"+spec+"'; referrenced jclouds spec '"+jcloudsSpec+"' cannot be resolved");
+ } else if (!JcloudsLocation.class.isAssignableFrom(jcloudsLocationSpec.get().getType())) {
+ throw new IllegalArgumentException("Invalid location '"+spec+"'; referrenced spec '"+jcloudsSpec+"' of type "+jcloudsLocationSpec.get().getType()+" rather than JcloudsLocation");
}
- // For everything in brooklyn.properties, only use things with correct prefix (and remove that prefix).
- // But for everything passed in via locationFlags, pass those as-is.
- // TODO Should revisit the locationFlags: where are these actually used? Reason accepting properties without
- // full prefix is that the map's context is explicitly this location, rather than being generic properties.
- Map allProperties = getAllProperties(registry, globalProperties);
- Map jcloudsProperties = new JcloudsPropertiesFromBrooklynProperties().getJcloudsProperties(providerOrApi, regionName, namedLocation, allProperties);
- jcloudsProperties.putAll(locationFlags);
- jcloudsProperties.putAll(argsMap);
+ List<?> hostIds;
- String jcloudsSpec = "jclouds:"+providerOrApi + (regionName != null ? ":"+regionName : "") + (endpoint != null ? ":"+endpoint : "");
- JcloudsLocation jcloudsLocation = (JcloudsLocation) registry.resolve(jcloudsSpec, jcloudsProperties);
+ if (hosts instanceof String) {
+ if (((String) hosts).isEmpty()) {
+ hostIds = ImmutableList.of();
+ } else {
+ hostIds = WildcardGlobs.getGlobsAfterBraceExpansion("{"+hosts+"}",
+ true /* numeric */, /* no quote support though */ PhraseTreatment.NOT_A_SPECIAL_CHAR, PhraseTreatment.NOT_A_SPECIAL_CHAR);
+ }
+ } else if (hosts instanceof Iterable) {
+ hostIds = ImmutableList.copyOf((Iterable<?>)hosts);
+ } else {
+ throw new IllegalArgumentException("Invalid location '"+spec+"'; at least one host must be defined");
+ }
+ if (hostIds.isEmpty()) {
+ throw new IllegalArgumentException("Invalid location '"+spec+"'; at least one host must be defined");
+ }
- List<String> hostIdentifiers = WildcardGlobs.getGlobsAfterBraceExpansion("{"+hosts+"}",
- true /* numeric */, /* no quote support though */ PhraseTreatment.NOT_A_SPECIAL_CHAR, PhraseTreatment.NOT_A_SPECIAL_CHAR);
- List<JcloudsSshMachineLocation> machines = Lists.newArrayList();
-
- for (String hostIdentifier : hostIdentifiers) {
- Map<?, ?> machineFlags = MutableMap.builder()
- .put("id", hostIdentifier)
- .putIfNotNull("user", user)
- .putIfNotNull("privateKeyFile", privateKeyFile)
- .build();
- try {
- // TODO management of these machines may be odd, as it is passed in as a key as config to a spec
- JcloudsSshMachineLocation machine = jcloudsLocation.rebindMachine(jcloudsLocation.config().getBag().putAll(machineFlags));
- machines.add(machine);
- } catch (NoMachinesAvailableException e) {
- log.warn("Error rebinding to jclouds machine "+hostIdentifier+" in "+jcloudsLocation, e);
- Exceptions.propagate(e);
+ final List<Map<?,?>> machinesFlags = Lists.newArrayList();
+ for (Object hostId : hostIds) {
+ Map<?, ?> machineFlags;
+ if (hostId instanceof String) {
+ machineFlags = parseMachineFlags((String)hostId, config);
+ } else if (hostId instanceof Map) {
+ machineFlags = parseMachineFlags((Map<String,?>)hostId, config);
+ } else {
+ throw new IllegalArgumentException("Invalid host type '"+(hostId == null ? null : hostId.getClass().getName())+", referrenced in spec '"+spec);
}
+ machinesFlags.add(machineFlags);
}
+ Supplier<List<JcloudsMachineLocation>> machinesFactory = new Supplier<List<JcloudsMachineLocation>>() {
+ @Override
+ public List<JcloudsMachineLocation> get() {
+ // FIXME Can't change the parent of the JcloudsMachineLocation; but the FixedListMachineLocation will expect it to be the parent?
--- End diff --
Is this tracked in Jira?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] brooklyn-server pull request: BROOKLYN-259: Fix JcloudsByonLocatio...
Posted by aledsage <gi...@git.apache.org>.
Github user aledsage commented on a diff in the pull request:
https://github.com/apache/brooklyn-server/pull/132#discussion_r63903569
--- Diff: locations/jclouds/src/main/java/org/apache/brooklyn/location/jclouds/JcloudsByonLocationResolver.java ---
@@ -61,129 +56,140 @@
* @author aled
*/
@SuppressWarnings({"unchecked","rawtypes"})
-public class JcloudsByonLocationResolver implements LocationResolver {
+public class JcloudsByonLocationResolver extends AbstractLocationResolver implements LocationResolver {
public static final Logger log = LoggerFactory.getLogger(JcloudsByonLocationResolver.class);
public static final String BYON = "jcloudsByon";
- private static final Pattern PATTERN = Pattern.compile("("+BYON+"|"+BYON.toUpperCase()+")" + ":" + "\\((.*)\\)$");
-
- private ManagementContext managementContext;
-
@Override
- public void init(ManagementContext managementContext) {
- this.managementContext = checkNotNull(managementContext, "managementContext");
+ public boolean isEnabled() {
+ return LocationConfigUtils.isResolverPrefixEnabled(managementContext, getPrefix());
}
@Override
- public boolean isEnabled() {
- return LocationConfigUtils.isResolverPrefixEnabled(managementContext, getPrefix());
+ public String getPrefix() {
+ return BYON;
}
- // TODO Remove some duplication from JcloudsResolver; needs more careful review
@Override
- public LocationSpec<? extends Location> newLocationSpecFromString(String spec, Map<?, ?> locationFlags, LocationRegistry registry) {
- Map globalProperties = registry.getProperties();
+ protected Class<? extends Location> getLocationType() {
+ return FixedListMachineProvisioningLocation.class;
+ }
- Matcher matcher = PATTERN.matcher(spec);
- if (!matcher.matches()) {
- throw new IllegalArgumentException("Invalid location '"+spec+"'; must specify something like jcloudsByon(provider=\"aws-ec2\",region=\"us-east-1\",hosts=\"i-f2014593,i-d1234567\")");
- }
-
- String argsPart = matcher.group(2);
- Map<String, String> argsMap = KeyValueParser.parseMap(argsPart);
-
- // prefer args map over location flags
-
- String namedLocation = (String) locationFlags.get(LocationInternal.NAMED_SPEC_NAME.getName());
+ @Override
+ protected SpecParser getSpecParser() {
+ return new AbstractLocationResolver.SpecParser(getPrefix()).setExampleUsage("\"jcloudsByon(provider='aws-ec2',region='us-east-1',hosts='i-12345678,i-90123456')\"");
+ }
+
+ @Override
+ protected ConfigBag extractConfig(Map<?,?> locationFlags, String spec, final LocationRegistry registry) {
+ ConfigBag config = super.extractConfig(locationFlags, spec, registry);
- String providerOrApi = argsMap.containsKey("provider") ? argsMap.get("provider") : (String)locationFlags.get("provider");
+ String providerOrApi = (String) config.getStringKey("provider");
+ String regionName = (String) config.getStringKey("region");
+ String endpoint = (String) config.getStringKey("endpoint");
+ String namedLocation = (String) config.get(LocationInternal.NAMED_SPEC_NAME);
+ config.remove(LocationInternal.NAMED_SPEC_NAME.getName());
- String regionName = argsMap.containsKey("region") ? argsMap.get("region") : (String)locationFlags.get("region");
-
- String endpoint = argsMap.containsKey("endpoint") ? argsMap.get("endpoint") : (String)locationFlags.get("endpoint");
-
- String name = argsMap.containsKey("name") ? argsMap.get("name") : (String)locationFlags.get("name");
+ Object hosts = config.getStringKey("hosts");
+ config.remove("hosts");
- String user = argsMap.containsKey("user") ? argsMap.get("user") : (String)locationFlags.get("user");
+ Map jcloudsProperties = new JcloudsPropertiesFromBrooklynProperties().getJcloudsProperties(providerOrApi, regionName, namedLocation, config.getAllConfig());
+ config.putIfAbsent(jcloudsProperties);
- String privateKeyFile = argsMap.containsKey("privateKeyFile") ? argsMap.get("privateKeyFile") : (String)locationFlags.get("privateKeyFile");
-
- String hosts = argsMap.get("hosts");
-
if (Strings.isEmpty(providerOrApi)) {
throw new IllegalArgumentException("Invalid location '"+spec+"'; provider must be defined");
}
- if (hosts == null || hosts.isEmpty()) {
+ if (hosts == null || (hosts instanceof String && Strings.isBlank((String)hosts))) {
throw new IllegalArgumentException("Invalid location '"+spec+"'; at least one host must be defined");
}
- if (argsMap.containsKey("name") && (Strings.isEmpty(name))) {
- throw new IllegalArgumentException("Invalid location '"+spec+"'; if name supplied then value must be non-empty");
+
+ final String jcloudsSpec = "jclouds:"+providerOrApi + (regionName != null ? ":"+regionName : "") + (endpoint != null ? ":"+endpoint : "");
+ final Maybe<LocationSpec<? extends Location>> jcloudsLocationSpec = registry.getLocationSpec(jcloudsSpec, config.getAllConfig());
+ if (jcloudsLocationSpec.isAbsent()) {
--- End diff --
Changing to `if (jcloudsLocationSpec.isAbsentOrNull())`.
I really don't get why we're using `Maybe` here, rather than `Optional`. The contract would be far more sensible if `getLocationSpec` guaranteed to never return `Maybe.of(null)`! But we can't change the return type without breaking backwards compatibility :-(
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] brooklyn-server pull request: BROOKLYN-259: Fix JcloudsByonLocatio...
Posted by sjcorbett <gi...@git.apache.org>.
Github user sjcorbett commented on a diff in the pull request:
https://github.com/apache/brooklyn-server/pull/132#discussion_r62505309
--- Diff: locations/jclouds/src/main/java/org/apache/brooklyn/location/jclouds/JcloudsByonLocationResolver.java ---
@@ -61,129 +56,140 @@
* @author aled
*/
@SuppressWarnings({"unchecked","rawtypes"})
-public class JcloudsByonLocationResolver implements LocationResolver {
+public class JcloudsByonLocationResolver extends AbstractLocationResolver implements LocationResolver {
public static final Logger log = LoggerFactory.getLogger(JcloudsByonLocationResolver.class);
public static final String BYON = "jcloudsByon";
- private static final Pattern PATTERN = Pattern.compile("("+BYON+"|"+BYON.toUpperCase()+")" + ":" + "\\((.*)\\)$");
-
- private ManagementContext managementContext;
-
@Override
- public void init(ManagementContext managementContext) {
- this.managementContext = checkNotNull(managementContext, "managementContext");
+ public boolean isEnabled() {
+ return LocationConfigUtils.isResolverPrefixEnabled(managementContext, getPrefix());
}
@Override
- public boolean isEnabled() {
- return LocationConfigUtils.isResolverPrefixEnabled(managementContext, getPrefix());
+ public String getPrefix() {
+ return BYON;
}
- // TODO Remove some duplication from JcloudsResolver; needs more careful review
@Override
- public LocationSpec<? extends Location> newLocationSpecFromString(String spec, Map<?, ?> locationFlags, LocationRegistry registry) {
- Map globalProperties = registry.getProperties();
+ protected Class<? extends Location> getLocationType() {
+ return FixedListMachineProvisioningLocation.class;
+ }
- Matcher matcher = PATTERN.matcher(spec);
- if (!matcher.matches()) {
- throw new IllegalArgumentException("Invalid location '"+spec+"'; must specify something like jcloudsByon(provider=\"aws-ec2\",region=\"us-east-1\",hosts=\"i-f2014593,i-d1234567\")");
- }
-
- String argsPart = matcher.group(2);
- Map<String, String> argsMap = KeyValueParser.parseMap(argsPart);
-
- // prefer args map over location flags
-
- String namedLocation = (String) locationFlags.get(LocationInternal.NAMED_SPEC_NAME.getName());
+ @Override
+ protected SpecParser getSpecParser() {
+ return new AbstractLocationResolver.SpecParser(getPrefix()).setExampleUsage("\"jcloudsByon(provider='aws-ec2',region='us-east-1',hosts='i-12345678,i-90123456')\"");
+ }
+
+ @Override
+ protected ConfigBag extractConfig(Map<?,?> locationFlags, String spec, final LocationRegistry registry) {
+ ConfigBag config = super.extractConfig(locationFlags, spec, registry);
- String providerOrApi = argsMap.containsKey("provider") ? argsMap.get("provider") : (String)locationFlags.get("provider");
+ String providerOrApi = (String) config.getStringKey("provider");
+ String regionName = (String) config.getStringKey("region");
+ String endpoint = (String) config.getStringKey("endpoint");
+ String namedLocation = (String) config.get(LocationInternal.NAMED_SPEC_NAME);
+ config.remove(LocationInternal.NAMED_SPEC_NAME.getName());
- String regionName = argsMap.containsKey("region") ? argsMap.get("region") : (String)locationFlags.get("region");
-
- String endpoint = argsMap.containsKey("endpoint") ? argsMap.get("endpoint") : (String)locationFlags.get("endpoint");
-
- String name = argsMap.containsKey("name") ? argsMap.get("name") : (String)locationFlags.get("name");
+ Object hosts = config.getStringKey("hosts");
+ config.remove("hosts");
- String user = argsMap.containsKey("user") ? argsMap.get("user") : (String)locationFlags.get("user");
+ Map jcloudsProperties = new JcloudsPropertiesFromBrooklynProperties().getJcloudsProperties(providerOrApi, regionName, namedLocation, config.getAllConfig());
+ config.putIfAbsent(jcloudsProperties);
- String privateKeyFile = argsMap.containsKey("privateKeyFile") ? argsMap.get("privateKeyFile") : (String)locationFlags.get("privateKeyFile");
-
- String hosts = argsMap.get("hosts");
-
if (Strings.isEmpty(providerOrApi)) {
throw new IllegalArgumentException("Invalid location '"+spec+"'; provider must be defined");
}
- if (hosts == null || hosts.isEmpty()) {
+ if (hosts == null || (hosts instanceof String && Strings.isBlank((String)hosts))) {
throw new IllegalArgumentException("Invalid location '"+spec+"'; at least one host must be defined");
}
- if (argsMap.containsKey("name") && (Strings.isEmpty(name))) {
- throw new IllegalArgumentException("Invalid location '"+spec+"'; if name supplied then value must be non-empty");
+
+ final String jcloudsSpec = "jclouds:"+providerOrApi + (regionName != null ? ":"+regionName : "") + (endpoint != null ? ":"+endpoint : "");
+ final Maybe<LocationSpec<? extends Location>> jcloudsLocationSpec = registry.getLocationSpec(jcloudsSpec, config.getAllConfig());
+ if (jcloudsLocationSpec.isAbsent()) {
--- End diff --
Check `!isPresentAndNonNull` instead?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] brooklyn-server pull request: BROOKLYN-259: Fix JcloudsByonLocatio...
Posted by johnmccabe <gi...@git.apache.org>.
Github user johnmccabe commented on a diff in the pull request:
https://github.com/apache/brooklyn-server/pull/132#discussion_r62524649
--- Diff: locations/jclouds/src/main/java/org/apache/brooklyn/location/jclouds/JcloudsByonLocationResolver.java ---
@@ -61,129 +56,140 @@
* @author aled
*/
@SuppressWarnings({"unchecked","rawtypes"})
-public class JcloudsByonLocationResolver implements LocationResolver {
+public class JcloudsByonLocationResolver extends AbstractLocationResolver implements LocationResolver {
public static final Logger log = LoggerFactory.getLogger(JcloudsByonLocationResolver.class);
public static final String BYON = "jcloudsByon";
- private static final Pattern PATTERN = Pattern.compile("("+BYON+"|"+BYON.toUpperCase()+")" + ":" + "\\((.*)\\)$");
-
- private ManagementContext managementContext;
-
@Override
- public void init(ManagementContext managementContext) {
- this.managementContext = checkNotNull(managementContext, "managementContext");
+ public boolean isEnabled() {
+ return LocationConfigUtils.isResolverPrefixEnabled(managementContext, getPrefix());
}
@Override
- public boolean isEnabled() {
- return LocationConfigUtils.isResolverPrefixEnabled(managementContext, getPrefix());
+ public String getPrefix() {
+ return BYON;
}
- // TODO Remove some duplication from JcloudsResolver; needs more careful review
@Override
- public LocationSpec<? extends Location> newLocationSpecFromString(String spec, Map<?, ?> locationFlags, LocationRegistry registry) {
- Map globalProperties = registry.getProperties();
+ protected Class<? extends Location> getLocationType() {
+ return FixedListMachineProvisioningLocation.class;
+ }
- Matcher matcher = PATTERN.matcher(spec);
- if (!matcher.matches()) {
- throw new IllegalArgumentException("Invalid location '"+spec+"'; must specify something like jcloudsByon(provider=\"aws-ec2\",region=\"us-east-1\",hosts=\"i-f2014593,i-d1234567\")");
- }
-
- String argsPart = matcher.group(2);
- Map<String, String> argsMap = KeyValueParser.parseMap(argsPart);
-
- // prefer args map over location flags
-
- String namedLocation = (String) locationFlags.get(LocationInternal.NAMED_SPEC_NAME.getName());
+ @Override
+ protected SpecParser getSpecParser() {
+ return new AbstractLocationResolver.SpecParser(getPrefix()).setExampleUsage("\"jcloudsByon(provider='aws-ec2',region='us-east-1',hosts='i-12345678,i-90123456')\"");
+ }
+
+ @Override
+ protected ConfigBag extractConfig(Map<?,?> locationFlags, String spec, final LocationRegistry registry) {
+ ConfigBag config = super.extractConfig(locationFlags, spec, registry);
- String providerOrApi = argsMap.containsKey("provider") ? argsMap.get("provider") : (String)locationFlags.get("provider");
+ String providerOrApi = (String) config.getStringKey("provider");
+ String regionName = (String) config.getStringKey("region");
+ String endpoint = (String) config.getStringKey("endpoint");
+ String namedLocation = (String) config.get(LocationInternal.NAMED_SPEC_NAME);
+ config.remove(LocationInternal.NAMED_SPEC_NAME.getName());
- String regionName = argsMap.containsKey("region") ? argsMap.get("region") : (String)locationFlags.get("region");
-
- String endpoint = argsMap.containsKey("endpoint") ? argsMap.get("endpoint") : (String)locationFlags.get("endpoint");
-
- String name = argsMap.containsKey("name") ? argsMap.get("name") : (String)locationFlags.get("name");
+ Object hosts = config.getStringKey("hosts");
+ config.remove("hosts");
- String user = argsMap.containsKey("user") ? argsMap.get("user") : (String)locationFlags.get("user");
+ Map jcloudsProperties = new JcloudsPropertiesFromBrooklynProperties().getJcloudsProperties(providerOrApi, regionName, namedLocation, config.getAllConfig());
+ config.putIfAbsent(jcloudsProperties);
- String privateKeyFile = argsMap.containsKey("privateKeyFile") ? argsMap.get("privateKeyFile") : (String)locationFlags.get("privateKeyFile");
-
- String hosts = argsMap.get("hosts");
-
if (Strings.isEmpty(providerOrApi)) {
throw new IllegalArgumentException("Invalid location '"+spec+"'; provider must be defined");
}
- if (hosts == null || hosts.isEmpty()) {
+ if (hosts == null || (hosts instanceof String && Strings.isBlank((String)hosts))) {
throw new IllegalArgumentException("Invalid location '"+spec+"'; at least one host must be defined");
}
- if (argsMap.containsKey("name") && (Strings.isEmpty(name))) {
- throw new IllegalArgumentException("Invalid location '"+spec+"'; if name supplied then value must be non-empty");
+
+ final String jcloudsSpec = "jclouds:"+providerOrApi + (regionName != null ? ":"+regionName : "") + (endpoint != null ? ":"+endpoint : "");
+ final Maybe<LocationSpec<? extends Location>> jcloudsLocationSpec = registry.getLocationSpec(jcloudsSpec, config.getAllConfig());
+ if (jcloudsLocationSpec.isAbsent()) {
+ throw new IllegalArgumentException("Invalid location '"+spec+"'; referrenced jclouds spec '"+jcloudsSpec+"' cannot be resolved");
+ } else if (!JcloudsLocation.class.isAssignableFrom(jcloudsLocationSpec.get().getType())) {
+ throw new IllegalArgumentException("Invalid location '"+spec+"'; referrenced spec '"+jcloudsSpec+"' of type "+jcloudsLocationSpec.get().getType()+" rather than JcloudsLocation");
}
- // For everything in brooklyn.properties, only use things with correct prefix (and remove that prefix).
- // But for everything passed in via locationFlags, pass those as-is.
- // TODO Should revisit the locationFlags: where are these actually used? Reason accepting properties without
- // full prefix is that the map's context is explicitly this location, rather than being generic properties.
- Map allProperties = getAllProperties(registry, globalProperties);
- Map jcloudsProperties = new JcloudsPropertiesFromBrooklynProperties().getJcloudsProperties(providerOrApi, regionName, namedLocation, allProperties);
- jcloudsProperties.putAll(locationFlags);
- jcloudsProperties.putAll(argsMap);
+ List<?> hostIds;
- String jcloudsSpec = "jclouds:"+providerOrApi + (regionName != null ? ":"+regionName : "") + (endpoint != null ? ":"+endpoint : "");
- JcloudsLocation jcloudsLocation = (JcloudsLocation) registry.resolve(jcloudsSpec, jcloudsProperties);
+ if (hosts instanceof String) {
+ if (((String) hosts).isEmpty()) {
+ hostIds = ImmutableList.of();
+ } else {
+ hostIds = WildcardGlobs.getGlobsAfterBraceExpansion("{"+hosts+"}",
+ true /* numeric */, /* no quote support though */ PhraseTreatment.NOT_A_SPECIAL_CHAR, PhraseTreatment.NOT_A_SPECIAL_CHAR);
+ }
+ } else if (hosts instanceof Iterable) {
+ hostIds = ImmutableList.copyOf((Iterable<?>)hosts);
+ } else {
+ throw new IllegalArgumentException("Invalid location '"+spec+"'; at least one host must be defined");
+ }
+ if (hostIds.isEmpty()) {
+ throw new IllegalArgumentException("Invalid location '"+spec+"'; at least one host must be defined");
+ }
- List<String> hostIdentifiers = WildcardGlobs.getGlobsAfterBraceExpansion("{"+hosts+"}",
- true /* numeric */, /* no quote support though */ PhraseTreatment.NOT_A_SPECIAL_CHAR, PhraseTreatment.NOT_A_SPECIAL_CHAR);
- List<JcloudsSshMachineLocation> machines = Lists.newArrayList();
-
- for (String hostIdentifier : hostIdentifiers) {
- Map<?, ?> machineFlags = MutableMap.builder()
- .put("id", hostIdentifier)
- .putIfNotNull("user", user)
- .putIfNotNull("privateKeyFile", privateKeyFile)
- .build();
- try {
- // TODO management of these machines may be odd, as it is passed in as a key as config to a spec
- JcloudsSshMachineLocation machine = jcloudsLocation.rebindMachine(jcloudsLocation.config().getBag().putAll(machineFlags));
- machines.add(machine);
- } catch (NoMachinesAvailableException e) {
- log.warn("Error rebinding to jclouds machine "+hostIdentifier+" in "+jcloudsLocation, e);
- Exceptions.propagate(e);
+ final List<Map<?,?>> machinesFlags = Lists.newArrayList();
+ for (Object hostId : hostIds) {
+ Map<?, ?> machineFlags;
+ if (hostId instanceof String) {
+ machineFlags = parseMachineFlags((String)hostId, config);
+ } else if (hostId instanceof Map) {
+ machineFlags = parseMachineFlags((Map<String,?>)hostId, config);
+ } else {
+ throw new IllegalArgumentException("Invalid host type '"+(hostId == null ? null : hostId.getClass().getName())+", referrenced in spec '"+spec);
}
+ machinesFlags.add(machineFlags);
}
+ Supplier<List<JcloudsMachineLocation>> machinesFactory = new Supplier<List<JcloudsMachineLocation>>() {
+ @Override
+ public List<JcloudsMachineLocation> get() {
+ // FIXME Can't change the parent of the JcloudsMachineLocation; but the FixedListMachineLocation will expect it to be the parent?
+ // We'll get errors on rebind. Need to reproduce/investigate.
+ List<JcloudsMachineLocation> result = Lists.newArrayList();
+ JcloudsLocation jcloudsLocation = (JcloudsLocation) managementContext.getLocationManager().createLocation(jcloudsLocationSpec.get());
+ for (Map<?,?> machineFlags : machinesFlags) {
+ try {
+ JcloudsMachineLocation machine = jcloudsLocation.registerMachine(jcloudsLocation.config().getBag().putAll(machineFlags));
+ result.add(machine);
+ } catch (NoMachinesAvailableException e) {
+ Map<?,?> sanitizedMachineFlags = Sanitizer.sanitize(machineFlags);
+ log.warn("Error rebinding to jclouds machine "+sanitizedMachineFlags+" in "+jcloudsLocation, e);
+ Exceptions.propagate(e);
+ }
+ }
+
+ log.debug("Created machines for jclouds BYON location: "+result);
+ return result;
+ }
+ };
+
ConfigBag flags = ConfigBag.newInstance(jcloudsProperties);
--- End diff --
Are we doing anything with this?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] brooklyn-server pull request: BROOKLYN-259: Fix JcloudsByonLocationResolver
Posted by aledsage <gi...@git.apache.org>.
Github user aledsage commented on the pull request:
https://github.com/apache/brooklyn-server/pull/132
Great, thanks @johnmccabe - merging now!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] brooklyn-server pull request: BROOKLYN-259: Fix JcloudsByonLocatio...
Posted by sjcorbett <gi...@git.apache.org>.
Github user sjcorbett commented on a diff in the pull request:
https://github.com/apache/brooklyn-server/pull/132#discussion_r62503400
--- Diff: core/src/main/java/org/apache/brooklyn/location/byon/FixedListMachineProvisioningLocation.java ---
@@ -147,8 +168,21 @@ public void init() {
}
config().set(MACHINE_SPECS, (List<LocationSpec<? extends MachineLocation>>) null);
+ Supplier<? extends List<? extends MachineLocation>> initialMachinesFactory = getConfig(INITIAL_MACHINES_FACTORY);
+ if (initialMachinesFactory != null) {
+ List<? extends MachineLocation> initialMachines = initialMachinesFactory.get();
+ if (initialMachines != null) {
+ for (MachineLocation machine : initialMachines) {
+ @SuppressWarnings("unchecked")
+ T castMachine = (T) machine;
+ machines.add(castMachine);
--- End diff --
Probably worth checking that `machine` is not null too.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] brooklyn-server pull request: BROOKLYN-259: Fix JcloudsByonLocatio...
Posted by aledsage <gi...@git.apache.org>.
Github user aledsage commented on a diff in the pull request:
https://github.com/apache/brooklyn-server/pull/132#discussion_r63905627
--- Diff: locations/jclouds/src/main/java/org/apache/brooklyn/location/jclouds/JcloudsByonLocationResolver.java ---
@@ -61,129 +56,140 @@
* @author aled
*/
@SuppressWarnings({"unchecked","rawtypes"})
-public class JcloudsByonLocationResolver implements LocationResolver {
+public class JcloudsByonLocationResolver extends AbstractLocationResolver implements LocationResolver {
public static final Logger log = LoggerFactory.getLogger(JcloudsByonLocationResolver.class);
public static final String BYON = "jcloudsByon";
- private static final Pattern PATTERN = Pattern.compile("("+BYON+"|"+BYON.toUpperCase()+")" + ":" + "\\((.*)\\)$");
-
- private ManagementContext managementContext;
-
@Override
- public void init(ManagementContext managementContext) {
- this.managementContext = checkNotNull(managementContext, "managementContext");
+ public boolean isEnabled() {
+ return LocationConfigUtils.isResolverPrefixEnabled(managementContext, getPrefix());
}
@Override
- public boolean isEnabled() {
- return LocationConfigUtils.isResolverPrefixEnabled(managementContext, getPrefix());
+ public String getPrefix() {
+ return BYON;
}
- // TODO Remove some duplication from JcloudsResolver; needs more careful review
@Override
- public LocationSpec<? extends Location> newLocationSpecFromString(String spec, Map<?, ?> locationFlags, LocationRegistry registry) {
- Map globalProperties = registry.getProperties();
+ protected Class<? extends Location> getLocationType() {
+ return FixedListMachineProvisioningLocation.class;
+ }
- Matcher matcher = PATTERN.matcher(spec);
- if (!matcher.matches()) {
- throw new IllegalArgumentException("Invalid location '"+spec+"'; must specify something like jcloudsByon(provider=\"aws-ec2\",region=\"us-east-1\",hosts=\"i-f2014593,i-d1234567\")");
- }
-
- String argsPart = matcher.group(2);
- Map<String, String> argsMap = KeyValueParser.parseMap(argsPart);
-
- // prefer args map over location flags
-
- String namedLocation = (String) locationFlags.get(LocationInternal.NAMED_SPEC_NAME.getName());
+ @Override
+ protected SpecParser getSpecParser() {
+ return new AbstractLocationResolver.SpecParser(getPrefix()).setExampleUsage("\"jcloudsByon(provider='aws-ec2',region='us-east-1',hosts='i-12345678,i-90123456')\"");
+ }
+
+ @Override
+ protected ConfigBag extractConfig(Map<?,?> locationFlags, String spec, final LocationRegistry registry) {
+ ConfigBag config = super.extractConfig(locationFlags, spec, registry);
- String providerOrApi = argsMap.containsKey("provider") ? argsMap.get("provider") : (String)locationFlags.get("provider");
+ String providerOrApi = (String) config.getStringKey("provider");
+ String regionName = (String) config.getStringKey("region");
+ String endpoint = (String) config.getStringKey("endpoint");
+ String namedLocation = (String) config.get(LocationInternal.NAMED_SPEC_NAME);
+ config.remove(LocationInternal.NAMED_SPEC_NAME.getName());
- String regionName = argsMap.containsKey("region") ? argsMap.get("region") : (String)locationFlags.get("region");
-
- String endpoint = argsMap.containsKey("endpoint") ? argsMap.get("endpoint") : (String)locationFlags.get("endpoint");
-
- String name = argsMap.containsKey("name") ? argsMap.get("name") : (String)locationFlags.get("name");
+ Object hosts = config.getStringKey("hosts");
+ config.remove("hosts");
- String user = argsMap.containsKey("user") ? argsMap.get("user") : (String)locationFlags.get("user");
+ Map jcloudsProperties = new JcloudsPropertiesFromBrooklynProperties().getJcloudsProperties(providerOrApi, regionName, namedLocation, config.getAllConfig());
+ config.putIfAbsent(jcloudsProperties);
- String privateKeyFile = argsMap.containsKey("privateKeyFile") ? argsMap.get("privateKeyFile") : (String)locationFlags.get("privateKeyFile");
-
- String hosts = argsMap.get("hosts");
-
if (Strings.isEmpty(providerOrApi)) {
throw new IllegalArgumentException("Invalid location '"+spec+"'; provider must be defined");
}
- if (hosts == null || hosts.isEmpty()) {
+ if (hosts == null || (hosts instanceof String && Strings.isBlank((String)hosts))) {
throw new IllegalArgumentException("Invalid location '"+spec+"'; at least one host must be defined");
}
- if (argsMap.containsKey("name") && (Strings.isEmpty(name))) {
- throw new IllegalArgumentException("Invalid location '"+spec+"'; if name supplied then value must be non-empty");
+
+ final String jcloudsSpec = "jclouds:"+providerOrApi + (regionName != null ? ":"+regionName : "") + (endpoint != null ? ":"+endpoint : "");
+ final Maybe<LocationSpec<? extends Location>> jcloudsLocationSpec = registry.getLocationSpec(jcloudsSpec, config.getAllConfig());
+ if (jcloudsLocationSpec.isAbsent()) {
+ throw new IllegalArgumentException("Invalid location '"+spec+"'; referrenced jclouds spec '"+jcloudsSpec+"' cannot be resolved");
+ } else if (!JcloudsLocation.class.isAssignableFrom(jcloudsLocationSpec.get().getType())) {
+ throw new IllegalArgumentException("Invalid location '"+spec+"'; referrenced spec '"+jcloudsSpec+"' of type "+jcloudsLocationSpec.get().getType()+" rather than JcloudsLocation");
}
- // For everything in brooklyn.properties, only use things with correct prefix (and remove that prefix).
- // But for everything passed in via locationFlags, pass those as-is.
- // TODO Should revisit the locationFlags: where are these actually used? Reason accepting properties without
- // full prefix is that the map's context is explicitly this location, rather than being generic properties.
- Map allProperties = getAllProperties(registry, globalProperties);
- Map jcloudsProperties = new JcloudsPropertiesFromBrooklynProperties().getJcloudsProperties(providerOrApi, regionName, namedLocation, allProperties);
- jcloudsProperties.putAll(locationFlags);
- jcloudsProperties.putAll(argsMap);
+ List<?> hostIds;
- String jcloudsSpec = "jclouds:"+providerOrApi + (regionName != null ? ":"+regionName : "") + (endpoint != null ? ":"+endpoint : "");
- JcloudsLocation jcloudsLocation = (JcloudsLocation) registry.resolve(jcloudsSpec, jcloudsProperties);
+ if (hosts instanceof String) {
+ if (((String) hosts).isEmpty()) {
+ hostIds = ImmutableList.of();
+ } else {
+ hostIds = WildcardGlobs.getGlobsAfterBraceExpansion("{"+hosts+"}",
+ true /* numeric */, /* no quote support though */ PhraseTreatment.NOT_A_SPECIAL_CHAR, PhraseTreatment.NOT_A_SPECIAL_CHAR);
+ }
+ } else if (hosts instanceof Iterable) {
+ hostIds = ImmutableList.copyOf((Iterable<?>)hosts);
+ } else {
+ throw new IllegalArgumentException("Invalid location '"+spec+"'; at least one host must be defined");
+ }
+ if (hostIds.isEmpty()) {
+ throw new IllegalArgumentException("Invalid location '"+spec+"'; at least one host must be defined");
+ }
- List<String> hostIdentifiers = WildcardGlobs.getGlobsAfterBraceExpansion("{"+hosts+"}",
- true /* numeric */, /* no quote support though */ PhraseTreatment.NOT_A_SPECIAL_CHAR, PhraseTreatment.NOT_A_SPECIAL_CHAR);
- List<JcloudsSshMachineLocation> machines = Lists.newArrayList();
-
- for (String hostIdentifier : hostIdentifiers) {
- Map<?, ?> machineFlags = MutableMap.builder()
- .put("id", hostIdentifier)
- .putIfNotNull("user", user)
- .putIfNotNull("privateKeyFile", privateKeyFile)
- .build();
- try {
- // TODO management of these machines may be odd, as it is passed in as a key as config to a spec
- JcloudsSshMachineLocation machine = jcloudsLocation.rebindMachine(jcloudsLocation.config().getBag().putAll(machineFlags));
- machines.add(machine);
- } catch (NoMachinesAvailableException e) {
- log.warn("Error rebinding to jclouds machine "+hostIdentifier+" in "+jcloudsLocation, e);
- Exceptions.propagate(e);
+ final List<Map<?,?>> machinesFlags = Lists.newArrayList();
+ for (Object hostId : hostIds) {
+ Map<?, ?> machineFlags;
+ if (hostId instanceof String) {
+ machineFlags = parseMachineFlags((String)hostId, config);
+ } else if (hostId instanceof Map) {
+ machineFlags = parseMachineFlags((Map<String,?>)hostId, config);
+ } else {
+ throw new IllegalArgumentException("Invalid host type '"+(hostId == null ? null : hostId.getClass().getName())+", referrenced in spec '"+spec);
}
+ machinesFlags.add(machineFlags);
}
+ Supplier<List<JcloudsMachineLocation>> machinesFactory = new Supplier<List<JcloudsMachineLocation>>() {
+ @Override
+ public List<JcloudsMachineLocation> get() {
+ // FIXME Can't change the parent of the JcloudsMachineLocation; but the FixedListMachineLocation will expect it to be the parent?
--- End diff --
Unfortunately not, and unfortunately I can't find the log where we encountered this! I tried to reproduce in a simple unit test but didn't manage. Will need to investigate further.
You're right that a vague FIXME comment in the code is not the right thing. I'll remove the fixme, and try to reproduce properly so I can open a jira issue.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] brooklyn-server pull request: BROOKLYN-259: Fix JcloudsByonLocatio...
Posted by johnmccabe <gi...@git.apache.org>.
Github user johnmccabe commented on the pull request:
https://github.com/apache/brooklyn-server/pull/132#issuecomment-221911240
@aledsage testing this at the moment
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] brooklyn-server pull request: BROOKLYN-259: Fix JcloudsByonLocatio...
Posted by johnmccabe <gi...@git.apache.org>.
Github user johnmccabe commented on a diff in the pull request:
https://github.com/apache/brooklyn-server/pull/132#discussion_r62522025
--- Diff: locations/jclouds/src/test/java/org/apache/brooklyn/location/jclouds/JcloudsByonLocationResolverStubbedTest.java ---
@@ -0,0 +1,157 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.brooklyn.location.jclouds;
+
+import static org.testng.Assert.assertEquals;
+
+import java.util.Collection;
+import java.util.Map;
+import java.util.Set;
+
+import org.apache.brooklyn.api.location.Location;
+import org.apache.brooklyn.api.location.LocationSpec;
+import org.apache.brooklyn.api.location.MachineLocation;
+import org.apache.brooklyn.core.internal.BrooklynProperties;
+import org.apache.brooklyn.core.mgmt.internal.LocalManagementContext;
+import org.apache.brooklyn.core.test.entity.LocalManagementContextForTests;
+import org.apache.brooklyn.location.byon.FixedListMachineProvisioningLocation;
+import org.jclouds.compute.domain.ComputeMetadata;
+import org.jclouds.compute.domain.NodeMetadata;
+import org.jclouds.compute.domain.NodeMetadata.Status;
+import org.jclouds.compute.domain.NodeMetadataBuilder;
+import org.jclouds.compute.domain.Template;
+import org.jclouds.domain.LoginCredentials;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.testng.annotations.Test;
+
+import com.google.common.base.Predicate;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.ImmutableSet;
+import com.google.common.collect.Iterables;
+
+public class JcloudsByonLocationResolverStubbedTest extends AbstractJcloudsStubbedLiveTest {
+
+ @SuppressWarnings("unused")
+ private static final Logger log = LoggerFactory.getLogger(JcloudsByonLocationResolverStubbedTest.class);
+
+ private final String nodeId = "mynodeid";
+ private final String nodePublicAddress = "173.194.32.123";
+ private final String nodePrivateAddress = "172.168.10.11";
+
+ protected LocalManagementContext newManagementContext() {
+ // This really is stubbed; no live access to the cloud
+ LocalManagementContext result = LocalManagementContextForTests.builder(true).build();
+ BrooklynProperties brooklynProperties = result.getBrooklynProperties();
+ brooklynProperties.put("brooklyn.location.jclouds."+SOFTLAYER_PROVIDER+".identity", "myidentity");
+ brooklynProperties.put("brooklyn.location.jclouds."+SOFTLAYER_PROVIDER+".credential", "mycredential");
+ return result;
+
+ }
+
+ @Override
+ protected NodeCreator newNodeCreator() {
+ return new NodeCreator() {
+ @Override
+ public Set<? extends NodeMetadata> listNodesDetailsMatching(Predicate<ComputeMetadata> filter) {
+ NodeMetadata result = new NodeMetadataBuilder()
+ .id(nodeId)
+ .credentials(LoginCredentials.builder().identity("dummy").credential("dummy").build())
+ .loginPort(22)
+ .status(Status.RUNNING)
+ .publicAddresses(ImmutableList.of(nodePublicAddress))
+ .privateAddresses(ImmutableList.of(nodePrivateAddress))
+ .build();
+ return ImmutableSet.copyOf(Iterables.filter(ImmutableList.of(result), filter));
+ }
+ @Override
+ protected NodeMetadata newNode(String group, Template template) {
+ throw new UnsupportedOperationException();
+ }
+ };
+ }
+
+ @Test
+ public void testResolvesHostInSpec() throws Exception {
+ String spec = "jcloudsByon:(provider=\""+SOFTLAYER_PROVIDER+"\",region=\""+SOFTLAYER_AMS01_REGION_NAME+"\",user=\"myuser\",password=\"mypassword\",hosts=\""+nodeId+"\")";
+ Map<?,?> specFlags = ImmutableMap.of(JcloudsLocationConfig.COMPUTE_SERVICE_REGISTRY, computeServiceRegistry);
+
+ FixedListMachineProvisioningLocation<MachineLocation> location = getLocationManaged(spec, specFlags);
+
+ JcloudsSshMachineLocation machine = (JcloudsSshMachineLocation) Iterables.getOnlyElement(location.getAllMachines());
+ assertEquals(machine.getJcloudsId(), nodeId);
+ assertEquals(machine.getPublicAddresses(), ImmutableSet.of(nodePublicAddress));
+ assertEquals(machine.getPrivateAddresses(), ImmutableSet.of(nodePrivateAddress));
+
+ // TODO what user/password should we expect? Fails below because has "dummy":
+// assertEquals(machine.getUser(), "myuser");
--- End diff --
This is whats throwing the NPE mentioned earlier, previously it was these user/password values that were being used.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] brooklyn-server pull request: BROOKLYN-259: Fix JcloudsByonLocationResolver
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/brooklyn-server/pull/132
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] brooklyn-server pull request: BROOKLYN-259: Fix JcloudsByonLocatio...
Posted by sjcorbett <gi...@git.apache.org>.
Github user sjcorbett commented on a diff in the pull request:
https://github.com/apache/brooklyn-server/pull/132#discussion_r62502931
--- Diff: core/src/main/java/org/apache/brooklyn/location/byon/FixedListMachineProvisioningLocation.java ---
@@ -103,10 +105,29 @@
@Beta
@SuppressWarnings("serial")
public static final ConfigKey<List<LocationSpec<? extends MachineLocation>>> MACHINE_SPECS = ConfigKeys.newConfigKey(
- new TypeToken<List<LocationSpec<? extends MachineLocation>>>() {},
- "byon.machineSpecs",
- "Specs of machines that should be immediatly instantiated on init",
- ImmutableList.<LocationSpec<? extends MachineLocation>>of());
+ new TypeToken<List<LocationSpec<? extends MachineLocation>>>() {},
+ "byon.machineSpecs",
+ "Specs of machines that should be immediatly instantiated on init",
+ ImmutableList.<LocationSpec<? extends MachineLocation>>of());
+
+ /**
+ * The initialMachinesFactory allows {@code JcloudsByonLocationResolver} to work, to defer
+ * instantiating the {@code JcloudsLocation} and the {@code JcloudsMachineLocation} instances.
+ * (Important because the caller might not use the spec and thus might not unmanage the machine
+ * instances).
+ *
+ * We clear the initialMachinesFactory in init, so they will never be persisted. This will help
+ * with backwards compatibility if we change how this is done.
+ *
+ * By the end of init(), the {@link #machines} will contain the full list of locations.
+ */
+ @Beta
+ @SuppressWarnings("serial")
+ public static final ConfigKey<Supplier<? extends List<? extends MachineLocation>>> INITIAL_MACHINES_FACTORY = ConfigKeys.newConfigKey(
+ new TypeToken<Supplier<? extends List<? extends MachineLocation>>>() {},
+ "byon.initialMachinesFactory",
+ "Factory for creating the machines that should be immediatly instantiated on init",
--- End diff --
`immediatly`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] brooklyn-server pull request: BROOKLYN-259: Fix JcloudsByonLocatio...
Posted by sjcorbett <gi...@git.apache.org>.
Github user sjcorbett commented on a diff in the pull request:
https://github.com/apache/brooklyn-server/pull/132#discussion_r62503507
--- Diff: core/src/main/java/org/apache/brooklyn/location/byon/FixedListMachineProvisioningLocation.java ---
@@ -147,8 +168,21 @@ public void init() {
}
config().set(MACHINE_SPECS, (List<LocationSpec<? extends MachineLocation>>) null);
+ Supplier<? extends List<? extends MachineLocation>> initialMachinesFactory = getConfig(INITIAL_MACHINES_FACTORY);
+ if (initialMachinesFactory != null) {
+ List<? extends MachineLocation> initialMachines = initialMachinesFactory.get();
+ if (initialMachines != null) {
+ for (MachineLocation machine : initialMachines) {
+ @SuppressWarnings("unchecked")
+ T castMachine = (T) machine;
+ machines.add(castMachine);
--- End diff --
See that's handled further anyway.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] brooklyn-server pull request: BROOKLYN-259: Fix JcloudsByonLocatio...
Posted by aledsage <gi...@git.apache.org>.
Github user aledsage commented on the pull request:
https://github.com/apache/brooklyn-server/pull/132#issuecomment-220478329
Rebasing fixed the problem in my test where `assertEquals(machine.getUser(), "myuser");` was failing (so that is now uncommented). I hope that whatever fixed that issue will have fixed @johnmccabe 's "NPE when deploying an app to a JcloudsByonLocation".
For "NPE when deploying a docker infrastructure via Clocker", that has been fixed in a separate PR by the looks of it (certainly we guard against that NPE). See commit bee832d139bdaffffb859278ef6eeddd93085838.
I've confirmed that it works with username using `JcloudsByonLocationResolverAwsLiveTest` (though I needed to fix another couple of things there - see second commit).
@johnmccabe can you review / test again with this latest code when you get a chance?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] brooklyn-server pull request: BROOKLYN-259: Fix JcloudsByonLocationResolver
Posted by aledsage <gi...@git.apache.org>.
Github user aledsage commented on the pull request:
https://github.com/apache/brooklyn-server/pull/132
@johnmccabe did you get a chance to test this? Any comments? Can we merge it?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] brooklyn-server pull request: BROOKLYN-259: Fix JcloudsByonLocatio...
Posted by johnmccabe <gi...@git.apache.org>.
Github user johnmccabe commented on the pull request:
https://github.com/apache/brooklyn-server/pull/132#issuecomment-217896393
Throwing an NPE when deploying an app to a JcloudsByonLocation that was working before this PR:
```
2016-05-09 14:30:30,074 WARN o.a.b.c.m.i.LocalLocationManager [brooklyn-execmanager-K1kS1uK9-466]: Failed to create location using spec LocationSpec{type=class org.apache.brooklyn.location.byon.FixedListMachineProvisioningLocation}@4c055676 (rethrowing)
java.lang.NullPointerException: null
at org.apache.brooklyn.location.jclouds.JcloudsLocation.createJcloudsSshMachineLocation(JcloudsLocation.java:2319) ~[brooklyn-locations-jclouds-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at org.apache.brooklyn.location.jclouds.JcloudsLocation.registerJcloudsSshMachineLocation(JcloudsLocation.java:2255) ~[brooklyn-locations-jclouds-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at org.apache.brooklyn.location.jclouds.JcloudsLocation.registerMachineLocation(JcloudsLocation.java:2106) ~[brooklyn-locations-jclouds-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at org.apache.brooklyn.location.jclouds.JcloudsLocation.registerMachine(JcloudsLocation.java:2097) ~[brooklyn-locations-jclouds-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at org.apache.brooklyn.location.jclouds.JcloudsByonLocationResolver$1.get(JcloudsByonLocationResolver.java:156) ~[brooklyn-locations-jclouds-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at org.apache.brooklyn.location.jclouds.JcloudsByonLocationResolver$1.get(JcloudsByonLocationResolver.java:147) ~[brooklyn-locations-jclouds-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at org.apache.brooklyn.location.byon.FixedListMachineProvisioningLocation.init(FixedListMachineProvisioningLocation.java:173) ~[brooklyn-core-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at org.apache.brooklyn.core.objs.proxy.InternalLocationFactory.createLocation(InternalLocationFactory.java:137) ~[brooklyn-core-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at org.apache.brooklyn.core.mgmt.internal.LocalLocationManager.createLocation(LocalLocationManager.java:94) ~[brooklyn-core-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at org.apache.brooklyn.core.objs.proxy.InternalEntityFactory$1.run(InternalEntityFactory.java:331) [brooklyn-core-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_99]
at org.apache.brooklyn.util.core.task.BasicExecutionManager$SubmissionCallable.call(BasicExecutionManager.java:519) [brooklyn-core-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at java.util.concurrent.FutureTask.run(FutureTask.java:262) [na:1.7.0_99]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_99]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_99]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_99]
```
Example location:
```
brooklyn.location.named.my-named-byon=jcloudsByon:(provider="cloudstack",endpoint="xxxxxxxxxxxxxx",region="xxxxxxxxxxxxxx",networkName="xxxxxxxxxxxxxx",user="root",password="xxxxxxxxxxxxxx",hosts="xxxxxxxxxxxxxx,yyyyyyyyyyyyy,zzzzzzzzzzzzz")
brooklyn.location.named.bt-beta-byon.identity=xxxxxxx
brooklyn.location.named.bt-beta-byon.credential=xxxxxxx
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] brooklyn-server pull request: BROOKLYN-259: Fix JcloudsByonLocatio...
Posted by johnmccabe <gi...@git.apache.org>.
Github user johnmccabe commented on the pull request:
https://github.com/apache/brooklyn-server/pull/132#issuecomment-218096768
Spoke too soon on the location working with privateKey, throws an NPE when deploying a docker infrastructure via Clocker:
```
Caused by: java.lang.NullPointerException: id
at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:229) ~[guava-16.0.1.jar:na]
at org.jclouds.compute.internal.BaseComputeService.getImage(BaseComputeService.java:410) ~[jclouds-compute-1.9.2.jar:1.9.2]
at org.apache.brooklyn.location.jclouds.JcloudsSshMachineLocation.getOptionalImage(JcloudsSshMachineLocation.java:226) ~[brooklyn-locations-jclouds-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at org.apache.brooklyn.location.jclouds.JcloudsSshMachineLocation.getOptionalOperatingSystem(JcloudsSshMachineLocation.java:504) ~[brooklyn-locations-jclouds-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at org.apache.brooklyn.location.jclouds.JcloudsSshMachineLocation.inferMachineDetails(JcloudsSshMachineLocation.java:522) ~[brooklyn-locations-jclouds-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at org.apache.brooklyn.location.ssh.SshMachineLocation.getMachineDetails(SshMachineLocation.java:1056) ~[brooklyn-core-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at clocker.docker.entity.DockerHostSshDriver.install(DockerHostSshDriver.java:183) ~[brooklyn-clocker-docker-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
at org.apache.brooklyn.entity.software.base.AbstractSoftwareProcessDriver$2$6.run(AbstractSoftwareProcessDriver.java:159) ~[brooklyn-software-base-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) ~[na:1.7.0_99]
at org.apache.brooklyn.util.core.task.DynamicSequentialTask$DstJob.call(DynamicSequentialTask.java:359) ~[brooklyn-core-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
... 5 common frames omitted
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] brooklyn-server pull request: BROOKLYN-259: Fix JcloudsByonLocationResolver
Posted by johnmccabe <gi...@git.apache.org>.
Github user johnmccabe commented on the pull request:
https://github.com/apache/brooklyn-server/pull/132
Rebuild appears to have done the trick, in the process of deploying a simple test app.
```
location: my-jclouds-byon
services:
- type: org.apache.brooklyn.entity.webapp.tomcat.TomcatServer
id: tomcat
name: Tomcat
war: http://tomcat.apache.org/tomcat-6.0-doc/appdev/sample/sample.war
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] brooklyn-server pull request: BROOKLYN-259: Fix JcloudsByonLocationResolver
Posted by johnmccabe <gi...@git.apache.org>.
Github user johnmccabe commented on the pull request:
https://github.com/apache/brooklyn-server/pull/132
Hi @aledsage, sorry for the delay getting back to you, the env I had originally observed this on is still inaccessible so I had to recreate, unfortunately, still seeing an NPE when attempting to deploy to the following location (`userCredentials.getUser()` again):
```
brooklyn.location.named.my-jclouds-byon=jcloudsByon:(provider="cloudstack",endpoint="https://my-cloudstack-endpoint.domain.com/client/api",region="my-region-id",networkName="my-network-id",user="root",password="supersecretrootpassword",hosts="my-existing-host-id")
brooklyn.location.named.my-jclouds-byon.identity=MY_API_KEY
brooklyn.location.named.my-jclouds-byon.credential=MY_SECRET_KEY
```
```
2016-06-01 11:30:04,141 WARN o.a.b.c.m.i.LocalLocationManager [brooklyn-execmanager-qQtAY4JU-1]: Failed to create location using spec LocationSpec{type=class org.apache.brooklyn.location.byon.FixedListMachineProvisioningLocation}@33a9098e (rethrowing)
java.lang.NullPointerException: null
at org.apache.brooklyn.location.jclouds.JcloudsLocation.createJcloudsSshMachineLocation(JcloudsLocation.java:2319) ~[brooklyn-locations-jclouds-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at org.apache.brooklyn.location.jclouds.JcloudsLocation.registerJcloudsSshMachineLocation(JcloudsLocation.java:2255) ~[brooklyn-locations-jclouds-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at org.apache.brooklyn.location.jclouds.JcloudsLocation.registerMachineLocation(JcloudsLocation.java:2106) ~[brooklyn-locations-jclouds-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at org.apache.brooklyn.location.jclouds.JcloudsLocation.registerMachine(JcloudsLocation.java:2097) ~[brooklyn-locations-jclouds-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at org.apache.brooklyn.location.jclouds.JcloudsByonLocationResolver$1.get(JcloudsByonLocationResolver.java:156) ~[brooklyn-locations-jclouds-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at org.apache.brooklyn.location.jclouds.JcloudsByonLocationResolver$1.get(JcloudsByonLocationResolver.java:147) ~[brooklyn-locations-jclouds-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at org.apache.brooklyn.location.byon.FixedListMachineProvisioningLocation.init(FixedListMachineProvisioningLocation.java:173) ~[brooklyn-core-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at org.apache.brooklyn.core.objs.proxy.InternalLocationFactory.createLocation(InternalLocationFactory.java:137) ~[brooklyn-core-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at org.apache.brooklyn.core.mgmt.internal.LocalLocationManager.createLocation(LocalLocationManager.java:94) ~[brooklyn-core-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at org.apache.brooklyn.core.objs.proxy.InternalEntityFactory$1.run(InternalEntityFactory.java:331) [brooklyn-core-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_79]
at org.apache.brooklyn.util.core.task.BasicExecutionManager$SubmissionCallable.call(BasicExecutionManager.java:519) [brooklyn-core-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at java.util.concurrent.FutureTask.run(FutureTask.java:262) [na:1.7.0_79]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_79]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_79]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_79]
```
I'm going to rebuild to sanity check my install (I've built clocker against the #132 brooklyn-server branch).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---