You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@brooklyn.apache.org by aledsage <gi...@git.apache.org> on 2014/09/18 11:26:41 UTC

[GitHub] incubator-brooklyn pull request: Feature: transform state

GitHub user aledsage opened a pull request:

    https://github.com/apache/incubator-brooklyn/pull/177

    Feature: transform state

    For transforming persisted state, e.g. if a class has been renamed.
    
    For reviewing, probably best to go through one commit at a time.
    
    For the CLI support, this is really to get early feedback. It needs improvements, and I haven't written any docs.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/aledsage/incubator-brooklyn feature/transform-state

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/incubator-brooklyn/pull/177.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #177
    
----
commit bba5f3e545f3b91fdee78b2b99d122344194187f
Author: Aled Sage <al...@gmail.com>
Date:   2014-09-10T09:28:30Z

    Adds persister.loadMementoRawData
    
    - Refactors BrooklynMementoPersisterToObjectStore to remove duplication
    - Adds BrooklynMementoRawData, which has the contents of the raw
      blobs/files for persistence (rather than deserialised mementos)
    - Adds BrooklynMementoPersisterToObjectStore.loadMementoRawData
    - Adds BrooklynMementoPersisterToObjectStore.checkpoint(BrooklynMementoRawData)

commit 82ddabb013e3114195b53c8a9afc05e5459d2155
Author: Aled Sage <al...@gmail.com>
Date:   2014-09-11T12:00:26Z

    copy-state CLI: use BrooklynMementoRawData

commit 348cc3fa5ec693894c4ef8e3da729c5fed743475
Author: Aled Sage <al...@gmail.com>
Date:   2014-09-11T12:27:43Z

    CLI copy-state supports uploading to any location
    
    - previously could only copy to the filesystem, rather than to
      any object store

commit dd10a151b821cbad6837d4712b5417810591e304
Author: Aled Sage <al...@gmail.com>
Date:   2014-09-15T17:15:14Z

    Transformations of persisted state

commit 9dd50dacb52ec7b02796e85e3b428c64dda535af
Author: Aled Sage <al...@gmail.com>
Date:   2014-09-15T17:15:42Z

    CLI copy-state: support transformations

commit d911997ecb0c67f037552bd8859bbf2d1d5455d9
Author: Aled Sage <al...@gmail.com>
Date:   2014-09-15T17:16:06Z

    Transformations of BrooklynMemento (persisted state)

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-brooklyn pull request: Feature: transform state

Posted by neykov <gi...@git.apache.org>.
Github user neykov commented on a diff in the pull request:

    https://github.com/apache/incubator-brooklyn/pull/177#discussion_r17963845
  
    --- Diff: core/src/main/java/brooklyn/entity/rebind/transformer/CompoundTransformer.java ---
    @@ -0,0 +1,159 @@
    +package brooklyn.entity.rebind.transformer;
    +
    +import static com.google.common.base.Preconditions.checkNotNull;
    +
    +import java.util.Collection;
    +import java.util.Map;
    +
    +import brooklyn.entity.rebind.BrooklynObjectType;
    +import brooklyn.entity.rebind.RebindExceptionHandler;
    +import brooklyn.entity.rebind.persister.BrooklynMementoPersisterToObjectStore;
    +import brooklyn.entity.rebind.transformer.impl.XsltTransformer;
    +import brooklyn.entity.rebind.transformer.impl.XsltTransformerTest;
    +import brooklyn.mementos.BrooklynMementoRawData;
    +import brooklyn.util.ResourceUtils;
    +import brooklyn.util.collections.MutableMap;
    +import brooklyn.util.text.TemplateProcessor;
    +
    +import com.google.api.client.repackaged.com.google.common.annotations.VisibleForTesting;
    +import com.google.common.annotations.Beta;
    +import com.google.common.collect.ArrayListMultimap;
    +import com.google.common.collect.ImmutableMap;
    +import com.google.common.collect.Multimap;
    +
    +@Beta
    +public class CompoundTransformer {
    +    
    +    public static final CompoundTransformer NOOP = builder().build();
    +    
    +    // TODO Does not yet handle BrooklynMementoTransformer, for changing an entire BrooklynMemento.
    +    // Need to do refactoring in RebindManager and BrooklynMementoPersisterToObjectStore to convert 
    +    // from a BrooklynMementoRawData to a BrooklynMemento.
    +    
    +    public static Builder builder() {
    +        return new Builder();
    +    }
    +
    +    public static class Builder {
    +        private final Multimap<BrooklynObjectType, RawDataTransformer> rawDataTransformers = ArrayListMultimap.<BrooklynObjectType, RawDataTransformer>create();
    +        
    +        public Builder rawDataTransformer(RawDataTransformer val) {
    +            for (BrooklynObjectType type : BrooklynObjectType.values()) {
    +                rawDataTransformer(type, val);
    +            }
    +            return this;
    +        }
    +        public Builder rawDataTransformer(BrooklynObjectType type, RawDataTransformer val) {
    +            rawDataTransformers.put(checkNotNull(type, "type"), checkNotNull(val, "val"));
    +            return this;
    +        }
    +        public Builder renameType(String oldVal, String newVal) {
    +            // xstream format for inner classes is like <brooklyn.entity.rebind.transformer.CompoundTransformerTest_-OrigType>
    +            oldVal = toXstreamClassnameFormat(oldVal);
    +            newVal = toXstreamClassnameFormat(newVal);
    +            
    +            String xsltTemplate = ResourceUtils.create(XsltTransformerTest.class).getResourceAsString("classpath://brooklyn/entity/rebind/transformer/renameType.xslt");
    +            String xslt = TemplateProcessor.processTemplateContents(xsltTemplate, ImmutableMap.of("old_val", oldVal, "new_val", newVal));
    +            return xsltTransformer(xslt);
    +        }
    +        public Builder renameClass(String oldVal, String newVal) {
    +            // xstream format for inner classes is like <brooklyn.entity.rebind.transformer.CompoundTransfor<merTest_-OrigType>
    +            oldVal = toXstreamClassnameFormat(oldVal);
    +            newVal = toXstreamClassnameFormat(newVal);
    +            
    +            String xsltTemplate = ResourceUtils.create(XsltTransformerTest.class).getResourceAsString("classpath://brooklyn/entity/rebind/transformer/renameClass.xslt");
    +            String xslt = TemplateProcessor.processTemplateContents(xsltTemplate, ImmutableMap.of("old_val", oldVal, "new_val", newVal));
    +            return xsltTransformer(xslt);
    +        }
    +        public Builder renameField(String clazz, String oldVal, String newVal) {
    +            // xstream format for inner classes is like <brooklyn.entity.rebind.transformer.CompoundTransfor<merTest_-OrigType>
    +            clazz = toXstreamClassnameFormat(clazz);
    +            oldVal = toXstreamClassnameFormat(oldVal);
    +            newVal = toXstreamClassnameFormat(newVal);
    +            
    +            String xsltTemplate = ResourceUtils.create(XsltTransformerTest.class).getResourceAsString("classpath://brooklyn/entity/rebind/transformer/renameClass.xslt");
    +            String xslt = TemplateProcessor.processTemplateContents(xsltTemplate, ImmutableMap.of("class_name", clazz, "old_val", oldVal, "new_val", newVal));
    +            return xsltTransformer(xslt);
    +        }
    +        public Builder xsltTransformer(String xslt) {
    +            XsltTransformer xsltTransformer = new XsltTransformer(xslt);
    +            for (BrooklynObjectType type : BrooklynObjectType.values()) {
    +                rawDataTransformer(type, xsltTransformer);
    +            }
    +            return this;
    +        }
    +        public Builder xsltTransformer(BrooklynObjectType type, String xslt) {
    +            XsltTransformer xsltTransformer = new XsltTransformer(xslt);
    +            rawDataTransformer(type, xsltTransformer);
    +            return this;
    +        }
    +        private String toXstreamClassnameFormat(String val) {
    +            return (val.contains("$")) ? val.replace("$", "_-") : val;
    +        }
    +        public CompoundTransformer build() {
    +            return new CompoundTransformer(this);
    +        }
    +    }
    +
    +    private final Multimap<BrooklynObjectType, RawDataTransformer> rawDataTransformers;
    +    
    +    protected CompoundTransformer(Builder builder) {
    +        rawDataTransformers = builder.rawDataTransformers;
    +    }
    +
    +    public BrooklynMementoRawData transform(BrooklynMementoPersisterToObjectStore reader, RebindExceptionHandler exceptionHandler) throws Exception {
    +        BrooklynMementoRawData rawData = reader.loadMementoRawData(exceptionHandler);
    +        return transform(rawData);
    +    }
    +    
    +    public BrooklynMementoRawData transform(BrooklynMementoRawData rawData) throws Exception {
    +        Map<String, String> entities = MutableMap.copyOf(rawData.getEntities());
    +        Map<String, String> locations = MutableMap.copyOf(rawData.getLocations());
    +        Map<String, String> policies = MutableMap.copyOf(rawData.getPolicies());
    +        Map<String, String> enrichers = MutableMap.copyOf(rawData.getEnrichers());
    +
    +        for (BrooklynObjectType type : BrooklynObjectType.values()) {
    +            Collection<RawDataTransformer> transformers = rawDataTransformers.get(type);
    --- End diff --
    
    I imagine there would be cases where the order of applying the transformers is important. We should apply them in the order they appear in the YAML file (parsing the YAML to a Map could be another source of reordering).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-brooklyn pull request: Feature: transform state

Posted by ahgittin <gi...@git.apache.org>.
Github user ahgittin commented on the pull request:

    https://github.com/apache/incubator-brooklyn/pull/177#issuecomment-56852076
  
    merged in 320c1c752a0085f360cd688d0cc8ff8113a9fa65 , along with my fixes.
    
    i'm curious whether there is a better way to deal with merge conflict resolutions.  maybe better to commit merge including the conflicts, then fix the conflicts in a separate commit?  `git` seems to encourage fixing as part of the merge commit, but this gives no good way to inspect the fixes...


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-brooklyn pull request: Feature: transform state

Posted by neykov <gi...@git.apache.org>.
Github user neykov commented on a diff in the pull request:

    https://github.com/apache/incubator-brooklyn/pull/177#discussion_r17806178
  
    --- Diff: core/src/main/java/brooklyn/entity/rebind/transformer/CompoundTransformer.java ---
    @@ -0,0 +1,159 @@
    +package brooklyn.entity.rebind.transformer;
    +
    +import static com.google.common.base.Preconditions.checkNotNull;
    +
    +import java.util.Collection;
    +import java.util.Map;
    +
    +import brooklyn.entity.rebind.BrooklynObjectType;
    +import brooklyn.entity.rebind.RebindExceptionHandler;
    +import brooklyn.entity.rebind.persister.BrooklynMementoPersisterToObjectStore;
    +import brooklyn.entity.rebind.transformer.impl.XsltTransformer;
    +import brooklyn.entity.rebind.transformer.impl.XsltTransformerTest;
    +import brooklyn.mementos.BrooklynMementoRawData;
    +import brooklyn.util.ResourceUtils;
    +import brooklyn.util.collections.MutableMap;
    +import brooklyn.util.text.TemplateProcessor;
    +
    +import com.google.api.client.repackaged.com.google.common.annotations.VisibleForTesting;
    +import com.google.common.annotations.Beta;
    +import com.google.common.collect.ArrayListMultimap;
    +import com.google.common.collect.ImmutableMap;
    +import com.google.common.collect.Multimap;
    +
    +@Beta
    +public class CompoundTransformer {
    +    
    +    public static final CompoundTransformer NOOP = builder().build();
    +    
    +    // TODO Does not yet handle BrooklynMementoTransformer, for changing an entire BrooklynMemento.
    +    // Need to do refactoring in RebindManager and BrooklynMementoPersisterToObjectStore to convert 
    +    // from a BrooklynMementoRawData to a BrooklynMemento.
    +    
    +    public static Builder builder() {
    +        return new Builder();
    +    }
    +
    +    public static class Builder {
    +        private final Multimap<BrooklynObjectType, RawDataTransformer> rawDataTransformers = ArrayListMultimap.<BrooklynObjectType, RawDataTransformer>create();
    +        
    +        public Builder rawDataTransformer(RawDataTransformer val) {
    +            for (BrooklynObjectType type : BrooklynObjectType.values()) {
    +                rawDataTransformer(type, val);
    +            }
    +            return this;
    +        }
    +        public Builder rawDataTransformer(BrooklynObjectType type, RawDataTransformer val) {
    +            rawDataTransformers.put(checkNotNull(type, "type"), checkNotNull(val, "val"));
    +            return this;
    +        }
    +        public Builder renameType(String oldVal, String newVal) {
    +            // xstream format for inner classes is like <brooklyn.entity.rebind.transformer.CompoundTransformerTest_-OrigType>
    +            oldVal = toXstreamClassnameFormat(oldVal);
    +            newVal = toXstreamClassnameFormat(newVal);
    +            
    +            String xsltTemplate = ResourceUtils.create(XsltTransformerTest.class).getResourceAsString("classpath://brooklyn/entity/rebind/transformer/renameType.xslt");
    +            String xslt = TemplateProcessor.processTemplateContents(xsltTemplate, ImmutableMap.of("old_val", oldVal, "new_val", newVal));
    +            return xsltTransformer(xslt);
    +        }
    +        public Builder renameClass(String oldVal, String newVal) {
    +            // xstream format for inner classes is like <brooklyn.entity.rebind.transformer.CompoundTransfor<merTest_-OrigType>
    +            oldVal = toXstreamClassnameFormat(oldVal);
    +            newVal = toXstreamClassnameFormat(newVal);
    +            
    +            String xsltTemplate = ResourceUtils.create(XsltTransformerTest.class).getResourceAsString("classpath://brooklyn/entity/rebind/transformer/renameClass.xslt");
    +            String xslt = TemplateProcessor.processTemplateContents(xsltTemplate, ImmutableMap.of("old_val", oldVal, "new_val", newVal));
    +            return xsltTransformer(xslt);
    +        }
    +        public Builder renameField(String clazz, String oldVal, String newVal) {
    +            // xstream format for inner classes is like <brooklyn.entity.rebind.transformer.CompoundTransfor<merTest_-OrigType>
    +            clazz = toXstreamClassnameFormat(clazz);
    +            oldVal = toXstreamClassnameFormat(oldVal);
    +            newVal = toXstreamClassnameFormat(newVal);
    +            
    +            String xsltTemplate = ResourceUtils.create(XsltTransformerTest.class).getResourceAsString("classpath://brooklyn/entity/rebind/transformer/renameClass.xslt");
    --- End diff --
    
    Should be renameField.xslt


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-brooklyn pull request: Feature: transform state

Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:

    https://github.com/apache/incubator-brooklyn/pull/177


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-brooklyn pull request: Feature: transform state

Posted by neykov <gi...@git.apache.org>.
Github user neykov commented on a diff in the pull request:

    https://github.com/apache/incubator-brooklyn/pull/177#discussion_r17963382
  
    --- Diff: core/src/main/java/brooklyn/entity/rebind/transformer/CompoundTransformerLoader.java ---
    @@ -0,0 +1,65 @@
    +package brooklyn.entity.rebind.transformer;
    +
    +import java.util.Map;
    +
    +import brooklyn.entity.rebind.transformer.CompoundTransformer.Builder;
    +import brooklyn.entity.rebind.transformer.impl.XsltTransformerTest;
    +import brooklyn.util.ResourceUtils;
    +import brooklyn.util.exceptions.Exceptions;
    +import brooklyn.util.text.TemplateProcessor;
    +import brooklyn.util.yaml.Yamls;
    +
    +import com.google.common.annotations.Beta;
    +import com.google.common.collect.ImmutableMap;
    +import com.google.common.collect.Iterables;
    +
    +@Beta
    +public class CompoundTransformerLoader {
    +
    +    // TODO Improve error handing so get nicer errors.
    +    // TODO Improve names (e.g. always camel case?)
    +    // TODO Pass in classloader for reflectively loading rawDataTransformer?
    +    
    +    public static CompoundTransformer load(String contents) {
    +        CompoundTransformer.Builder builder = CompoundTransformer.builder();
    +        Iterable<Object> toplevel = Yamls.parseAll(contents);
    +        Map<String, Map<?,?>> rules = (Map<String, Map<?,?>>) ((Map<?,?>)Iterables.getOnlyElement(toplevel));
    +        for (Map.Entry<String, Map<?,?>> entry : rules.entrySet()) {
    +            addRule(builder, entry.getKey(), entry.getValue());
    +        }
    +        return builder.build();
    +    }
    +
    +    private static void addRule(Builder builder, String name, Map<?,?> args) {
    +        if (name.equals("renameClass")) {
    +            String oldVal = (String) args.get("old_val");
    +            String newVal = (String) args.get("new_val");
    +            builder.renameClass(oldVal, newVal);
    +        } else if (name.equals("renameType")) {
    +            String oldVal = (String) args.get("old_val");
    +            String newVal = (String) args.get("new_val");
    +            builder.renameType(oldVal, newVal);
    +        } else if (name.equals("renameField")) {
    +            String clazz = (String) args.get("class_name");
    +            String oldVal = (String) args.get("old_val");
    +            String newVal = (String) args.get("new_val");
    +            builder.renameField(clazz, oldVal, newVal);
    +        } else if (name.equals("xslt")) {
    +            String url = (String) args.get("url");
    +            Map<String,?> substitutions = (Map<String, ?>) args.get("substitutions");
    +            String xsltTemplate = ResourceUtils.create(XsltTransformerTest.class).getResourceAsString(url);
    +            String xslt = TemplateProcessor.processTemplateContents(xsltTemplate, substitutions == null ? ImmutableMap.<String, String>of() : substitutions);
    +            builder.xsltTransformer(xslt);
    --- End diff --
    
    I think that passing the parameters to the XSLT script would be more flexible/useful than doing a String transformation beforehand. It would even make the processTemplateContents step superfluous. The existing xslt snippets can easily be transformed to accept parameters - I can do the changes if we go that way.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-brooklyn pull request: Feature: transform state

Posted by neykov <gi...@git.apache.org>.
Github user neykov commented on a diff in the pull request:

    https://github.com/apache/incubator-brooklyn/pull/177#discussion_r17963106
  
    --- Diff: core/src/main/java/brooklyn/entity/rebind/transformer/CompoundTransformerLoader.java ---
    @@ -0,0 +1,65 @@
    +package brooklyn.entity.rebind.transformer;
    +
    +import java.util.Map;
    +
    +import brooklyn.entity.rebind.transformer.CompoundTransformer.Builder;
    +import brooklyn.entity.rebind.transformer.impl.XsltTransformerTest;
    +import brooklyn.util.ResourceUtils;
    +import brooklyn.util.exceptions.Exceptions;
    +import brooklyn.util.text.TemplateProcessor;
    +import brooklyn.util.yaml.Yamls;
    +
    +import com.google.common.annotations.Beta;
    +import com.google.common.collect.ImmutableMap;
    +import com.google.common.collect.Iterables;
    +
    +@Beta
    +public class CompoundTransformerLoader {
    +
    +    // TODO Improve error handing so get nicer errors.
    +    // TODO Improve names (e.g. always camel case?)
    +    // TODO Pass in classloader for reflectively loading rawDataTransformer?
    +    
    +    public static CompoundTransformer load(String contents) {
    +        CompoundTransformer.Builder builder = CompoundTransformer.builder();
    +        Iterable<Object> toplevel = Yamls.parseAll(contents);
    +        Map<String, Map<?,?>> rules = (Map<String, Map<?,?>>) ((Map<?,?>)Iterables.getOnlyElement(toplevel));
    --- End diff --
    
    Warning:
    Type safety: Unchecked cast from Map<capture#1-of ?,capture#2-of ?> to Map<String,Map<?,?>>


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-brooklyn pull request: Feature: transform state

Posted by ahgittin <gi...@git.apache.org>.
Github user ahgittin commented on a diff in the pull request:

    https://github.com/apache/incubator-brooklyn/pull/177#discussion_r18041059
  
    --- Diff: core/src/main/java/brooklyn/entity/rebind/transformer/CompoundTransformerLoader.java ---
    @@ -0,0 +1,65 @@
    +package brooklyn.entity.rebind.transformer;
    +
    +import java.util.Map;
    +
    +import brooklyn.entity.rebind.transformer.CompoundTransformer.Builder;
    +import brooklyn.entity.rebind.transformer.impl.XsltTransformerTest;
    +import brooklyn.util.ResourceUtils;
    +import brooklyn.util.exceptions.Exceptions;
    +import brooklyn.util.text.TemplateProcessor;
    +import brooklyn.util.yaml.Yamls;
    +
    +import com.google.common.annotations.Beta;
    +import com.google.common.collect.ImmutableMap;
    +import com.google.common.collect.Iterables;
    +
    +@Beta
    +public class CompoundTransformerLoader {
    +
    +    // TODO Improve error handing so get nicer errors.
    +    // TODO Improve names (e.g. always camel case?)
    +    // TODO Pass in classloader for reflectively loading rawDataTransformer?
    +    
    +    public static CompoundTransformer load(String contents) {
    +        CompoundTransformer.Builder builder = CompoundTransformer.builder();
    +        Iterable<Object> toplevel = Yamls.parseAll(contents);
    +        Map<String, Map<?,?>> rules = (Map<String, Map<?,?>>) ((Map<?,?>)Iterables.getOnlyElement(toplevel));
    +        for (Map.Entry<String, Map<?,?>> entry : rules.entrySet()) {
    +            addRule(builder, entry.getKey(), entry.getValue());
    +        }
    +        return builder.build();
    +    }
    +
    +    private static void addRule(Builder builder, String name, Map<?,?> args) {
    +        if (name.equals("renameClass")) {
    +            String oldVal = (String) args.get("old_val");
    +            String newVal = (String) args.get("new_val");
    +            builder.renameClass(oldVal, newVal);
    +        } else if (name.equals("renameType")) {
    +            String oldVal = (String) args.get("old_val");
    +            String newVal = (String) args.get("new_val");
    +            builder.renameType(oldVal, newVal);
    +        } else if (name.equals("renameField")) {
    +            String clazz = (String) args.get("class_name");
    +            String oldVal = (String) args.get("old_val");
    +            String newVal = (String) args.get("new_val");
    +            builder.renameField(clazz, oldVal, newVal);
    +        } else if (name.equals("xslt")) {
    +            String url = (String) args.get("url");
    +            Map<String,?> substitutions = (Map<String, ?>) args.get("substitutions");
    +            String xsltTemplate = ResourceUtils.create(XsltTransformerTest.class).getResourceAsString(url);
    +            String xslt = TemplateProcessor.processTemplateContents(xsltTemplate, substitutions == null ? ImmutableMap.<String, String>of() : substitutions);
    +            builder.xsltTransformer(xslt);
    --- End diff --
    
    I'm on the fence here.  Our freemarker support is pretty good here, we can set that up to access entity config, etc.  So that seems pragmatic.
    
    OTOH parameters is the normal way with xslt, so that's standard.
    
    I'm not sure what params we need here, I think that would sway me.  For now however I lean towards keeping it as is, with a TODO (which I'll add).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-brooklyn pull request: Feature: transform state

Posted by ahgittin <gi...@git.apache.org>.
Github user ahgittin commented on a diff in the pull request:

    https://github.com/apache/incubator-brooklyn/pull/177#discussion_r18043995
  
    --- Diff: core/src/main/java/brooklyn/entity/rebind/transformer/CompoundTransformer.java ---
    @@ -0,0 +1,159 @@
    +package brooklyn.entity.rebind.transformer;
    +
    +import static com.google.common.base.Preconditions.checkNotNull;
    +
    +import java.util.Collection;
    +import java.util.Map;
    +
    +import brooklyn.entity.rebind.BrooklynObjectType;
    +import brooklyn.entity.rebind.RebindExceptionHandler;
    +import brooklyn.entity.rebind.persister.BrooklynMementoPersisterToObjectStore;
    +import brooklyn.entity.rebind.transformer.impl.XsltTransformer;
    +import brooklyn.entity.rebind.transformer.impl.XsltTransformerTest;
    +import brooklyn.mementos.BrooklynMementoRawData;
    +import brooklyn.util.ResourceUtils;
    +import brooklyn.util.collections.MutableMap;
    +import brooklyn.util.text.TemplateProcessor;
    +
    +import com.google.api.client.repackaged.com.google.common.annotations.VisibleForTesting;
    +import com.google.common.annotations.Beta;
    +import com.google.common.collect.ArrayListMultimap;
    +import com.google.common.collect.ImmutableMap;
    +import com.google.common.collect.Multimap;
    +
    +@Beta
    +public class CompoundTransformer {
    +    
    +    public static final CompoundTransformer NOOP = builder().build();
    +    
    +    // TODO Does not yet handle BrooklynMementoTransformer, for changing an entire BrooklynMemento.
    +    // Need to do refactoring in RebindManager and BrooklynMementoPersisterToObjectStore to convert 
    +    // from a BrooklynMementoRawData to a BrooklynMemento.
    +    
    +    public static Builder builder() {
    +        return new Builder();
    +    }
    +
    +    public static class Builder {
    +        private final Multimap<BrooklynObjectType, RawDataTransformer> rawDataTransformers = ArrayListMultimap.<BrooklynObjectType, RawDataTransformer>create();
    +        
    +        public Builder rawDataTransformer(RawDataTransformer val) {
    +            for (BrooklynObjectType type : BrooklynObjectType.values()) {
    +                rawDataTransformer(type, val);
    +            }
    +            return this;
    +        }
    +        public Builder rawDataTransformer(BrooklynObjectType type, RawDataTransformer val) {
    +            rawDataTransformers.put(checkNotNull(type, "type"), checkNotNull(val, "val"));
    +            return this;
    +        }
    +        public Builder renameType(String oldVal, String newVal) {
    +            // xstream format for inner classes is like <brooklyn.entity.rebind.transformer.CompoundTransformerTest_-OrigType>
    +            oldVal = toXstreamClassnameFormat(oldVal);
    +            newVal = toXstreamClassnameFormat(newVal);
    +            
    +            String xsltTemplate = ResourceUtils.create(XsltTransformerTest.class).getResourceAsString("classpath://brooklyn/entity/rebind/transformer/renameType.xslt");
    +            String xslt = TemplateProcessor.processTemplateContents(xsltTemplate, ImmutableMap.of("old_val", oldVal, "new_val", newVal));
    +            return xsltTransformer(xslt);
    +        }
    +        public Builder renameClass(String oldVal, String newVal) {
    --- End diff --
    
    the `<` in `Transofr<merTest` is a typo right?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-brooklyn pull request: Feature: transform state

Posted by neykov <gi...@git.apache.org>.
Github user neykov commented on a diff in the pull request:

    https://github.com/apache/incubator-brooklyn/pull/177#discussion_r17963995
  
    --- Diff: core/src/main/java/brooklyn/entity/rebind/transformer/CompoundTransformerLoader.java ---
    @@ -0,0 +1,65 @@
    +package brooklyn.entity.rebind.transformer;
    +
    +import java.util.Map;
    +
    +import brooklyn.entity.rebind.transformer.CompoundTransformer.Builder;
    +import brooklyn.entity.rebind.transformer.impl.XsltTransformerTest;
    +import brooklyn.util.ResourceUtils;
    +import brooklyn.util.exceptions.Exceptions;
    +import brooklyn.util.text.TemplateProcessor;
    +import brooklyn.util.yaml.Yamls;
    +
    +import com.google.common.annotations.Beta;
    +import com.google.common.collect.ImmutableMap;
    +import com.google.common.collect.Iterables;
    +
    +@Beta
    +public class CompoundTransformerLoader {
    +
    +    // TODO Improve error handing so get nicer errors.
    +    // TODO Improve names (e.g. always camel case?)
    +    // TODO Pass in classloader for reflectively loading rawDataTransformer?
    +    
    +    public static CompoundTransformer load(String contents) {
    +        CompoundTransformer.Builder builder = CompoundTransformer.builder();
    +        Iterable<Object> toplevel = Yamls.parseAll(contents);
    +        Map<String, Map<?,?>> rules = (Map<String, Map<?,?>>) ((Map<?,?>)Iterables.getOnlyElement(toplevel));
    +        for (Map.Entry<String, Map<?,?>> entry : rules.entrySet()) {
    +            addRule(builder, entry.getKey(), entry.getValue());
    +        }
    +        return builder.build();
    +    }
    +
    +    private static void addRule(Builder builder, String name, Map<?,?> args) {
    +        if (name.equals("renameClass")) {
    +            String oldVal = (String) args.get("old_val");
    +            String newVal = (String) args.get("new_val");
    +            builder.renameClass(oldVal, newVal);
    +        } else if (name.equals("renameType")) {
    +            String oldVal = (String) args.get("old_val");
    +            String newVal = (String) args.get("new_val");
    +            builder.renameType(oldVal, newVal);
    +        } else if (name.equals("renameField")) {
    +            String clazz = (String) args.get("class_name");
    +            String oldVal = (String) args.get("old_val");
    +            String newVal = (String) args.get("new_val");
    +            builder.renameField(clazz, oldVal, newVal);
    +        } else if (name.equals("xslt")) {
    +            String url = (String) args.get("url");
    +            Map<String,?> substitutions = (Map<String, ?>) args.get("substitutions");
    +            String xsltTemplate = ResourceUtils.create(XsltTransformerTest.class).getResourceAsString(url);
    +            String xslt = TemplateProcessor.processTemplateContents(xsltTemplate, substitutions == null ? ImmutableMap.<String, String>of() : substitutions);
    +            builder.xsltTransformer(xslt);
    +        } else if (name.equals("rawDataTransformer")) {
    +            String type = (String) args.get("type");
    +            try {
    +                Class<?> clazz = CompoundTransformerLoader.class.getClassLoader().loadClass(type);
    +                builder.rawDataTransformer((RawDataTransformer) clazz.newInstance());
    --- End diff --
    
    Can we pass the parameters to the raw transformer as well?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-brooklyn pull request: Feature: transform state

Posted by ahgittin <gi...@git.apache.org>.
Github user ahgittin commented on a diff in the pull request:

    https://github.com/apache/incubator-brooklyn/pull/177#discussion_r17847823
  
    --- Diff: core/src/main/java/brooklyn/entity/rebind/transformer/CompoundTransformer.java ---
    @@ -0,0 +1,159 @@
    +package brooklyn.entity.rebind.transformer;
    +
    +import static com.google.common.base.Preconditions.checkNotNull;
    +
    +import java.util.Collection;
    +import java.util.Map;
    +
    +import brooklyn.entity.rebind.BrooklynObjectType;
    +import brooklyn.entity.rebind.RebindExceptionHandler;
    +import brooklyn.entity.rebind.persister.BrooklynMementoPersisterToObjectStore;
    +import brooklyn.entity.rebind.transformer.impl.XsltTransformer;
    +import brooklyn.entity.rebind.transformer.impl.XsltTransformerTest;
    --- End diff --
    
    won't compile w test dependency in main class


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-brooklyn pull request: Feature: transform state

Posted by aledsage <gi...@git.apache.org>.
Github user aledsage commented on a diff in the pull request:

    https://github.com/apache/incubator-brooklyn/pull/177#discussion_r18063320
  
    --- Diff: core/src/main/java/brooklyn/entity/rebind/persister/BrooklynMementoPersisterToObjectStore.java ---
    @@ -279,134 +386,56 @@ public void run() {
                     throw new CompoundRuntimeException("Problem loading mementos", exceptions);
                 }
             }
    -
    -        BrooklynMementoManifest result = builder.build();
    -
    -        if (LOG.isDebugEnabled()) {
    -            LOG.debug("Loaded memento manifest; took {}; {} entities, {} locations, {} policies, {} enrichers, from {}", new Object[]{
    -                     Time.makeTimeStringRounded(stopwatch.elapsed(TimeUnit.MILLISECONDS)), result.getEntityIdToType().size(), 
    -                     result.getLocationIdToType().size(), result.getPolicyIdToType().size(), result.getEnricherIdToType().size(),
    -                     objectStore.getSummaryName() });
    -        }
    -
    -        if (result.getEntityIdToType().size() != entitySubPathList.size()) {
    -            LOG.error("Lost an entity?!");
    -        }
    -        
    -        return result;
         }
     
    -    @Override
    -    public BrooklynMemento loadMemento(LookupContext lookupContext, final RebindExceptionHandler exceptionHandler) throws IOException {
    +    @Beta
    +    public void checkpoint(BrooklynMementoRawData newMemento, PersistenceExceptionHandler exceptionHandler) {
             if (!running) {
    -            throw new IllegalStateException("Persister not running; cannot load memento from " + objectStore.getSummaryName());
    -        }
    -        Stopwatch stopwatch = Stopwatch.createStarted();
    -
    -        List<String> entitySubPathList;
    -        List<String> locationSubPathList;
    -        List<String> policySubPathList;
    -        List<String> enricherSubPathList;
    -        try {
    -            entitySubPathList = objectStore.listContentsWithSubPath("entities");
    -            locationSubPathList = objectStore.listContentsWithSubPath("locations");
    -            policySubPathList = objectStore.listContentsWithSubPath("policies");
    -            enricherSubPathList = objectStore.listContentsWithSubPath("enrichers");
    -        } catch (Exception e) {
    -            Exceptions.propagateIfFatal(e);
    -            exceptionHandler.onLoadMementoFailed(BrooklynObjectType.UNKNOWN, "Failed to list files", e);
    -            throw new IllegalStateException("Failed to list memento files in "+objectStore+": "+e, e);
    +            if (LOG.isDebugEnabled()) LOG.debug("Ignoring checkpointing entire memento, because not running");
    +            return;
             }
             
    -        LOG.debug("Loading persisted state: {} entities, {} locations, {} policies, {} enrichers, from {}", new Object[]{
    -            entitySubPathList.size(), locationSubPathList.size(), policySubPathList.size(), enricherSubPathList.size(),
    -            objectStore.getSummaryName() });
    -
    -        final BrooklynMementoImpl.Builder builder = BrooklynMementoImpl.builder();
    -        serializer.setLookupContext(lookupContext);
    -        
    -        List<ListenableFuture<?>> futures = Lists.newArrayList();
    -
    -        class MementoLoader implements Runnable {
    -            private final String subPath;
    -            private final BrooklynObjectType type;
    -            public MementoLoader(String subPath, BrooklynObjectType type) {
    -                this.subPath = subPath;
    -                this.type = type;
    -            }
    -            public void run() {
    -                try {
    -                    Memento memento = (Memento) serializer.fromString(read(subPath));
    -                    if (memento == null) {
    -                        LOG.warn("No "+type.toString().toLowerCase()+"-memento deserialized from " + subPath + "; ignoring and continuing");
    -                    } else {
    -                        builder.memento(memento);
    -                    }
    -                } catch (Exception e) {
    -                    exceptionHandler.onLoadMementoFailed(type, "Memento "+subPath, e);
    -                }
    -            }
    +        try {
    +            lock.writeLock().lockInterruptibly();
    +        } catch (InterruptedException e) {
    +            throw Exceptions.propagate(e);
             }
    -
             try {
    -            for (final String subPath : entitySubPathList) {
    -                futures.add(executor.submit(new MementoLoader(subPath, BrooklynObjectType.ENTITY)));
    +            objectStore.prepareForMasterUse();
    +            
    +            Stopwatch stopwatch = Stopwatch.createStarted();
    +            List<ListenableFuture<?>> futures = Lists.newArrayList();
    +            
    +            for (Map.Entry<String, String> entry : newMemento.getEntities().entrySet()) {
    +                futures.add(asyncPersist("entities", BrooklynObjectType.ENTITY, entry.getKey(), entry.getValue(), exceptionHandler));
                 }
    -            for (final String subPath : locationSubPathList) {
    -                futures.add(executor.submit(new MementoLoader(subPath, BrooklynObjectType.LOCATION)));
    +            for (Map.Entry<String, String> entry : newMemento.getLocations().entrySet()) {
    +                futures.add(asyncPersist("locations", BrooklynObjectType.LOCATION, entry.getKey(), entry.getValue(), exceptionHandler));
                 }
    -            for (final String subPath : policySubPathList) {
    -                futures.add(executor.submit(new MementoLoader(subPath, BrooklynObjectType.POLICY)));
    +            for (Map.Entry<String, String> entry : newMemento.getPolicies().entrySet()) {
    +                futures.add(asyncPersist("policies", BrooklynObjectType.POLICY, entry.getKey(), entry.getValue(), exceptionHandler));
                 }
    -            for (final String subPath : enricherSubPathList) {
    -                futures.add(executor.submit(new MementoLoader(subPath, BrooklynObjectType.ENRICHER)));
    +            for (Map.Entry<String, String> entry : newMemento.getEnrichers().entrySet()) {
    +                futures.add(asyncPersist("enrichers", BrooklynObjectType.ENRICHER, entry.getKey(), entry.getValue(), exceptionHandler));
                 }
                 
                 try {
    -                // Wait for all, failing fast if any exceptions.
    +                // Wait for all the tasks to complete or fail, rather than aborting on the first failure.
    +                // But then propagate failure if any fail. (hence the two calls).
    +                Futures.successfulAsList(futures).get();
                     Futures.allAsList(futures).get();
                 } catch (Exception e) {
    -                Exceptions.propagateIfFatal(e);
    -                
    -                List<Exception> exceptions = Lists.newArrayList();
    -                
    -                for (ListenableFuture<?> future : futures) {
    -                    if (future.isDone()) {
    -                        try {
    -                            future.get();
    -                        } catch (InterruptedException e2) {
    -                            throw Exceptions.propagate(e2);
    -                        } catch (ExecutionException e2) {
    -                            LOG.warn("Problem loading memento", e2);
    -                            exceptions.add(e2);
    -                        }
    -                        future.cancel(true);
    -                    }
    -                }
    -                if (exceptions.isEmpty()) {
    -                    throw Exceptions.propagate(e);
    -                } else {
    -                    // Normally there should be at lesat one failure; otherwise all.get() would not have failed.
    -                    throw new CompoundRuntimeException("Problem loading mementos", exceptions);
    --- End diff --
    
    @ahgittin not sure why I changed this! If we're now not getting logging of the individual problems, then definitely agree should restore this.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-brooklyn pull request: Feature: transform state

Posted by ahgittin <gi...@git.apache.org>.
Github user ahgittin commented on a diff in the pull request:

    https://github.com/apache/incubator-brooklyn/pull/177#discussion_r18041619
  
    --- Diff: core/src/main/java/brooklyn/entity/rebind/transformer/CompoundTransformer.java ---
    @@ -0,0 +1,159 @@
    +package brooklyn.entity.rebind.transformer;
    +
    +import static com.google.common.base.Preconditions.checkNotNull;
    +
    +import java.util.Collection;
    +import java.util.Map;
    +
    +import brooklyn.entity.rebind.BrooklynObjectType;
    +import brooklyn.entity.rebind.RebindExceptionHandler;
    +import brooklyn.entity.rebind.persister.BrooklynMementoPersisterToObjectStore;
    +import brooklyn.entity.rebind.transformer.impl.XsltTransformer;
    +import brooklyn.entity.rebind.transformer.impl.XsltTransformerTest;
    +import brooklyn.mementos.BrooklynMementoRawData;
    +import brooklyn.util.ResourceUtils;
    +import brooklyn.util.collections.MutableMap;
    +import brooklyn.util.text.TemplateProcessor;
    +
    +import com.google.api.client.repackaged.com.google.common.annotations.VisibleForTesting;
    +import com.google.common.annotations.Beta;
    +import com.google.common.collect.ArrayListMultimap;
    +import com.google.common.collect.ImmutableMap;
    +import com.google.common.collect.Multimap;
    +
    +@Beta
    +public class CompoundTransformer {
    +    
    +    public static final CompoundTransformer NOOP = builder().build();
    +    
    +    // TODO Does not yet handle BrooklynMementoTransformer, for changing an entire BrooklynMemento.
    +    // Need to do refactoring in RebindManager and BrooklynMementoPersisterToObjectStore to convert 
    +    // from a BrooklynMementoRawData to a BrooklynMemento.
    +    
    +    public static Builder builder() {
    +        return new Builder();
    +    }
    +
    +    public static class Builder {
    +        private final Multimap<BrooklynObjectType, RawDataTransformer> rawDataTransformers = ArrayListMultimap.<BrooklynObjectType, RawDataTransformer>create();
    +        
    +        public Builder rawDataTransformer(RawDataTransformer val) {
    +            for (BrooklynObjectType type : BrooklynObjectType.values()) {
    +                rawDataTransformer(type, val);
    +            }
    +            return this;
    +        }
    +        public Builder rawDataTransformer(BrooklynObjectType type, RawDataTransformer val) {
    +            rawDataTransformers.put(checkNotNull(type, "type"), checkNotNull(val, "val"));
    +            return this;
    +        }
    +        public Builder renameType(String oldVal, String newVal) {
    +            // xstream format for inner classes is like <brooklyn.entity.rebind.transformer.CompoundTransformerTest_-OrigType>
    +            oldVal = toXstreamClassnameFormat(oldVal);
    +            newVal = toXstreamClassnameFormat(newVal);
    +            
    +            String xsltTemplate = ResourceUtils.create(XsltTransformerTest.class).getResourceAsString("classpath://brooklyn/entity/rebind/transformer/renameType.xslt");
    +            String xslt = TemplateProcessor.processTemplateContents(xsltTemplate, ImmutableMap.of("old_val", oldVal, "new_val", newVal));
    +            return xsltTransformer(xslt);
    +        }
    +        public Builder renameClass(String oldVal, String newVal) {
    +            // xstream format for inner classes is like <brooklyn.entity.rebind.transformer.CompoundTransfor<merTest_-OrigType>
    +            oldVal = toXstreamClassnameFormat(oldVal);
    +            newVal = toXstreamClassnameFormat(newVal);
    +            
    +            String xsltTemplate = ResourceUtils.create(XsltTransformerTest.class).getResourceAsString("classpath://brooklyn/entity/rebind/transformer/renameClass.xslt");
    +            String xslt = TemplateProcessor.processTemplateContents(xsltTemplate, ImmutableMap.of("old_val", oldVal, "new_val", newVal));
    +            return xsltTransformer(xslt);
    +        }
    +        public Builder renameField(String clazz, String oldVal, String newVal) {
    +            // xstream format for inner classes is like <brooklyn.entity.rebind.transformer.CompoundTransfor<merTest_-OrigType>
    +            clazz = toXstreamClassnameFormat(clazz);
    +            oldVal = toXstreamClassnameFormat(oldVal);
    +            newVal = toXstreamClassnameFormat(newVal);
    +            
    +            String xsltTemplate = ResourceUtils.create(XsltTransformerTest.class).getResourceAsString("classpath://brooklyn/entity/rebind/transformer/renameClass.xslt");
    +            String xslt = TemplateProcessor.processTemplateContents(xsltTemplate, ImmutableMap.of("class_name", clazz, "old_val", oldVal, "new_val", newVal));
    +            return xsltTransformer(xslt);
    +        }
    +        public Builder xsltTransformer(String xslt) {
    +            XsltTransformer xsltTransformer = new XsltTransformer(xslt);
    +            for (BrooklynObjectType type : BrooklynObjectType.values()) {
    +                rawDataTransformer(type, xsltTransformer);
    +            }
    +            return this;
    +        }
    +        public Builder xsltTransformer(BrooklynObjectType type, String xslt) {
    +            XsltTransformer xsltTransformer = new XsltTransformer(xslt);
    +            rawDataTransformer(type, xsltTransformer);
    +            return this;
    +        }
    +        private String toXstreamClassnameFormat(String val) {
    +            return (val.contains("$")) ? val.replace("$", "_-") : val;
    +        }
    +        public CompoundTransformer build() {
    +            return new CompoundTransformer(this);
    +        }
    +    }
    +
    +    private final Multimap<BrooklynObjectType, RawDataTransformer> rawDataTransformers;
    +    
    +    protected CompoundTransformer(Builder builder) {
    +        rawDataTransformers = builder.rawDataTransformers;
    +    }
    +
    +    public BrooklynMementoRawData transform(BrooklynMementoPersisterToObjectStore reader, RebindExceptionHandler exceptionHandler) throws Exception {
    +        BrooklynMementoRawData rawData = reader.loadMementoRawData(exceptionHandler);
    +        return transform(rawData);
    +    }
    +    
    +    public BrooklynMementoRawData transform(BrooklynMementoRawData rawData) throws Exception {
    +        Map<String, String> entities = MutableMap.copyOf(rawData.getEntities());
    +        Map<String, String> locations = MutableMap.copyOf(rawData.getLocations());
    +        Map<String, String> policies = MutableMap.copyOf(rawData.getPolicies());
    +        Map<String, String> enrichers = MutableMap.copyOf(rawData.getEnrichers());
    +
    +        for (BrooklynObjectType type : BrooklynObjectType.values()) {
    +            Collection<RawDataTransformer> transformers = rawDataTransformers.get(type);
    --- End diff --
    
    good point.  OTOH entities usually refer to adjuncts, not the other way round, so not sure.  i think we fix this if we hit errors.  have added TODO in code:
    
            // TODO @neykov asks whether transformers should be run in registration order,
            // rather than in type order.  TBD.  (would be an easy change.)



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-brooklyn pull request: Feature: transform state

Posted by ahgittin <gi...@git.apache.org>.
Github user ahgittin commented on the pull request:

    https://github.com/apache/incubator-brooklyn/pull/177#issuecomment-56842752
  
    v good points in your list @neykov .  i think view this as a first pass.
    
    @aledsage i am updating this on top of catalog items work.  lots of conflicts there, as you've added new memento data store.  not complicated, but quite extensive.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-brooklyn pull request: Feature: transform state

Posted by ahgittin <gi...@git.apache.org>.
Github user ahgittin commented on a diff in the pull request:

    https://github.com/apache/incubator-brooklyn/pull/177#discussion_r18038726
  
    --- Diff: core/src/main/java/brooklyn/entity/rebind/persister/BrooklynMementoPersisterToObjectStore.java ---
    @@ -279,134 +386,56 @@ public void run() {
                     throw new CompoundRuntimeException("Problem loading mementos", exceptions);
                 }
             }
    -
    -        BrooklynMementoManifest result = builder.build();
    -
    -        if (LOG.isDebugEnabled()) {
    -            LOG.debug("Loaded memento manifest; took {}; {} entities, {} locations, {} policies, {} enrichers, from {}", new Object[]{
    -                     Time.makeTimeStringRounded(stopwatch.elapsed(TimeUnit.MILLISECONDS)), result.getEntityIdToType().size(), 
    -                     result.getLocationIdToType().size(), result.getPolicyIdToType().size(), result.getEnricherIdToType().size(),
    -                     objectStore.getSummaryName() });
    -        }
    -
    -        if (result.getEntityIdToType().size() != entitySubPathList.size()) {
    -            LOG.error("Lost an entity?!");
    -        }
    -        
    -        return result;
         }
     
    -    @Override
    -    public BrooklynMemento loadMemento(LookupContext lookupContext, final RebindExceptionHandler exceptionHandler) throws IOException {
    +    @Beta
    +    public void checkpoint(BrooklynMementoRawData newMemento, PersistenceExceptionHandler exceptionHandler) {
             if (!running) {
    -            throw new IllegalStateException("Persister not running; cannot load memento from " + objectStore.getSummaryName());
    -        }
    -        Stopwatch stopwatch = Stopwatch.createStarted();
    -
    -        List<String> entitySubPathList;
    -        List<String> locationSubPathList;
    -        List<String> policySubPathList;
    -        List<String> enricherSubPathList;
    -        try {
    -            entitySubPathList = objectStore.listContentsWithSubPath("entities");
    -            locationSubPathList = objectStore.listContentsWithSubPath("locations");
    -            policySubPathList = objectStore.listContentsWithSubPath("policies");
    -            enricherSubPathList = objectStore.listContentsWithSubPath("enrichers");
    -        } catch (Exception e) {
    -            Exceptions.propagateIfFatal(e);
    -            exceptionHandler.onLoadMementoFailed(BrooklynObjectType.UNKNOWN, "Failed to list files", e);
    -            throw new IllegalStateException("Failed to list memento files in "+objectStore+": "+e, e);
    +            if (LOG.isDebugEnabled()) LOG.debug("Ignoring checkpointing entire memento, because not running");
    +            return;
             }
             
    -        LOG.debug("Loading persisted state: {} entities, {} locations, {} policies, {} enrichers, from {}", new Object[]{
    -            entitySubPathList.size(), locationSubPathList.size(), policySubPathList.size(), enricherSubPathList.size(),
    -            objectStore.getSummaryName() });
    -
    -        final BrooklynMementoImpl.Builder builder = BrooklynMementoImpl.builder();
    -        serializer.setLookupContext(lookupContext);
    -        
    -        List<ListenableFuture<?>> futures = Lists.newArrayList();
    -
    -        class MementoLoader implements Runnable {
    -            private final String subPath;
    -            private final BrooklynObjectType type;
    -            public MementoLoader(String subPath, BrooklynObjectType type) {
    -                this.subPath = subPath;
    -                this.type = type;
    -            }
    -            public void run() {
    -                try {
    -                    Memento memento = (Memento) serializer.fromString(read(subPath));
    -                    if (memento == null) {
    -                        LOG.warn("No "+type.toString().toLowerCase()+"-memento deserialized from " + subPath + "; ignoring and continuing");
    -                    } else {
    -                        builder.memento(memento);
    -                    }
    -                } catch (Exception e) {
    -                    exceptionHandler.onLoadMementoFailed(type, "Memento "+subPath, e);
    -                }
    -            }
    +        try {
    +            lock.writeLock().lockInterruptibly();
    +        } catch (InterruptedException e) {
    +            throw Exceptions.propagate(e);
             }
    -
             try {
    -            for (final String subPath : entitySubPathList) {
    -                futures.add(executor.submit(new MementoLoader(subPath, BrooklynObjectType.ENTITY)));
    +            objectStore.prepareForMasterUse();
    +            
    +            Stopwatch stopwatch = Stopwatch.createStarted();
    +            List<ListenableFuture<?>> futures = Lists.newArrayList();
    +            
    +            for (Map.Entry<String, String> entry : newMemento.getEntities().entrySet()) {
    +                futures.add(asyncPersist("entities", BrooklynObjectType.ENTITY, entry.getKey(), entry.getValue(), exceptionHandler));
                 }
    -            for (final String subPath : locationSubPathList) {
    -                futures.add(executor.submit(new MementoLoader(subPath, BrooklynObjectType.LOCATION)));
    +            for (Map.Entry<String, String> entry : newMemento.getLocations().entrySet()) {
    +                futures.add(asyncPersist("locations", BrooklynObjectType.LOCATION, entry.getKey(), entry.getValue(), exceptionHandler));
                 }
    -            for (final String subPath : policySubPathList) {
    -                futures.add(executor.submit(new MementoLoader(subPath, BrooklynObjectType.POLICY)));
    +            for (Map.Entry<String, String> entry : newMemento.getPolicies().entrySet()) {
    +                futures.add(asyncPersist("policies", BrooklynObjectType.POLICY, entry.getKey(), entry.getValue(), exceptionHandler));
                 }
    -            for (final String subPath : enricherSubPathList) {
    -                futures.add(executor.submit(new MementoLoader(subPath, BrooklynObjectType.ENRICHER)));
    +            for (Map.Entry<String, String> entry : newMemento.getEnrichers().entrySet()) {
    +                futures.add(asyncPersist("enrichers", BrooklynObjectType.ENRICHER, entry.getKey(), entry.getValue(), exceptionHandler));
                 }
                 
                 try {
    -                // Wait for all, failing fast if any exceptions.
    +                // Wait for all the tasks to complete or fail, rather than aborting on the first failure.
    +                // But then propagate failure if any fail. (hence the two calls).
    +                Futures.successfulAsList(futures).get();
                     Futures.allAsList(futures).get();
                 } catch (Exception e) {
    -                Exceptions.propagateIfFatal(e);
    -                
    -                List<Exception> exceptions = Lists.newArrayList();
    -                
    -                for (ListenableFuture<?> future : futures) {
    -                    if (future.isDone()) {
    -                        try {
    -                            future.get();
    -                        } catch (InterruptedException e2) {
    -                            throw Exceptions.propagate(e2);
    -                        } catch (ExecutionException e2) {
    -                            LOG.warn("Problem loading memento", e2);
    -                            exceptions.add(e2);
    -                        }
    -                        future.cancel(true);
    -                    }
    -                }
    -                if (exceptions.isEmpty()) {
    -                    throw Exceptions.propagate(e);
    -                } else {
    -                    // Normally there should be at lesat one failure; otherwise all.get() would not have failed.
    -                    throw new CompoundRuntimeException("Problem loading mementos", exceptions);
    --- End diff --
    
    I really like the way `CompoundRuntimeException` reports things, with a smart summary when there are many errors.
    
    That coupled with the `LOG.warn` above means we get very good logging on rebind failures.
    
    @aledsage Do you think the reporting in your new scheme is just as good?  Or did you do it for code cleanliness?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-brooklyn pull request: Feature: transform state

Posted by neykov <gi...@git.apache.org>.
Github user neykov commented on the pull request:

    https://github.com/apache/incubator-brooklyn/pull/177#issuecomment-56665694
  
    Completed the review. Very useful functionality. I like the YAML approach to defining the transformers. 
    
    I'd like to put down in writing some thoughts, not directly related to this PR. 
      * There are use-cases where access to the whole collection of items is needed. A single transformer could need multiple passes or a more complex strategy for updates. For example creating, deleting items, splitting classes, transforms related to object IDs (references).
      * The person administering the brooklyn instance most certainly doesn't know enough of the entity internals and changes to be able to write the transforms needed for updating to a new version. He can see that there is a problem with rebinding and even check the error message and pass it on. It is the entity creators' obligation and ability to create the transforms. Would be great if we allow bundling the transforms along with the entities in the same deliverable. This way the administrator could pass the problem to the entity developer who will create a new version of the package and push it back to the brooklyn instance to resolve the rebinding problem. For entities not distributed with brooklyn this is even more important.
      * If we bundle the transforms with the entities and don't have a person hand picking them for each specific installation then we have the issues of ordering and which ones to apply. When third party entities get involved then it becomes even more complex.
      * We have discussed a functionality which, as a part of CI, to test rebinding against a real-world storage location for early warning. Not sure if this is a good idea, having a step as part of the maven build which automatically serializes each new entity it finds and during testing tries to rebind it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-brooklyn pull request: Feature: transform state

Posted by aledsage <gi...@git.apache.org>.
Github user aledsage commented on a diff in the pull request:

    https://github.com/apache/incubator-brooklyn/pull/177#discussion_r18062967
  
    --- Diff: core/src/main/java/brooklyn/entity/rebind/transformer/CompoundTransformer.java ---
    @@ -0,0 +1,159 @@
    +package brooklyn.entity.rebind.transformer;
    +
    +import static com.google.common.base.Preconditions.checkNotNull;
    +
    +import java.util.Collection;
    +import java.util.Map;
    +
    +import brooklyn.entity.rebind.BrooklynObjectType;
    +import brooklyn.entity.rebind.RebindExceptionHandler;
    +import brooklyn.entity.rebind.persister.BrooklynMementoPersisterToObjectStore;
    +import brooklyn.entity.rebind.transformer.impl.XsltTransformer;
    +import brooklyn.entity.rebind.transformer.impl.XsltTransformerTest;
    +import brooklyn.mementos.BrooklynMementoRawData;
    +import brooklyn.util.ResourceUtils;
    +import brooklyn.util.collections.MutableMap;
    +import brooklyn.util.text.TemplateProcessor;
    +
    +import com.google.api.client.repackaged.com.google.common.annotations.VisibleForTesting;
    +import com.google.common.annotations.Beta;
    +import com.google.common.collect.ArrayListMultimap;
    +import com.google.common.collect.ImmutableMap;
    +import com.google.common.collect.Multimap;
    +
    +@Beta
    +public class CompoundTransformer {
    +    
    +    public static final CompoundTransformer NOOP = builder().build();
    +    
    +    // TODO Does not yet handle BrooklynMementoTransformer, for changing an entire BrooklynMemento.
    +    // Need to do refactoring in RebindManager and BrooklynMementoPersisterToObjectStore to convert 
    +    // from a BrooklynMementoRawData to a BrooklynMemento.
    +    
    +    public static Builder builder() {
    +        return new Builder();
    +    }
    +
    +    public static class Builder {
    +        private final Multimap<BrooklynObjectType, RawDataTransformer> rawDataTransformers = ArrayListMultimap.<BrooklynObjectType, RawDataTransformer>create();
    +        
    +        public Builder rawDataTransformer(RawDataTransformer val) {
    +            for (BrooklynObjectType type : BrooklynObjectType.values()) {
    +                rawDataTransformer(type, val);
    +            }
    +            return this;
    +        }
    +        public Builder rawDataTransformer(BrooklynObjectType type, RawDataTransformer val) {
    +            rawDataTransformers.put(checkNotNull(type, "type"), checkNotNull(val, "val"));
    +            return this;
    +        }
    +        public Builder renameType(String oldVal, String newVal) {
    +            // xstream format for inner classes is like <brooklyn.entity.rebind.transformer.CompoundTransformerTest_-OrigType>
    +            oldVal = toXstreamClassnameFormat(oldVal);
    +            newVal = toXstreamClassnameFormat(newVal);
    +            
    +            String xsltTemplate = ResourceUtils.create(XsltTransformerTest.class).getResourceAsString("classpath://brooklyn/entity/rebind/transformer/renameType.xslt");
    +            String xslt = TemplateProcessor.processTemplateContents(xsltTemplate, ImmutableMap.of("old_val", oldVal, "new_val", newVal));
    +            return xsltTransformer(xslt);
    +        }
    +        public Builder renameClass(String oldVal, String newVal) {
    +            // xstream format for inner classes is like <brooklyn.entity.rebind.transformer.CompoundTransfor<merTest_-OrigType>
    +            oldVal = toXstreamClassnameFormat(oldVal);
    +            newVal = toXstreamClassnameFormat(newVal);
    +            
    +            String xsltTemplate = ResourceUtils.create(XsltTransformerTest.class).getResourceAsString("classpath://brooklyn/entity/rebind/transformer/renameClass.xslt");
    +            String xslt = TemplateProcessor.processTemplateContents(xsltTemplate, ImmutableMap.of("old_val", oldVal, "new_val", newVal));
    +            return xsltTransformer(xslt);
    +        }
    +        public Builder renameField(String clazz, String oldVal, String newVal) {
    +            // xstream format for inner classes is like <brooklyn.entity.rebind.transformer.CompoundTransfor<merTest_-OrigType>
    +            clazz = toXstreamClassnameFormat(clazz);
    +            oldVal = toXstreamClassnameFormat(oldVal);
    +            newVal = toXstreamClassnameFormat(newVal);
    +            
    +            String xsltTemplate = ResourceUtils.create(XsltTransformerTest.class).getResourceAsString("classpath://brooklyn/entity/rebind/transformer/renameClass.xslt");
    +            String xslt = TemplateProcessor.processTemplateContents(xsltTemplate, ImmutableMap.of("class_name", clazz, "old_val", oldVal, "new_val", newVal));
    +            return xsltTransformer(xslt);
    +        }
    +        public Builder xsltTransformer(String xslt) {
    +            XsltTransformer xsltTransformer = new XsltTransformer(xslt);
    +            for (BrooklynObjectType type : BrooklynObjectType.values()) {
    +                rawDataTransformer(type, xsltTransformer);
    +            }
    +            return this;
    +        }
    +        public Builder xsltTransformer(BrooklynObjectType type, String xslt) {
    +            XsltTransformer xsltTransformer = new XsltTransformer(xslt);
    +            rawDataTransformer(type, xsltTransformer);
    +            return this;
    +        }
    +        private String toXstreamClassnameFormat(String val) {
    +            return (val.contains("$")) ? val.replace("$", "_-") : val;
    +        }
    +        public CompoundTransformer build() {
    +            return new CompoundTransformer(this);
    +        }
    +    }
    +
    +    private final Multimap<BrooklynObjectType, RawDataTransformer> rawDataTransformers;
    +    
    +    protected CompoundTransformer(Builder builder) {
    +        rawDataTransformers = builder.rawDataTransformers;
    +    }
    +
    +    public BrooklynMementoRawData transform(BrooklynMementoPersisterToObjectStore reader, RebindExceptionHandler exceptionHandler) throws Exception {
    +        BrooklynMementoRawData rawData = reader.loadMementoRawData(exceptionHandler);
    +        return transform(rawData);
    +    }
    +    
    +    public BrooklynMementoRawData transform(BrooklynMementoRawData rawData) throws Exception {
    +        Map<String, String> entities = MutableMap.copyOf(rawData.getEntities());
    +        Map<String, String> locations = MutableMap.copyOf(rawData.getLocations());
    +        Map<String, String> policies = MutableMap.copyOf(rawData.getPolicies());
    +        Map<String, String> enrichers = MutableMap.copyOf(rawData.getEnrichers());
    +
    +        for (BrooklynObjectType type : BrooklynObjectType.values()) {
    +            Collection<RawDataTransformer> transformers = rawDataTransformers.get(type);
    --- End diff --
    
    Within a given type, they will be applied in registration order. If two rules apply to different types, then they shouldn't try to modify the same file. So agree with @ahgittin that it doesn't matter.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-brooklyn pull request: Feature: transform state

Posted by neykov <gi...@git.apache.org>.
Github user neykov commented on a diff in the pull request:

    https://github.com/apache/incubator-brooklyn/pull/177#discussion_r17963097
  
    --- Diff: core/src/main/java/brooklyn/entity/rebind/transformer/CompoundTransformerLoader.java ---
    @@ -0,0 +1,65 @@
    +package brooklyn.entity.rebind.transformer;
    +
    +import java.util.Map;
    +
    +import brooklyn.entity.rebind.transformer.CompoundTransformer.Builder;
    +import brooklyn.entity.rebind.transformer.impl.XsltTransformerTest;
    +import brooklyn.util.ResourceUtils;
    +import brooklyn.util.exceptions.Exceptions;
    +import brooklyn.util.text.TemplateProcessor;
    +import brooklyn.util.yaml.Yamls;
    +
    +import com.google.common.annotations.Beta;
    +import com.google.common.collect.ImmutableMap;
    +import com.google.common.collect.Iterables;
    +
    +@Beta
    +public class CompoundTransformerLoader {
    +
    +    // TODO Improve error handing so get nicer errors.
    +    // TODO Improve names (e.g. always camel case?)
    +    // TODO Pass in classloader for reflectively loading rawDataTransformer?
    +    
    +    public static CompoundTransformer load(String contents) {
    +        CompoundTransformer.Builder builder = CompoundTransformer.builder();
    +        Iterable<Object> toplevel = Yamls.parseAll(contents);
    +        Map<String, Map<?,?>> rules = (Map<String, Map<?,?>>) ((Map<?,?>)Iterables.getOnlyElement(toplevel));
    +        for (Map.Entry<String, Map<?,?>> entry : rules.entrySet()) {
    +            addRule(builder, entry.getKey(), entry.getValue());
    +        }
    +        return builder.build();
    +    }
    +
    +    private static void addRule(Builder builder, String name, Map<?,?> args) {
    +        if (name.equals("renameClass")) {
    +            String oldVal = (String) args.get("old_val");
    +            String newVal = (String) args.get("new_val");
    +            builder.renameClass(oldVal, newVal);
    +        } else if (name.equals("renameType")) {
    +            String oldVal = (String) args.get("old_val");
    +            String newVal = (String) args.get("new_val");
    +            builder.renameType(oldVal, newVal);
    +        } else if (name.equals("renameField")) {
    +            String clazz = (String) args.get("class_name");
    +            String oldVal = (String) args.get("old_val");
    +            String newVal = (String) args.get("new_val");
    +            builder.renameField(clazz, oldVal, newVal);
    +        } else if (name.equals("xslt")) {
    +            String url = (String) args.get("url");
    +            Map<String,?> substitutions = (Map<String, ?>) args.get("substitutions");
    --- End diff --
    
    I get a warning in Eclipse:
    Type safety: Unchecked cast from capture#22-of ? to Map<String,?>


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-brooklyn pull request: Feature: transform state

Posted by ahgittin <gi...@git.apache.org>.
Github user ahgittin commented on the pull request:

    https://github.com/apache/incubator-brooklyn/pull/177#issuecomment-56849595
  
    rat bites again, `XsltTransformerTest` (fixed, before merge this time!).
    @cloudbees we miss you /cc @richardcloudsoft @grkvlt !


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-brooklyn pull request: Feature: transform state

Posted by ahgittin <gi...@git.apache.org>.
Github user ahgittin commented on a diff in the pull request:

    https://github.com/apache/incubator-brooklyn/pull/177#discussion_r18044210
  
    --- Diff: core/src/main/java/brooklyn/entity/rebind/transformer/CompoundTransformer.java ---
    @@ -0,0 +1,159 @@
    +package brooklyn.entity.rebind.transformer;
    +
    +import static com.google.common.base.Preconditions.checkNotNull;
    +
    +import java.util.Collection;
    +import java.util.Map;
    +
    +import brooklyn.entity.rebind.BrooklynObjectType;
    +import brooklyn.entity.rebind.RebindExceptionHandler;
    +import brooklyn.entity.rebind.persister.BrooklynMementoPersisterToObjectStore;
    +import brooklyn.entity.rebind.transformer.impl.XsltTransformer;
    +import brooklyn.entity.rebind.transformer.impl.XsltTransformerTest;
    --- End diff --
    
    only used to resolve `classpath:` urls, so easily replaced (i've done this)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-brooklyn pull request: Feature: transform state

Posted by neykov <gi...@git.apache.org>.
Github user neykov commented on a diff in the pull request:

    https://github.com/apache/incubator-brooklyn/pull/177#discussion_r17967344
  
    --- Diff: usage/cli/src/main/java/brooklyn/cli/Main.java ---
    @@ -682,28 +683,34 @@ public ToStringHelper string() {
             public String localBrooklynProperties;
     
             @Option(name = { "--persistenceDir" }, title = "persistence dir",
    -                description = "The directory to read/write persisted state (or container name if using an object store)")
    +                description = "The directory to read persisted state (or container name if using an object store)")
             public String persistenceDir;
     
             @Option(name = { "--persistenceLocation" }, title = "persistence location",
    -            description = "The location spec for an object store to read/write persisted state")
    +            description = "The location spec for an object store to read persisted state")
             public String persistenceLocation;
         
             @Option(name = { "--destinationDir" }, required = true, title = "destination dir",
                     description = "The directory to copy persistence data to")
                 public String destinationDir;
             
    +        @Option(name = { "--destinationLocation" }, title = "persistence location",
    +                description = "The location spec for an object store to copy data to")
    --- End diff --
    
    When reading the descriptions I am left with the impression that destinationDir and destinationLocation are alternative options rather than complementary.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-brooklyn pull request: Feature: transform state

Posted by neykov <gi...@git.apache.org>.
Github user neykov commented on a diff in the pull request:

    https://github.com/apache/incubator-brooklyn/pull/177#discussion_r17964491
  
    --- Diff: core/src/test/java/brooklyn/entity/rebind/transformer/impl/XsltTransformerTest.java ---
    @@ -0,0 +1,117 @@
    +package brooklyn.entity.rebind.transformer.impl;
    +
    +import static org.testng.Assert.assertEquals;
    +
    +import org.testng.annotations.Test;
    +
    +import brooklyn.entity.rebind.transformer.impl.XsltTransformer;
    +import brooklyn.entity.rebind.transformer.impl.XsltTransformerTest;
    +import brooklyn.util.ResourceUtils;
    +import brooklyn.util.text.TemplateProcessor;
    +
    +import com.google.common.collect.ImmutableMap;
    +
    +public class XsltTransformerTest {
    +
    +    @Test
    +    public void testRenameType() throws Exception {
    +        String xsltTemplate = ResourceUtils.create(XsltTransformerTest.class).getResourceAsString("classpath://brooklyn/entity/rebind/transformer/renameType.xslt");
    +        String xslt = TemplateProcessor.processTemplateContents(xsltTemplate, ImmutableMap.of("old_val", "mytype.Before", "new_val", "mytype.After"));
    +        String input = 
    +                "<entity myattrib=\"myval\">"+"\n"+
    --- End diff --
    
    Single quotes can be used to avoid escapes, but fine as is.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-brooklyn pull request: Feature: transform state

Posted by neykov <gi...@git.apache.org>.
Github user neykov commented on a diff in the pull request:

    https://github.com/apache/incubator-brooklyn/pull/177#discussion_r17963054
  
    --- Diff: core/src/main/java/brooklyn/entity/rebind/transformer/CompoundTransformerLoader.java ---
    @@ -0,0 +1,65 @@
    +package brooklyn.entity.rebind.transformer;
    +
    +import java.util.Map;
    +
    +import brooklyn.entity.rebind.transformer.CompoundTransformer.Builder;
    +import brooklyn.entity.rebind.transformer.impl.XsltTransformerTest;
    +import brooklyn.util.ResourceUtils;
    +import brooklyn.util.exceptions.Exceptions;
    +import brooklyn.util.text.TemplateProcessor;
    +import brooklyn.util.yaml.Yamls;
    +
    +import com.google.common.annotations.Beta;
    +import com.google.common.collect.ImmutableMap;
    +import com.google.common.collect.Iterables;
    +
    +@Beta
    +public class CompoundTransformerLoader {
    +
    +    // TODO Improve error handing so get nicer errors.
    +    // TODO Improve names (e.g. always camel case?)
    +    // TODO Pass in classloader for reflectively loading rawDataTransformer?
    +    
    +    public static CompoundTransformer load(String contents) {
    +        CompoundTransformer.Builder builder = CompoundTransformer.builder();
    +        Iterable<Object> toplevel = Yamls.parseAll(contents);
    +        Map<String, Map<?,?>> rules = (Map<String, Map<?,?>>) ((Map<?,?>)Iterables.getOnlyElement(toplevel));
    +        for (Map.Entry<String, Map<?,?>> entry : rules.entrySet()) {
    +            addRule(builder, entry.getKey(), entry.getValue());
    +        }
    +        return builder.build();
    +    }
    +
    +    private static void addRule(Builder builder, String name, Map<?,?> args) {
    +        if (name.equals("renameClass")) {
    +            String oldVal = (String) args.get("old_val");
    +            String newVal = (String) args.get("new_val");
    +            builder.renameClass(oldVal, newVal);
    +        } else if (name.equals("renameType")) {
    +            String oldVal = (String) args.get("old_val");
    +            String newVal = (String) args.get("new_val");
    +            builder.renameType(oldVal, newVal);
    +        } else if (name.equals("renameField")) {
    +            String clazz = (String) args.get("class_name");
    +            String oldVal = (String) args.get("old_val");
    +            String newVal = (String) args.get("new_val");
    +            builder.renameField(clazz, oldVal, newVal);
    +        } else if (name.equals("xslt")) {
    +            String url = (String) args.get("url");
    +            Map<String,?> substitutions = (Map<String, ?>) args.get("substitutions");
    +            String xsltTemplate = ResourceUtils.create(XsltTransformerTest.class).getResourceAsString(url);
    +            String xslt = TemplateProcessor.processTemplateContents(xsltTemplate, substitutions == null ? ImmutableMap.<String, String>of() : substitutions);
    --- End diff --
    
    No need for the null check, processTemplateContents handles null arguments.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-brooklyn pull request: Feature: transform state

Posted by neykov <gi...@git.apache.org>.
Github user neykov commented on a diff in the pull request:

    https://github.com/apache/incubator-brooklyn/pull/177#discussion_r17961266
  
    --- Diff: usage/cli/src/main/java/brooklyn/cli/Main.java ---
    @@ -724,8 +731,10 @@ public Void call() throws Exception {
                 }
                 
                 try {
    -                BrooklynMemento memento = launcher.retrieveState();
    -                launcher.persistState(memento, destinationDirF);
    +                BrooklynMementoRawData memento = launcher.retrieveState();
    +                CompoundTransformer transformer = loadTransformer(transformations);
    +                BrooklynMementoRawData newMemento = transformer.transform(memento);
    +                launcher.persistState(memento, destinationDir, destinationLocation);
    --- End diff --
    
    Should persist newMemento instead.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-brooklyn pull request: Feature: transform state

Posted by ahgittin <gi...@git.apache.org>.
Github user ahgittin commented on the pull request:

    https://github.com/apache/incubator-brooklyn/pull/177#issuecomment-56848812
  
    finished working through merge conflicts (mainly around catalog items) and reviewing; i'm going to merge shortly, once tests pass, and assuming you don't object to the assumptions and comments i've made above @aledsage @neykov 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-brooklyn pull request: Feature: transform state

Posted by ahgittin <gi...@git.apache.org>.
Github user ahgittin commented on a diff in the pull request:

    https://github.com/apache/incubator-brooklyn/pull/177#discussion_r18042623
  
    --- Diff: core/src/main/java/brooklyn/entity/rebind/transformer/CompoundTransformer.java ---
    @@ -0,0 +1,159 @@
    +package brooklyn.entity.rebind.transformer;
    +
    +import static com.google.common.base.Preconditions.checkNotNull;
    +
    +import java.util.Collection;
    +import java.util.Map;
    +
    +import brooklyn.entity.rebind.BrooklynObjectType;
    +import brooklyn.entity.rebind.RebindExceptionHandler;
    +import brooklyn.entity.rebind.persister.BrooklynMementoPersisterToObjectStore;
    +import brooklyn.entity.rebind.transformer.impl.XsltTransformer;
    +import brooklyn.entity.rebind.transformer.impl.XsltTransformerTest;
    +import brooklyn.mementos.BrooklynMementoRawData;
    +import brooklyn.util.ResourceUtils;
    +import brooklyn.util.collections.MutableMap;
    +import brooklyn.util.text.TemplateProcessor;
    +
    +import com.google.api.client.repackaged.com.google.common.annotations.VisibleForTesting;
    +import com.google.common.annotations.Beta;
    +import com.google.common.collect.ArrayListMultimap;
    +import com.google.common.collect.ImmutableMap;
    +import com.google.common.collect.Multimap;
    +
    +@Beta
    +public class CompoundTransformer {
    +    
    +    public static final CompoundTransformer NOOP = builder().build();
    +    
    +    // TODO Does not yet handle BrooklynMementoTransformer, for changing an entire BrooklynMemento.
    +    // Need to do refactoring in RebindManager and BrooklynMementoPersisterToObjectStore to convert 
    +    // from a BrooklynMementoRawData to a BrooklynMemento.
    +    
    +    public static Builder builder() {
    +        return new Builder();
    +    }
    +
    +    public static class Builder {
    +        private final Multimap<BrooklynObjectType, RawDataTransformer> rawDataTransformers = ArrayListMultimap.<BrooklynObjectType, RawDataTransformer>create();
    +        
    +        public Builder rawDataTransformer(RawDataTransformer val) {
    +            for (BrooklynObjectType type : BrooklynObjectType.values()) {
    +                rawDataTransformer(type, val);
    +            }
    +            return this;
    +        }
    +        public Builder rawDataTransformer(BrooklynObjectType type, RawDataTransformer val) {
    +            rawDataTransformers.put(checkNotNull(type, "type"), checkNotNull(val, "val"));
    +            return this;
    +        }
    +        public Builder renameType(String oldVal, String newVal) {
    +            // xstream format for inner classes is like <brooklyn.entity.rebind.transformer.CompoundTransformerTest_-OrigType>
    +            oldVal = toXstreamClassnameFormat(oldVal);
    +            newVal = toXstreamClassnameFormat(newVal);
    +            
    +            String xsltTemplate = ResourceUtils.create(XsltTransformerTest.class).getResourceAsString("classpath://brooklyn/entity/rebind/transformer/renameType.xslt");
    +            String xslt = TemplateProcessor.processTemplateContents(xsltTemplate, ImmutableMap.of("old_val", oldVal, "new_val", newVal));
    +            return xsltTransformer(xslt);
    +        }
    +        public Builder renameClass(String oldVal, String newVal) {
    +            // xstream format for inner classes is like <brooklyn.entity.rebind.transformer.CompoundTransfor<merTest_-OrigType>
    +            oldVal = toXstreamClassnameFormat(oldVal);
    +            newVal = toXstreamClassnameFormat(newVal);
    +            
    +            String xsltTemplate = ResourceUtils.create(XsltTransformerTest.class).getResourceAsString("classpath://brooklyn/entity/rebind/transformer/renameClass.xslt");
    +            String xslt = TemplateProcessor.processTemplateContents(xsltTemplate, ImmutableMap.of("old_val", oldVal, "new_val", newVal));
    +            return xsltTransformer(xslt);
    +        }
    +        public Builder renameField(String clazz, String oldVal, String newVal) {
    +            // xstream format for inner classes is like <brooklyn.entity.rebind.transformer.CompoundTransfor<merTest_-OrigType>
    +            clazz = toXstreamClassnameFormat(clazz);
    +            oldVal = toXstreamClassnameFormat(oldVal);
    +            newVal = toXstreamClassnameFormat(newVal);
    +            
    +            String xsltTemplate = ResourceUtils.create(XsltTransformerTest.class).getResourceAsString("classpath://brooklyn/entity/rebind/transformer/renameClass.xslt");
    +            String xslt = TemplateProcessor.processTemplateContents(xsltTemplate, ImmutableMap.of("class_name", clazz, "old_val", oldVal, "new_val", newVal));
    +            return xsltTransformer(xslt);
    +        }
    +        public Builder xsltTransformer(String xslt) {
    +            XsltTransformer xsltTransformer = new XsltTransformer(xslt);
    +            for (BrooklynObjectType type : BrooklynObjectType.values()) {
    +                rawDataTransformer(type, xsltTransformer);
    +            }
    +            return this;
    +        }
    +        public Builder xsltTransformer(BrooklynObjectType type, String xslt) {
    +            XsltTransformer xsltTransformer = new XsltTransformer(xslt);
    +            rawDataTransformer(type, xsltTransformer);
    +            return this;
    +        }
    +        private String toXstreamClassnameFormat(String val) {
    +            return (val.contains("$")) ? val.replace("$", "_-") : val;
    +        }
    +        public CompoundTransformer build() {
    +            return new CompoundTransformer(this);
    +        }
    +    }
    +
    +    private final Multimap<BrooklynObjectType, RawDataTransformer> rawDataTransformers;
    +    
    +    protected CompoundTransformer(Builder builder) {
    +        rawDataTransformers = builder.rawDataTransformers;
    +    }
    +
    +    public BrooklynMementoRawData transform(BrooklynMementoPersisterToObjectStore reader, RebindExceptionHandler exceptionHandler) throws Exception {
    +        BrooklynMementoRawData rawData = reader.loadMementoRawData(exceptionHandler);
    +        return transform(rawData);
    +    }
    +    
    +    public BrooklynMementoRawData transform(BrooklynMementoRawData rawData) throws Exception {
    +        Map<String, String> entities = MutableMap.copyOf(rawData.getEntities());
    +        Map<String, String> locations = MutableMap.copyOf(rawData.getLocations());
    +        Map<String, String> policies = MutableMap.copyOf(rawData.getPolicies());
    +        Map<String, String> enrichers = MutableMap.copyOf(rawData.getEnrichers());
    +
    +        for (BrooklynObjectType type : BrooklynObjectType.values()) {
    +            Collection<RawDataTransformer> transformers = rawDataTransformers.get(type);
    --- End diff --
    
    actually they're all strings here aren't they so it doesn't matter


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---