You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@brooklyn.apache.org by ahgittin <gi...@git.apache.org> on 2014/11/20 13:53:22 UTC

[GitHub] incubator-brooklyn pull request: Feeds etc

GitHub user ahgittin opened a pull request:

    https://github.com/apache/incubator-brooklyn/pull/352

    Feeds etc

    Add support for feed de-duplication, and a couple other goodies -- base64 tools and allow `${config['brooklyn.property']}` to be used in `setup.script.vars`

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/ahgittin/incubator-brooklyn feeds-etc

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/incubator-brooklyn/pull/352.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #352
    
----
commit ae6924543241897943c8d998d57fc3cb6d964442
Author: Alex Heneveld <al...@cloudsoftcorp.com>
Date:   2014-11-19T15:40:01Z

    utils for encoding base64 data strings

commit 92f775f89c93b620c3a1e9e73952ca095620dee0
Author: Alex Heneveld <al...@cloudsoftcorp.com>
Date:   2014-11-19T21:07:57Z

    allow brooklyn.properties to be used in setup.script.vars

commit 2af342266f7263c8415df212abfbb36cee99f8a8
Author: Alex Heneveld <al...@cloudsoftcorp.com>
Date:   2014-11-19T14:40:12Z

    feed de-duplication
    
    prevents problems, especially on rebind, where if the same feed is added multiple times, we get multiple running instances.
    works primarily by using unique tags which are now inferred for feeds.
    these can be explicitly set, if the default de-dupe behaviour is not right,
    but in most cases it will be right, using a combo of the source and the sensor it writes to.
    (since most sensors will only be written to by one source this is more than enough!)
    also adds lots of equals and hashcodes, as this is used secondarily when there is no unique tag
    (but now there always is so much of this is not needed)

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-brooklyn pull request: Feeds etc

Posted by aledsage <gi...@git.apache.org>.
Github user aledsage commented on a diff in the pull request:

    https://github.com/apache/incubator-brooklyn/pull/352#discussion_r20651865
  
    --- Diff: utils/common/src/main/java/brooklyn/util/net/Urls.java ---
    @@ -28,10 +28,13 @@
     
     import javax.annotation.Nullable;
     
    +import org.yaml.snakeyaml.external.biz.base64Coder.Base64Coder;
    --- End diff --
    
    I'd prefer us to use guava utilities where available, unless there's a compelling reason to do otherwise - see http://docs.guava-libraries.googlecode.com/git/javadoc/com/google/common/io/BaseEncoding.html#base64(), or `base64Url()` if that is more appropriate.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-brooklyn pull request: Feeds etc

Posted by neykov <gi...@git.apache.org>.
Github user neykov commented on a diff in the pull request:

    https://github.com/apache/incubator-brooklyn/pull/352#discussion_r20643844
  
    --- Diff: core/src/main/java/brooklyn/event/feed/http/HttpPollConfig.java ---
    @@ -159,9 +160,12 @@ public URI buildUri(URI baseUri, Map<String,String> baseUriVars) {
             return MutableMap.<K,V>builder().putAll(map1).putAll(map2).build();
         }
     
    +    @Override protected String toStringBaseName() { return "http"; }
    +    @Override protected String toStringPollSource() { return suburl; }
         @Override
    -    public String toString() {
    -        return "http["+suburl+"]";
    +    protected MutableList<Object> equalsFields() {
    +        return super.equalsFields().appendIfNotNull(method).appendIfNotNull(vars).appendIfNotNull(headers)
    --- End diff --
    
    Is IfNotNull the right approach? Better to have positional fields even if they are null. Or is it a future-proof approach for new versions?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-brooklyn pull request: Feeds etc

Posted by neykov <gi...@git.apache.org>.
Github user neykov commented on the pull request:

    https://github.com/apache/incubator-brooklyn/pull/352#issuecomment-64203879
  
    With my testcase I still get multiplying feeds, but seems there's no way around it - the value of my supplier is a random value making the heuristic deep comparison useless.
    
    Good to merge.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-brooklyn pull request: Feeds etc

Posted by neykov <gi...@git.apache.org>.
Github user neykov commented on a diff in the pull request:

    https://github.com/apache/incubator-brooklyn/pull/352#discussion_r20790748
  
    --- Diff: core/src/main/java/brooklyn/entity/basic/AbstractEntity.java ---
    @@ -1246,20 +1258,31 @@ public void addEnricher(Enricher enricher) {
                 return null;
             }
         }
    -    private <T extends EntityAdjunct> T findApparentlyEqual(Collection<? extends T> itemsCopy, T newItem) {
    -        // TODO workaround for issue where enrichers can get added multiple times on rebind,
    -        // if it's added in onBecomingManager or connectSensors; the right fix will be more disciplined about how/where these are added
    -        // (easier done when sensor feeds are persisted)
    +    private <T extends EntityAdjunct> T findApparentlyEqual(Collection<? extends T> itemsCopy, T newItem, boolean transferUniqueTag) {
    --- End diff --
    
    `transferUniqueTag` is not used in the body


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-brooklyn pull request: Feeds etc

Posted by neykov <gi...@git.apache.org>.
Github user neykov commented on a diff in the pull request:

    https://github.com/apache/incubator-brooklyn/pull/352#discussion_r20643543
  
    --- Diff: core/src/main/java/brooklyn/event/feed/AbstractFeed.java ---
    @@ -79,7 +83,39 @@ public void setEntity(EntityLocal entity) {
                 ((EntityInternal)entity).feeds().addFeed(this);
             }
         }
    +
    +    protected void initUniqueTag(String uniqueTag, Object ...valsForDefault) {
    +        if (Strings.isNonBlank(uniqueTag)) this.uniqueTag = uniqueTag;
    +        else if (Strings.isBlank(this.uniqueTag)) this.uniqueTag = getDefaultUniqueTag(valsForDefault);
    +    }
    +
    +    @Override
    +    public String getUniqueTag() {
    +        if (Strings.isBlank(uniqueTag)) initUniqueTag(null);
    +        return super.getUniqueTag();
    +    }
         
    +    protected String getDefaultUniqueTag(Object ...valsForDefault) {
    +        StringBuilder sb = new StringBuilder();
    +        sb.append(JavaClassNames.simpleClassName(this));
    +        if (valsForDefault.length==0) {
    +            sb.append("@");
    +            sb.append(hashCode());
    +        } else if (valsForDefault.length==1 && valsForDefault[0] instanceof Collection){
    +            sb.append(Strings.toUniqueString(valsForDefault[0], 80));
    +        } else {
    +            sb.append("[");
    +            boolean first = true;
    +            for (Object x: valsForDefault) {
    --- End diff --
    
    Why reimplement toString for list, you can use Arrays.asList() and go to the above case (nitpick).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-brooklyn pull request: Feeds etc

Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:

    https://github.com/apache/incubator-brooklyn/pull/352


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-brooklyn pull request: Feeds etc

Posted by ahgittin <gi...@git.apache.org>.
Github user ahgittin commented on the pull request:

    https://github.com/apache/incubator-brooklyn/pull/352#issuecomment-63985571
  
    @neykov @aledsage I've addressed the problems.  It looks like `uniqueTag` wasn't being persisted (for policies and enrichers either, but there was some backup logic to derive it by the looks of it).  Better now, I think.  See detailed notes on https://github.com/ahgittin/incubator-brooklyn/commit/ff0abc34b317907f79cf26bb4664da6324bb90ee .


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-brooklyn pull request: Feeds etc

Posted by neykov <gi...@git.apache.org>.
Github user neykov commented on a diff in the pull request:

    https://github.com/apache/incubator-brooklyn/pull/352#discussion_r20643345
  
    --- Diff: core/src/main/java/brooklyn/event/feed/AbstractFeed.java ---
    @@ -79,7 +83,39 @@ public void setEntity(EntityLocal entity) {
                 ((EntityInternal)entity).feeds().addFeed(this);
             }
         }
    +
    +    protected void initUniqueTag(String uniqueTag, Object ...valsForDefault) {
    +        if (Strings.isNonBlank(uniqueTag)) this.uniqueTag = uniqueTag;
    +        else if (Strings.isBlank(this.uniqueTag)) this.uniqueTag = getDefaultUniqueTag(valsForDefault);
    --- End diff --
    
    no need for if isBlank


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-brooklyn pull request: Feeds etc

Posted by neykov <gi...@git.apache.org>.
Github user neykov commented on the pull request:

    https://github.com/apache/incubator-brooklyn/pull/352#issuecomment-63813156
  
    A small correction to the last paragraph - we should warn only if there are existing feeds after a rebind and the new feed doesn't match any  of them.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-brooklyn pull request: Feeds etc

Posted by neykov <gi...@git.apache.org>.
Github user neykov commented on the pull request:

    https://github.com/apache/incubator-brooklyn/pull/352#issuecomment-63812970
  
    The code looks good, but the IBM case still doesn't work. The rebind code doesn't add the items to the feeds collection so the deduplication logic doesn't kick in.
    
    I propose that we warn for items which are added after a rebind, supposedly all feeds are constructed only the first time the entity is run. We could mark the entity as rebinded with a tag to distinguish this case.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-brooklyn pull request: Feeds etc

Posted by ahgittin <gi...@git.apache.org>.
Github user ahgittin commented on a diff in the pull request:

    https://github.com/apache/incubator-brooklyn/pull/352#discussion_r20720594
  
    --- Diff: utils/common/src/main/java/brooklyn/util/net/Urls.java ---
    @@ -28,10 +28,13 @@
     
     import javax.annotation.Nullable;
     
    +import org.yaml.snakeyaml.external.biz.base64Coder.Base64Coder;
    --- End diff --
    
    good idea.  gone with guava's `base64()`, that seems the norm.  didn't know it was there so thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-brooklyn pull request: Feeds etc

Posted by ahgittin <gi...@git.apache.org>.
Github user ahgittin commented on a diff in the pull request:

    https://github.com/apache/incubator-brooklyn/pull/352#discussion_r20646470
  
    --- Diff: core/src/main/java/brooklyn/event/feed/http/HttpPollConfig.java ---
    @@ -159,9 +160,12 @@ public URI buildUri(URI baseUri, Map<String,String> baseUriVars) {
             return MutableMap.<K,V>builder().putAll(map1).putAll(map2).build();
         }
     
    +    @Override protected String toStringBaseName() { return "http"; }
    +    @Override protected String toStringPollSource() { return suburl; }
         @Override
    -    public String toString() {
    -        return "http["+suburl+"]";
    +    protected MutableList<Object> equalsFields() {
    +        return super.equalsFields().appendIfNotNull(method).appendIfNotNull(vars).appendIfNotNull(headers)
    --- End diff --
    
    don't think we need positional parameters, and not sure we filter for `null` on the caller side


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-brooklyn pull request: Feeds etc

Posted by ahgittin <gi...@git.apache.org>.
Github user ahgittin commented on a diff in the pull request:

    https://github.com/apache/incubator-brooklyn/pull/352#discussion_r20646451
  
    --- Diff: core/src/main/java/brooklyn/event/feed/AbstractFeed.java ---
    @@ -79,7 +83,39 @@ public void setEntity(EntityLocal entity) {
                 ((EntityInternal)entity).feeds().addFeed(this);
             }
         }
    +
    +    protected void initUniqueTag(String uniqueTag, Object ...valsForDefault) {
    +        if (Strings.isNonBlank(uniqueTag)) this.uniqueTag = uniqueTag;
    +        else if (Strings.isBlank(this.uniqueTag)) this.uniqueTag = getDefaultUniqueTag(valsForDefault);
    +    }
    +
    +    @Override
    +    public String getUniqueTag() {
    +        if (Strings.isBlank(uniqueTag)) initUniqueTag(null);
    +        return super.getUniqueTag();
    +    }
         
    +    protected String getDefaultUniqueTag(Object ...valsForDefault) {
    +        StringBuilder sb = new StringBuilder();
    +        sb.append(JavaClassNames.simpleClassName(this));
    +        if (valsForDefault.length==0) {
    +            sb.append("@");
    +            sb.append(hashCode());
    +        } else if (valsForDefault.length==1 && valsForDefault[0] instanceof Collection){
    +            sb.append(Strings.toUniqueString(valsForDefault[0], 80));
    +        } else {
    +            sb.append("[");
    +            boolean first = true;
    +            for (Object x: valsForDefault) {
    --- End diff --
    
    If we didn't need to do `Strings.toUniqueString(x, 80)` then yes, but we want that behaviour in case one of the items is a set


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---