You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@metron.apache.org by nickwallen <gi...@git.apache.org> on 2016/12/03 01:21:17 UTC

[GitHub] incubator-metron pull request #387: METRON-606

GitHub user nickwallen opened a pull request:

    https://github.com/apache/incubator-metron/pull/387

    METRON-606

    [METRON-606](https://issues.apache.org/jira/browse/METRON-606)
    
    The Profiler is repetitively overwriting previously written values. Where I would expect to see a series of multiple values in Hbase, one written per flush, there is only a single value. This same value is being repetitively overwritten with new data.
    
    The following changes are included in the PR.
    
    * Refactored the ProfileBuilderBolt so that the logic for building a profile no longer lives in the bolt itself, but in a separate ProfileBuilder.  The bolt simply maintains a cache of ProfileBuilder instances. 
    
    * Ensured that ProfileMeasurement objects are not reused, which reduces the likelihood of another defect like this in the future.
    
    * Added a Clock interface that allows me to advance time in the unit tests.  This was necessary for testing this specific defect.  This will also help with METRON-590 in dealing with processing versus event time.
    
    * Added a specific unit test for this defect.
    
    * Added a unit test for the recent 'tickUpdate' addition to the Profiler.
    
    The change was tested on a live AWS cluster along with the 'Quick Dev' environment.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/nickwallen/incubator-metron METRON-606

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/incubator-metron/pull/387.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #387
    
----
commit 4e9fc6307356e9dcb71ec1d9ed78644b1e7f3eb3
Author: Nick Allen <ni...@nickallen.org>
Date:   2016-11-25T23:24:22Z

    METRON-606 Extracted ProfileBuilder logic from the ProfileBuilderBolt.

commit deaaf4cb5066e1c03f20b45bb62a577724cdf22f
Author: Nick Allen <ni...@nickallen.org>
Date:   2016-12-02T23:07:43Z

    METRON-606 Ensure that a new profile measurement is created on each flush

commit cad4674df2eaf67bcfbc8de3cc8fa74a8f1a2b77
Author: Nick Allen <ni...@nickallen.org>
Date:   2016-12-02T23:34:46Z

    METRON-606 Added a unit test specifically for METRON-606

commit d5bfa09248018ad009ac20ccf4032082f4c06d74
Author: Nick Allen <ni...@nickallen.org>
Date:   2016-12-02T23:53:04Z

    METRON-606 Added missing ASF license headers

commit 8b0d84336503898300cf17ddc8ac78c8c25932d3
Author: Nick Allen <ni...@nickallen.org>
Date:   2016-12-03T01:15:57Z

    METRON-606 Exception message can cause an NPE

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #387: METRON-606 Profiler Overwriting Previously Writ...

Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:

    https://github.com/apache/incubator-metron/pull/387
  
    Ok, I'm in.  +1 by inspection.  Great job tracking it down, @nickwallen 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #387: METRON-606 Profiler Overwriting Previously Writ...

Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:

    https://github.com/apache/incubator-metron/pull/387
  
    Did this get tested in vagrant?  
    Specifically, did we try the following things:
    * A global profile
    * A non-global profile
    * 2 profiles that share the same variable names in their update logic



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #387: METRON-606 Profiler Overwriting Previously Writ...

Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:

    https://github.com/apache/incubator-metron/pull/387
  
    I had hit the first two, but not the last.  I just setup an additional manual test that covers all of these scenarios.  More specifically, here are the profiles that I defined.  
    
    ```
    {
      "profiles": [
        {
          "profile": "counter",
          "foreach": "'global'",
          "init":  { "s": 0 },
          "update":  { "s": "s + 1" },
          "result":  "s"
        },
        {
          "profile": "by-ip",
          "foreach": "ip_src_addr",
          "init":  { "s": 0 },
          "update":  { "s": "s + 1" },
          "result":  "s"
        },
        {
          "profile": "always-zero",
          "foreach": "ip_src_addr",
          "init":  { "s": 0 },
          "update":  { "s": 0 },
          "result":  "s"
        }
      ]
    }
    ```
    
    One of the profiles counts messages that it sees "globally", the next counts the number of messages by source IP, and the last likes to always zero-out the same variable named "s", just to see if it can muck with the other two profiles.  
    
    The `always-zero` profile as expected remains 0.  I can see additional values being written as time progresses.
    ```
    [Stellar]>>> PROFILE_GET("always-zero","72.34.49.86", 5, "MINUTES")
    [0, 0]
    [Stellar]>>> PROFILE_GET("always-zero","72.34.49.86", 5, "MINUTES")
    [0, 0, 0]
    ```
    
    All of these profiles share the same variable 's', so if there was some kind of bad intermingling, I would expect the `always-zero` profile to seriously muck things up for the other two profiles.  But I do not see any of that.  
    
    I can see new values being added to the `by-ip` profile.
    ```
    [Stellar]>>> PROFILE_GET("by-ip","72.34.49.86", 5, "MINUTES")
    [10.0, 106.0]
    [Stellar]>>> PROFILE_GET("by-ip","72.34.49.86", 5, "MINUTES")
    [10.0, 106.0, 58.0]
    
    [Stellar]>>> PROFILE_GET("by-ip","192.168.138.158", 5, "MINUTES")
    [296.0, 303.0, 302.0]
    [Stellar]>>> PROFILE_GET("by-ip","192.168.138.158", 7, "MINUTES")
    [296.0, 303.0, 302.0, 303.0]
    ```
    
    The "global" profile is also behaving properly.
    ```
    [Stellar]>>> PROFILE_GET("counter","global", 5, "MINUTES")
    [833.0, 319.0, 746.0]
    [Stellar]>>> PROFILE_GET("counter","global", 6, "MINUTES")
    [833.0, 319.0, 746.0, 1538.0]
    ```



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #387: METRON-606 Profiler Overwriting Previous...

Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:

    https://github.com/apache/incubator-metron/pull/387


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---