You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@metron.apache.org by nickwallen <gi...@git.apache.org> on 2016/08/17 20:20:36 UTC

[GitHub] incubator-metron pull request #215: METRON-377 Enable Profiles that Use Non-...

GitHub user nickwallen opened a pull request:

    https://github.com/apache/incubator-metron/pull/215

    METRON-377 Enable Profiles that Use Non-Single Pass Summary Functions

    Note: This change depends on #208, #212, #213, #214 . The diff will be easier to grok once those PRs are merged.
    
    ### [METRON-377](https://issues.apache.org/jira/browse/METRON-377)
    
    As of METRON-372 and #214 , Profiles can be built using many statistical summaries that only require a single-pass over the data.  This is less memory intensive and more scalable for high volume loads.
    
    Unfortunately, not all functions can be calculated in a single pass.  In particular, the skewness, ketosis and percentiles require all data to be stored in memory for the calculation to occur.  The platform was enhanced so that a user can leverage skewness, ketosis and percentiles. 
    
    ### Changes
    
    The `STATS_INIT` function was enhanced to accept a `window_size`.  This defines the number of input data elements that are maintained in memory.  
    
    If the `window_size` is greater than 0, a rolling window of the most recent `window_size` elements is maintained in memory.  The skewness, ketosis and percentiles are calculated over this rolling window.  The `window_size` must be >0 otherwise these values cannot be calculated.
    
    If a user supplies a `window_size` equal to 0 then the more efficient implementation that does not maintain a rolling window in memory is used.  Of course, in this case, the skewness, ketosis, and percentiles cannot be calculated.
    
    The following additional functions were also added.
    * STATS_KETOSIS
    * STATS_SKEWNESS
    * STATS_PERCENTILES
    
    Another integration test was added to validate this end-to-end also.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/nickwallen/incubator-metron METRON-377

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/incubator-metron/pull/215.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #215
    
----
commit 11ef29bb73c8363b1c905f597be496929b77888f
Author: Nick Allen <ni...@nickallen.org>
Date:   2016-07-29T20:53:49Z

    METRON-309 Create a normalcy profiler

commit 28560b36273271c438c3309df610a4c4306ffdde
Author: Nick Allen <ni...@nickallen.org>
Date:   2016-08-11T20:46:00Z

    METRON-309 Corrected a typo in the 'Getting Started' instructions

commit d55d32efeea38f52bd9d1095aa6f8817bc15f0a5
Author: Nick Allen <ni...@nickallen.org>
Date:   2016-08-12T13:16:19Z

    METRON-309 Altered based on Stellar Unification changes

commit ca81339e0f326dc04110eafd74fd0879dfe1a029
Author: Nick Allen <ni...@nickallen.org>
Date:   2016-08-12T13:23:06Z

    METRON-309 Need to set the kafka broker in the Profiler topology properties

commit 712a1083c21eb8f7c0d81511d432c054792ebe9b
Author: Nick Allen <ni...@nickallen.org>
Date:   2016-08-12T15:17:47Z

    METRON-309 Updated examples to use latest Stellar binary functions

commit 01aa1f972324f97b231b7d9a26f122891a1b6d50
Author: Nick Allen <ni...@nickallen.org>
Date:   2016-08-15T18:03:49Z

    METRON-309 Fixed the README examples and added each as an integration test.

commit d9e2c5e568292c4a08c6c6c314d3c2e8a9ea38be
Author: Nick Allen <ni...@nickallen.org>
Date:   2016-08-15T20:08:26Z

    METRON-367 Enhance Profiler to Support Multiple Numeric Types

commit b9366ff97dbc8968ac44a2ba8dfe7bca43e7adc6
Author: Nick Allen <ni...@nickallen.org>
Date:   2016-08-15T20:32:51Z

    METRON-368 Simplify Profile Configuration with Sensible Defaults

commit 486e8138efbc64342b0bcf405adae521c55d4e33
Author: Nick Allen <ni...@nickallen.org>
Date:   2016-08-16T16:51:07Z

    METRON-309 Removed legacy classes from Stellar Unification that are no longer needed

commit c3ce806606c0903fd62819e4a29ea91adb1979be
Author: Nick Allen <ni...@nickallen.org>
Date:   2016-08-16T16:34:26Z

    METRON-372 Refactored the Stellar functions for clarity

commit eaf5255143d7c09ee8735d3c7d4fa34a58c25831
Author: Nick Allen <ni...@nickallen.org>
Date:   2016-08-16T19:49:13Z

    METRON-372 Added summary statistics functions to Stellar

commit 2e4844cf7e4a98c3594fac542bcf0f32d938b0b2
Author: Nick Allen <ni...@nickallen.org>
Date:   2016-08-16T20:01:13Z

    METRON-372 Updated example to show use of new STATS_x functions

commit 8025994f5858a2e332f4a2ee325ab9cffd82b092
Author: Nick Allen <ni...@nickallen.org>
Date:   2016-08-17T19:39:53Z

    METRON-377 Enable Profiles that Use Non-Single Pass Summary Functions

commit 62a3a0f48d5c792c85486014cfa45e2c4620362c
Author: Nick Allen <ni...@nickallen.org>
Date:   2016-08-17T20:18:24Z

    METRON-377 Added another integration test

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #215: METRON-377 Enable Profiles that Use Non-Single ...

Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:

    https://github.com/apache/incubator-metron/pull/215
  
    Replaces by #226 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #215: METRON-377 Enable Profiles that Use Non-...

Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen closed the pull request at:

    https://github.com/apache/incubator-metron/pull/215


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #215: METRON-377 Enable Profiles that Use Non-Single ...

Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:

    https://github.com/apache/incubator-metron/pull/215
  
    did you mean Kurtosis?  Ketosis is a metabolic process, not a stats function. ;)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #215: METRON-377 Enable Profiles that Use Non-Single ...

Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:

    https://github.com/apache/incubator-metron/pull/215
  
    Ha. Yes.  Fortunately, the code is right.  Just my description in the PR is wrong.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---