You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@metron.apache.org by nickwallen <gi...@git.apache.org> on 2016/08/16 20:32:43 UTC
[GitHub] incubator-metron pull request #214: METRON-372 Enhance Statistical Operation...
GitHub user nickwallen opened a pull request:
https://github.com/apache/incubator-metron/pull/214
METRON-372 Enhance Statistical Operations Available for Use with the Profiler
### [METRON-372](https://issues.apache.org/jira/browse/METRON-372)
Only basic math functions are currently available in Stellar for use with the Profiler. This makes life difficult for users to create even basic profiles like a running average.
This can be seen in the following example where the average must be calculated manually in Stellar. A variable `sum` and `cnt` must be maintained and then used to calculate the average.
```
{
"profile": "example3",
"foreach": "ip_src_addr",
"onlyif": "protocol == 'HTTP'",
"init": {
"sum": 0.0,
"cnt": 0.0
},
"update": {
"sum": "sum + resp_body_len",
"cnt": "cnt + 1"
},
"result": "sum / cnt"
}
```
This change introduces a series of summary functions that make creating profiles much simpler for the user. Instead of re-implementing the calculation of an average in Stellar, this leverages Commons Math to perform all the heavy lifting.
The example above for calculating an average can be re-defined as follows.
```
{
"profile": "example3",
"foreach": "ip_src_addr",
"onlyif": "protocol == 'HTTP'",
"init": { "s": "STATS_INIT(0)" },
"update": { "_": "STATS_ADD(length, s)" },
"result": "STATS_MEAN(s)"
}
```
The following summary functions are supported. These are all statistics that can be calculated in a single pass. This means that none of the values being summarized are stored in memory.
* count
* mean
* geometric mean
* max
* min
* sum
* population variance
* variance
* second moment
* quadratic mean
* standard deviation
* sum of logs
* sum of squares.
Note: This change depends on METRON-309.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/nickwallen/incubator-metron METRON-372
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-metron/pull/214.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #214
----
commit 11ef29bb73c8363b1c905f597be496929b77888f
Author: Nick Allen <ni...@nickallen.org>
Date: 2016-07-29T20:53:49Z
METRON-309 Create a normalcy profiler
commit 28560b36273271c438c3309df610a4c4306ffdde
Author: Nick Allen <ni...@nickallen.org>
Date: 2016-08-11T20:46:00Z
METRON-309 Corrected a typo in the 'Getting Started' instructions
commit d55d32efeea38f52bd9d1095aa6f8817bc15f0a5
Author: Nick Allen <ni...@nickallen.org>
Date: 2016-08-12T13:16:19Z
METRON-309 Altered based on Stellar Unification changes
commit ca81339e0f326dc04110eafd74fd0879dfe1a029
Author: Nick Allen <ni...@nickallen.org>
Date: 2016-08-12T13:23:06Z
METRON-309 Need to set the kafka broker in the Profiler topology properties
commit 712a1083c21eb8f7c0d81511d432c054792ebe9b
Author: Nick Allen <ni...@nickallen.org>
Date: 2016-08-12T15:17:47Z
METRON-309 Updated examples to use latest Stellar binary functions
commit 01aa1f972324f97b231b7d9a26f122891a1b6d50
Author: Nick Allen <ni...@nickallen.org>
Date: 2016-08-15T18:03:49Z
METRON-309 Fixed the README examples and added each as an integration test.
commit d9e2c5e568292c4a08c6c6c314d3c2e8a9ea38be
Author: Nick Allen <ni...@nickallen.org>
Date: 2016-08-15T20:08:26Z
METRON-367 Enhance Profiler to Support Multiple Numeric Types
commit b9366ff97dbc8968ac44a2ba8dfe7bca43e7adc6
Author: Nick Allen <ni...@nickallen.org>
Date: 2016-08-15T20:32:51Z
METRON-368 Simplify Profile Configuration with Sensible Defaults
commit 486e8138efbc64342b0bcf405adae521c55d4e33
Author: Nick Allen <ni...@nickallen.org>
Date: 2016-08-16T16:51:07Z
METRON-309 Removed legacy classes from Stellar Unification that are no longer needed
commit c3ce806606c0903fd62819e4a29ea91adb1979be
Author: Nick Allen <ni...@nickallen.org>
Date: 2016-08-16T16:34:26Z
METRON-372 Refactored the Stellar functions for clarity
commit eaf5255143d7c09ee8735d3c7d4fa34a58c25831
Author: Nick Allen <ni...@nickallen.org>
Date: 2016-08-16T19:49:13Z
METRON-372 Added summary statistics functions to Stellar
commit 06118874b28deaf62fb4b454672c6bce21ea39ee
Author: Nick Allen <ni...@nickallen.org>
Date: 2016-08-16T20:01:13Z
METRON-372 Updated example to show use of new STATS_x functions
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #214: METRON-372 Enhance Statistical Operation...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen closed the pull request at:
https://github.com/apache/incubator-metron/pull/214
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #214: METRON-372 Enhance Statistical Operations Avail...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:
https://github.com/apache/incubator-metron/pull/214
Closing in favor of #225
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---