You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@metron.apache.org by cestella <gi...@git.apache.org> on 2016/11/10 20:27:34 UTC
[GitHub] incubator-metron pull request #352: METRON-562: Add rudimentary statistical ...
GitHub user cestella opened a pull request:
https://github.com/apache/incubator-metron/pull/352
METRON-562: Add rudimentary statistical outlier detection
With the advent of the profiler, we can now capture state. Furthermore, with Stellar, we can capture statistical summaries. We should provide rudimentary outlier detection functionality in the form of Stellar functions that can operate on captured state from the profiler.
To begin, we should enable simple outlier tests using distance from a central measure such as Median Absolute Deviation (see http://www.itl.nist.gov/div898/handbook/eda/section3/eda35h.htm).
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/cestella/incubator-metron outlier_models_mad
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-metron/pull/352.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #352
----
commit 754c4b46fbd92b5b3ecea24324b1ca7e36834ab6
Author: cstella <ce...@gmail.com>
Date: 2016-11-10T16:16:29Z
Initial cut.
commit 9f39aedec3da8bc994a39b5e9585da2d012bc07b
Author: cstella <ce...@gmail.com>
Date: 2016-11-10T20:20:48Z
Added rudimentary statistical outlier detection.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #352: METRON-562: Add rudimentary statistical ...
Posted by cestella <gi...@git.apache.org>.
Github user cestella closed the pull request at:
https://github.com/apache/incubator-metron/pull/352
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #352: METRON-562: Add rudimentary statistical ...
Posted by cestella <gi...@git.apache.org>.
GitHub user cestella reopened a pull request:
https://github.com/apache/incubator-metron/pull/352
METRON-562: Add rudimentary statistical outlier detection
With the advent of the profiler, we can now capture state. Furthermore, with Stellar, we can capture statistical summaries. We should provide rudimentary outlier detection functionality in the form of Stellar functions that can operate on captured state from the profiler.
To begin, we should enable simple outlier tests using distance from a central measure such as Median Absolute Deviation (see http://www.itl.nist.gov/div898/handbook/eda/section3/eda35h.htm).
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/cestella/incubator-metron outlier_models_mad
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-metron/pull/352.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #352
----
commit 754c4b46fbd92b5b3ecea24324b1ca7e36834ab6
Author: cstella <ce...@gmail.com>
Date: 2016-11-10T16:16:29Z
Initial cut.
commit 9f39aedec3da8bc994a39b5e9585da2d012bc07b
Author: cstella <ce...@gmail.com>
Date: 2016-11-10T20:20:48Z
Added rudimentary statistical outlier detection.
commit a06b1e0c3bb95557427ed52ab7614d49d3c5a61e
Author: cstella <ce...@gmail.com>
Date: 2016-11-11T19:59:06Z
Fixed tests, I think.
commit a20b7731124860dd916a2f3fd1b235fa6cf0bf50
Author: cstella <ce...@gmail.com>
Date: 2016-11-11T20:43:22Z
Update to properly update on the tick.
commit 4b22e47c3cf79b9f360df427feb6dfc5919fca1c
Author: cstella <ce...@gmail.com>
Date: 2016-11-11T21:29:52Z
forgot guava.
commit 7ad9ad76dd9d2ef6607abed400d2e8852989984f
Author: cstella <ce...@gmail.com>
Date: 2016-11-11T23:18:13Z
Fixed profile bolt issue.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #352: METRON-562: Add rudimentary statistical ...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-metron/pull/352
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #352: METRON-562: Add rudimentary statistical ...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/352#discussion_r87657821
--- Diff: metron-analytics/metron-profiler/src/main/java/org/apache/metron/profiler/bolt/ProfileBuilderBolt.java ---
@@ -246,6 +247,8 @@ private void flush(Tuple tickTuple) {
// clear the execution state to prepare for the next window
executor.clearState();
+ profileConfig.getTickUpdate().forEach((var, expr) -> executor.assign(var, expr, new HashMap<>()));
--- End diff --
I do not need to access the message, but rather just the internal state. I have made a modification which does what I think it should do with some comments. If you see issues, I'd love it if you could let me have it. :)
Taking a step back, the purpose of this is in situations where we need to do things like capture a trailing window of state. In the instance of Median Absolute Deviation it is required because we need not just the distribution of the values, but also the distribution of the DEVIATION of the value from the median (which is the median of the values of the trailing window). That window can be updated upon tick. So, for instance, if we want 3 hours lookback, at hour 4, we'd have hours 1, 2 and 3. At hour 5, we'd want hours 2, 3, and 4 and so on. Constructing this trailing window lead me to this. I could have captured this state internally in memory, but that had the issue that internal state for the window would be removed for the entire trailing window upon worker crash/restart.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #352: METRON-562: Add rudimentary statistical ...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/352#discussion_r87668638
--- Diff: metron-analytics/metron-profiler/src/main/java/org/apache/metron/profiler/bolt/ProfileBuilderBolt.java ---
@@ -243,8 +244,18 @@ private void flush(Tuple tickTuple) {
measurement.setValue(result);
emit(measurement, tickTuple);
+ // Execute the update with the old state
+ Map<String, String> tickUpdate = profileConfig.getTickUpdate();
+ Map<String, Object> state = executor.getState();
+ if(tickUpdate != null) {
+ tickUpdate.forEach((var, expr) -> executor.assign(var, expr, executor.getState()));
--- End diff --
The 3rd argument to executor.assign is called 'transientState'. It exists only to make additional state available to the expression; additional state that is not already maintained by the executor itself. The normal use for it is when you want to make the fields in a telemetry message available to the expression.
Setting transientState=executor.getState() does not make sense here. It is exactly the same as passing nothing at all. The state in executor.getState() is already exposed to the expression.
Hopefully, I am describing this well enough. As another feeble attempt at explanation, the following two expression are equivalent.
```
tickUpdate.forEach((var, expr) -> executor.assign(var, expr, executor.getState()));
```
```
tickUpdate.forEach((var, expr) -> executor.assign(var, expr, null));
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #352: METRON-562: Add rudimentary statistical outlier...
Posted by james-sirota <gi...@git.apache.org>.
Github user james-sirota commented on the issue:
https://github.com/apache/incubator-metron/pull/352
+ 1. i got it to work. one point of feedback, though. a lot of telemetry data distributions tend to be heavy in the tails. not all are normally distributed. i would augment your Gaussian test data to also include something from a heavy-tailed distribution of your choice..... https://docs.scipy.org/doc/numpy/reference/generated/numpy.random.standard_t.html maybe?
I know your method will work with both because I have experience using it in production with real data. but i think the tests should reflect it as well.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #352: METRON-562: Add rudimentary statistical ...
Posted by james-sirota <gi...@git.apache.org>.
Github user james-sirota commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/352#discussion_r88757736
--- Diff: metron-analytics/metron-statistics/src/main/java/org/apache/metron/statistics/outlier/MedianAbsoluteDeviationFunctions.java ---
@@ -0,0 +1,193 @@
+/*
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+package org.apache.metron.statistics.outlier;
+
+import org.apache.metron.common.dsl.Context;
+import org.apache.metron.common.dsl.ParseException;
+import org.apache.metron.common.dsl.Stellar;
+import org.apache.metron.common.dsl.StellarFunction;
+import org.apache.metron.statistics.OnlineStatisticsProvider;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Optional;
+
+public class MedianAbsoluteDeviationFunctions {
+ public static class State {
+ OnlineStatisticsProvider tickMedianProvider;
+ OnlineStatisticsProvider tickMADProvider;
+ OnlineStatisticsProvider windowMedianProvider;
+ OnlineStatisticsProvider windowMADProvider;
+
+ public State() {
+ tickMedianProvider = new OnlineStatisticsProvider();
+ tickMADProvider = new OnlineStatisticsProvider();
+ windowMedianProvider = new OnlineStatisticsProvider();
+ windowMADProvider = new OnlineStatisticsProvider();
+ }
+
+ public State(Optional<List<State>> previousStates, Optional<State> currentState)
+ {
+ tickMedianProvider = new OnlineStatisticsProvider();
+ tickMADProvider = new OnlineStatisticsProvider();
+ windowMedianProvider = currentState.isPresent()?currentState.get().tickMedianProvider:new OnlineStatisticsProvider();
+ windowMADProvider = currentState.isPresent()?currentState.get().tickMADProvider:new OnlineStatisticsProvider();
+ for(State s : previousStates.orElse(new ArrayList<>())) {
+ windowMedianProvider = (OnlineStatisticsProvider) windowMedianProvider.merge(s.tickMedianProvider);
+ windowMADProvider = (OnlineStatisticsProvider) windowMADProvider.merge(s.tickMADProvider);
+ }
+ }
+
+ public void add(Double d) {
+ if(!Double.isNaN(d)) {
+ tickMedianProvider.addValue(d);
+ double deviation = Math.abs(d - windowMedianProvider.getPercentile(50));
+ windowMedianProvider.addValue(d);
+ if(!Double.isNaN(deviation)) {
+ windowMADProvider.addValue(deviation);
+ tickMADProvider.addValue(deviation);
+ }
+ }
+ }
+ }
+
+ @Stellar(namespace="OUTLIER"
+ ,name="MAD_STATE_MERGE"
+ ,description="Update the statistical state required to compute the Median Absolute Deviation"
+ ,params= {
+ "[state] - A list of Median Absolute Deviation States to merge. Generally these are states across time."
+ ,"currentState? - The current state (optional)"
+ }
+ ,returns="The Median Absolute Deviation state."
+ )
+ public static class StateUpdate implements StellarFunction{
+
+ @Override
+ public Object apply(List<Object> args, Context context) throws ParseException {
+ State state = null;
+ List<State> states = (List<State>) args.get(0);
+ State currentState = null;
+ if(args.size() > 1) {
+ currentState = (State) args.get(1);
+ }
+ state = new State(Optional.ofNullable(states), Optional.ofNullable(currentState));
+ return state;
+ }
+
+ @Override
+ public void initialize(Context context) {
+
+ }
+
+ @Override
+ public boolean isInitialized() {
+ return true;
+ }
+ }
+
+ @Stellar(namespace="OUTLIER"
+ ,name="MAD_ADD"
+ ,params= {
+ "state - The MAD state"
+ , "value - The numeric value to add"
+ }
+ ,description="Add a piece of data to the state."
+ ,returns="The MAD state."
+ )
+ public static class PointUpdate implements StellarFunction{
+
+ @Override
+ public Object apply(List<Object> args, Context context) throws ParseException {
+ State state = (State) args.get(0);
+ Object o = args.get(1);
+ List<Double> data = new ArrayList<>();
+ if(o != null) {
+ if (o instanceof List) {
+ for (Object datum : (List<Object>) o) {
+ Number n = (Number) datum;
+ data.add(n.doubleValue());
+ }
+ } else {
+ Number n = (Number)o;
+ data.add(n.doubleValue());
+ }
+ }
+ if(state != null) {
+ for(Double d : data) {
+ state.add(d);
+ }
+ }
+ return state;
+ }
+
+ @Override
+ public void initialize(Context context) {
+
+ }
+
+ @Override
+ public boolean isInitialized() {
+ return true;
+ }
+ }
+
+ @Stellar(namespace="OUTLIER"
+ ,name="MAD_SCORE"
+ ,params = {
+ "state - The MAD state"
+ ,"value - The value to score"
+ ,"scale? - Optionally the scale to use when computing the modified z-score. Default is 0.6745, see the first page of http://web.ipac.caltech.edu/staff/fmasci/home/astro_refs/BetterThanMAD.pdf"
+ }
+ ,description="Get the modified z-score normalized by the MAD: scale * | x_i - median(X) | / MAD. See the first page of http://web.ipac.caltech.edu/staff/fmasci/home/astro_refs/BetterThanMAD.pdf"
+ ,returns="The modified z-score."
+ )
+ public static class Score implements StellarFunction{
+
+ @Override
+ public Object apply(List<Object> args, Context context) throws ParseException {
+ double scale = 0.6745;
--- End diff --
this is 1/b from the caltech paper
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #352: METRON-562: Add rudimentary statistical ...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/352#discussion_r87670483
--- Diff: metron-analytics/metron-profiler/src/main/java/org/apache/metron/profiler/bolt/ProfileBuilderBolt.java ---
@@ -243,8 +244,18 @@ private void flush(Tuple tickTuple) {
measurement.setValue(result);
emit(measurement, tickTuple);
+ // Execute the update with the old state
+ Map<String, String> tickUpdate = profileConfig.getTickUpdate();
+ Map<String, Object> state = executor.getState();
+ if(tickUpdate != null) {
+ tickUpdate.forEach((var, expr) -> executor.assign(var, expr, executor.getState()));
--- End diff --
This is what you had in a prior iteration that does what you are describing.
```
profileConfig.getTickUpdate().forEach((var, expr) -> executor.assign(var, expr, new HashMap<>()));
```
The only change that i suggested previously was to make the `result` available to the `tick` expressions, which I think could be generally useful.
```
profileConfig.getTickUpdate().forEach((var, expr) -> executor.assign(var, expr, Collections.singletonMap("result", result));
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #352: METRON-562: Add rudimentary statistical ...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/352#discussion_r88905131
--- Diff: metron-analytics/metron-statistics/src/main/java/org/apache/metron/statistics/outlier/MedianAbsoluteDeviationFunctions.java ---
@@ -0,0 +1,193 @@
+/*
--- End diff --
Agreed; given the structure, I think the profiler client itself should have a LRU cache that `PROFILE_GET` should pull from before hitting HBase. I'll make the JIRA.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #352: METRON-562: Add rudimentary statistical ...
Posted by cestella <gi...@git.apache.org>.
Github user cestella closed the pull request at:
https://github.com/apache/incubator-metron/pull/352
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #352: METRON-562: Add rudimentary statistical ...
Posted by james-sirota <gi...@git.apache.org>.
Github user james-sirota commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/352#discussion_r88757617
--- Diff: metron-analytics/metron-statistics/README.md ---
@@ -0,0 +1,346 @@
+# Statistics and Mathematical Functions
+
+A variety of non-trivial and advanced analytics make use of statistics
+and advanced mathematical functions. Particular, capturing the
+statistical snapshots in a scalable way can open up doors for more
+advanced analytics such as outlier analysis. As such, this project is
+aimed at capturing a robust set of statistical functions and
+statistical-based algorithms in the form of Stellar functions. These
+functions can be used from everywhere where Stellar is used.
+
+##Stellar Functions
+
+### Mathematical Functions
+* `ABS`
+ * Description: Returns the absolute value of a number.
+ * Input:
+ * number - The number to take the absolute value of
+ * Returns: The absolute value of the number passed in.
+
+
+### Distributional Statistics
+
+* `STATS_ADD`
+ * Description: Adds one or more input values to those that are used to calculate the summary statistics.
+ * Input:
+ * stats - The Stellar statistics object. If null, then a new one is initialized.
+ * value+ - One or more numbers to add
+ * Returns: A Stellar statistics object
+* `STATS_COUNT`
+ * Description: Calculates the count of the values accumulated (or in the window if a window is used).
+ * Input:
+ * stats - The Stellar statistics object
+ * Returns: The count of the values in the window or NaN if the statistics object is null.
+* `STATS_GEOMETRIC_MEAN`
+ * Description: Calculates the geometric mean of the accumulated values (or in the window if a window is used). See http://commons.apache.org/proper/commons-math/userguide/stat.html#a1.2_Descriptive_statistics
+ * Input:
+ * stats - The Stellar statistics object
+ * Returns: The geometric mean of the values in the window or NaN if the statistics object is null.
+* `STATS_INIT`
+ * Description: Initializes a statistics object
+ * Input:
+ * window_size - The number of input data values to maintain in a rolling window in memory. If window_size is equal to 0, then no rolling window is maintained. Using no rolling window is less memory intensive, but cannot calculate certain statistics like percentiles and kurtosis.
+ * Returns: A Stellar statistics object
+* `STATS_KURTOSIS`
+ * Description: Calculates the kurtosis of the accumulated values (or in the window if a window is used). See http://commons.apache.org/proper/commons-math/userguide/stat.html#a1.2_Descriptive_statistics
+ * Input:
+ * stats - The Stellar statistics object
+ * Returns: The kurtosis of the values in the window or NaN if the statistics object is null.
+* `STATS_MAX`
+ * Description: Calculates the maximum of the accumulated values (or in the window if a window is used).
+ * Input:
+ * stats - The Stellar statistics object
+ * Returns: The maximum of the accumulated values in the window or NaN if the statistics object is null.
+* `STATS_MEAN`
+ * Description: Calculates the mean of the accumulated values (or in the window if a window is used).
+ * Input:
+ * stats - The Stellar statistics object
+ * Returns: The mean of the values in the window or NaN if the statistics object is null.
+* `STATS_MERGE`
+ * Description: Merges statistics objects.
+ * Input:
+ * statistics - A list of statistics objects
+ * Returns: A Stellar statistics object
+* `STATS_MIN`
+ * Description: Calculates the minimum of the accumulated values (or in the window if a window is used).
+ * Input:
+ * stats - The Stellar statistics object
+ * Returns: The minimum of the accumulated values in the window or NaN if the statistics object is null.
+* `STATS_PERCENTILE`
+ * Description: Computes the p'th percentile of the accumulated values (or in the window if a window is used).
+ * Input:
+ * stats - The Stellar statistics object
+ * p - a double where 0 <= p < 1 representing the percentile
+ * Returns: The p'th percentile of the data or NaN if the statistics object is null
+* `STATS_POPULATION_VARIANCE`
+ * Description: Calculates the population variance of the accumulated values (or in the window if a window is used). See http://commons.apache.org/proper/commons-math/userguide/stat.html#a1.2_Descriptive_statistics
+ * Input:
+ * stats - The Stellar statistics object
+ * Returns: The population variance of the values in the window or NaN if the statistics object is null.
+* `STATS_QUADRATIC_MEAN`
+ * Description: Calculates the quadratic mean of the accumulated values (or in the window if a window is used). See http://commons.apache.org/proper/commons-math/userguide/stat.html#a1.2_Descriptive_statistics
+ * Input:
+ * stats - The Stellar statistics object
+ * Returns: The quadratic mean of the values in the window or NaN if the statistics object is null.
+* `STATS_SD`
+ * Description: Calculates the standard deviation of the accumulated values (or in the window if a window is used). See http://commons.apache.org/proper/commons-math/userguide/stat.html#a1.2_Descriptive_statistics
+ * Input:
+ * stats - The Stellar statistics object
+ * Returns: The standard deviation of the values in the window or NaN if the statistics object is null.
+* `STATS_SKEWNESS`
+ * Description: Calculates the skewness of the accumulated values (or in the window if a window is used). See http://commons.apache.org/proper/commons-math/userguide/stat.html#a1.2_Descriptive_statistics
+ * Input:
+ * stats - The Stellar statistics object
+ * Returns: The skewness of the values in the window or NaN if the statistics object is null.
+* `STATS_SUM`
+ * Description: Calculates the sum of the accumulated values (or in the window if a window is used).
+ * Input:
+ * stats - The Stellar statistics object
+ * Returns: The sum of the values in the window or NaN if the statistics object is null.
+* `STATS_SUM_LOGS`
+ * Description: Calculates the sum of the (natural) log of the accumulated values (or in the window if a window is used). See http://commons.apache.org/proper/commons-math/userguide/stat.html#a1.2_Descriptive_statistics
+ * Input:
+ * stats - The Stellar statistics object
+ * Returns: The sum of the (natural) log of the values in the window or NaN if the statistics object is null.
+* `STATS_SUM_SQUARES`
+ * Description: Calculates the sum of the squares of the accumulated values (or in the window if a window is used).
+ * Input:
+ * stats - The Stellar statistics object
+ * Returns: The sum of the squares of the values in the window or NaN if the statistics object is null.
+* `STATS_VARIANCE`
+ * Description: Calculates the variance of the accumulated values (or in the window if a window is used). See http://commons.apache.org/proper/commons-math/userguide/stat.html#a1.2_Descriptive_statistics
+ * Input:
+ * stats - The Stellar statistics object
+ * Returns: The variance of the values in the window or NaN if the statistics object is null.
+
+### Statistical Outlier Detection
+
+* `OUTLIER_MAD_STATE_MERGE`
+ * Description: Update the statistical state required to compute the Median Absolute Deviation.
+ * Input:
+ * [state] - A list of Median Absolute Deviation States to merge. Generally these are states across time.
+ * currentState? - The current state (optional)
+ * Returns: The Median Absolute Deviation state
+* `OUTLIER_MAD_ADD`
+ * Description: Add a piece of data to the state.
+ * Input:
+ * state - The MAD state
+ * value - The numeric value to add
+ * Returns: The MAD state
+* `OUTLIER_MAD_SCORE`
+ * Description: Get the modified z-score normalized by the MAD: scale * | x_i - median(X) | / MAD. See the first page of http://web.ipac.caltech.edu/staff/fmasci/home/astro_refs/BetterThanMAD.pdf
+ * Input:
+ * state - The MAD state
+ * value - The numeric value to score
+ * scale? - Optionally the scale to use when computing the modified z-score. Default is `0.6745`, see the first page of http://web.ipac.caltech.edu/staff/fmasci/home/astro_refs/BetterThanMAD.pdf
+ * Returns: The modified z-score
+
+# Outlier Analysis
+
+A common desire is to find anomalies in numerical data. To that end,
+we have some simple statistical anomaly detectors.
+
+## Median Absolute Deviation
+
+Much has been written about this robust estimator. See the first page
+of http://web.ipac.caltech.edu/staff/fmasci/home/astro_refs/BetterThanMAD.pdf
+for a good coverage of the good and the bad of MAD. The usage, however
+is fairly straightforward:
+* Gather the statistical state required to compute the MAD
+ * The distribution of the values of a univariate random variable over time.
+ * The distribution of the absolute deviations of the values from the median.
+* Use this statistical state to score unseen values. The higher the score, the more unlike the previously seen data the value is.
+
+There are a couple of issues which make MAD a bit hard to compute.
+First, the statistical state requires computing median, which can be
+computationally expensive to compute exactly. To get around this, we
+use the OnlineStatisticalProvider to compute a sketch rather than the
+exact median. Secondly, the statistical state for seasonal data should
+be limited to a fixed, trailing window. We do this by ensuring that the
+MAD state is mergeable and able to be queried from within the Profiler.
+
+### Example
+
+We will create a dummy data stream of gaussian noise to illustrate how
+to use the MAD functionality along with the profiler to tag messages as
+outliers or not.
+
+To do this, we will create a
+* data generator
+* parser
+* profiler profile
+* enrichment and threat triage
+
+#### Data Generator
+
+We can create a simple python script to generate a stream of gaussian
+noise at the frequency of one message per second as a python script
+which should be saved at `~/rand_gen.py`:
+```
+#!/usr/bin/python
+import random
+import sys
+import time
+def main():
+ mu = float(sys.argv[1])
+ sigma = float(sys.argv[2])
+ freq_s = int(sys.argv[3])
+ while True:
+ print str(random.gauss(mu, sigma))
+ sys.stdout.flush()
+ time.sleep(freq_s)
+
+if __name__ == '__main__':
+ main()
+```
+
+This script will take the following as arguments:
+* The mean of the data generated
+* The standard deviation of the data generated
+* The frequency (in seconds) of the data generated
+
+#### The Parser
+
+We will create a parser that will take the single numbers in and create
+a message with a field called `value` in them using the `CSVParser`.
+
+Add the following file to
+`$METRON_HOME/config/zookeeper/parsers/mad.json`:
+```
+{
+ "parserClassName" : "org.apache.metron.parsers.csv.CSVParser"
+ ,"sensorTopic" : "mad"
+ ,"parserConfig" : {
+ "columns" : {
+ "value_str" : 0
+ }
+ }
+ ,"fieldTransformations" : [
+ {
+ "transformation" : "STELLAR"
+ ,"output" : [ "value" ]
+ ,"config" : {
+ "value" : "TO_DOUBLE(value_str)"
+ }
+ }
+ ]
+}
+```
+
+#### Enrichment and Threat Intel
+
+We will set a threat triage level of `10` if a message generates a outlier score of more than 3.5.
+This cutoff will depend on your data and should be adjusted based on the
+assumed underlying distribution. Note that under the assumptions of
+normality, MAD will act as a robust estimator of the standard deviation, so the cutoff
+should be considered the number of standard deviations away. For other
+distributions, there are other interpretations which will make sense in
+the context of measuring the "degree different". See
+http://eurekastatistics.com/using-the-median-absolute-deviation-to-find-outliers/
+for a brief discussion of this.
+
+Create the following in
+`$METRON_HOME/config/zookeeper/enrichments/mad.json`:
+
+```
+{
+ "index": "mad",
+ "batchSize": 1,
+ "enrichment": {
+ "fieldMap": {
+ "stellar" : {
+ "config" : {
+ "parser_score" : "OUTLIER_MAD_SCORE(OUTLIER_MAD_STATE_MERGE(
+PROFILE_GET( 'sketchy_mad', 'global', 10, 'MINUTES') ), value)"
+ ,"is_alert" : "if parser_score > 3.5 then true else is_alert"
+ }
+ }
+ }
+ ,"fieldToTypeMap": { }
+ },
+ "threatIntel": {
+ "fieldMap": { },
+ "fieldToTypeMap": { },
+ "triageConfig" : {
+ "riskLevelRules" : {
+ "parser_score > 3.5" : 10
--- End diff --
Slick integration of model and triage function
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #352: METRON-562: Add rudimentary statistical outlier...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/352
Note: I'm still in the process of testing this on vagrant and I owe a README.md with an example.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #352: METRON-562: Add rudimentary statistical ...
Posted by james-sirota <gi...@git.apache.org>.
Github user james-sirota commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/352#discussion_r88757519
--- Diff: metron-analytics/metron-statistics/README.md ---
@@ -0,0 +1,346 @@
+# Statistics and Mathematical Functions
+
+A variety of non-trivial and advanced analytics make use of statistics
+and advanced mathematical functions. Particular, capturing the
+statistical snapshots in a scalable way can open up doors for more
+advanced analytics such as outlier analysis. As such, this project is
+aimed at capturing a robust set of statistical functions and
+statistical-based algorithms in the form of Stellar functions. These
+functions can be used from everywhere where Stellar is used.
+
+##Stellar Functions
+
+### Mathematical Functions
+* `ABS`
+ * Description: Returns the absolute value of a number.
+ * Input:
+ * number - The number to take the absolute value of
+ * Returns: The absolute value of the number passed in.
+
+
+### Distributional Statistics
+
+* `STATS_ADD`
+ * Description: Adds one or more input values to those that are used to calculate the summary statistics.
+ * Input:
+ * stats - The Stellar statistics object. If null, then a new one is initialized.
+ * value+ - One or more numbers to add
+ * Returns: A Stellar statistics object
+* `STATS_COUNT`
+ * Description: Calculates the count of the values accumulated (or in the window if a window is used).
+ * Input:
+ * stats - The Stellar statistics object
+ * Returns: The count of the values in the window or NaN if the statistics object is null.
+* `STATS_GEOMETRIC_MEAN`
+ * Description: Calculates the geometric mean of the accumulated values (or in the window if a window is used). See http://commons.apache.org/proper/commons-math/userguide/stat.html#a1.2_Descriptive_statistics
+ * Input:
+ * stats - The Stellar statistics object
+ * Returns: The geometric mean of the values in the window or NaN if the statistics object is null.
+* `STATS_INIT`
+ * Description: Initializes a statistics object
+ * Input:
+ * window_size - The number of input data values to maintain in a rolling window in memory. If window_size is equal to 0, then no rolling window is maintained. Using no rolling window is less memory intensive, but cannot calculate certain statistics like percentiles and kurtosis.
+ * Returns: A Stellar statistics object
+* `STATS_KURTOSIS`
+ * Description: Calculates the kurtosis of the accumulated values (or in the window if a window is used). See http://commons.apache.org/proper/commons-math/userguide/stat.html#a1.2_Descriptive_statistics
+ * Input:
+ * stats - The Stellar statistics object
+ * Returns: The kurtosis of the values in the window or NaN if the statistics object is null.
+* `STATS_MAX`
+ * Description: Calculates the maximum of the accumulated values (or in the window if a window is used).
+ * Input:
+ * stats - The Stellar statistics object
+ * Returns: The maximum of the accumulated values in the window or NaN if the statistics object is null.
+* `STATS_MEAN`
+ * Description: Calculates the mean of the accumulated values (or in the window if a window is used).
+ * Input:
+ * stats - The Stellar statistics object
+ * Returns: The mean of the values in the window or NaN if the statistics object is null.
+* `STATS_MERGE`
+ * Description: Merges statistics objects.
+ * Input:
+ * statistics - A list of statistics objects
+ * Returns: A Stellar statistics object
+* `STATS_MIN`
+ * Description: Calculates the minimum of the accumulated values (or in the window if a window is used).
+ * Input:
+ * stats - The Stellar statistics object
+ * Returns: The minimum of the accumulated values in the window or NaN if the statistics object is null.
+* `STATS_PERCENTILE`
+ * Description: Computes the p'th percentile of the accumulated values (or in the window if a window is used).
+ * Input:
+ * stats - The Stellar statistics object
+ * p - a double where 0 <= p < 1 representing the percentile
+ * Returns: The p'th percentile of the data or NaN if the statistics object is null
+* `STATS_POPULATION_VARIANCE`
+ * Description: Calculates the population variance of the accumulated values (or in the window if a window is used). See http://commons.apache.org/proper/commons-math/userguide/stat.html#a1.2_Descriptive_statistics
+ * Input:
+ * stats - The Stellar statistics object
+ * Returns: The population variance of the values in the window or NaN if the statistics object is null.
+* `STATS_QUADRATIC_MEAN`
+ * Description: Calculates the quadratic mean of the accumulated values (or in the window if a window is used). See http://commons.apache.org/proper/commons-math/userguide/stat.html#a1.2_Descriptive_statistics
+ * Input:
+ * stats - The Stellar statistics object
+ * Returns: The quadratic mean of the values in the window or NaN if the statistics object is null.
+* `STATS_SD`
+ * Description: Calculates the standard deviation of the accumulated values (or in the window if a window is used). See http://commons.apache.org/proper/commons-math/userguide/stat.html#a1.2_Descriptive_statistics
+ * Input:
+ * stats - The Stellar statistics object
+ * Returns: The standard deviation of the values in the window or NaN if the statistics object is null.
+* `STATS_SKEWNESS`
+ * Description: Calculates the skewness of the accumulated values (or in the window if a window is used). See http://commons.apache.org/proper/commons-math/userguide/stat.html#a1.2_Descriptive_statistics
+ * Input:
+ * stats - The Stellar statistics object
+ * Returns: The skewness of the values in the window or NaN if the statistics object is null.
+* `STATS_SUM`
+ * Description: Calculates the sum of the accumulated values (or in the window if a window is used).
+ * Input:
+ * stats - The Stellar statistics object
+ * Returns: The sum of the values in the window or NaN if the statistics object is null.
+* `STATS_SUM_LOGS`
+ * Description: Calculates the sum of the (natural) log of the accumulated values (or in the window if a window is used). See http://commons.apache.org/proper/commons-math/userguide/stat.html#a1.2_Descriptive_statistics
+ * Input:
+ * stats - The Stellar statistics object
+ * Returns: The sum of the (natural) log of the values in the window or NaN if the statistics object is null.
+* `STATS_SUM_SQUARES`
+ * Description: Calculates the sum of the squares of the accumulated values (or in the window if a window is used).
+ * Input:
+ * stats - The Stellar statistics object
+ * Returns: The sum of the squares of the values in the window or NaN if the statistics object is null.
+* `STATS_VARIANCE`
+ * Description: Calculates the variance of the accumulated values (or in the window if a window is used). See http://commons.apache.org/proper/commons-math/userguide/stat.html#a1.2_Descriptive_statistics
+ * Input:
+ * stats - The Stellar statistics object
+ * Returns: The variance of the values in the window or NaN if the statistics object is null.
+
+### Statistical Outlier Detection
+
+* `OUTLIER_MAD_STATE_MERGE`
+ * Description: Update the statistical state required to compute the Median Absolute Deviation.
+ * Input:
+ * [state] - A list of Median Absolute Deviation States to merge. Generally these are states across time.
+ * currentState? - The current state (optional)
+ * Returns: The Median Absolute Deviation state
+* `OUTLIER_MAD_ADD`
+ * Description: Add a piece of data to the state.
+ * Input:
+ * state - The MAD state
+ * value - The numeric value to add
+ * Returns: The MAD state
+* `OUTLIER_MAD_SCORE`
+ * Description: Get the modified z-score normalized by the MAD: scale * | x_i - median(X) | / MAD. See the first page of http://web.ipac.caltech.edu/staff/fmasci/home/astro_refs/BetterThanMAD.pdf
+ * Input:
+ * state - The MAD state
+ * value - The numeric value to score
+ * scale? - Optionally the scale to use when computing the modified z-score. Default is `0.6745`, see the first page of http://web.ipac.caltech.edu/staff/fmasci/home/astro_refs/BetterThanMAD.pdf
+ * Returns: The modified z-score
+
+# Outlier Analysis
+
+A common desire is to find anomalies in numerical data. To that end,
+we have some simple statistical anomaly detectors.
+
+## Median Absolute Deviation
+
+Much has been written about this robust estimator. See the first page
+of http://web.ipac.caltech.edu/staff/fmasci/home/astro_refs/BetterThanMAD.pdf
+for a good coverage of the good and the bad of MAD. The usage, however
+is fairly straightforward:
+* Gather the statistical state required to compute the MAD
+ * The distribution of the values of a univariate random variable over time.
+ * The distribution of the absolute deviations of the values from the median.
+* Use this statistical state to score unseen values. The higher the score, the more unlike the previously seen data the value is.
+
+There are a couple of issues which make MAD a bit hard to compute.
+First, the statistical state requires computing median, which can be
+computationally expensive to compute exactly. To get around this, we
+use the OnlineStatisticalProvider to compute a sketch rather than the
+exact median. Secondly, the statistical state for seasonal data should
+be limited to a fixed, trailing window. We do this by ensuring that the
+MAD state is mergeable and able to be queried from within the Profiler.
+
+### Example
+
+We will create a dummy data stream of gaussian noise to illustrate how
+to use the MAD functionality along with the profiler to tag messages as
+outliers or not.
+
+To do this, we will create a
+* data generator
+* parser
+* profiler profile
+* enrichment and threat triage
+
+#### Data Generator
+
+We can create a simple python script to generate a stream of gaussian
+noise at the frequency of one message per second as a python script
+which should be saved at `~/rand_gen.py`:
+```
+#!/usr/bin/python
+import random
+import sys
+import time
+def main():
+ mu = float(sys.argv[1])
+ sigma = float(sys.argv[2])
+ freq_s = int(sys.argv[3])
+ while True:
+ print str(random.gauss(mu, sigma))
+ sys.stdout.flush()
+ time.sleep(freq_s)
+
+if __name__ == '__main__':
+ main()
+```
+
+This script will take the following as arguments:
+* The mean of the data generated
+* The standard deviation of the data generated
+* The frequency (in seconds) of the data generated
+
+#### The Parser
+
+We will create a parser that will take the single numbers in and create
+a message with a field called `value` in them using the `CSVParser`.
+
+Add the following file to
+`$METRON_HOME/config/zookeeper/parsers/mad.json`:
+```
+{
+ "parserClassName" : "org.apache.metron.parsers.csv.CSVParser"
+ ,"sensorTopic" : "mad"
+ ,"parserConfig" : {
+ "columns" : {
+ "value_str" : 0
+ }
+ }
+ ,"fieldTransformations" : [
+ {
+ "transformation" : "STELLAR"
+ ,"output" : [ "value" ]
+ ,"config" : {
+ "value" : "TO_DOUBLE(value_str)"
+ }
+ }
+ ]
+}
+```
+
+#### Enrichment and Threat Intel
+
+We will set a threat triage level of `10` if a message generates a outlier score of more than 3.5.
+This cutoff will depend on your data and should be adjusted based on the
+assumed underlying distribution. Note that under the assumptions of
+normality, MAD will act as a robust estimator of the standard deviation, so the cutoff
+should be considered the number of standard deviations away. For other
+distributions, there are other interpretations which will make sense in
+the context of measuring the "degree different". See
+http://eurekastatistics.com/using-the-median-absolute-deviation-to-find-outliers/
+for a brief discussion of this.
+
+Create the following in
+`$METRON_HOME/config/zookeeper/enrichments/mad.json`:
+
+```
+{
+ "index": "mad",
+ "batchSize": 1,
+ "enrichment": {
+ "fieldMap": {
+ "stellar" : {
+ "config" : {
+ "parser_score" : "OUTLIER_MAD_SCORE(OUTLIER_MAD_STATE_MERGE(
+PROFILE_GET( 'sketchy_mad', 'global', 10, 'MINUTES') ), value)"
+ ,"is_alert" : "if parser_score > 3.5 then true else is_alert"
--- End diff --
Can 3.5 be pulled out into a hyper parameter?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #352: METRON-562: Add rudimentary statistical outlier...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/352
I want to also summarize a couple of my thoughts that I had in the course of creating this PR. It appears to me that there are a number of these sorts of outlier analysis models that can be split into a common pattern:
* Gather state via the profiler.
* Operate on that state during the scoring mechanism
For these z-score based statistical outlier analysis techniques/models, that state comes in the form of a statistical summary or multiple statistical summaries in the case of median absolute deviation. The scoring component is literally composing a modified z-score (scaled by MAD). The question arises as to whether or not this model is better served in MaaS or as a stellar function. Where I have come down in this PR is that I feel these very simple z-score or deviation-from-a-central-moment style outlier models fit better within Stellar due to their simplicity from a computational perspective. Furthermore, it would seem to be less than ideal performance-wise to perform a network hop to a REST interface in MaaS to do simple subtraction and division.
An alternate architecture here would be to use MaaS and have the model pull the appropriate statistical context from the profiler's store in HBase. I am sensitive to the possibility that the placement of model application within Metron may be a point of confusion and a consistent message is important. I decided to go with simplicity and performance for the moment, but I'd love to hear if there is opposition in the audience.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #352: METRON-562: Add rudimentary statistical ...
Posted by cestella <gi...@git.apache.org>.
GitHub user cestella reopened a pull request:
https://github.com/apache/incubator-metron/pull/352
METRON-562: Add rudimentary statistical outlier detection
With the advent of the profiler, we can now capture state. Furthermore, with Stellar, we can capture statistical summaries. We should provide rudimentary outlier detection functionality in the form of Stellar functions that can operate on captured state from the profiler.
To begin, we should enable simple outlier tests using distance from a central measure such as Median Absolute Deviation (see http://www.itl.nist.gov/div898/handbook/eda/section3/eda35h.htm).
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/cestella/incubator-metron outlier_models_mad
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-metron/pull/352.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #352
----
commit 754c4b46fbd92b5b3ecea24324b1ca7e36834ab6
Author: cstella <ce...@gmail.com>
Date: 2016-11-10T16:16:29Z
Initial cut.
commit 9f39aedec3da8bc994a39b5e9585da2d012bc07b
Author: cstella <ce...@gmail.com>
Date: 2016-11-10T20:20:48Z
Added rudimentary statistical outlier detection.
commit a06b1e0c3bb95557427ed52ab7614d49d3c5a61e
Author: cstella <ce...@gmail.com>
Date: 2016-11-11T19:59:06Z
Fixed tests, I think.
commit a20b7731124860dd916a2f3fd1b235fa6cf0bf50
Author: cstella <ce...@gmail.com>
Date: 2016-11-11T20:43:22Z
Update to properly update on the tick.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #352: METRON-562: Add rudimentary statistical ...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/352#discussion_r87669635
--- Diff: metron-analytics/metron-profiler/src/main/java/org/apache/metron/profiler/bolt/ProfileBuilderBolt.java ---
@@ -243,8 +244,18 @@ private void flush(Tuple tickTuple) {
measurement.setValue(result);
emit(measurement, tickTuple);
+ // Execute the update with the old state
+ Map<String, String> tickUpdate = profileConfig.getTickUpdate();
+ Map<String, Object> state = executor.getState();
+ if(tickUpdate != null) {
+ tickUpdate.forEach((var, expr) -> executor.assign(var, expr, executor.getState()));
--- End diff --
Ok, I'm confused. When we do the update on a per-tuple basis, `executor.assign` is called on every expression that will be updated in the config. Let's say my expression is `"s" : "STATS_ADD(s, value)"` Now, obviously, within context there is data from my message (`value`) and message from the existing state (the `s` variable). In the case of a tick update, I'd like to have available just the variables from the existing state (e.g. the `s` variable in my example above). How would I go about accomplishing that? I assumed that `s` would be held in the `executor.getState()`, but maybe not.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #352: METRON-562: Add rudimentary statistical ...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/352#discussion_r87650462
--- Diff: metron-analytics/metron-profiler/src/main/java/org/apache/metron/profiler/bolt/ProfileBuilderBolt.java ---
@@ -246,6 +247,8 @@ private void flush(Tuple tickTuple) {
// clear the execution state to prepare for the next window
executor.clearState();
+ profileConfig.getTickUpdate().forEach((var, expr) -> executor.assign(var, expr, new HashMap<>()));
--- End diff --
At this point in execution, during a flush operation, we don't have a telemetry message. It is the tick tuple that drives the flush. We won't have a telemetry message to offer to the 'tick' expressions.
With how you have it now, the 'tick' expressions will be able to reference all of the state being maintained by the 'init' and 'update' expressions.
The one missing piece that I can think of is that the 'tick' expressions cannot access the result of the 'result' expression. It might be useful to offer that up to the 'tick' expressions.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #352: METRON-562: Add rudimentary statistical ...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/352#discussion_r87647426
--- Diff: metron-analytics/metron-profiler/src/main/java/org/apache/metron/profiler/bolt/ProfileBuilderBolt.java ---
@@ -246,6 +247,8 @@ private void flush(Tuple tickTuple) {
// clear the execution state to prepare for the next window
executor.clearState();
+ profileConfig.getTickUpdate().forEach((var, expr) -> executor.assign(var, expr, new HashMap<>()));
--- End diff --
@nickwallen My intent here is to add a hook in the profiler on tick. Would you take a look and make sure I'm doing this right? I'm interested specifically in the empty hashmap here. I'd like to ensure that the current message is being passed in on tick, but I fear it won't be the case.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #352: METRON-562: Add rudimentary statistical ...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/352#discussion_r87670020
--- Diff: metron-analytics/metron-profiler/src/main/java/org/apache/metron/profiler/bolt/ProfileBuilderBolt.java ---
@@ -243,8 +244,18 @@ private void flush(Tuple tickTuple) {
measurement.setValue(result);
emit(measurement, tickTuple);
+ // Execute the update with the old state
+ Map<String, String> tickUpdate = profileConfig.getTickUpdate();
+ Map<String, Object> state = executor.getState();
+ if(tickUpdate != null) {
+ tickUpdate.forEach((var, expr) -> executor.assign(var, expr, executor.getState()));
--- End diff --
What you had previously, before this latest commit, does exactly that.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #352: METRON-562: Add rudimentary statistical outlier...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/352
I worked out an example in vagrant. Verified that the statistical and profiler functions can be called from all the relevant places:
* The REPL
* The parser topology
* The enrichment topology
To test this yourself, please check out the README under `metron-analytics/metron-statistics`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #352: METRON-562: Add rudimentary statistical ...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/352#discussion_r87670584
--- Diff: metron-analytics/metron-profiler/src/main/java/org/apache/metron/profiler/bolt/ProfileBuilderBolt.java ---
@@ -243,8 +244,18 @@ private void flush(Tuple tickTuple) {
measurement.setValue(result);
emit(measurement, tickTuple);
+ // Execute the update with the old state
+ Map<String, String> tickUpdate = profileConfig.getTickUpdate();
+ Map<String, Object> state = executor.getState();
+ if(tickUpdate != null) {
+ tickUpdate.forEach((var, expr) -> executor.assign(var, expr, executor.getState()));
+ }
// clear the execution state to prepare for the next window
executor.clearState();
+ //make sure that we bring along the update state
+ if(tickUpdate != null) {
+ tickUpdate.forEach((var, expr) -> executor.getState().put(var, state.get(var)));
--- End diff --
Ok. So we are taking on the risk of premature worker death.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #352: METRON-562: Add rudimentary statistical ...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/352#discussion_r87669665
--- Diff: metron-analytics/metron-profiler/src/main/java/org/apache/metron/profiler/bolt/ProfileBuilderBolt.java ---
@@ -243,8 +244,18 @@ private void flush(Tuple tickTuple) {
measurement.setValue(result);
emit(measurement, tickTuple);
+ // Execute the update with the old state
+ Map<String, String> tickUpdate = profileConfig.getTickUpdate();
+ Map<String, Object> state = executor.getState();
+ if(tickUpdate != null) {
+ tickUpdate.forEach((var, expr) -> executor.assign(var, expr, executor.getState()));
+ }
// clear the execution state to prepare for the next window
executor.clearState();
+ //make sure that we bring along the update state
+ if(tickUpdate != null) {
+ tickUpdate.forEach((var, expr) -> executor.getState().put(var, state.get(var)));
--- End diff --
The effect of this is to carry forward the values of all 'tick' expressions past a flush. So you effectively have a set of variables that are never cleared or reset.
I thought you had said that you did not want to take this approach due to the risk of losing that state if a worker dies. Am I misunderstanding this?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #352: METRON-562: Add rudimentary statistical ...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/352#discussion_r87671704
--- Diff: metron-analytics/metron-profiler/src/main/java/org/apache/metron/profiler/bolt/ProfileBuilderBolt.java ---
@@ -243,8 +244,18 @@ private void flush(Tuple tickTuple) {
measurement.setValue(result);
emit(measurement, tickTuple);
+ // Execute the update with the old state
+ Map<String, String> tickUpdate = profileConfig.getTickUpdate();
+ Map<String, Object> state = executor.getState();
+ if(tickUpdate != null) {
+ tickUpdate.forEach((var, expr) -> executor.assign(var, expr, executor.getState()));
--- End diff --
and this is after the state is cleared, right?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #352: METRON-562: Add rudimentary statistical ...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/352#discussion_r87669865
--- Diff: metron-analytics/metron-profiler/src/main/java/org/apache/metron/profiler/bolt/ProfileBuilderBolt.java ---
@@ -243,8 +244,18 @@ private void flush(Tuple tickTuple) {
measurement.setValue(result);
emit(measurement, tickTuple);
+ // Execute the update with the old state
+ Map<String, String> tickUpdate = profileConfig.getTickUpdate();
+ Map<String, Object> state = executor.getState();
+ if(tickUpdate != null) {
+ tickUpdate.forEach((var, expr) -> executor.assign(var, expr, executor.getState()));
+ }
// clear the execution state to prepare for the next window
executor.clearState();
+ //make sure that we bring along the update state
+ if(tickUpdate != null) {
+ tickUpdate.forEach((var, expr) -> executor.getState().put(var, state.get(var)));
--- End diff --
It would carry forward just the results from the expressions run on the tick update.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #352: METRON-562: Add rudimentary statistical ...
Posted by cestella <gi...@git.apache.org>.
Github user cestella closed the pull request at:
https://github.com/apache/incubator-metron/pull/352
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #352: METRON-562: Add rudimentary statistical ...
Posted by cestella <gi...@git.apache.org>.
GitHub user cestella reopened a pull request:
https://github.com/apache/incubator-metron/pull/352
METRON-562: Add rudimentary statistical outlier detection
With the advent of the profiler, we can now capture state. Furthermore, with Stellar, we can capture statistical summaries. We should provide rudimentary outlier detection functionality in the form of Stellar functions that can operate on captured state from the profiler.
To begin, we should enable simple outlier tests using distance from a central measure such as Median Absolute Deviation (see http://www.itl.nist.gov/div898/handbook/eda/section3/eda35h.htm).
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/cestella/incubator-metron outlier_models_mad
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-metron/pull/352.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #352
----
commit 754c4b46fbd92b5b3ecea24324b1ca7e36834ab6
Author: cstella <ce...@gmail.com>
Date: 2016-11-10T16:16:29Z
Initial cut.
commit 9f39aedec3da8bc994a39b5e9585da2d012bc07b
Author: cstella <ce...@gmail.com>
Date: 2016-11-10T20:20:48Z
Added rudimentary statistical outlier detection.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #352: METRON-562: Add rudimentary statistical ...
Posted by james-sirota <gi...@git.apache.org>.
Github user james-sirota commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/352#discussion_r88757991
--- Diff: metron-analytics/metron-statistics/src/main/java/org/apache/metron/statistics/outlier/MedianAbsoluteDeviationFunctions.java ---
@@ -0,0 +1,193 @@
+/*
--- End diff --
We need to file a follow-on Jira to add some caching to this
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #352: METRON-562: Add rudimentary statistical outlier...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/352
@james-sirota Thanks for running it up! Agreed about the long tailed distributions; MAD should have interpretations for non-normally distributed data, but it we could stand generating a t-distribution in the unit tests for sure.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #352: METRON-562: Add rudimentary statistical outlier...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/352
One more thing, I also added a new lifecycle hook for the profiler. Now you can update the state on the tick. This is needed for MAD because we will need to refresh the statistical summary of the lookback window that we are using to compute the MAD. This should become more clear upon the creation of a README with a worked example, which I owe y'all.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #352: METRON-562: Add rudimentary statistical ...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/352#discussion_r87674055
--- Diff: metron-analytics/metron-profiler/src/main/java/org/apache/metron/profiler/bolt/ProfileBuilderBolt.java ---
@@ -243,8 +244,18 @@ private void flush(Tuple tickTuple) {
measurement.setValue(result);
emit(measurement, tickTuple);
+ // Execute the update with the old state
+ Map<String, String> tickUpdate = profileConfig.getTickUpdate();
+ Map<String, Object> state = executor.getState();
+ if(tickUpdate != null) {
+ tickUpdate.forEach((var, expr) -> executor.assign(var, expr, executor.getState()));
--- End diff --
Awesome, updated. Look ok now?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---