You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@metron.apache.org by nickwallen <gi...@git.apache.org> on 2016/08/23 17:38:59 UTC
[GitHub] incubator-metron pull request #226: METRON-377 Enable Profiles that Use Non-...
GitHub user nickwallen opened a pull request:
https://github.com/apache/incubator-metron/pull/226
METRON-377 Enable Profiles that Use Non-Single Pass Summary Functions
### [METRON-377](https://issues.apache.org/jira/browse/METRON-377)
This pull request replaces #215 .
As of METRON-372 , Profiles can be built using many statistical summaries that only require a single-pass over the data. This is less memory intensive and more scalable for high volume loads.
Unfortunately, not all functions can be calculated in a single pass. In particular, the skewness, ketosis and percentiles require all data to be stored in memory for the calculation to occur. The platform was enhanced so that a user can leverage skewness, ketosis and percentiles.
### Changes
The `STATS_INIT` function was enhanced to accept a `window_size`. This defines the number of input data elements that are maintained in memory.
If the `window_size` is greater than 0, a rolling window of the most recent `window_size` elements is maintained in memory. The skewness, ketosis and percentiles are calculated over this rolling window. The `window_size` must be >0 otherwise these values cannot be calculated.
If a user supplies a `window_size` equal to 0 then the more efficient implementation that does not maintain a rolling window in memory is used. Of course, in this case, the skewness, ketosis, and percentiles cannot be calculated.
The following additional functions were also added.
* STATS_KURTOSIS
* STATS_SKEWNESS
* STATS_PERCENTILES
Another integration test was added to validate this end-to-end also.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/nickwallen/incubator-metron METRON-377-2
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-metron/pull/226.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #226
----
commit f256fe7461a0a8273b0a9f9e2d10a01c7c53c473
Author: Nick Allen <ni...@nickallen.org>
Date: 2016-08-23T17:03:15Z
METRON-372 Enhance Statistical Operations Available for Use with the Profiler
commit 37d7c3813b4c142fff351cc5b020393a97a5d4d2
Author: Nick Allen <ni...@nickallen.org>
Date: 2016-08-23T17:18:26Z
METRON-377 Enable Profiles that Use Non-Single Pass Summary Functions
commit b6284ced44fd3f41945ad9e68267cd84b5aad782
Author: Nick Allen <ni...@nickallen.org>
Date: 2016-08-23T17:35:24Z
METRON-377 Added an integration tests to use new stats functionality
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #226: METRON-377 Enable Profiles that Use Non-...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/226#discussion_r76092175
--- Diff: metron-platform/metron-common/src/test/java/org/apache/metron/common/stellar/StellarStatisticsFunctionsTest.java ---
@@ -0,0 +1,244 @@
+/*
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.metron.common.stellar;
+
+import org.apache.commons.math3.stat.descriptive.DescriptiveStatistics;
+import org.apache.commons.math3.stat.descriptive.SummaryStatistics;
+import org.apache.metron.common.dsl.Context;
+import org.apache.metron.common.dsl.ParseException;
+import org.apache.metron.common.dsl.StellarFunctions;
+import org.junit.Before;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotNull;
+
+/**
+ * Tests the statistical summary functions of Stellar.
+ */
+@RunWith(Parameterized.class)
+public class StellarStatisticsFunctionsTest {
+
+ private List<Double> values;
+ private Map<String, Object> variables;
+ private DescriptiveStatistics stats;
+ private SummaryStatistics summaryStats;
+ private int windowSize;
+
+ public StellarStatisticsFunctionsTest(int windowSize) {
+ this.windowSize = windowSize;
+ }
+
+ @Parameterized.Parameters
+ public static Collection<Object[]> data() {
+ // each test will be run against these values for windowSize
+ return Arrays.asList(new Object[][] {{ 0 }, { 100 }});
+ }
+
+ /**
+ * Runs a Stellar expression.
+ * @param expr The expression to run.
+ * @param variables The variables available to the expression.
+ */
+ private static Object run(String expr, Map<String, Object> variables) {
+ StellarProcessor processor = new StellarProcessor();
+ return processor.parse(expr, x -> variables.get(x), StellarFunctions.FUNCTION_RESOLVER(), Context.EMPTY_CONTEXT());
--- End diff --
Yes. Will be fixed once I merge in all those changes from METRON-372. Almost there
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #226: METRON-377 Enable Profiles that Use Non-...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/226#discussion_r76103558
--- Diff: metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/functions/StellarStatistics.java ---
@@ -0,0 +1,207 @@
+/*
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.metron.common.dsl.functions;
+
+import org.apache.commons.lang.NotImplementedException;
+import org.apache.commons.math3.stat.descriptive.DescriptiveStatistics;
+import org.apache.commons.math3.stat.descriptive.SummaryStatistics;
+
+import java.io.Serializable;
+
+/**
+ * Provides basic summary statistics to Stellar.
+ *
+ * Used as an adapter to provide a single interface to two underlying Commons
+ * Math classes that provide summary statistics.
+ */
+public class StellarStatistics implements Serializable {
+
+ /**
+ * DescriptiveStatistics stores a rolling window of input data elements
+ * which is then used to calculate the summary statistic. There are some
+ * summary statistics like kurtosis and percentiles, that can only
+ * be calculated with this.
+ *
+ * This implementation is used if the windowSize > 0.
+ */
+ private DescriptiveStatistics descStats;
+
+ /**
+ * SummaryStatistics can be used for summary statistics that only
+ * require a single pass over the input data. This has the advantage
+ * of being less memory intensive, but not all summary statistics are
+ * available.
+ *
+ * This implementation is used if the windowSize == 0.
+ */
+ private SummaryStatistics summStats;
+
+ /**
+ * @param windowSize The number of input data elements to maintain in memory. If
+ * windowSize == 0, then no data elements will be maintained in
+ * memory.
+ */
+ public StellarStatistics(int windowSize) {
+
+ // only one of the underlying implementation classes will be used at a time
+ if(windowSize > 0) {
+ descStats = new DescriptiveStatistics(windowSize);
+ } else {
+ summStats = new SummaryStatistics();
+ }
+ }
+
+ public void addValue(double value) {
+ if(descStats != null) {
+ descStats.addValue(value);
+ } else {
+ summStats.addValue(value);
+ }
+ }
+
+ public long getCount() {
+ if(descStats != null) {
+ return descStats.getN();
+ } else {
+ return summStats.getN();
+ }
+ }
+
+ public double getMin() {
+ if(descStats != null) {
+ return descStats.getMin();
+ } else {
+ return summStats.getMin();
+ }
+ }
+
+ public double getMax() {
+ if(descStats != null) {
+ return descStats.getMax();
+ } else {
+ return summStats.getMax();
+ }
+ }
+
+ public double getMean() {
+ if(descStats != null) {
+ return descStats.getMean();
+ } else {
+ return summStats.getMean();
+ }
+ }
+
+ public double getSum() {
+ if(descStats != null) {
+ return descStats.getSum();
+ } else {
+ return summStats.getSum();
+ }
+ }
+
+ public double getVariance() {
+ if(descStats != null) {
+ return descStats.getVariance();
+ } else {
+ return summStats.getVariance();
+ }
+ }
+
+ public double getStandardDeviation() {
+ if(descStats != null) {
+ return descStats.getStandardDeviation();
+ } else {
+ return summStats.getStandardDeviation();
+ }
+ }
+
+ public double getGeometricMean() {
+ if(descStats != null) {
+ return descStats.getGeometricMean();
+ } else {
+ return summStats.getGeometricMean();
+ }
+ }
+
+ public double getPopulationVariance() {
+ if(descStats != null) {
+ return descStats.getPopulationVariance();
+ } else {
+ return summStats.getPopulationVariance();
+ }
+ }
+
+ public double getSecondMoment() {
+ if(descStats != null) {
+ throw new NotImplementedException("second moment not available if 'windowSize' > 0");
--- End diff --
@cestella pointed out that second moment is the same as variance. The naming between DescriptiveStatistics and SummaryStatistics just differs. Just added a commit that removes `STATS_SECOND_MOMENT` as it is redundant. Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #226: METRON-377 Enable Profiles that Use Non-...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/226#discussion_r76101432
--- Diff: metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/functions/StellarStatistics.java ---
@@ -0,0 +1,207 @@
+/*
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.metron.common.dsl.functions;
+
+import org.apache.commons.lang.NotImplementedException;
+import org.apache.commons.math3.stat.descriptive.DescriptiveStatistics;
+import org.apache.commons.math3.stat.descriptive.SummaryStatistics;
+
+import java.io.Serializable;
+
+/**
+ * Provides basic summary statistics to Stellar.
+ *
+ * Used as an adapter to provide a single interface to two underlying Commons
+ * Math classes that provide summary statistics.
+ */
+public class StellarStatistics implements Serializable {
+
+ /**
+ * DescriptiveStatistics stores a rolling window of input data elements
+ * which is then used to calculate the summary statistic. There are some
+ * summary statistics like kurtosis and percentiles, that can only
+ * be calculated with this.
+ *
+ * This implementation is used if the windowSize > 0.
+ */
+ private DescriptiveStatistics descStats;
+
+ /**
+ * SummaryStatistics can be used for summary statistics that only
+ * require a single pass over the input data. This has the advantage
+ * of being less memory intensive, but not all summary statistics are
+ * available.
+ *
+ * This implementation is used if the windowSize == 0.
+ */
+ private SummaryStatistics summStats;
+
+ /**
+ * @param windowSize The number of input data elements to maintain in memory. If
+ * windowSize == 0, then no data elements will be maintained in
+ * memory.
+ */
+ public StellarStatistics(int windowSize) {
+
+ // only one of the underlying implementation classes will be used at a time
+ if(windowSize > 0) {
+ descStats = new DescriptiveStatistics(windowSize);
+ } else {
+ summStats = new SummaryStatistics();
+ }
+ }
+
+ public void addValue(double value) {
+ if(descStats != null) {
+ descStats.addValue(value);
+ } else {
+ summStats.addValue(value);
+ }
+ }
+
+ public long getCount() {
+ if(descStats != null) {
+ return descStats.getN();
+ } else {
+ return summStats.getN();
+ }
+ }
+
+ public double getMin() {
+ if(descStats != null) {
+ return descStats.getMin();
+ } else {
+ return summStats.getMin();
+ }
+ }
+
+ public double getMax() {
+ if(descStats != null) {
+ return descStats.getMax();
+ } else {
+ return summStats.getMax();
+ }
+ }
+
+ public double getMean() {
+ if(descStats != null) {
+ return descStats.getMean();
+ } else {
+ return summStats.getMean();
+ }
+ }
+
+ public double getSum() {
+ if(descStats != null) {
+ return descStats.getSum();
+ } else {
+ return summStats.getSum();
+ }
+ }
+
+ public double getVariance() {
+ if(descStats != null) {
+ return descStats.getVariance();
+ } else {
+ return summStats.getVariance();
+ }
+ }
+
+ public double getStandardDeviation() {
+ if(descStats != null) {
+ return descStats.getStandardDeviation();
+ } else {
+ return summStats.getStandardDeviation();
+ }
+ }
+
+ public double getGeometricMean() {
+ if(descStats != null) {
+ return descStats.getGeometricMean();
+ } else {
+ return summStats.getGeometricMean();
+ }
+ }
+
+ public double getPopulationVariance() {
+ if(descStats != null) {
+ return descStats.getPopulationVariance();
+ } else {
+ return summStats.getPopulationVariance();
+ }
+ }
+
+ public double getSecondMoment() {
+ if(descStats != null) {
+ throw new NotImplementedException("second moment not available if 'windowSize' > 0");
--- End diff --
Any reason why we're not implementing the second moment for SummaryStatistics? It should be the square of the standard deviation, right?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #226: METRON-377 Enable Profiles that Use Non-Single ...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:
https://github.com/apache/incubator-metron/pull/226
The PR has been merged and re-tested. Ready for a look-see.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #226: METRON-377 Enable Profiles that Use Non-Single ...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/226
Looks good, +1 pending green build.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #226: METRON-377 Enable Profiles that Use Non-...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/226#discussion_r76100309
--- Diff: metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/functions/StellarStatistics.java ---
@@ -0,0 +1,207 @@
+/*
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.metron.common.dsl.functions;
+
+import org.apache.commons.lang.NotImplementedException;
+import org.apache.commons.math3.stat.descriptive.DescriptiveStatistics;
+import org.apache.commons.math3.stat.descriptive.SummaryStatistics;
+
+import java.io.Serializable;
+
+/**
+ * Provides basic summary statistics to Stellar.
+ *
+ * Used as an adapter to provide a single interface to two underlying Commons
+ * Math classes that provide summary statistics.
+ */
+public class StellarStatistics implements Serializable {
+
+ /**
+ * DescriptiveStatistics stores a rolling window of input data elements
+ * which is then used to calculate the summary statistic. There are some
+ * summary statistics like kurtosis and percentiles, that can only
+ * be calculated with this.
+ *
+ * This implementation is used if the windowSize > 0.
+ */
+ private DescriptiveStatistics descStats;
+
+ /**
+ * SummaryStatistics can be used for summary statistics that only
+ * require a single pass over the input data. This has the advantage
+ * of being less memory intensive, but not all summary statistics are
+ * available.
+ *
+ * This implementation is used if the windowSize == 0.
+ */
+ private SummaryStatistics summStats;
+
+ /**
+ * @param windowSize The number of input data elements to maintain in memory. If
+ * windowSize == 0, then no data elements will be maintained in
+ * memory.
+ */
+ public StellarStatistics(int windowSize) {
--- End diff --
Can we move the functions for StellarStatistics into their own interface and have two implementations, one backed by SummaryStatistics and one backed by DescriptiveStatistics? We will likely have a third implementation, ProbabalisticStatistics that supports approximate distributional stats while not requiring a window size and I'd like it to be as simple as implementing an interface, not adding to a branch at every method.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #226: METRON-377 Enable Profiles that Use Non-...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/226#discussion_r75933922
--- Diff: metron-platform/metron-common/src/test/java/org/apache/metron/common/stellar/StellarStatisticsFunctionsTest.java ---
@@ -0,0 +1,244 @@
+/*
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.metron.common.stellar;
+
+import org.apache.commons.math3.stat.descriptive.DescriptiveStatistics;
+import org.apache.commons.math3.stat.descriptive.SummaryStatistics;
+import org.apache.metron.common.dsl.Context;
+import org.apache.metron.common.dsl.ParseException;
+import org.apache.metron.common.dsl.StellarFunctions;
+import org.junit.Before;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotNull;
+
+/**
+ * Tests the statistical summary functions of Stellar.
+ */
+@RunWith(Parameterized.class)
+public class StellarStatisticsFunctionsTest {
+
+ private List<Double> values;
+ private Map<String, Object> variables;
+ private DescriptiveStatistics stats;
+ private SummaryStatistics summaryStats;
+ private int windowSize;
+
+ public StellarStatisticsFunctionsTest(int windowSize) {
+ this.windowSize = windowSize;
+ }
+
+ @Parameterized.Parameters
+ public static Collection<Object[]> data() {
+ // each test will be run against these values for windowSize
+ return Arrays.asList(new Object[][] {{ 0 }, { 100 }});
+ }
+
+ /**
+ * Runs a Stellar expression.
+ * @param expr The expression to run.
+ * @param variables The variables available to the expression.
+ */
+ private static Object run(String expr, Map<String, Object> variables) {
+ StellarProcessor processor = new StellarProcessor();
+ return processor.parse(expr, x -> variables.get(x), StellarFunctions.FUNCTION_RESOLVER(), Context.EMPTY_CONTEXT());
--- End diff --
Can we make sure that we run Stellar's `validate()` function on the expressions passed in.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #226: METRON-377 Enable Profiles that Use Non-...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-metron/pull/226
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---