You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by feynmanliang <gi...@git.apache.org> on 2015/02/21 23:31:21 UTC
[GitHub] spark pull request: A/B testing
GitHub user feynmanliang opened a pull request:
https://github.com/apache/spark/pull/4716
A/B testing
Implementation of A/B testing using Streaming API.
This contribution is my original work and I you license the work to the project under the project's open source license.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/feynmanliang/spark ab_testing
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/4716.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #4716
----
commit 105401a89216516565236f59a66a22cc91830686
Author: Feynman Liang <fe...@gmail.com>
Date: 2015-01-10T19:36:27Z
Add broken implementation of AB testing.
commit cb73e790c435a4819fb62bc6c37717f4b882aee4
Author: Feynman Liang <fe...@gmail.com>
Date: 2015-01-10T21:07:29Z
Fix AB testing implementation and add unit tests.
commit e0d5beccf54914ebdc5663dbe4ba71944f3183e2
Author: Feynman Liang <fe...@gmail.com>
Date: 2015-01-10T22:54:26Z
Extract t-testing code out of OnlineABTesting.
commit 2100de641a2e86efeaa0f559500c7ced6f7d51a9
Author: Feynman Liang <fe...@gmail.com>
Date: 2015-01-11T04:56:30Z
Add peace period for dropping first k entries of each A/B group.
commit 708380e980ed46ac1beb7665f7854fcf36ebc403
Author: Feynman Liang <fe...@gmail.com>
Date: 2015-01-11T05:09:18Z
Add numDim to MultivariateOnlineSummarizer.
commit ec7f700fbca15d84bba126edaaa50d53ce5fc7be
Author: Feynman Liang <fe...@gmail.com>
Date: 2015-01-11T06:02:41Z
Refactored ABTestingMethod into sealed trait.
commit 3f19e15aa3b7056262b601686643ed962846cdc3
Author: Feynman Liang <fe...@gmail.com>
Date: 2015-01-11T06:29:49Z
Add (non-sliding) testing window functionality.
commit c56f9237aa81a70e8572e2ecb851ebaf5cdfa473
Author: Feynman Liang <fe...@gmail.com>
Date: 2015-01-11T15:19:46Z
Fix peace period implementation.
commit 0d738815eb1cd49096112d8be7e9124345af0604
Author: Feynman Liang <fe...@gmail.com>
Date: 2015-01-11T17:31:05Z
Fix test window batching.
commit abf59d5e8f817f847af77aef7514fb740dbbf69d
Author: Feynman Liang <fe...@gmail.com>
Date: 2015-01-11T17:56:15Z
Handle (inelegantly) closure capture for ABTestMethod
commit e05eaaf3bb21bbed4c123d9ec6514e84ae75adcb
Author: Feynman Liang <fe...@gmail.com>
Date: 2015-01-11T18:20:19Z
Improve handling of OnlineABTestMethod closure by moving DStream processing method into Serializable class.
commit 964a555746273a3afa542e34fdc6b86be60a5db9
Author: Feynman Liang <fe...@gmail.com>
Date: 2015-01-11T18:52:37Z
Fixed flaky peacePeriod test.
commit 79c1d44c6232b0a4af5df4dc14cdc83919cfdea9
Author: Feynman Liang <fe...@gmail.com>
Date: 2015-01-11T20:39:58Z
Add ScalaDocs and format to style guide.
commit e030c12337dce99abcf26f7d02c5d00a78f58c9b
Author: Feynman Liang <fe...@gmail.com>
Date: 2015-01-12T00:02:20Z
Add OnlineABTestExample.
commit e8e1f82b16fbdd8446e21b32bb39b413e1ae30d1
Author: Feynman Liang <fe...@gmail.com>
Date: 2015-01-12T00:03:12Z
Format code to style guide.
commit 17eef4eb22d918198dd03f2a931f009863fadcf5
Author: Feynman Liang <fe...@gmail.com>
Date: 2015-01-19T04:43:36Z
Switch MultivariateOnlineSummarizer to univariate StatsCounter.
commit a2ad38be8a77eef045581282b3dbc9d6a1544870
Author: Feynman Liang <fe...@gmail.com>
Date: 2015-01-19T14:45:15Z
Reduce number of passes in pairSummaries.
commit 4bb8636e5317a542ff0b29270548bd933199c6eb
Author: Feynman Liang <fe...@gmail.com>
Date: 2015-01-19T14:45:41Z
Add test for behavior when missing data from one group.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3147][MLLib] A/B testing
Posted by feynmanliang <gi...@git.apache.org>.
Github user feynmanliang commented on the pull request:
https://github.com/apache/spark/pull/4716#issuecomment-75543022
`[error] * abstract method numDim()Int in interface org.apache.spark.mllib.stat.MultivariateStatisticalSummary does not have a correspondent in old version`
Would it be bett
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3147][MLLib][Streaming] Streaming 2-sam...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/4716#issuecomment-141346945
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3147][MLLib][Streaming] Streaming 2-sam...
Posted by feynmanliang <gi...@git.apache.org>.
Github user feynmanliang commented on a diff in the pull request:
https://github.com/apache/spark/pull/4716#discussion_r39897100
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/stat/test/StreamingTest.scala ---
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.stat.test
+
+import org.apache.spark.Logging
+import org.apache.spark.annotation.{Experimental, Since}
+import org.apache.spark.streaming.dstream.DStream
+import org.apache.spark.util.StatCounter
+
+/**
+ * :: Experimental ::
+ * Performs online 2-sample significance testing for a stream of (Boolean, Double) pairs. The
+ * Boolean identifies which sample each observation comes from, and the Double is the numeric value
+ * of the observation.
+ *
+ * To address novelty affects, the `peacePeriod` specifies a set number of initial
+ * [[org.apache.spark.rdd.RDD]] batches of the [[DStream]] to be dropped from significance testing.
+ *
+ * The `windowSize` sets the number of batches each significance test is to be performed over. The
+ * window is sliding with a stride length of 1 batch. Setting windowSize to 0 will perform
+ * cumulative processing, using all batches seen so far.
+ *
+ * Different tests may be used for assessing statistical significance depending on assumptions
+ * satisfied by data. For more details, see [[StreamingTestMethod]]. The `testMethod` specifies
+ * which test will be used.
+ *
+ * Use a builder pattern to construct a streaming test in an application, for example:
+ * ```
+ * val model = new OnlineABTest()
+ * .setPeacePeriod(10)
+ * .setWindowSize(0)
+ * .setTestMethod("welch")
+ * .registerStream(DStream)
+ * ```
+ */
+@Experimental
+@Since("1.6.0")
+class StreamingTest(
+ @Since("1.6.0") var peacePeriod: Int = 0,
+ @Since("1.6.0") var windowSize: Int = 0,
+ @Since("1.6.0") var testMethod: StreamingTestMethod = WelchTTest)
+ extends Logging with Serializable {
+
+ /** Set the number of initial batches to ignore. */
+ @Since("1.6.0")
+ def setPeacePeriod(peacePeriod: Int): this.type = {
+ this.peacePeriod = peacePeriod
+ this
+ }
+
+ /**
+ * Set the number of batches to compute significance tests over.
+ * A value of 0 will use all batches seen so far.
+ */
+ @Since("1.6.0")
+ def setWindowSize(windowSize: Int): this.type = {
+ this.windowSize = windowSize
+ this
+ }
+
+ /** Set the statistical method used for significance testing. */
+ @Since("1.6.0")
+ def setTestMethod(method: String): this.type = {
+ this.testMethod = StreamingTestMethod.getTestMethodFromName(method)
+ this
+ }
+
+ /**
+ * Register a [[DStream]] of values for significance testing.
+ *
+ * @param data stream of (key,value) pairs where the key is the group membership (control or
+ * treatment) and the value is the numerical metric to test for significance
+ * @return stream of significance testing results
+ */
+ @Since("1.6.0")
+ def registerStream(data: DStream[(Boolean, Double)]): DStream[StreamingTestResult] = {
+ val dataAfterPeacePeriod = dropPeacePeriod(data)
+ val summarizedData = summarizeByKeyAndWindow(dataAfterPeacePeriod)
+ val pairedSummaries = pairSummaries(summarizedData)
+ val testResults = testMethod.doTest(pairedSummaries)
+
+ testResults
+ }
+
+ /** Drop all batches inside the peace period. */
+ private[stat] def dropPeacePeriod(
+ data: DStream[(Boolean, Double)]): DStream[(Boolean, Double)] = {
+ data.transform { (rdd, time) =>
+ if (time.milliseconds > data.slideDuration.milliseconds * peacePeriod) {
+ rdd
+ } else {
+ rdd.filter(_ => false) // TODO: Is there a better way to drop a RDD from a DStream?
+ }
+ }
+ }
+
+ /** Compute summary statistics over each key and the specified test window size. */
+ private[stat] def summarizeByKeyAndWindow(
+ data: DStream[(Boolean, Double)]): DStream[(Boolean, StatCounter)] = {
+ if (this.windowSize == 0) {
+ data.updateStateByKey[StatCounter](
+ (newValues: Seq[Double], oldSummary: Option[StatCounter]) => {
+ val newSummary = oldSummary.getOrElse(new StatCounter())
+ newSummary.merge(newValues)
+ Some(newSummary)
+ })
+ } else {
+ val windowDuration = data.slideDuration * this.windowSize
+ data
+ .groupByKeyAndWindow(windowDuration)
+ .mapValues { values =>
+ val summary = new StatCounter()
+ values.foreach(value => summary.merge(value))
+ summary
+ }
+ }
+ }
+
+ /**
+ * Transform a stream of summaries into pairs representing summary statistics for group A and
+ * group B up to this batch.
+ */
+ private[stat] def pairSummaries(summarizedData: DStream[(Boolean, StatCounter)])
+ : DStream[(StatCounter, StatCounter)] = {
+ summarizedData
+ .map[(Int, StatCounter)](x => (0, x._2))
+ .groupByKey() // Iterable[StatCounter] should be length two, one for each A/B group
--- End diff --
OK
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3147][MLLib][Streaming] Streaming 2-sam...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/4716#issuecomment-141562018
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3147][MLLib][Streaming] Streaming 2-sam...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/4716#issuecomment-141353561
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3147][MLLib][Streaming] Streaming 2-sam...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/4716#discussion_r39824237
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/stat/test/StreamingTest.scala ---
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.stat.test
+
+import org.apache.spark.Logging
+import org.apache.spark.annotation.{Experimental, Since}
+import org.apache.spark.streaming.dstream.DStream
+import org.apache.spark.util.StatCounter
+
+/**
+ * :: Experimental ::
+ * Performs online 2-sample significance testing for a stream of (Boolean, Double) pairs. The
+ * Boolean identifies which sample each observation comes from, and the Double is the numeric value
+ * of the observation.
+ *
+ * To address novelty affects, the `peacePeriod` specifies a set number of initial
+ * [[org.apache.spark.rdd.RDD]] batches of the [[DStream]] to be dropped from significance testing.
+ *
+ * The `windowSize` sets the number of batches each significance test is to be performed over. The
+ * window is sliding with a stride length of 1 batch. Setting windowSize to 0 will perform
+ * cumulative processing, using all batches seen so far.
+ *
+ * Different tests may be used for assessing statistical significance depending on assumptions
+ * satisfied by data. For more details, see [[StreamingTestMethod]]. The `testMethod` specifies
+ * which test will be used.
+ *
+ * Use a builder pattern to construct a streaming test in an application, for example:
+ * ```
--- End diff --
use `{{{` for example code
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3147][MLLib][Streaming] Streaming 2-sam...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/4716#discussion_r39824556
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/stat/test/StreamingTestMethod.scala ---
@@ -0,0 +1,165 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.stat.test
+
+import java.io.Serializable
+
+import scala.language.implicitConversions
+import scala.math.pow
+
+import com.twitter.chill.MeatLocker
+import org.apache.commons.math3.stat.descriptive.StatisticalSummaryValues
+import org.apache.commons.math3.stat.inference.TTest
+
+import org.apache.spark.Logging
+import org.apache.spark.streaming.dstream.DStream
+import org.apache.spark.util.StatCounter
+
+/**
+ * Significance testing methods for [[StreamingTest]]. New 2-sample statistical significance tests
+ * should extend [[StreamingTestMethod]] and introduce a new entry in
+ * [[StreamingTestMethod.TEST_NAME_TO_OBJECT]]
+ */
+private[stat] sealed trait StreamingTestMethod extends Serializable {
+
+ val MethodName: String
+ val NullHypothesis: String
+
+ protected type SummaryPairStream =
+ DStream[(StatCounter, StatCounter)]
+
+ /**
+ * Perform streaming 2-sample statistical significance testing.
+ *
+ * @param sampleSummaries stream pairs of summary statistics for the 2 samples
+ * @return stream of rest results
+ */
+ def doTest(sampleSummaries: SummaryPairStream): DStream[StreamingTestResult]
+
+ /**
+ * Implicit adapter to convert between streaming summary statistics type and the type required by
+ * the t-testing libraries.
+ */
+ protected implicit def toApacheCommonsStats(
+ summaryStats: StatCounter): StatisticalSummaryValues = {
+ new StatisticalSummaryValues(
+ summaryStats.mean,
+ summaryStats.variance,
+ summaryStats.count,
+ summaryStats.max,
+ summaryStats.min,
+ summaryStats.mean * summaryStats.count
+ )
+ }
+}
+
+/**
+ * Performs Welch's 2-sample t-test. The null hypothesis is that the two data sets have equal mean.
+ * This test does not assume equal variance between the two samples and does not assume equal
+ * sample size.
+ *
+ * More information: http://en.wikipedia.org/wiki/Welch%27s_t_test
+ */
+private[stat] object WelchTTest extends StreamingTestMethod with Logging {
+
+ final val MethodName = "Welch's 2-sample T-test"
+ final val NullHypothesis = "Both groups have same mean"
+
+ private final val TTester = MeatLocker(new TTest())
+
+ def doTest(data: SummaryPairStream): DStream[StreamingTestResult] =
+ data.map[StreamingTestResult]((test _).tupled)
+
+ private def test(
+ statsA: StatCounter,
+ statsB: StatCounter): StreamingTestResult = {
+ def welchDF(sample1: StatisticalSummaryValues, sample2: StatisticalSummaryValues): Double = {
+ val s1 = sample1.getVariance
+ val n1 = sample1.getN
+ val s2 = sample2.getVariance
+ val n2 = sample2.getN
+
+ val a = pow(s1, 2) / n1
+ val b = pow(s2, 2) / n2
+
+ pow(a + b, 2) / ((pow(a, 2) / (n1 - 1)) + (pow(b, 2) / (n2 - 1)))
+ }
+
+ new StreamingTestResult(
+ TTester.get.tTest(statsA, statsB),
+ welchDF(statsA, statsB),
+ TTester.get.t(statsA, statsB),
+ MethodName,
+ NullHypothesis
+ )
+ }
+}
+
+/**
+ * Performs Students's 2-sample t-test. The null hypothesis is that the two data sets have equal
+ * mean. This test assumes equal variance between the two samples and does not assume equal sample
+ * size. For unequal variances, Welch's t-test should be used instead.
+ *
+ * More information: http://en.wikipedia.org/wiki/Student%27s_t-test
+ */
+private[stat] object StudentTTest extends StreamingTestMethod with Logging {
+
+ final val MethodName = "Student's 2-sample T-test"
--- End diff --
`t-test`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3147][MLLib][Streaming] Streaming 2-sam...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/4716#discussion_r39824239
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/stat/test/StreamingTest.scala ---
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.stat.test
+
+import org.apache.spark.Logging
+import org.apache.spark.annotation.{Experimental, Since}
+import org.apache.spark.streaming.dstream.DStream
+import org.apache.spark.util.StatCounter
+
+/**
+ * :: Experimental ::
+ * Performs online 2-sample significance testing for a stream of (Boolean, Double) pairs. The
+ * Boolean identifies which sample each observation comes from, and the Double is the numeric value
+ * of the observation.
+ *
+ * To address novelty affects, the `peacePeriod` specifies a set number of initial
+ * [[org.apache.spark.rdd.RDD]] batches of the [[DStream]] to be dropped from significance testing.
+ *
+ * The `windowSize` sets the number of batches each significance test is to be performed over. The
+ * window is sliding with a stride length of 1 batch. Setting windowSize to 0 will perform
+ * cumulative processing, using all batches seen so far.
+ *
+ * Different tests may be used for assessing statistical significance depending on assumptions
+ * satisfied by data. For more details, see [[StreamingTestMethod]]. The `testMethod` specifies
+ * which test will be used.
+ *
+ * Use a builder pattern to construct a streaming test in an application, for example:
+ * ```
+ * val model = new OnlineABTest()
--- End diff --
`StreamingTest`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3147][MLLib][Streaming] Streaming 2-sam...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4716#issuecomment-139663024
[Test build #42353 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42353/console) for PR 4716 at commit [`60b2e57`](https://github.com/apache/spark/commit/60b2e57026febcb68e459983ba3164281a47f636).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class MinMaxScaler(JavaEstimator, HasInputCol, HasOutputCol):`
* `class MinMaxScalerModel(JavaModel):`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3147][MLLib][Streaming] Streaming 2-sam...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/4716
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3147][MLLib][Streaming] Streaming 2-sam...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/4716#discussion_r39824269
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/stat/test/StreamingTest.scala ---
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.stat.test
+
+import org.apache.spark.Logging
+import org.apache.spark.annotation.{Experimental, Since}
+import org.apache.spark.streaming.dstream.DStream
+import org.apache.spark.util.StatCounter
+
+/**
+ * :: Experimental ::
+ * Performs online 2-sample significance testing for a stream of (Boolean, Double) pairs. The
+ * Boolean identifies which sample each observation comes from, and the Double is the numeric value
+ * of the observation.
+ *
+ * To address novelty affects, the `peacePeriod` specifies a set number of initial
+ * [[org.apache.spark.rdd.RDD]] batches of the [[DStream]] to be dropped from significance testing.
+ *
+ * The `windowSize` sets the number of batches each significance test is to be performed over. The
+ * window is sliding with a stride length of 1 batch. Setting windowSize to 0 will perform
+ * cumulative processing, using all batches seen so far.
+ *
+ * Different tests may be used for assessing statistical significance depending on assumptions
+ * satisfied by data. For more details, see [[StreamingTestMethod]]. The `testMethod` specifies
+ * which test will be used.
+ *
+ * Use a builder pattern to construct a streaming test in an application, for example:
+ * ```
+ * val model = new OnlineABTest()
+ * .setPeacePeriod(10)
+ * .setWindowSize(0)
+ * .setTestMethod("welch")
+ * .registerStream(DStream)
+ * ```
+ */
+@Experimental
+@Since("1.6.0")
+class StreamingTest(
+ @Since("1.6.0") var peacePeriod: Int = 0,
+ @Since("1.6.0") var windowSize: Int = 0,
+ @Since("1.6.0") var testMethod: StreamingTestMethod = WelchTTest)
+ extends Logging with Serializable {
+
+ /** Set the number of initial batches to ignore. */
+ @Since("1.6.0")
+ def setPeacePeriod(peacePeriod: Int): this.type = {
+ this.peacePeriod = peacePeriod
+ this
+ }
+
+ /**
+ * Set the number of batches to compute significance tests over.
+ * A value of 0 will use all batches seen so far.
+ */
+ @Since("1.6.0")
+ def setWindowSize(windowSize: Int): this.type = {
+ this.windowSize = windowSize
+ this
+ }
+
+ /** Set the statistical method used for significance testing. */
+ @Since("1.6.0")
+ def setTestMethod(method: String): this.type = {
+ this.testMethod = StreamingTestMethod.getTestMethodFromName(method)
+ this
+ }
+
+ /**
+ * Register a [[DStream]] of values for significance testing.
+ *
+ * @param data stream of (key,value) pairs where the key is the group membership (control or
--- End diff --
document clearly whether `true` means control or experiment
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3147][MLLib] A/B testing
Posted by freeman-lab <gi...@git.apache.org>.
Github user freeman-lab commented on the pull request:
https://github.com/apache/spark/pull/4716#issuecomment-90807615
@mengxr @feynmanliang sure thing! This looks really cool, will try to go through it in the next couple days.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3147][MLLib][Streaming] Streaming 2-sam...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the pull request:
https://github.com/apache/spark/pull/4716#issuecomment-141346526
test this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3147][MLLib][Streaming] Streaming 2-sam...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/4716#issuecomment-139647413
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3147][MLLib][Streaming] Streaming 2-sam...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/4716#issuecomment-141562049
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3147][MLLib][Streaming] Streaming 2-sam...
Posted by feynmanliang <gi...@git.apache.org>.
Github user feynmanliang commented on a diff in the pull request:
https://github.com/apache/spark/pull/4716#discussion_r39897117
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/stat/test/StreamingTest.scala ---
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.stat.test
+
+import org.apache.spark.Logging
+import org.apache.spark.annotation.{Experimental, Since}
+import org.apache.spark.streaming.dstream.DStream
+import org.apache.spark.util.StatCounter
+
+/**
+ * :: Experimental ::
+ * Performs online 2-sample significance testing for a stream of (Boolean, Double) pairs. The
+ * Boolean identifies which sample each observation comes from, and the Double is the numeric value
+ * of the observation.
+ *
+ * To address novelty affects, the `peacePeriod` specifies a set number of initial
+ * [[org.apache.spark.rdd.RDD]] batches of the [[DStream]] to be dropped from significance testing.
+ *
+ * The `windowSize` sets the number of batches each significance test is to be performed over. The
+ * window is sliding with a stride length of 1 batch. Setting windowSize to 0 will perform
+ * cumulative processing, using all batches seen so far.
+ *
+ * Different tests may be used for assessing statistical significance depending on assumptions
+ * satisfied by data. For more details, see [[StreamingTestMethod]]. The `testMethod` specifies
+ * which test will be used.
+ *
+ * Use a builder pattern to construct a streaming test in an application, for example:
+ * ```
+ * val model = new OnlineABTest()
+ * .setPeacePeriod(10)
+ * .setWindowSize(0)
+ * .setTestMethod("welch")
+ * .registerStream(DStream)
+ * ```
+ */
+@Experimental
+@Since("1.6.0")
+class StreamingTest(
+ @Since("1.6.0") var peacePeriod: Int = 0,
+ @Since("1.6.0") var windowSize: Int = 0,
+ @Since("1.6.0") var testMethod: StreamingTestMethod = WelchTTest)
+ extends Logging with Serializable {
+
+ /** Set the number of initial batches to ignore. */
+ @Since("1.6.0")
+ def setPeacePeriod(peacePeriod: Int): this.type = {
+ this.peacePeriod = peacePeriod
+ this
+ }
+
+ /**
+ * Set the number of batches to compute significance tests over.
+ * A value of 0 will use all batches seen so far.
+ */
+ @Since("1.6.0")
+ def setWindowSize(windowSize: Int): this.type = {
+ this.windowSize = windowSize
+ this
+ }
+
+ /** Set the statistical method used for significance testing. */
+ @Since("1.6.0")
+ def setTestMethod(method: String): this.type = {
+ this.testMethod = StreamingTestMethod.getTestMethodFromName(method)
+ this
+ }
+
+ /**
+ * Register a [[DStream]] of values for significance testing.
+ *
+ * @param data stream of (key,value) pairs where the key is the group membership (control or
+ * treatment) and the value is the numerical metric to test for significance
+ * @return stream of significance testing results
+ */
+ @Since("1.6.0")
+ def registerStream(data: DStream[(Boolean, Double)]): DStream[StreamingTestResult] = {
+ val dataAfterPeacePeriod = dropPeacePeriod(data)
+ val summarizedData = summarizeByKeyAndWindow(dataAfterPeacePeriod)
+ val pairedSummaries = pairSummaries(summarizedData)
+ val testResults = testMethod.doTest(pairedSummaries)
+
+ testResults
+ }
+
+ /** Drop all batches inside the peace period. */
+ private[stat] def dropPeacePeriod(
+ data: DStream[(Boolean, Double)]): DStream[(Boolean, Double)] = {
+ data.transform { (rdd, time) =>
+ if (time.milliseconds > data.slideDuration.milliseconds * peacePeriod) {
+ rdd
+ } else {
+ rdd.filter(_ => false) // TODO: Is there a better way to drop a RDD from a DStream?
+ }
+ }
+ }
+
+ /** Compute summary statistics over each key and the specified test window size. */
+ private[stat] def summarizeByKeyAndWindow(
+ data: DStream[(Boolean, Double)]): DStream[(Boolean, StatCounter)] = {
+ if (this.windowSize == 0) {
+ data.updateStateByKey[StatCounter](
+ (newValues: Seq[Double], oldSummary: Option[StatCounter]) => {
+ val newSummary = oldSummary.getOrElse(new StatCounter())
+ newSummary.merge(newValues)
+ Some(newSummary)
+ })
+ } else {
+ val windowDuration = data.slideDuration * this.windowSize
+ data
+ .groupByKeyAndWindow(windowDuration)
+ .mapValues { values =>
+ val summary = new StatCounter()
+ values.foreach(value => summary.merge(value))
+ summary
+ }
+ }
+ }
+
+ /**
+ * Transform a stream of summaries into pairs representing summary statistics for group A and
+ * group B up to this batch.
+ */
+ private[stat] def pairSummaries(summarizedData: DStream[(Boolean, StatCounter)])
+ : DStream[(StatCounter, StatCounter)] = {
+ summarizedData
+ .map[(Int, StatCounter)](x => (0, x._2))
+ .groupByKey() // Iterable[StatCounter] should be length two, one for each A/B group
+ .map(x => (x._2.head, x._2.last) )
--- End diff --
OK
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3147][MLLib] A/B testing
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/4716#issuecomment-75701760
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27880/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3147][MLLib] A/B testing
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4716#issuecomment-75701749
[Test build #27880 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27880/consoleFull) for PR 4716 at commit [`7ce63c8`](https://github.com/apache/spark/commit/7ce63c813bb2b82b4771ffbc6d7ec5bb5900ce94).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3147][MLLib][Streaming] Streaming 2-sam...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/4716#discussion_r39824241
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/stat/test/StreamingTest.scala ---
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.stat.test
+
+import org.apache.spark.Logging
+import org.apache.spark.annotation.{Experimental, Since}
+import org.apache.spark.streaming.dstream.DStream
+import org.apache.spark.util.StatCounter
+
+/**
+ * :: Experimental ::
+ * Performs online 2-sample significance testing for a stream of (Boolean, Double) pairs. The
+ * Boolean identifies which sample each observation comes from, and the Double is the numeric value
+ * of the observation.
+ *
+ * To address novelty affects, the `peacePeriod` specifies a set number of initial
+ * [[org.apache.spark.rdd.RDD]] batches of the [[DStream]] to be dropped from significance testing.
+ *
+ * The `windowSize` sets the number of batches each significance test is to be performed over. The
+ * window is sliding with a stride length of 1 batch. Setting windowSize to 0 will perform
+ * cumulative processing, using all batches seen so far.
+ *
+ * Different tests may be used for assessing statistical significance depending on assumptions
+ * satisfied by data. For more details, see [[StreamingTestMethod]]. The `testMethod` specifies
+ * which test will be used.
+ *
+ * Use a builder pattern to construct a streaming test in an application, for example:
+ * ```
+ * val model = new OnlineABTest()
+ * .setPeacePeriod(10)
+ * .setWindowSize(0)
+ * .setTestMethod("welch")
+ * .registerStream(DStream)
+ * ```
+ */
+@Experimental
+@Since("1.6.0")
+class StreamingTest(
--- End diff --
add since version to constructor as well: `class StreamingTest @Since("1.6.0") (`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3147][MLLib][Streaming] Streaming A/B t...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/4716#issuecomment-139640196
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3147][MLLib][Streaming] Streaming 2-sam...
Posted by feynmanliang <gi...@git.apache.org>.
Github user feynmanliang commented on a diff in the pull request:
https://github.com/apache/spark/pull/4716#discussion_r39896437
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/stat/test/StreamingTest.scala ---
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.stat.test
+
+import org.apache.spark.Logging
+import org.apache.spark.annotation.{Experimental, Since}
+import org.apache.spark.streaming.dstream.DStream
+import org.apache.spark.util.StatCounter
+
+/**
+ * :: Experimental ::
+ * Performs online 2-sample significance testing for a stream of (Boolean, Double) pairs. The
+ * Boolean identifies which sample each observation comes from, and the Double is the numeric value
+ * of the observation.
+ *
+ * To address novelty affects, the `peacePeriod` specifies a set number of initial
+ * [[org.apache.spark.rdd.RDD]] batches of the [[DStream]] to be dropped from significance testing.
+ *
+ * The `windowSize` sets the number of batches each significance test is to be performed over. The
+ * window is sliding with a stride length of 1 batch. Setting windowSize to 0 will perform
+ * cumulative processing, using all batches seen so far.
+ *
+ * Different tests may be used for assessing statistical significance depending on assumptions
+ * satisfied by data. For more details, see [[StreamingTestMethod]]. The `testMethod` specifies
+ * which test will be used.
+ *
+ * Use a builder pattern to construct a streaming test in an application, for example:
+ * ```
+ * val model = new OnlineABTest()
+ * .setPeacePeriod(10)
+ * .setWindowSize(0)
+ * .setTestMethod("welch")
+ * .registerStream(DStream)
+ * ```
+ */
+@Experimental
+@Since("1.6.0")
+class StreamingTest(
+ @Since("1.6.0") var peacePeriod: Int = 0,
+ @Since("1.6.0") var windowSize: Int = 0,
+ @Since("1.6.0") var testMethod: StreamingTestMethod = WelchTTest)
+ extends Logging with Serializable {
+
+ /** Set the number of initial batches to ignore. */
--- End diff --
OK
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3147][MLLib][Streaming] Streaming 2-sam...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4716#issuecomment-141572238
[Test build #42689 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42689/console) for PR 4716 at commit [`ba71bfa`](https://github.com/apache/spark/commit/ba71bfad58d6aedb193e0f7b0cf32747d6a59ce2).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3147][MLLib][Streaming] Streaming A/B t...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/4716#issuecomment-139640243
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3147][MLLib][Streaming] Streaming 2-sam...
Posted by feynmanliang <gi...@git.apache.org>.
Github user feynmanliang commented on a diff in the pull request:
https://github.com/apache/spark/pull/4716#discussion_r39897325
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/stat/test/StreamingTestMethod.scala ---
@@ -0,0 +1,165 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.stat.test
+
+import java.io.Serializable
+
+import scala.language.implicitConversions
+import scala.math.pow
+
+import com.twitter.chill.MeatLocker
+import org.apache.commons.math3.stat.descriptive.StatisticalSummaryValues
+import org.apache.commons.math3.stat.inference.TTest
+
+import org.apache.spark.Logging
+import org.apache.spark.streaming.dstream.DStream
+import org.apache.spark.util.StatCounter
+
+/**
+ * Significance testing methods for [[StreamingTest]]. New 2-sample statistical significance tests
+ * should extend [[StreamingTestMethod]] and introduce a new entry in
+ * [[StreamingTestMethod.TEST_NAME_TO_OBJECT]]
+ */
+private[stat] sealed trait StreamingTestMethod extends Serializable {
+
+ val MethodName: String
+ val NullHypothesis: String
+
+ protected type SummaryPairStream =
+ DStream[(StatCounter, StatCounter)]
+
+ /**
+ * Perform streaming 2-sample statistical significance testing.
+ *
+ * @param sampleSummaries stream pairs of summary statistics for the 2 samples
+ * @return stream of rest results
+ */
+ def doTest(sampleSummaries: SummaryPairStream): DStream[StreamingTestResult]
+
+ /**
+ * Implicit adapter to convert between streaming summary statistics type and the type required by
+ * the t-testing libraries.
+ */
+ protected implicit def toApacheCommonsStats(
+ summaryStats: StatCounter): StatisticalSummaryValues = {
+ new StatisticalSummaryValues(
+ summaryStats.mean,
+ summaryStats.variance,
+ summaryStats.count,
+ summaryStats.max,
+ summaryStats.min,
+ summaryStats.mean * summaryStats.count
+ )
+ }
+}
+
+/**
+ * Performs Welch's 2-sample t-test. The null hypothesis is that the two data sets have equal mean.
+ * This test does not assume equal variance between the two samples and does not assume equal
+ * sample size.
+ *
+ * More information: http://en.wikipedia.org/wiki/Welch%27s_t_test
+ */
+private[stat] object WelchTTest extends StreamingTestMethod with Logging {
+
+ final val MethodName = "Welch's 2-sample T-test"
+ final val NullHypothesis = "Both groups have same mean"
+
+ private final val TTester = MeatLocker(new TTest())
+
+ def doTest(data: SummaryPairStream): DStream[StreamingTestResult] =
+ data.map[StreamingTestResult]((test _).tupled)
+
+ private def test(
+ statsA: StatCounter,
+ statsB: StatCounter): StreamingTestResult = {
+ def welchDF(sample1: StatisticalSummaryValues, sample2: StatisticalSummaryValues): Double = {
+ val s1 = sample1.getVariance
+ val n1 = sample1.getN
+ val s2 = sample2.getVariance
+ val n2 = sample2.getN
+
+ val a = pow(s1, 2) / n1
+ val b = pow(s2, 2) / n2
+
+ pow(a + b, 2) / ((pow(a, 2) / (n1 - 1)) + (pow(b, 2) / (n2 - 1)))
+ }
+
+ new StreamingTestResult(
+ TTester.get.tTest(statsA, statsB),
+ welchDF(statsA, statsB),
+ TTester.get.t(statsA, statsB),
+ MethodName,
+ NullHypothesis
+ )
+ }
+}
+
+/**
+ * Performs Students's 2-sample t-test. The null hypothesis is that the two data sets have equal
+ * mean. This test assumes equal variance between the two samples and does not assume equal sample
+ * size. For unequal variances, Welch's t-test should be used instead.
+ *
+ * More information: http://en.wikipedia.org/wiki/Student%27s_t-test
+ */
+private[stat] object StudentTTest extends StreamingTestMethod with Logging {
+
+ final val MethodName = "Student's 2-sample T-test"
--- End diff --
OK
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3147][MLLib][Streaming] Streaming 2-sam...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/4716#discussion_r39824392
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/stat/test/StreamingTest.scala ---
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.stat.test
+
+import org.apache.spark.Logging
+import org.apache.spark.annotation.{Experimental, Since}
+import org.apache.spark.streaming.dstream.DStream
+import org.apache.spark.util.StatCounter
+
+/**
+ * :: Experimental ::
+ * Performs online 2-sample significance testing for a stream of (Boolean, Double) pairs. The
+ * Boolean identifies which sample each observation comes from, and the Double is the numeric value
+ * of the observation.
+ *
+ * To address novelty affects, the `peacePeriod` specifies a set number of initial
+ * [[org.apache.spark.rdd.RDD]] batches of the [[DStream]] to be dropped from significance testing.
+ *
+ * The `windowSize` sets the number of batches each significance test is to be performed over. The
+ * window is sliding with a stride length of 1 batch. Setting windowSize to 0 will perform
+ * cumulative processing, using all batches seen so far.
+ *
+ * Different tests may be used for assessing statistical significance depending on assumptions
+ * satisfied by data. For more details, see [[StreamingTestMethod]]. The `testMethod` specifies
+ * which test will be used.
+ *
+ * Use a builder pattern to construct a streaming test in an application, for example:
+ * ```
+ * val model = new OnlineABTest()
+ * .setPeacePeriod(10)
+ * .setWindowSize(0)
+ * .setTestMethod("welch")
+ * .registerStream(DStream)
+ * ```
+ */
+@Experimental
+@Since("1.6.0")
+class StreamingTest(
+ @Since("1.6.0") var peacePeriod: Int = 0,
+ @Since("1.6.0") var windowSize: Int = 0,
+ @Since("1.6.0") var testMethod: StreamingTestMethod = WelchTTest)
+ extends Logging with Serializable {
+
+ /** Set the number of initial batches to ignore. */
+ @Since("1.6.0")
+ def setPeacePeriod(peacePeriod: Int): this.type = {
+ this.peacePeriod = peacePeriod
+ this
+ }
+
+ /**
+ * Set the number of batches to compute significance tests over.
+ * A value of 0 will use all batches seen so far.
+ */
+ @Since("1.6.0")
+ def setWindowSize(windowSize: Int): this.type = {
+ this.windowSize = windowSize
+ this
+ }
+
+ /** Set the statistical method used for significance testing. */
+ @Since("1.6.0")
+ def setTestMethod(method: String): this.type = {
+ this.testMethod = StreamingTestMethod.getTestMethodFromName(method)
+ this
+ }
+
+ /**
+ * Register a [[DStream]] of values for significance testing.
+ *
+ * @param data stream of (key,value) pairs where the key is the group membership (control or
+ * treatment) and the value is the numerical metric to test for significance
+ * @return stream of significance testing results
+ */
+ @Since("1.6.0")
+ def registerStream(data: DStream[(Boolean, Double)]): DStream[StreamingTestResult] = {
+ val dataAfterPeacePeriod = dropPeacePeriod(data)
+ val summarizedData = summarizeByKeyAndWindow(dataAfterPeacePeriod)
+ val pairedSummaries = pairSummaries(summarizedData)
+ val testResults = testMethod.doTest(pairedSummaries)
+
+ testResults
+ }
+
+ /** Drop all batches inside the peace period. */
+ private[stat] def dropPeacePeriod(
+ data: DStream[(Boolean, Double)]): DStream[(Boolean, Double)] = {
+ data.transform { (rdd, time) =>
+ if (time.milliseconds > data.slideDuration.milliseconds * peacePeriod) {
+ rdd
+ } else {
+ rdd.filter(_ => false) // TODO: Is there a better way to drop a RDD from a DStream?
+ }
+ }
+ }
+
+ /** Compute summary statistics over each key and the specified test window size. */
+ private[stat] def summarizeByKeyAndWindow(
+ data: DStream[(Boolean, Double)]): DStream[(Boolean, StatCounter)] = {
+ if (this.windowSize == 0) {
+ data.updateStateByKey[StatCounter](
+ (newValues: Seq[Double], oldSummary: Option[StatCounter]) => {
+ val newSummary = oldSummary.getOrElse(new StatCounter())
+ newSummary.merge(newValues)
+ Some(newSummary)
+ })
+ } else {
+ val windowDuration = data.slideDuration * this.windowSize
+ data
+ .groupByKeyAndWindow(windowDuration)
+ .mapValues { values =>
+ val summary = new StatCounter()
+ values.foreach(value => summary.merge(value))
+ summary
+ }
+ }
+ }
+
+ /**
+ * Transform a stream of summaries into pairs representing summary statistics for group A and
+ * group B up to this batch.
+ */
+ private[stat] def pairSummaries(summarizedData: DStream[(Boolean, StatCounter)])
+ : DStream[(StatCounter, StatCounter)] = {
+ summarizedData
+ .map[(Int, StatCounter)](x => (0, x._2))
+ .groupByKey() // Iterable[StatCounter] should be length two, one for each A/B group
--- End diff --
A/B -> control/experiment
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3147][MLLib] A/B testing
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4716#issuecomment-75695983
[Test build #27880 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27880/consoleFull) for PR 4716 at commit [`7ce63c8`](https://github.com/apache/spark/commit/7ce63c813bb2b82b4771ffbc6d7ec5bb5900ce94).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3147][MLLib][Streaming] Streaming 2-sam...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/4716#issuecomment-139653470
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3147][MLLib][Streaming] Streaming 2-sam...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/4716#issuecomment-139649904
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3147][MLLib][Streaming] Streaming 2-sam...
Posted by feynmanliang <gi...@git.apache.org>.
Github user feynmanliang commented on a diff in the pull request:
https://github.com/apache/spark/pull/4716#discussion_r39896849
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/stat/test/StreamingTest.scala ---
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.stat.test
+
+import org.apache.spark.Logging
+import org.apache.spark.annotation.{Experimental, Since}
+import org.apache.spark.streaming.dstream.DStream
+import org.apache.spark.util.StatCounter
+
+/**
+ * :: Experimental ::
+ * Performs online 2-sample significance testing for a stream of (Boolean, Double) pairs. The
+ * Boolean identifies which sample each observation comes from, and the Double is the numeric value
+ * of the observation.
+ *
+ * To address novelty affects, the `peacePeriod` specifies a set number of initial
+ * [[org.apache.spark.rdd.RDD]] batches of the [[DStream]] to be dropped from significance testing.
+ *
+ * The `windowSize` sets the number of batches each significance test is to be performed over. The
+ * window is sliding with a stride length of 1 batch. Setting windowSize to 0 will perform
+ * cumulative processing, using all batches seen so far.
+ *
+ * Different tests may be used for assessing statistical significance depending on assumptions
+ * satisfied by data. For more details, see [[StreamingTestMethod]]. The `testMethod` specifies
+ * which test will be used.
+ *
+ * Use a builder pattern to construct a streaming test in an application, for example:
+ * ```
+ * val model = new OnlineABTest()
+ * .setPeacePeriod(10)
+ * .setWindowSize(0)
+ * .setTestMethod("welch")
+ * .registerStream(DStream)
+ * ```
+ */
+@Experimental
+@Since("1.6.0")
+class StreamingTest(
+ @Since("1.6.0") var peacePeriod: Int = 0,
+ @Since("1.6.0") var windowSize: Int = 0,
+ @Since("1.6.0") var testMethod: StreamingTestMethod = WelchTTest)
+ extends Logging with Serializable {
+
+ /** Set the number of initial batches to ignore. */
+ @Since("1.6.0")
+ def setPeacePeriod(peacePeriod: Int): this.type = {
+ this.peacePeriod = peacePeriod
+ this
+ }
+
+ /**
+ * Set the number of batches to compute significance tests over.
+ * A value of 0 will use all batches seen so far.
+ */
+ @Since("1.6.0")
+ def setWindowSize(windowSize: Int): this.type = {
+ this.windowSize = windowSize
+ this
+ }
+
+ /** Set the statistical method used for significance testing. */
+ @Since("1.6.0")
+ def setTestMethod(method: String): this.type = {
+ this.testMethod = StreamingTestMethod.getTestMethodFromName(method)
+ this
+ }
+
+ /**
+ * Register a [[DStream]] of values for significance testing.
+ *
+ * @param data stream of (key,value) pairs where the key is the group membership (control or
+ * treatment) and the value is the numerical metric to test for significance
+ * @return stream of significance testing results
+ */
+ @Since("1.6.0")
+ def registerStream(data: DStream[(Boolean, Double)]): DStream[StreamingTestResult] = {
+ val dataAfterPeacePeriod = dropPeacePeriod(data)
+ val summarizedData = summarizeByKeyAndWindow(dataAfterPeacePeriod)
+ val pairedSummaries = pairSummaries(summarizedData)
+ val testResults = testMethod.doTest(pairedSummaries)
+
+ testResults
+ }
+
+ /** Drop all batches inside the peace period. */
+ private[stat] def dropPeacePeriod(
+ data: DStream[(Boolean, Double)]): DStream[(Boolean, Double)] = {
+ data.transform { (rdd, time) =>
+ if (time.milliseconds > data.slideDuration.milliseconds * peacePeriod) {
+ rdd
+ } else {
+ rdd.filter(_ => false) // TODO: Is there a better way to drop a RDD from a DStream?
--- End diff --
OK
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3147][MLLib][Streaming] Streaming 2-sam...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/4716#discussion_r39824515
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/stat/test/StreamingTestMethod.scala ---
@@ -0,0 +1,165 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.stat.test
+
+import java.io.Serializable
+
+import scala.language.implicitConversions
+import scala.math.pow
+
+import com.twitter.chill.MeatLocker
+import org.apache.commons.math3.stat.descriptive.StatisticalSummaryValues
+import org.apache.commons.math3.stat.inference.TTest
+
+import org.apache.spark.Logging
+import org.apache.spark.streaming.dstream.DStream
+import org.apache.spark.util.StatCounter
+
+/**
+ * Significance testing methods for [[StreamingTest]]. New 2-sample statistical significance tests
+ * should extend [[StreamingTestMethod]] and introduce a new entry in
+ * [[StreamingTestMethod.TEST_NAME_TO_OBJECT]]
+ */
+private[stat] sealed trait StreamingTestMethod extends Serializable {
+
+ val MethodName: String
--- End diff --
`MethodName` ->` methodName`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3147][MLLib][Streaming] Streaming 2-sam...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/4716#issuecomment-141572449
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42689/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3147][MLLib][Streaming] Streaming 2-sam...
Posted by feynmanliang <gi...@git.apache.org>.
Github user feynmanliang commented on a diff in the pull request:
https://github.com/apache/spark/pull/4716#discussion_r39897481
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/stat/test/TestResult.scala ---
@@ -115,3 +115,25 @@ class KolmogorovSmirnovTestResult private[stat] (
"Kolmogorov-Smirnov test summary:\n" + super.toString
}
}
+
+/**
+ * :: Experimental ::
+ * Object containing the test results for streaming testing.
+ */
+@Experimental
+@Since("1.6.0")
+private[stat] class StreamingTestResult(
--- End diff --
OK
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3147][MLLib][Streaming] Streaming 2-sam...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/4716#issuecomment-139663153
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42353/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3147][MLLib][Streaming] Streaming 2-sam...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/4716#issuecomment-139649906
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42351/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3147][MLLib] A/B testing
Posted by feynmanliang <gi...@git.apache.org>.
Github user feynmanliang closed the pull request at:
https://github.com/apache/spark/pull/4716
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3147][MLLib] A/B testing
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the pull request:
https://github.com/apache/spark/pull/4716#issuecomment-75502716
add to whitelist
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3147][MLLib][Streaming] Streaming 2-sam...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4716#issuecomment-139649853
[Test build #42351 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42351/console) for PR 4716 at commit [`2493418`](https://github.com/apache/spark/commit/249341874112485c26c5e1965a74c60c443d13cd).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class MultilayerPerceptronClassifier(JavaEstimator, HasFeaturesCol, HasLabelCol, HasPredictionCol,`
* `class MultilayerPerceptronClassificationModel(JavaModel):`
* `class MinMaxScaler(JavaEstimator, HasInputCol, HasOutputCol):`
* `class MinMaxScalerModel(JavaModel):`
* ` ("thresholds", "Thresholds in multi-class classification to adjust the probability of " +`
* `class HasElasticNetParam(Params):`
* `class HasFitIntercept(Params):`
* `class HasStandardization(Params):`
* `class HasThresholds(Params):`
* ` thresholds = Param(Params._dummy(), "thresholds", "Thresholds in multi-class classification to adjust the probability of predicting each class. Array must have length equal to the number of classes, with values >= 0. The class with largest value p/t is predicted, where p is the original probability of that class and t is the class' threshold.")`
* ` self.thresholds = Param(self, "thresholds", "Thresholds in multi-class classification to adjust the probability of predicting each class. Array must have length equal to the number of classes, with values >= 0. The class with largest value p/t is predicted, where p is the original probability of that class and t is the class' threshold.")`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3147][MLLib][Streaming] Streaming 2-sam...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4716#issuecomment-141347889
[Test build #42642 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42642/consoleFull) for PR 4716 at commit [`60b2e57`](https://github.com/apache/spark/commit/60b2e57026febcb68e459983ba3164281a47f636).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3147][MLLib][Streaming] Streaming 2-sam...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/4716#issuecomment-141353562
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42642/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3147][MLLib] A/B testing
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the pull request:
https://github.com/apache/spark/pull/4716#issuecomment-90734623
@freeman-lab Do you want to make a pass on this PR?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: A/B testing
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/4716#issuecomment-75396732
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3147][MLLib] A/B testing
Posted by feynmanliang <gi...@git.apache.org>.
GitHub user feynmanliang reopened a pull request:
https://github.com/apache/spark/pull/4716
[SPARK-3147][MLLib] A/B testing
Implementation of A/B testing using Streaming API.
This contribution is my original work and I license the work to the project under the project's open source license.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/feynmanliang/spark ab_testing
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/4716.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #4716
----
commit 105401a89216516565236f59a66a22cc91830686
Author: Feynman Liang <fe...@gmail.com>
Date: 2015-01-10T19:36:27Z
Add broken implementation of AB testing.
commit cb73e790c435a4819fb62bc6c37717f4b882aee4
Author: Feynman Liang <fe...@gmail.com>
Date: 2015-01-10T21:07:29Z
Fix AB testing implementation and add unit tests.
commit e0d5beccf54914ebdc5663dbe4ba71944f3183e2
Author: Feynman Liang <fe...@gmail.com>
Date: 2015-01-10T22:54:26Z
Extract t-testing code out of OnlineABTesting.
commit 2100de641a2e86efeaa0f559500c7ced6f7d51a9
Author: Feynman Liang <fe...@gmail.com>
Date: 2015-01-11T04:56:30Z
Add peace period for dropping first k entries of each A/B group.
commit 708380e980ed46ac1beb7665f7854fcf36ebc403
Author: Feynman Liang <fe...@gmail.com>
Date: 2015-01-11T05:09:18Z
Add numDim to MultivariateOnlineSummarizer.
commit ec7f700fbca15d84bba126edaaa50d53ce5fc7be
Author: Feynman Liang <fe...@gmail.com>
Date: 2015-01-11T06:02:41Z
Refactored ABTestingMethod into sealed trait.
commit 3f19e15aa3b7056262b601686643ed962846cdc3
Author: Feynman Liang <fe...@gmail.com>
Date: 2015-01-11T06:29:49Z
Add (non-sliding) testing window functionality.
commit c56f9237aa81a70e8572e2ecb851ebaf5cdfa473
Author: Feynman Liang <fe...@gmail.com>
Date: 2015-01-11T15:19:46Z
Fix peace period implementation.
commit 0d738815eb1cd49096112d8be7e9124345af0604
Author: Feynman Liang <fe...@gmail.com>
Date: 2015-01-11T17:31:05Z
Fix test window batching.
commit abf59d5e8f817f847af77aef7514fb740dbbf69d
Author: Feynman Liang <fe...@gmail.com>
Date: 2015-01-11T17:56:15Z
Handle (inelegantly) closure capture for ABTestMethod
commit e05eaaf3bb21bbed4c123d9ec6514e84ae75adcb
Author: Feynman Liang <fe...@gmail.com>
Date: 2015-01-11T18:20:19Z
Improve handling of OnlineABTestMethod closure by moving DStream processing method into Serializable class.
commit 964a555746273a3afa542e34fdc6b86be60a5db9
Author: Feynman Liang <fe...@gmail.com>
Date: 2015-01-11T18:52:37Z
Fixed flaky peacePeriod test.
commit 79c1d44c6232b0a4af5df4dc14cdc83919cfdea9
Author: Feynman Liang <fe...@gmail.com>
Date: 2015-01-11T20:39:58Z
Add ScalaDocs and format to style guide.
commit e030c12337dce99abcf26f7d02c5d00a78f58c9b
Author: Feynman Liang <fe...@gmail.com>
Date: 2015-01-12T00:02:20Z
Add OnlineABTestExample.
commit e8e1f82b16fbdd8446e21b32bb39b413e1ae30d1
Author: Feynman Liang <fe...@gmail.com>
Date: 2015-01-12T00:03:12Z
Format code to style guide.
commit 17eef4eb22d918198dd03f2a931f009863fadcf5
Author: Feynman Liang <fe...@gmail.com>
Date: 2015-01-19T04:43:36Z
Switch MultivariateOnlineSummarizer to univariate StatsCounter.
commit a2ad38be8a77eef045581282b3dbc9d6a1544870
Author: Feynman Liang <fe...@gmail.com>
Date: 2015-01-19T14:45:15Z
Reduce number of passes in pairSummaries.
commit 4bb8636e5317a542ff0b29270548bd933199c6eb
Author: Feynman Liang <fe...@gmail.com>
Date: 2015-01-19T14:45:41Z
Add test for behavior when missing data from one group.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3147][MLLib][Streaming] Streaming 2-sam...
Posted by feynmanliang <gi...@git.apache.org>.
Github user feynmanliang commented on a diff in the pull request:
https://github.com/apache/spark/pull/4716#discussion_r39896160
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/stat/test/StreamingTest.scala ---
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.stat.test
+
+import org.apache.spark.Logging
+import org.apache.spark.annotation.{Experimental, Since}
+import org.apache.spark.streaming.dstream.DStream
+import org.apache.spark.util.StatCounter
+
+/**
+ * :: Experimental ::
+ * Performs online 2-sample significance testing for a stream of (Boolean, Double) pairs. The
+ * Boolean identifies which sample each observation comes from, and the Double is the numeric value
+ * of the observation.
+ *
+ * To address novelty affects, the `peacePeriod` specifies a set number of initial
+ * [[org.apache.spark.rdd.RDD]] batches of the [[DStream]] to be dropped from significance testing.
+ *
+ * The `windowSize` sets the number of batches each significance test is to be performed over. The
+ * window is sliding with a stride length of 1 batch. Setting windowSize to 0 will perform
+ * cumulative processing, using all batches seen so far.
+ *
+ * Different tests may be used for assessing statistical significance depending on assumptions
+ * satisfied by data. For more details, see [[StreamingTestMethod]]. The `testMethod` specifies
+ * which test will be used.
+ *
+ * Use a builder pattern to construct a streaming test in an application, for example:
+ * ```
+ * val model = new OnlineABTest()
--- End diff --
OK
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3147][MLLib][Streaming] Streaming 2-sam...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4716#issuecomment-141353499
[Test build #42642 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42642/console) for PR 4716 at commit [`60b2e57`](https://github.com/apache/spark/commit/60b2e57026febcb68e459983ba3164281a47f636).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3147][MLLib][Streaming] Streaming 2-sam...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/4716#issuecomment-141572446
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3147][MLLib][Streaming] Streaming 2-sam...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/4716#discussion_r39824543
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/stat/test/StreamingTestMethod.scala ---
@@ -0,0 +1,165 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.stat.test
+
+import java.io.Serializable
+
+import scala.language.implicitConversions
+import scala.math.pow
+
+import com.twitter.chill.MeatLocker
+import org.apache.commons.math3.stat.descriptive.StatisticalSummaryValues
+import org.apache.commons.math3.stat.inference.TTest
+
+import org.apache.spark.Logging
+import org.apache.spark.streaming.dstream.DStream
+import org.apache.spark.util.StatCounter
+
+/**
+ * Significance testing methods for [[StreamingTest]]. New 2-sample statistical significance tests
+ * should extend [[StreamingTestMethod]] and introduce a new entry in
+ * [[StreamingTestMethod.TEST_NAME_TO_OBJECT]]
+ */
+private[stat] sealed trait StreamingTestMethod extends Serializable {
+
+ val MethodName: String
+ val NullHypothesis: String
+
+ protected type SummaryPairStream =
+ DStream[(StatCounter, StatCounter)]
+
+ /**
+ * Perform streaming 2-sample statistical significance testing.
+ *
+ * @param sampleSummaries stream pairs of summary statistics for the 2 samples
+ * @return stream of rest results
+ */
+ def doTest(sampleSummaries: SummaryPairStream): DStream[StreamingTestResult]
+
+ /**
+ * Implicit adapter to convert between streaming summary statistics type and the type required by
+ * the t-testing libraries.
+ */
+ protected implicit def toApacheCommonsStats(
+ summaryStats: StatCounter): StatisticalSummaryValues = {
+ new StatisticalSummaryValues(
+ summaryStats.mean,
+ summaryStats.variance,
+ summaryStats.count,
+ summaryStats.max,
+ summaryStats.min,
+ summaryStats.mean * summaryStats.count
+ )
+ }
+}
+
+/**
+ * Performs Welch's 2-sample t-test. The null hypothesis is that the two data sets have equal mean.
+ * This test does not assume equal variance between the two samples and does not assume equal
+ * sample size.
+ *
+ * More information: http://en.wikipedia.org/wiki/Welch%27s_t_test
+ */
+private[stat] object WelchTTest extends StreamingTestMethod with Logging {
+
+ final val MethodName = "Welch's 2-sample T-test"
+ final val NullHypothesis = "Both groups have same mean"
+
+ private final val TTester = MeatLocker(new TTest())
--- End diff --
`tTester`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3147][MLLib][Streaming] Streaming 2-sam...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/4716#discussion_r39824244
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/stat/test/StreamingTest.scala ---
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.stat.test
+
+import org.apache.spark.Logging
+import org.apache.spark.annotation.{Experimental, Since}
+import org.apache.spark.streaming.dstream.DStream
+import org.apache.spark.util.StatCounter
+
+/**
+ * :: Experimental ::
+ * Performs online 2-sample significance testing for a stream of (Boolean, Double) pairs. The
+ * Boolean identifies which sample each observation comes from, and the Double is the numeric value
+ * of the observation.
+ *
+ * To address novelty affects, the `peacePeriod` specifies a set number of initial
+ * [[org.apache.spark.rdd.RDD]] batches of the [[DStream]] to be dropped from significance testing.
+ *
+ * The `windowSize` sets the number of batches each significance test is to be performed over. The
+ * window is sliding with a stride length of 1 batch. Setting windowSize to 0 will perform
+ * cumulative processing, using all batches seen so far.
+ *
+ * Different tests may be used for assessing statistical significance depending on assumptions
+ * satisfied by data. For more details, see [[StreamingTestMethod]]. The `testMethod` specifies
+ * which test will be used.
+ *
+ * Use a builder pattern to construct a streaming test in an application, for example:
+ * ```
+ * val model = new OnlineABTest()
+ * .setPeacePeriod(10)
+ * .setWindowSize(0)
+ * .setTestMethod("welch")
+ * .registerStream(DStream)
+ * ```
+ */
+@Experimental
+@Since("1.6.0")
+class StreamingTest(
+ @Since("1.6.0") var peacePeriod: Int = 0,
--- End diff --
The default values are not Java friendly. Since we already have setters, we can make a default constructor with no arguments.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3147][MLLib] A/B testing
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/4716#issuecomment-75511267
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27851/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3147][MLLib] A/B testing
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the pull request:
https://github.com/apache/spark/pull/4716#issuecomment-75580259
Let's remove `numDim`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3147][MLLib][Streaming] Streaming 2-sam...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the pull request:
https://github.com/apache/spark/pull/4716#issuecomment-142095949
LGTM. Merged into master. Thanks! Sorry for the long delay!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3147][MLLib] A/B testing
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the pull request:
https://github.com/apache/spark/pull/4716#issuecomment-75502732
ok to test
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3147][MLLib][Streaming] Streaming 2-sam...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/4716#discussion_r39824247
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/stat/test/StreamingTest.scala ---
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.stat.test
+
+import org.apache.spark.Logging
+import org.apache.spark.annotation.{Experimental, Since}
+import org.apache.spark.streaming.dstream.DStream
+import org.apache.spark.util.StatCounter
+
+/**
+ * :: Experimental ::
+ * Performs online 2-sample significance testing for a stream of (Boolean, Double) pairs. The
+ * Boolean identifies which sample each observation comes from, and the Double is the numeric value
+ * of the observation.
+ *
+ * To address novelty affects, the `peacePeriod` specifies a set number of initial
+ * [[org.apache.spark.rdd.RDD]] batches of the [[DStream]] to be dropped from significance testing.
+ *
+ * The `windowSize` sets the number of batches each significance test is to be performed over. The
+ * window is sliding with a stride length of 1 batch. Setting windowSize to 0 will perform
+ * cumulative processing, using all batches seen so far.
+ *
+ * Different tests may be used for assessing statistical significance depending on assumptions
+ * satisfied by data. For more details, see [[StreamingTestMethod]]. The `testMethod` specifies
+ * which test will be used.
+ *
+ * Use a builder pattern to construct a streaming test in an application, for example:
+ * ```
+ * val model = new OnlineABTest()
+ * .setPeacePeriod(10)
+ * .setWindowSize(0)
+ * .setTestMethod("welch")
+ * .registerStream(DStream)
+ * ```
+ */
+@Experimental
+@Since("1.6.0")
+class StreamingTest(
+ @Since("1.6.0") var peacePeriod: Int = 0,
+ @Since("1.6.0") var windowSize: Int = 0,
+ @Since("1.6.0") var testMethod: StreamingTestMethod = WelchTTest)
+ extends Logging with Serializable {
+
+ /** Set the number of initial batches to ignore. */
--- End diff --
document default value
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3147][MLLib][Streaming] Streaming 2-sam...
Posted by feynmanliang <gi...@git.apache.org>.
Github user feynmanliang commented on a diff in the pull request:
https://github.com/apache/spark/pull/4716#discussion_r39896288
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/stat/test/StreamingTest.scala ---
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.stat.test
+
+import org.apache.spark.Logging
+import org.apache.spark.annotation.{Experimental, Since}
+import org.apache.spark.streaming.dstream.DStream
+import org.apache.spark.util.StatCounter
+
+/**
+ * :: Experimental ::
+ * Performs online 2-sample significance testing for a stream of (Boolean, Double) pairs. The
+ * Boolean identifies which sample each observation comes from, and the Double is the numeric value
+ * of the observation.
+ *
+ * To address novelty affects, the `peacePeriod` specifies a set number of initial
+ * [[org.apache.spark.rdd.RDD]] batches of the [[DStream]] to be dropped from significance testing.
+ *
+ * The `windowSize` sets the number of batches each significance test is to be performed over. The
+ * window is sliding with a stride length of 1 batch. Setting windowSize to 0 will perform
+ * cumulative processing, using all batches seen so far.
+ *
+ * Different tests may be used for assessing statistical significance depending on assumptions
+ * satisfied by data. For more details, see [[StreamingTestMethod]]. The `testMethod` specifies
+ * which test will be used.
+ *
+ * Use a builder pattern to construct a streaming test in an application, for example:
+ * ```
+ * val model = new OnlineABTest()
+ * .setPeacePeriod(10)
+ * .setWindowSize(0)
+ * .setTestMethod("welch")
+ * .registerStream(DStream)
+ * ```
+ */
+@Experimental
+@Since("1.6.0")
+class StreamingTest(
+ @Since("1.6.0") var peacePeriod: Int = 0,
--- End diff --
OK
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3147][MLLib][Streaming] Streaming A/B t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4716#issuecomment-139643242
[Test build #42351 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42351/consoleFull) for PR 4716 at commit [`2493418`](https://github.com/apache/spark/commit/249341874112485c26c5e1965a74c60c443d13cd).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3147][MLLib][Streaming] Streaming 2-sam...
Posted by feynmanliang <gi...@git.apache.org>.
Github user feynmanliang commented on a diff in the pull request:
https://github.com/apache/spark/pull/4716#discussion_r39896998
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/stat/test/StreamingTest.scala ---
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.stat.test
+
+import org.apache.spark.Logging
+import org.apache.spark.annotation.{Experimental, Since}
+import org.apache.spark.streaming.dstream.DStream
+import org.apache.spark.util.StatCounter
+
+/**
+ * :: Experimental ::
+ * Performs online 2-sample significance testing for a stream of (Boolean, Double) pairs. The
+ * Boolean identifies which sample each observation comes from, and the Double is the numeric value
+ * of the observation.
+ *
+ * To address novelty affects, the `peacePeriod` specifies a set number of initial
+ * [[org.apache.spark.rdd.RDD]] batches of the [[DStream]] to be dropped from significance testing.
+ *
+ * The `windowSize` sets the number of batches each significance test is to be performed over. The
+ * window is sliding with a stride length of 1 batch. Setting windowSize to 0 will perform
+ * cumulative processing, using all batches seen so far.
+ *
+ * Different tests may be used for assessing statistical significance depending on assumptions
+ * satisfied by data. For more details, see [[StreamingTestMethod]]. The `testMethod` specifies
+ * which test will be used.
+ *
+ * Use a builder pattern to construct a streaming test in an application, for example:
+ * ```
+ * val model = new OnlineABTest()
+ * .setPeacePeriod(10)
+ * .setWindowSize(0)
+ * .setTestMethod("welch")
+ * .registerStream(DStream)
+ * ```
+ */
+@Experimental
+@Since("1.6.0")
+class StreamingTest(
+ @Since("1.6.0") var peacePeriod: Int = 0,
+ @Since("1.6.0") var windowSize: Int = 0,
+ @Since("1.6.0") var testMethod: StreamingTestMethod = WelchTTest)
+ extends Logging with Serializable {
+
+ /** Set the number of initial batches to ignore. */
+ @Since("1.6.0")
+ def setPeacePeriod(peacePeriod: Int): this.type = {
+ this.peacePeriod = peacePeriod
+ this
+ }
+
+ /**
+ * Set the number of batches to compute significance tests over.
+ * A value of 0 will use all batches seen so far.
+ */
+ @Since("1.6.0")
+ def setWindowSize(windowSize: Int): this.type = {
+ this.windowSize = windowSize
+ this
+ }
+
+ /** Set the statistical method used for significance testing. */
+ @Since("1.6.0")
+ def setTestMethod(method: String): this.type = {
+ this.testMethod = StreamingTestMethod.getTestMethodFromName(method)
+ this
+ }
+
+ /**
+ * Register a [[DStream]] of values for significance testing.
+ *
+ * @param data stream of (key,value) pairs where the key is the group membership (control or
+ * treatment) and the value is the numerical metric to test for significance
+ * @return stream of significance testing results
+ */
+ @Since("1.6.0")
+ def registerStream(data: DStream[(Boolean, Double)]): DStream[StreamingTestResult] = {
+ val dataAfterPeacePeriod = dropPeacePeriod(data)
+ val summarizedData = summarizeByKeyAndWindow(dataAfterPeacePeriod)
+ val pairedSummaries = pairSummaries(summarizedData)
+ val testResults = testMethod.doTest(pairedSummaries)
+
+ testResults
+ }
+
+ /** Drop all batches inside the peace period. */
+ private[stat] def dropPeacePeriod(
+ data: DStream[(Boolean, Double)]): DStream[(Boolean, Double)] = {
+ data.transform { (rdd, time) =>
+ if (time.milliseconds > data.slideDuration.milliseconds * peacePeriod) {
+ rdd
+ } else {
+ rdd.filter(_ => false) // TODO: Is there a better way to drop a RDD from a DStream?
+ }
+ }
+ }
+
+ /** Compute summary statistics over each key and the specified test window size. */
+ private[stat] def summarizeByKeyAndWindow(
+ data: DStream[(Boolean, Double)]): DStream[(Boolean, StatCounter)] = {
+ if (this.windowSize == 0) {
+ data.updateStateByKey[StatCounter](
+ (newValues: Seq[Double], oldSummary: Option[StatCounter]) => {
+ val newSummary = oldSummary.getOrElse(new StatCounter())
+ newSummary.merge(newValues)
+ Some(newSummary)
+ })
+ } else {
+ val windowDuration = data.slideDuration * this.windowSize
+ data
+ .groupByKeyAndWindow(windowDuration)
+ .mapValues { values =>
+ val summary = new StatCounter()
+ values.foreach(value => summary.merge(value))
+ summary
+ }
+ }
+ }
+
+ /**
+ * Transform a stream of summaries into pairs representing summary statistics for group A and
--- End diff --
OK
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3147][MLLib][Streaming] Streaming 2-sam...
Posted by feynmanliang <gi...@git.apache.org>.
Github user feynmanliang commented on a diff in the pull request:
https://github.com/apache/spark/pull/4716#discussion_r39896449
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/stat/test/StreamingTest.scala ---
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.stat.test
+
+import org.apache.spark.Logging
+import org.apache.spark.annotation.{Experimental, Since}
+import org.apache.spark.streaming.dstream.DStream
+import org.apache.spark.util.StatCounter
+
+/**
+ * :: Experimental ::
+ * Performs online 2-sample significance testing for a stream of (Boolean, Double) pairs. The
+ * Boolean identifies which sample each observation comes from, and the Double is the numeric value
+ * of the observation.
+ *
+ * To address novelty affects, the `peacePeriod` specifies a set number of initial
+ * [[org.apache.spark.rdd.RDD]] batches of the [[DStream]] to be dropped from significance testing.
+ *
+ * The `windowSize` sets the number of batches each significance test is to be performed over. The
+ * window is sliding with a stride length of 1 batch. Setting windowSize to 0 will perform
+ * cumulative processing, using all batches seen so far.
+ *
+ * Different tests may be used for assessing statistical significance depending on assumptions
+ * satisfied by data. For more details, see [[StreamingTestMethod]]. The `testMethod` specifies
+ * which test will be used.
+ *
+ * Use a builder pattern to construct a streaming test in an application, for example:
+ * ```
+ * val model = new OnlineABTest()
+ * .setPeacePeriod(10)
+ * .setWindowSize(0)
+ * .setTestMethod("welch")
+ * .registerStream(DStream)
+ * ```
+ */
+@Experimental
+@Since("1.6.0")
+class StreamingTest(
+ @Since("1.6.0") var peacePeriod: Int = 0,
+ @Since("1.6.0") var windowSize: Int = 0,
+ @Since("1.6.0") var testMethod: StreamingTestMethod = WelchTTest)
+ extends Logging with Serializable {
+
+ /** Set the number of initial batches to ignore. */
+ @Since("1.6.0")
+ def setPeacePeriod(peacePeriod: Int): this.type = {
+ this.peacePeriod = peacePeriod
+ this
+ }
+
+ /**
+ * Set the number of batches to compute significance tests over.
+ * A value of 0 will use all batches seen so far.
+ */
+ @Since("1.6.0")
+ def setWindowSize(windowSize: Int): this.type = {
+ this.windowSize = windowSize
+ this
+ }
+
+ /** Set the statistical method used for significance testing. */
--- End diff --
OK
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3147][MLLib][Streaming] Streaming 2-sam...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/4716#discussion_r39824390
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/stat/test/StreamingTest.scala ---
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.stat.test
+
+import org.apache.spark.Logging
+import org.apache.spark.annotation.{Experimental, Since}
+import org.apache.spark.streaming.dstream.DStream
+import org.apache.spark.util.StatCounter
+
+/**
+ * :: Experimental ::
+ * Performs online 2-sample significance testing for a stream of (Boolean, Double) pairs. The
+ * Boolean identifies which sample each observation comes from, and the Double is the numeric value
+ * of the observation.
+ *
+ * To address novelty affects, the `peacePeriod` specifies a set number of initial
+ * [[org.apache.spark.rdd.RDD]] batches of the [[DStream]] to be dropped from significance testing.
+ *
+ * The `windowSize` sets the number of batches each significance test is to be performed over. The
+ * window is sliding with a stride length of 1 batch. Setting windowSize to 0 will perform
+ * cumulative processing, using all batches seen so far.
+ *
+ * Different tests may be used for assessing statistical significance depending on assumptions
+ * satisfied by data. For more details, see [[StreamingTestMethod]]. The `testMethod` specifies
+ * which test will be used.
+ *
+ * Use a builder pattern to construct a streaming test in an application, for example:
+ * ```
+ * val model = new OnlineABTest()
+ * .setPeacePeriod(10)
+ * .setWindowSize(0)
+ * .setTestMethod("welch")
+ * .registerStream(DStream)
+ * ```
+ */
+@Experimental
+@Since("1.6.0")
+class StreamingTest(
+ @Since("1.6.0") var peacePeriod: Int = 0,
+ @Since("1.6.0") var windowSize: Int = 0,
+ @Since("1.6.0") var testMethod: StreamingTestMethod = WelchTTest)
+ extends Logging with Serializable {
+
+ /** Set the number of initial batches to ignore. */
+ @Since("1.6.0")
+ def setPeacePeriod(peacePeriod: Int): this.type = {
+ this.peacePeriod = peacePeriod
+ this
+ }
+
+ /**
+ * Set the number of batches to compute significance tests over.
+ * A value of 0 will use all batches seen so far.
+ */
+ @Since("1.6.0")
+ def setWindowSize(windowSize: Int): this.type = {
+ this.windowSize = windowSize
+ this
+ }
+
+ /** Set the statistical method used for significance testing. */
+ @Since("1.6.0")
+ def setTestMethod(method: String): this.type = {
+ this.testMethod = StreamingTestMethod.getTestMethodFromName(method)
+ this
+ }
+
+ /**
+ * Register a [[DStream]] of values for significance testing.
+ *
+ * @param data stream of (key,value) pairs where the key is the group membership (control or
+ * treatment) and the value is the numerical metric to test for significance
+ * @return stream of significance testing results
+ */
+ @Since("1.6.0")
+ def registerStream(data: DStream[(Boolean, Double)]): DStream[StreamingTestResult] = {
+ val dataAfterPeacePeriod = dropPeacePeriod(data)
+ val summarizedData = summarizeByKeyAndWindow(dataAfterPeacePeriod)
+ val pairedSummaries = pairSummaries(summarizedData)
+ val testResults = testMethod.doTest(pairedSummaries)
+
+ testResults
+ }
+
+ /** Drop all batches inside the peace period. */
+ private[stat] def dropPeacePeriod(
+ data: DStream[(Boolean, Double)]): DStream[(Boolean, Double)] = {
+ data.transform { (rdd, time) =>
+ if (time.milliseconds > data.slideDuration.milliseconds * peacePeriod) {
+ rdd
+ } else {
+ rdd.filter(_ => false) // TODO: Is there a better way to drop a RDD from a DStream?
+ }
+ }
+ }
+
+ /** Compute summary statistics over each key and the specified test window size. */
+ private[stat] def summarizeByKeyAndWindow(
+ data: DStream[(Boolean, Double)]): DStream[(Boolean, StatCounter)] = {
+ if (this.windowSize == 0) {
+ data.updateStateByKey[StatCounter](
+ (newValues: Seq[Double], oldSummary: Option[StatCounter]) => {
+ val newSummary = oldSummary.getOrElse(new StatCounter())
+ newSummary.merge(newValues)
+ Some(newSummary)
+ })
+ } else {
+ val windowDuration = data.slideDuration * this.windowSize
+ data
+ .groupByKeyAndWindow(windowDuration)
+ .mapValues { values =>
+ val summary = new StatCounter()
+ values.foreach(value => summary.merge(value))
+ summary
+ }
+ }
+ }
+
+ /**
+ * Transform a stream of summaries into pairs representing summary statistics for group A and
--- End diff --
A and B -> control and experiement
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3147][MLLib][Streaming] Streaming 2-sam...
Posted by feynmanliang <gi...@git.apache.org>.
Github user feynmanliang commented on a diff in the pull request:
https://github.com/apache/spark/pull/4716#discussion_r39896444
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/stat/test/StreamingTest.scala ---
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.stat.test
+
+import org.apache.spark.Logging
+import org.apache.spark.annotation.{Experimental, Since}
+import org.apache.spark.streaming.dstream.DStream
+import org.apache.spark.util.StatCounter
+
+/**
+ * :: Experimental ::
+ * Performs online 2-sample significance testing for a stream of (Boolean, Double) pairs. The
+ * Boolean identifies which sample each observation comes from, and the Double is the numeric value
+ * of the observation.
+ *
+ * To address novelty affects, the `peacePeriod` specifies a set number of initial
+ * [[org.apache.spark.rdd.RDD]] batches of the [[DStream]] to be dropped from significance testing.
+ *
+ * The `windowSize` sets the number of batches each significance test is to be performed over. The
+ * window is sliding with a stride length of 1 batch. Setting windowSize to 0 will perform
+ * cumulative processing, using all batches seen so far.
+ *
+ * Different tests may be used for assessing statistical significance depending on assumptions
+ * satisfied by data. For more details, see [[StreamingTestMethod]]. The `testMethod` specifies
+ * which test will be used.
+ *
+ * Use a builder pattern to construct a streaming test in an application, for example:
+ * ```
+ * val model = new OnlineABTest()
+ * .setPeacePeriod(10)
+ * .setWindowSize(0)
+ * .setTestMethod("welch")
+ * .registerStream(DStream)
+ * ```
+ */
+@Experimental
+@Since("1.6.0")
+class StreamingTest(
+ @Since("1.6.0") var peacePeriod: Int = 0,
+ @Since("1.6.0") var windowSize: Int = 0,
+ @Since("1.6.0") var testMethod: StreamingTestMethod = WelchTTest)
+ extends Logging with Serializable {
+
+ /** Set the number of initial batches to ignore. */
+ @Since("1.6.0")
+ def setPeacePeriod(peacePeriod: Int): this.type = {
+ this.peacePeriod = peacePeriod
+ this
+ }
+
+ /**
+ * Set the number of batches to compute significance tests over.
+ * A value of 0 will use all batches seen so far.
--- End diff --
OK
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3147][MLLib][Streaming] Streaming 2-sam...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/4716#issuecomment-139648564
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3147][MLLib][Streaming] Streaming 2-sam...
Posted by feynmanliang <gi...@git.apache.org>.
Github user feynmanliang commented on a diff in the pull request:
https://github.com/apache/spark/pull/4716#discussion_r39897181
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/stat/test/StreamingTestMethod.scala ---
@@ -0,0 +1,165 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.stat.test
+
+import java.io.Serializable
+
+import scala.language.implicitConversions
+import scala.math.pow
+
+import com.twitter.chill.MeatLocker
+import org.apache.commons.math3.stat.descriptive.StatisticalSummaryValues
+import org.apache.commons.math3.stat.inference.TTest
+
+import org.apache.spark.Logging
+import org.apache.spark.streaming.dstream.DStream
+import org.apache.spark.util.StatCounter
+
+/**
+ * Significance testing methods for [[StreamingTest]]. New 2-sample statistical significance tests
+ * should extend [[StreamingTestMethod]] and introduce a new entry in
+ * [[StreamingTestMethod.TEST_NAME_TO_OBJECT]]
+ */
+private[stat] sealed trait StreamingTestMethod extends Serializable {
+
+ val MethodName: String
+ val NullHypothesis: String
--- End diff --
OK
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3147][MLLib][Streaming] Streaming 2-sam...
Posted by feynmanliang <gi...@git.apache.org>.
Github user feynmanliang commented on a diff in the pull request:
https://github.com/apache/spark/pull/4716#discussion_r39896729
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/stat/test/StreamingTest.scala ---
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.stat.test
+
+import org.apache.spark.Logging
+import org.apache.spark.annotation.{Experimental, Since}
+import org.apache.spark.streaming.dstream.DStream
+import org.apache.spark.util.StatCounter
+
+/**
+ * :: Experimental ::
+ * Performs online 2-sample significance testing for a stream of (Boolean, Double) pairs. The
+ * Boolean identifies which sample each observation comes from, and the Double is the numeric value
+ * of the observation.
+ *
+ * To address novelty affects, the `peacePeriod` specifies a set number of initial
+ * [[org.apache.spark.rdd.RDD]] batches of the [[DStream]] to be dropped from significance testing.
+ *
+ * The `windowSize` sets the number of batches each significance test is to be performed over. The
+ * window is sliding with a stride length of 1 batch. Setting windowSize to 0 will perform
+ * cumulative processing, using all batches seen so far.
+ *
+ * Different tests may be used for assessing statistical significance depending on assumptions
+ * satisfied by data. For more details, see [[StreamingTestMethod]]. The `testMethod` specifies
+ * which test will be used.
+ *
+ * Use a builder pattern to construct a streaming test in an application, for example:
+ * ```
+ * val model = new OnlineABTest()
+ * .setPeacePeriod(10)
+ * .setWindowSize(0)
+ * .setTestMethod("welch")
+ * .registerStream(DStream)
+ * ```
+ */
+@Experimental
+@Since("1.6.0")
+class StreamingTest(
+ @Since("1.6.0") var peacePeriod: Int = 0,
+ @Since("1.6.0") var windowSize: Int = 0,
+ @Since("1.6.0") var testMethod: StreamingTestMethod = WelchTTest)
+ extends Logging with Serializable {
+
+ /** Set the number of initial batches to ignore. */
+ @Since("1.6.0")
+ def setPeacePeriod(peacePeriod: Int): this.type = {
+ this.peacePeriod = peacePeriod
+ this
+ }
+
+ /**
+ * Set the number of batches to compute significance tests over.
+ * A value of 0 will use all batches seen so far.
+ */
+ @Since("1.6.0")
+ def setWindowSize(windowSize: Int): this.type = {
+ this.windowSize = windowSize
+ this
+ }
+
+ /** Set the statistical method used for significance testing. */
+ @Since("1.6.0")
+ def setTestMethod(method: String): this.type = {
+ this.testMethod = StreamingTestMethod.getTestMethodFromName(method)
+ this
+ }
+
+ /**
+ * Register a [[DStream]] of values for significance testing.
+ *
+ * @param data stream of (key,value) pairs where the key is the group membership (control or
+ * treatment) and the value is the numerical metric to test for significance
+ * @return stream of significance testing results
+ */
+ @Since("1.6.0")
+ def registerStream(data: DStream[(Boolean, Double)]): DStream[StreamingTestResult] = {
+ val dataAfterPeacePeriod = dropPeacePeriod(data)
+ val summarizedData = summarizeByKeyAndWindow(dataAfterPeacePeriod)
+ val pairedSummaries = pairSummaries(summarizedData)
+ val testResults = testMethod.doTest(pairedSummaries)
--- End diff --
OK
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3147][MLLib] A/B testing
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4716#issuecomment-75503155
[Test build #27851 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27851/consoleFull) for PR 4716 at commit [`4bb8636`](https://github.com/apache/spark/commit/4bb8636e5317a542ff0b29270548bd933199c6eb).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3147][MLLib][Streaming] Streaming 2-sam...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/4716#discussion_r39824320
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/stat/test/StreamingTest.scala ---
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.stat.test
+
+import org.apache.spark.Logging
+import org.apache.spark.annotation.{Experimental, Since}
+import org.apache.spark.streaming.dstream.DStream
+import org.apache.spark.util.StatCounter
+
+/**
+ * :: Experimental ::
+ * Performs online 2-sample significance testing for a stream of (Boolean, Double) pairs. The
+ * Boolean identifies which sample each observation comes from, and the Double is the numeric value
+ * of the observation.
+ *
+ * To address novelty affects, the `peacePeriod` specifies a set number of initial
+ * [[org.apache.spark.rdd.RDD]] batches of the [[DStream]] to be dropped from significance testing.
+ *
+ * The `windowSize` sets the number of batches each significance test is to be performed over. The
+ * window is sliding with a stride length of 1 batch. Setting windowSize to 0 will perform
+ * cumulative processing, using all batches seen so far.
+ *
+ * Different tests may be used for assessing statistical significance depending on assumptions
+ * satisfied by data. For more details, see [[StreamingTestMethod]]. The `testMethod` specifies
+ * which test will be used.
+ *
+ * Use a builder pattern to construct a streaming test in an application, for example:
+ * ```
+ * val model = new OnlineABTest()
+ * .setPeacePeriod(10)
+ * .setWindowSize(0)
+ * .setTestMethod("welch")
+ * .registerStream(DStream)
+ * ```
+ */
+@Experimental
+@Since("1.6.0")
+class StreamingTest(
+ @Since("1.6.0") var peacePeriod: Int = 0,
+ @Since("1.6.0") var windowSize: Int = 0,
+ @Since("1.6.0") var testMethod: StreamingTestMethod = WelchTTest)
+ extends Logging with Serializable {
+
+ /** Set the number of initial batches to ignore. */
+ @Since("1.6.0")
+ def setPeacePeriod(peacePeriod: Int): this.type = {
+ this.peacePeriod = peacePeriod
+ this
+ }
+
+ /**
+ * Set the number of batches to compute significance tests over.
+ * A value of 0 will use all batches seen so far.
+ */
+ @Since("1.6.0")
+ def setWindowSize(windowSize: Int): this.type = {
+ this.windowSize = windowSize
+ this
+ }
+
+ /** Set the statistical method used for significance testing. */
+ @Since("1.6.0")
+ def setTestMethod(method: String): this.type = {
+ this.testMethod = StreamingTestMethod.getTestMethodFromName(method)
+ this
+ }
+
+ /**
+ * Register a [[DStream]] of values for significance testing.
+ *
+ * @param data stream of (key,value) pairs where the key is the group membership (control or
+ * treatment) and the value is the numerical metric to test for significance
+ * @return stream of significance testing results
+ */
+ @Since("1.6.0")
+ def registerStream(data: DStream[(Boolean, Double)]): DStream[StreamingTestResult] = {
+ val dataAfterPeacePeriod = dropPeacePeriod(data)
+ val summarizedData = summarizeByKeyAndWindow(dataAfterPeacePeriod)
+ val pairedSummaries = pairSummaries(summarizedData)
+ val testResults = testMethod.doTest(pairedSummaries)
+
+ testResults
+ }
+
+ /** Drop all batches inside the peace period. */
+ private[stat] def dropPeacePeriod(
+ data: DStream[(Boolean, Double)]): DStream[(Boolean, Double)] = {
+ data.transform { (rdd, time) =>
+ if (time.milliseconds > data.slideDuration.milliseconds * peacePeriod) {
+ rdd
+ } else {
+ rdd.filter(_ => false) // TODO: Is there a better way to drop a RDD from a DStream?
--- End diff --
you only need an empty `RDD[(Boolean, Double)]`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3147][MLLib] A/B testing
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4716#issuecomment-75511258
[Test build #27851 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27851/consoleFull) for PR 4716 at commit [`4bb8636`](https://github.com/apache/spark/commit/4bb8636e5317a542ff0b29270548bd933199c6eb).
* This patch **fails MiMa tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3147][MLLib][Streaming] Streaming 2-sam...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/4716#discussion_r39824516
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/stat/test/StreamingTestMethod.scala ---
@@ -0,0 +1,165 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.stat.test
+
+import java.io.Serializable
+
+import scala.language.implicitConversions
+import scala.math.pow
+
+import com.twitter.chill.MeatLocker
+import org.apache.commons.math3.stat.descriptive.StatisticalSummaryValues
+import org.apache.commons.math3.stat.inference.TTest
+
+import org.apache.spark.Logging
+import org.apache.spark.streaming.dstream.DStream
+import org.apache.spark.util.StatCounter
+
+/**
+ * Significance testing methods for [[StreamingTest]]. New 2-sample statistical significance tests
+ * should extend [[StreamingTestMethod]] and introduce a new entry in
+ * [[StreamingTestMethod.TEST_NAME_TO_OBJECT]]
+ */
+private[stat] sealed trait StreamingTestMethod extends Serializable {
+
+ val MethodName: String
+ val NullHypothesis: String
--- End diff --
`nullHypothesis`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3147][MLLib][Streaming] Streaming 2-sam...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4716#issuecomment-139648558
[Test build #42352 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42352/console) for PR 4716 at commit [`b81bb53`](https://github.com/apache/spark/commit/b81bb53ae79c19e2990a7781436b83d0b53ab1a4).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class MultilayerPerceptronClassifier(JavaEstimator, HasFeaturesCol, HasLabelCol, HasPredictionCol,`
* `class MultilayerPerceptronClassificationModel(JavaModel):`
* `class MinMaxScaler(JavaEstimator, HasInputCol, HasOutputCol):`
* `class MinMaxScalerModel(JavaModel):`
* `class StringIndexer(JavaEstimator, HasInputCol, HasOutputCol, HasHandleInvalid):`
* ` ("thresholds", "Thresholds in multi-class classification to adjust the probability of " +`
* `class HasHandleInvalid(Params):`
* `class HasElasticNetParam(Params):`
* `class HasFitIntercept(Params):`
* `class HasStandardization(Params):`
* `class HasThresholds(Params):`
* ` thresholds = Param(Params._dummy(), "thresholds", "Thresholds in multi-class classification to adjust the probability of predicting each class. Array must have length equal to the number of classes, with values >= 0. The class with largest value p/t is predicted, where p is the original probability of that class and t is the class' threshold.")`
* ` self.thresholds = Param(self, "thresholds", "Thresholds in multi-class classification to adjust the probability of predicting each class. Array must have length equal to the number of classes, with values >= 0. The class with largest value p/t is predicted, where p is the original probability of that class and t is the class' threshold.")`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3147][MLLib][Streaming] Streaming 2-sam...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4716#issuecomment-139647965
[Test build #42352 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42352/consoleFull) for PR 4716 at commit [`b81bb53`](https://github.com/apache/spark/commit/b81bb53ae79c19e2990a7781436b83d0b53ab1a4).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3147][MLLib][Streaming] Streaming 2-sam...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/4716#discussion_r39824264
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/stat/test/StreamingTest.scala ---
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.stat.test
+
+import org.apache.spark.Logging
+import org.apache.spark.annotation.{Experimental, Since}
+import org.apache.spark.streaming.dstream.DStream
+import org.apache.spark.util.StatCounter
+
+/**
+ * :: Experimental ::
+ * Performs online 2-sample significance testing for a stream of (Boolean, Double) pairs. The
+ * Boolean identifies which sample each observation comes from, and the Double is the numeric value
+ * of the observation.
+ *
+ * To address novelty affects, the `peacePeriod` specifies a set number of initial
+ * [[org.apache.spark.rdd.RDD]] batches of the [[DStream]] to be dropped from significance testing.
+ *
+ * The `windowSize` sets the number of batches each significance test is to be performed over. The
+ * window is sliding with a stride length of 1 batch. Setting windowSize to 0 will perform
+ * cumulative processing, using all batches seen so far.
+ *
+ * Different tests may be used for assessing statistical significance depending on assumptions
+ * satisfied by data. For more details, see [[StreamingTestMethod]]. The `testMethod` specifies
+ * which test will be used.
+ *
+ * Use a builder pattern to construct a streaming test in an application, for example:
+ * ```
+ * val model = new OnlineABTest()
+ * .setPeacePeriod(10)
+ * .setWindowSize(0)
+ * .setTestMethod("welch")
+ * .registerStream(DStream)
+ * ```
+ */
+@Experimental
+@Since("1.6.0")
+class StreamingTest(
+ @Since("1.6.0") var peacePeriod: Int = 0,
+ @Since("1.6.0") var windowSize: Int = 0,
+ @Since("1.6.0") var testMethod: StreamingTestMethod = WelchTTest)
+ extends Logging with Serializable {
+
+ /** Set the number of initial batches to ignore. */
+ @Since("1.6.0")
+ def setPeacePeriod(peacePeriod: Int): this.type = {
+ this.peacePeriod = peacePeriod
+ this
+ }
+
+ /**
+ * Set the number of batches to compute significance tests over.
+ * A value of 0 will use all batches seen so far.
+ */
+ @Since("1.6.0")
+ def setWindowSize(windowSize: Int): this.type = {
+ this.windowSize = windowSize
+ this
+ }
+
+ /** Set the statistical method used for significance testing. */
--- End diff --
document default value and available methods
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3147][MLLib][Streaming] Streaming 2-sam...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/4716#issuecomment-139647377
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3147][MLLib][Streaming] Streaming 2-sam...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/4716#discussion_r39824260
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/stat/test/StreamingTest.scala ---
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.stat.test
+
+import org.apache.spark.Logging
+import org.apache.spark.annotation.{Experimental, Since}
+import org.apache.spark.streaming.dstream.DStream
+import org.apache.spark.util.StatCounter
+
+/**
+ * :: Experimental ::
+ * Performs online 2-sample significance testing for a stream of (Boolean, Double) pairs. The
+ * Boolean identifies which sample each observation comes from, and the Double is the numeric value
+ * of the observation.
+ *
+ * To address novelty affects, the `peacePeriod` specifies a set number of initial
+ * [[org.apache.spark.rdd.RDD]] batches of the [[DStream]] to be dropped from significance testing.
+ *
+ * The `windowSize` sets the number of batches each significance test is to be performed over. The
+ * window is sliding with a stride length of 1 batch. Setting windowSize to 0 will perform
+ * cumulative processing, using all batches seen so far.
+ *
+ * Different tests may be used for assessing statistical significance depending on assumptions
+ * satisfied by data. For more details, see [[StreamingTestMethod]]. The `testMethod` specifies
+ * which test will be used.
+ *
+ * Use a builder pattern to construct a streaming test in an application, for example:
+ * ```
+ * val model = new OnlineABTest()
+ * .setPeacePeriod(10)
+ * .setWindowSize(0)
+ * .setTestMethod("welch")
+ * .registerStream(DStream)
+ * ```
+ */
+@Experimental
+@Since("1.6.0")
+class StreamingTest(
+ @Since("1.6.0") var peacePeriod: Int = 0,
+ @Since("1.6.0") var windowSize: Int = 0,
+ @Since("1.6.0") var testMethod: StreamingTestMethod = WelchTTest)
+ extends Logging with Serializable {
+
+ /** Set the number of initial batches to ignore. */
+ @Since("1.6.0")
+ def setPeacePeriod(peacePeriod: Int): this.type = {
+ this.peacePeriod = peacePeriod
+ this
+ }
+
+ /**
+ * Set the number of batches to compute significance tests over.
+ * A value of 0 will use all batches seen so far.
--- End diff --
document default value
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3147][MLLib][Streaming] Streaming 2-sam...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4716#issuecomment-139654748
[Test build #42353 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42353/consoleFull) for PR 4716 at commit [`60b2e57`](https://github.com/apache/spark/commit/60b2e57026febcb68e459983ba3164281a47f636).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3147][MLLib][Streaming] Streaming 2-sam...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/4716#discussion_r39824292
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/stat/test/StreamingTest.scala ---
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.stat.test
+
+import org.apache.spark.Logging
+import org.apache.spark.annotation.{Experimental, Since}
+import org.apache.spark.streaming.dstream.DStream
+import org.apache.spark.util.StatCounter
+
+/**
+ * :: Experimental ::
+ * Performs online 2-sample significance testing for a stream of (Boolean, Double) pairs. The
+ * Boolean identifies which sample each observation comes from, and the Double is the numeric value
+ * of the observation.
+ *
+ * To address novelty affects, the `peacePeriod` specifies a set number of initial
+ * [[org.apache.spark.rdd.RDD]] batches of the [[DStream]] to be dropped from significance testing.
+ *
+ * The `windowSize` sets the number of batches each significance test is to be performed over. The
+ * window is sliding with a stride length of 1 batch. Setting windowSize to 0 will perform
+ * cumulative processing, using all batches seen so far.
+ *
+ * Different tests may be used for assessing statistical significance depending on assumptions
+ * satisfied by data. For more details, see [[StreamingTestMethod]]. The `testMethod` specifies
+ * which test will be used.
+ *
+ * Use a builder pattern to construct a streaming test in an application, for example:
+ * ```
+ * val model = new OnlineABTest()
+ * .setPeacePeriod(10)
+ * .setWindowSize(0)
+ * .setTestMethod("welch")
+ * .registerStream(DStream)
+ * ```
+ */
+@Experimental
+@Since("1.6.0")
+class StreamingTest(
+ @Since("1.6.0") var peacePeriod: Int = 0,
+ @Since("1.6.0") var windowSize: Int = 0,
+ @Since("1.6.0") var testMethod: StreamingTestMethod = WelchTTest)
+ extends Logging with Serializable {
+
+ /** Set the number of initial batches to ignore. */
+ @Since("1.6.0")
+ def setPeacePeriod(peacePeriod: Int): this.type = {
+ this.peacePeriod = peacePeriod
+ this
+ }
+
+ /**
+ * Set the number of batches to compute significance tests over.
+ * A value of 0 will use all batches seen so far.
+ */
+ @Since("1.6.0")
+ def setWindowSize(windowSize: Int): this.type = {
+ this.windowSize = windowSize
+ this
+ }
+
+ /** Set the statistical method used for significance testing. */
+ @Since("1.6.0")
+ def setTestMethod(method: String): this.type = {
+ this.testMethod = StreamingTestMethod.getTestMethodFromName(method)
+ this
+ }
+
+ /**
+ * Register a [[DStream]] of values for significance testing.
+ *
+ * @param data stream of (key,value) pairs where the key is the group membership (control or
+ * treatment) and the value is the numerical metric to test for significance
+ * @return stream of significance testing results
+ */
+ @Since("1.6.0")
+ def registerStream(data: DStream[(Boolean, Double)]): DStream[StreamingTestResult] = {
+ val dataAfterPeacePeriod = dropPeacePeriod(data)
+ val summarizedData = summarizeByKeyAndWindow(dataAfterPeacePeriod)
+ val pairedSummaries = pairSummaries(summarizedData)
+ val testResults = testMethod.doTest(pairedSummaries)
--- End diff --
`val testResults = ` is not necessary
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3147][MLLib][Streaming] Streaming 2-sam...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/4716#discussion_r39824573
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/stat/test/TestResult.scala ---
@@ -115,3 +115,25 @@ class KolmogorovSmirnovTestResult private[stat] (
"Kolmogorov-Smirnov test summary:\n" + super.toString
}
}
+
+/**
+ * :: Experimental ::
+ * Object containing the test results for streaming testing.
+ */
+@Experimental
+@Since("1.6.0")
+private[stat] class StreamingTestResult(
--- End diff --
add @Since to constructor
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3147][MLLib][Streaming] Streaming 2-sam...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/4716#issuecomment-139663151
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3147][MLLib][Streaming] Streaming 2-sam...
Posted by feynmanliang <gi...@git.apache.org>.
Github user feynmanliang commented on a diff in the pull request:
https://github.com/apache/spark/pull/4716#discussion_r39896683
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/stat/test/StreamingTest.scala ---
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.stat.test
+
+import org.apache.spark.Logging
+import org.apache.spark.annotation.{Experimental, Since}
+import org.apache.spark.streaming.dstream.DStream
+import org.apache.spark.util.StatCounter
+
+/**
+ * :: Experimental ::
+ * Performs online 2-sample significance testing for a stream of (Boolean, Double) pairs. The
+ * Boolean identifies which sample each observation comes from, and the Double is the numeric value
+ * of the observation.
+ *
+ * To address novelty affects, the `peacePeriod` specifies a set number of initial
+ * [[org.apache.spark.rdd.RDD]] batches of the [[DStream]] to be dropped from significance testing.
+ *
+ * The `windowSize` sets the number of batches each significance test is to be performed over. The
+ * window is sliding with a stride length of 1 batch. Setting windowSize to 0 will perform
+ * cumulative processing, using all batches seen so far.
+ *
+ * Different tests may be used for assessing statistical significance depending on assumptions
+ * satisfied by data. For more details, see [[StreamingTestMethod]]. The `testMethod` specifies
+ * which test will be used.
+ *
+ * Use a builder pattern to construct a streaming test in an application, for example:
+ * ```
+ * val model = new OnlineABTest()
+ * .setPeacePeriod(10)
+ * .setWindowSize(0)
+ * .setTestMethod("welch")
+ * .registerStream(DStream)
+ * ```
+ */
+@Experimental
+@Since("1.6.0")
+class StreamingTest(
+ @Since("1.6.0") var peacePeriod: Int = 0,
+ @Since("1.6.0") var windowSize: Int = 0,
+ @Since("1.6.0") var testMethod: StreamingTestMethod = WelchTTest)
+ extends Logging with Serializable {
+
+ /** Set the number of initial batches to ignore. */
+ @Since("1.6.0")
+ def setPeacePeriod(peacePeriod: Int): this.type = {
+ this.peacePeriod = peacePeriod
+ this
+ }
+
+ /**
+ * Set the number of batches to compute significance tests over.
+ * A value of 0 will use all batches seen so far.
+ */
+ @Since("1.6.0")
+ def setWindowSize(windowSize: Int): this.type = {
+ this.windowSize = windowSize
+ this
+ }
+
+ /** Set the statistical method used for significance testing. */
+ @Since("1.6.0")
+ def setTestMethod(method: String): this.type = {
+ this.testMethod = StreamingTestMethod.getTestMethodFromName(method)
+ this
+ }
+
+ /**
+ * Register a [[DStream]] of values for significance testing.
+ *
+ * @param data stream of (key,value) pairs where the key is the group membership (control or
--- End diff --
I don't think it matters here since all the tests are symmetric; will revise to say "key represents sample group membership" since this no longer specific for A/B (control/treatment) testing
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3147][MLLib][Streaming] Streaming 2-sam...
Posted by feynmanliang <gi...@git.apache.org>.
Github user feynmanliang commented on a diff in the pull request:
https://github.com/apache/spark/pull/4716#discussion_r39897444
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/stat/test/StreamingTestMethod.scala ---
@@ -0,0 +1,165 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.stat.test
+
+import java.io.Serializable
+
+import scala.language.implicitConversions
+import scala.math.pow
+
+import com.twitter.chill.MeatLocker
+import org.apache.commons.math3.stat.descriptive.StatisticalSummaryValues
+import org.apache.commons.math3.stat.inference.TTest
+
+import org.apache.spark.Logging
+import org.apache.spark.streaming.dstream.DStream
+import org.apache.spark.util.StatCounter
+
+/**
+ * Significance testing methods for [[StreamingTest]]. New 2-sample statistical significance tests
+ * should extend [[StreamingTestMethod]] and introduce a new entry in
+ * [[StreamingTestMethod.TEST_NAME_TO_OBJECT]]
+ */
+private[stat] sealed trait StreamingTestMethod extends Serializable {
+
+ val MethodName: String
+ val NullHypothesis: String
+
+ protected type SummaryPairStream =
+ DStream[(StatCounter, StatCounter)]
+
+ /**
+ * Perform streaming 2-sample statistical significance testing.
+ *
+ * @param sampleSummaries stream pairs of summary statistics for the 2 samples
+ * @return stream of rest results
+ */
+ def doTest(sampleSummaries: SummaryPairStream): DStream[StreamingTestResult]
+
+ /**
+ * Implicit adapter to convert between streaming summary statistics type and the type required by
+ * the t-testing libraries.
+ */
+ protected implicit def toApacheCommonsStats(
+ summaryStats: StatCounter): StatisticalSummaryValues = {
+ new StatisticalSummaryValues(
+ summaryStats.mean,
+ summaryStats.variance,
+ summaryStats.count,
+ summaryStats.max,
+ summaryStats.min,
+ summaryStats.mean * summaryStats.count
+ )
+ }
+}
+
+/**
+ * Performs Welch's 2-sample t-test. The null hypothesis is that the two data sets have equal mean.
+ * This test does not assume equal variance between the two samples and does not assume equal
+ * sample size.
+ *
+ * More information: http://en.wikipedia.org/wiki/Welch%27s_t_test
+ */
+private[stat] object WelchTTest extends StreamingTestMethod with Logging {
+
+ final val MethodName = "Welch's 2-sample T-test"
+ final val NullHypothesis = "Both groups have same mean"
+
+ private final val TTester = MeatLocker(new TTest())
+
+ def doTest(data: SummaryPairStream): DStream[StreamingTestResult] =
+ data.map[StreamingTestResult]((test _).tupled)
+
+ private def test(
+ statsA: StatCounter,
+ statsB: StatCounter): StreamingTestResult = {
+ def welchDF(sample1: StatisticalSummaryValues, sample2: StatisticalSummaryValues): Double = {
+ val s1 = sample1.getVariance
+ val n1 = sample1.getN
+ val s2 = sample2.getVariance
+ val n2 = sample2.getN
+
+ val a = pow(s1, 2) / n1
+ val b = pow(s2, 2) / n2
+
+ pow(a + b, 2) / ((pow(a, 2) / (n1 - 1)) + (pow(b, 2) / (n2 - 1)))
+ }
+
+ new StreamingTestResult(
+ TTester.get.tTest(statsA, statsB),
+ welchDF(statsA, statsB),
+ TTester.get.t(statsA, statsB),
+ MethodName,
+ NullHypothesis
+ )
+ }
+}
+
+/**
+ * Performs Students's 2-sample t-test. The null hypothesis is that the two data sets have equal
+ * mean. This test assumes equal variance between the two samples and does not assume equal sample
+ * size. For unequal variances, Welch's t-test should be used instead.
+ *
+ * More information: http://en.wikipedia.org/wiki/Student%27s_t-test
+ */
+private[stat] object StudentTTest extends StreamingTestMethod with Logging {
+
+ final val MethodName = "Student's 2-sample T-test"
+ final val NullHypothesis = "Both groups have same mean"
+
+ private final val TTester = MeatLocker(new TTest())
+
+ def doTest(data: SummaryPairStream): DStream[StreamingTestResult] =
+ data.map[StreamingTestResult]((test _).tupled)
+
+ private def test(
+ statsA: StatCounter,
+ statsB: StatCounter): StreamingTestResult = {
+ def studentDF(sample1: StatisticalSummaryValues, sample2: StatisticalSummaryValues): Double =
+ sample1.getN + sample2.getN - 2
+
+ new StreamingTestResult(
+ TTester.get.homoscedasticTTest(statsA, statsB),
+ studentDF(statsA, statsB),
+ TTester.get.homoscedasticT(statsA, statsB),
+ MethodName,
+ NullHypothesis
+ )
+ }
+}
+
+/**
+ * Companion object holding supported [[StreamingTestMethod]] names and handles conversion between
+ * strings used in [[StreamingTest]] configuration and actual method implementation.
+ *
+ * Currently supported tests: `welch`, `student`.
+ */
+private[stat] object StreamingTestMethod {
+ // Note: after new `StreamingTestMethod`s are implemented, please update this map.
+ final val TEST_NAME_TO_OBJECT = Map(("welch", WelchTTest), ("student", StudentTTest))
--- End diff --
OK
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3147][MLLib][Streaming] Streaming 2-sam...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/4716#issuecomment-141346997
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3147][MLLib][Streaming] Streaming 2-sam...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4716#issuecomment-141562859
[Test build #42689 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42689/consoleFull) for PR 4716 at commit [`ba71bfa`](https://github.com/apache/spark/commit/ba71bfad58d6aedb193e0f7b0cf32747d6a59ce2).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3147][MLLib][Streaming] Streaming 2-sam...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/4716#discussion_r39824394
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/stat/test/StreamingTest.scala ---
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.stat.test
+
+import org.apache.spark.Logging
+import org.apache.spark.annotation.{Experimental, Since}
+import org.apache.spark.streaming.dstream.DStream
+import org.apache.spark.util.StatCounter
+
+/**
+ * :: Experimental ::
+ * Performs online 2-sample significance testing for a stream of (Boolean, Double) pairs. The
+ * Boolean identifies which sample each observation comes from, and the Double is the numeric value
+ * of the observation.
+ *
+ * To address novelty affects, the `peacePeriod` specifies a set number of initial
+ * [[org.apache.spark.rdd.RDD]] batches of the [[DStream]] to be dropped from significance testing.
+ *
+ * The `windowSize` sets the number of batches each significance test is to be performed over. The
+ * window is sliding with a stride length of 1 batch. Setting windowSize to 0 will perform
+ * cumulative processing, using all batches seen so far.
+ *
+ * Different tests may be used for assessing statistical significance depending on assumptions
+ * satisfied by data. For more details, see [[StreamingTestMethod]]. The `testMethod` specifies
+ * which test will be used.
+ *
+ * Use a builder pattern to construct a streaming test in an application, for example:
+ * ```
+ * val model = new OnlineABTest()
+ * .setPeacePeriod(10)
+ * .setWindowSize(0)
+ * .setTestMethod("welch")
+ * .registerStream(DStream)
+ * ```
+ */
+@Experimental
+@Since("1.6.0")
+class StreamingTest(
+ @Since("1.6.0") var peacePeriod: Int = 0,
+ @Since("1.6.0") var windowSize: Int = 0,
+ @Since("1.6.0") var testMethod: StreamingTestMethod = WelchTTest)
+ extends Logging with Serializable {
+
+ /** Set the number of initial batches to ignore. */
+ @Since("1.6.0")
+ def setPeacePeriod(peacePeriod: Int): this.type = {
+ this.peacePeriod = peacePeriod
+ this
+ }
+
+ /**
+ * Set the number of batches to compute significance tests over.
+ * A value of 0 will use all batches seen so far.
+ */
+ @Since("1.6.0")
+ def setWindowSize(windowSize: Int): this.type = {
+ this.windowSize = windowSize
+ this
+ }
+
+ /** Set the statistical method used for significance testing. */
+ @Since("1.6.0")
+ def setTestMethod(method: String): this.type = {
+ this.testMethod = StreamingTestMethod.getTestMethodFromName(method)
+ this
+ }
+
+ /**
+ * Register a [[DStream]] of values for significance testing.
+ *
+ * @param data stream of (key,value) pairs where the key is the group membership (control or
+ * treatment) and the value is the numerical metric to test for significance
+ * @return stream of significance testing results
+ */
+ @Since("1.6.0")
+ def registerStream(data: DStream[(Boolean, Double)]): DStream[StreamingTestResult] = {
+ val dataAfterPeacePeriod = dropPeacePeriod(data)
+ val summarizedData = summarizeByKeyAndWindow(dataAfterPeacePeriod)
+ val pairedSummaries = pairSummaries(summarizedData)
+ val testResults = testMethod.doTest(pairedSummaries)
+
+ testResults
+ }
+
+ /** Drop all batches inside the peace period. */
+ private[stat] def dropPeacePeriod(
+ data: DStream[(Boolean, Double)]): DStream[(Boolean, Double)] = {
+ data.transform { (rdd, time) =>
+ if (time.milliseconds > data.slideDuration.milliseconds * peacePeriod) {
+ rdd
+ } else {
+ rdd.filter(_ => false) // TODO: Is there a better way to drop a RDD from a DStream?
+ }
+ }
+ }
+
+ /** Compute summary statistics over each key and the specified test window size. */
+ private[stat] def summarizeByKeyAndWindow(
+ data: DStream[(Boolean, Double)]): DStream[(Boolean, StatCounter)] = {
+ if (this.windowSize == 0) {
+ data.updateStateByKey[StatCounter](
+ (newValues: Seq[Double], oldSummary: Option[StatCounter]) => {
+ val newSummary = oldSummary.getOrElse(new StatCounter())
+ newSummary.merge(newValues)
+ Some(newSummary)
+ })
+ } else {
+ val windowDuration = data.slideDuration * this.windowSize
+ data
+ .groupByKeyAndWindow(windowDuration)
+ .mapValues { values =>
+ val summary = new StatCounter()
+ values.foreach(value => summary.merge(value))
+ summary
+ }
+ }
+ }
+
+ /**
+ * Transform a stream of summaries into pairs representing summary statistics for group A and
+ * group B up to this batch.
+ */
+ private[stat] def pairSummaries(summarizedData: DStream[(Boolean, StatCounter)])
+ : DStream[(StatCounter, StatCounter)] = {
+ summarizedData
+ .map[(Int, StatCounter)](x => (0, x._2))
+ .groupByKey() // Iterable[StatCounter] should be length two, one for each A/B group
+ .map(x => (x._2.head, x._2.last) )
--- End diff --
`) ) -> `)`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3147][MLLib] A/B testing
Posted by feynmanliang <gi...@git.apache.org>.
Github user feynmanliang commented on the pull request:
https://github.com/apache/spark/pull/4716#issuecomment-75543539
`[error] * abstract method numDim()Int in interface org.apache.spark.mllib.stat.MultivariateStatisticalSummary does not have a correspondent in old version
`
Keep, or remove `numDim` from this patch completely since `MultivariateStatisticalSummary` has been replaced with `StatsCounter`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3147][MLLib][Streaming] Streaming 2-sam...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/4716#issuecomment-139653498
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3147][MLLib][Streaming] Streaming 2-sam...
Posted by feynmanliang <gi...@git.apache.org>.
Github user feynmanliang commented on a diff in the pull request:
https://github.com/apache/spark/pull/4716#discussion_r39897297
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/stat/test/StreamingTestMethod.scala ---
@@ -0,0 +1,165 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.stat.test
+
+import java.io.Serializable
+
+import scala.language.implicitConversions
+import scala.math.pow
+
+import com.twitter.chill.MeatLocker
+import org.apache.commons.math3.stat.descriptive.StatisticalSummaryValues
+import org.apache.commons.math3.stat.inference.TTest
+
+import org.apache.spark.Logging
+import org.apache.spark.streaming.dstream.DStream
+import org.apache.spark.util.StatCounter
+
+/**
+ * Significance testing methods for [[StreamingTest]]. New 2-sample statistical significance tests
+ * should extend [[StreamingTestMethod]] and introduce a new entry in
+ * [[StreamingTestMethod.TEST_NAME_TO_OBJECT]]
+ */
+private[stat] sealed trait StreamingTestMethod extends Serializable {
+
+ val MethodName: String
+ val NullHypothesis: String
+
+ protected type SummaryPairStream =
+ DStream[(StatCounter, StatCounter)]
+
+ /**
+ * Perform streaming 2-sample statistical significance testing.
+ *
+ * @param sampleSummaries stream pairs of summary statistics for the 2 samples
+ * @return stream of rest results
+ */
+ def doTest(sampleSummaries: SummaryPairStream): DStream[StreamingTestResult]
+
+ /**
+ * Implicit adapter to convert between streaming summary statistics type and the type required by
+ * the t-testing libraries.
+ */
+ protected implicit def toApacheCommonsStats(
+ summaryStats: StatCounter): StatisticalSummaryValues = {
+ new StatisticalSummaryValues(
+ summaryStats.mean,
+ summaryStats.variance,
+ summaryStats.count,
+ summaryStats.max,
+ summaryStats.min,
+ summaryStats.mean * summaryStats.count
+ )
+ }
+}
+
+/**
+ * Performs Welch's 2-sample t-test. The null hypothesis is that the two data sets have equal mean.
+ * This test does not assume equal variance between the two samples and does not assume equal
+ * sample size.
+ *
+ * More information: http://en.wikipedia.org/wiki/Welch%27s_t_test
+ */
+private[stat] object WelchTTest extends StreamingTestMethod with Logging {
+
+ final val MethodName = "Welch's 2-sample T-test"
--- End diff --
OK
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3147][MLLib][Streaming] Streaming 2-sam...
Posted by feynmanliang <gi...@git.apache.org>.
Github user feynmanliang commented on a diff in the pull request:
https://github.com/apache/spark/pull/4716#discussion_r39896148
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/stat/test/StreamingTest.scala ---
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.stat.test
+
+import org.apache.spark.Logging
+import org.apache.spark.annotation.{Experimental, Since}
+import org.apache.spark.streaming.dstream.DStream
+import org.apache.spark.util.StatCounter
+
+/**
+ * :: Experimental ::
+ * Performs online 2-sample significance testing for a stream of (Boolean, Double) pairs. The
+ * Boolean identifies which sample each observation comes from, and the Double is the numeric value
+ * of the observation.
+ *
+ * To address novelty affects, the `peacePeriod` specifies a set number of initial
+ * [[org.apache.spark.rdd.RDD]] batches of the [[DStream]] to be dropped from significance testing.
+ *
+ * The `windowSize` sets the number of batches each significance test is to be performed over. The
+ * window is sliding with a stride length of 1 batch. Setting windowSize to 0 will perform
+ * cumulative processing, using all batches seen so far.
+ *
+ * Different tests may be used for assessing statistical significance depending on assumptions
+ * satisfied by data. For more details, see [[StreamingTestMethod]]. The `testMethod` specifies
+ * which test will be used.
+ *
+ * Use a builder pattern to construct a streaming test in an application, for example:
+ * ```
--- End diff --
OK
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3147][MLLib][Streaming] Streaming 2-sam...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/4716#issuecomment-139648567
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42352/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3147][MLLib][Streaming] Streaming 2-sam...
Posted by feynmanliang <gi...@git.apache.org>.
Github user feynmanliang commented on a diff in the pull request:
https://github.com/apache/spark/pull/4716#discussion_r39897178
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/stat/test/StreamingTestMethod.scala ---
@@ -0,0 +1,165 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.stat.test
+
+import java.io.Serializable
+
+import scala.language.implicitConversions
+import scala.math.pow
+
+import com.twitter.chill.MeatLocker
+import org.apache.commons.math3.stat.descriptive.StatisticalSummaryValues
+import org.apache.commons.math3.stat.inference.TTest
+
+import org.apache.spark.Logging
+import org.apache.spark.streaming.dstream.DStream
+import org.apache.spark.util.StatCounter
+
+/**
+ * Significance testing methods for [[StreamingTest]]. New 2-sample statistical significance tests
+ * should extend [[StreamingTestMethod]] and introduce a new entry in
+ * [[StreamingTestMethod.TEST_NAME_TO_OBJECT]]
+ */
+private[stat] sealed trait StreamingTestMethod extends Serializable {
+
+ val MethodName: String
--- End diff --
OK
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3147][MLLib][Streaming] Streaming 2-sam...
Posted by feynmanliang <gi...@git.apache.org>.
Github user feynmanliang commented on a diff in the pull request:
https://github.com/apache/spark/pull/4716#discussion_r39896192
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/stat/test/StreamingTest.scala ---
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.stat.test
+
+import org.apache.spark.Logging
+import org.apache.spark.annotation.{Experimental, Since}
+import org.apache.spark.streaming.dstream.DStream
+import org.apache.spark.util.StatCounter
+
+/**
+ * :: Experimental ::
+ * Performs online 2-sample significance testing for a stream of (Boolean, Double) pairs. The
+ * Boolean identifies which sample each observation comes from, and the Double is the numeric value
+ * of the observation.
+ *
+ * To address novelty affects, the `peacePeriod` specifies a set number of initial
+ * [[org.apache.spark.rdd.RDD]] batches of the [[DStream]] to be dropped from significance testing.
+ *
+ * The `windowSize` sets the number of batches each significance test is to be performed over. The
+ * window is sliding with a stride length of 1 batch. Setting windowSize to 0 will perform
+ * cumulative processing, using all batches seen so far.
+ *
+ * Different tests may be used for assessing statistical significance depending on assumptions
+ * satisfied by data. For more details, see [[StreamingTestMethod]]. The `testMethod` specifies
+ * which test will be used.
+ *
+ * Use a builder pattern to construct a streaming test in an application, for example:
+ * ```
+ * val model = new OnlineABTest()
+ * .setPeacePeriod(10)
+ * .setWindowSize(0)
+ * .setTestMethod("welch")
+ * .registerStream(DStream)
+ * ```
+ */
+@Experimental
+@Since("1.6.0")
+class StreamingTest(
--- End diff --
OK
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3147][MLLib][Streaming] Streaming 2-sam...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/4716#discussion_r39824567
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/stat/test/StreamingTestMethod.scala ---
@@ -0,0 +1,165 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.stat.test
+
+import java.io.Serializable
+
+import scala.language.implicitConversions
+import scala.math.pow
+
+import com.twitter.chill.MeatLocker
+import org.apache.commons.math3.stat.descriptive.StatisticalSummaryValues
+import org.apache.commons.math3.stat.inference.TTest
+
+import org.apache.spark.Logging
+import org.apache.spark.streaming.dstream.DStream
+import org.apache.spark.util.StatCounter
+
+/**
+ * Significance testing methods for [[StreamingTest]]. New 2-sample statistical significance tests
+ * should extend [[StreamingTestMethod]] and introduce a new entry in
+ * [[StreamingTestMethod.TEST_NAME_TO_OBJECT]]
+ */
+private[stat] sealed trait StreamingTestMethod extends Serializable {
+
+ val MethodName: String
+ val NullHypothesis: String
+
+ protected type SummaryPairStream =
+ DStream[(StatCounter, StatCounter)]
+
+ /**
+ * Perform streaming 2-sample statistical significance testing.
+ *
+ * @param sampleSummaries stream pairs of summary statistics for the 2 samples
+ * @return stream of rest results
+ */
+ def doTest(sampleSummaries: SummaryPairStream): DStream[StreamingTestResult]
+
+ /**
+ * Implicit adapter to convert between streaming summary statistics type and the type required by
+ * the t-testing libraries.
+ */
+ protected implicit def toApacheCommonsStats(
+ summaryStats: StatCounter): StatisticalSummaryValues = {
+ new StatisticalSummaryValues(
+ summaryStats.mean,
+ summaryStats.variance,
+ summaryStats.count,
+ summaryStats.max,
+ summaryStats.min,
+ summaryStats.mean * summaryStats.count
+ )
+ }
+}
+
+/**
+ * Performs Welch's 2-sample t-test. The null hypothesis is that the two data sets have equal mean.
+ * This test does not assume equal variance between the two samples and does not assume equal
+ * sample size.
+ *
+ * More information: http://en.wikipedia.org/wiki/Welch%27s_t_test
+ */
+private[stat] object WelchTTest extends StreamingTestMethod with Logging {
+
+ final val MethodName = "Welch's 2-sample T-test"
+ final val NullHypothesis = "Both groups have same mean"
+
+ private final val TTester = MeatLocker(new TTest())
+
+ def doTest(data: SummaryPairStream): DStream[StreamingTestResult] =
+ data.map[StreamingTestResult]((test _).tupled)
+
+ private def test(
+ statsA: StatCounter,
+ statsB: StatCounter): StreamingTestResult = {
+ def welchDF(sample1: StatisticalSummaryValues, sample2: StatisticalSummaryValues): Double = {
+ val s1 = sample1.getVariance
+ val n1 = sample1.getN
+ val s2 = sample2.getVariance
+ val n2 = sample2.getN
+
+ val a = pow(s1, 2) / n1
+ val b = pow(s2, 2) / n2
+
+ pow(a + b, 2) / ((pow(a, 2) / (n1 - 1)) + (pow(b, 2) / (n2 - 1)))
+ }
+
+ new StreamingTestResult(
+ TTester.get.tTest(statsA, statsB),
+ welchDF(statsA, statsB),
+ TTester.get.t(statsA, statsB),
+ MethodName,
+ NullHypothesis
+ )
+ }
+}
+
+/**
+ * Performs Students's 2-sample t-test. The null hypothesis is that the two data sets have equal
+ * mean. This test assumes equal variance between the two samples and does not assume equal sample
+ * size. For unequal variances, Welch's t-test should be used instead.
+ *
+ * More information: http://en.wikipedia.org/wiki/Student%27s_t-test
+ */
+private[stat] object StudentTTest extends StreamingTestMethod with Logging {
+
+ final val MethodName = "Student's 2-sample T-test"
+ final val NullHypothesis = "Both groups have same mean"
+
+ private final val TTester = MeatLocker(new TTest())
+
+ def doTest(data: SummaryPairStream): DStream[StreamingTestResult] =
+ data.map[StreamingTestResult]((test _).tupled)
+
+ private def test(
+ statsA: StatCounter,
+ statsB: StatCounter): StreamingTestResult = {
+ def studentDF(sample1: StatisticalSummaryValues, sample2: StatisticalSummaryValues): Double =
+ sample1.getN + sample2.getN - 2
+
+ new StreamingTestResult(
+ TTester.get.homoscedasticTTest(statsA, statsB),
+ studentDF(statsA, statsB),
+ TTester.get.homoscedasticT(statsA, statsB),
+ MethodName,
+ NullHypothesis
+ )
+ }
+}
+
+/**
+ * Companion object holding supported [[StreamingTestMethod]] names and handles conversion between
+ * strings used in [[StreamingTest]] configuration and actual method implementation.
+ *
+ * Currently supported tests: `welch`, `student`.
+ */
+private[stat] object StreamingTestMethod {
+ // Note: after new `StreamingTestMethod`s are implemented, please update this map.
+ final val TEST_NAME_TO_OBJECT = Map(("welch", WelchTTest), ("student", StudentTTest))
--- End diff --
could be private
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3147][MLLib][Streaming] Streaming 2-sam...
Posted by feynmanliang <gi...@git.apache.org>.
Github user feynmanliang commented on a diff in the pull request:
https://github.com/apache/spark/pull/4716#discussion_r39897318
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/stat/test/StreamingTestMethod.scala ---
@@ -0,0 +1,165 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.stat.test
+
+import java.io.Serializable
+
+import scala.language.implicitConversions
+import scala.math.pow
+
+import com.twitter.chill.MeatLocker
+import org.apache.commons.math3.stat.descriptive.StatisticalSummaryValues
+import org.apache.commons.math3.stat.inference.TTest
+
+import org.apache.spark.Logging
+import org.apache.spark.streaming.dstream.DStream
+import org.apache.spark.util.StatCounter
+
+/**
+ * Significance testing methods for [[StreamingTest]]. New 2-sample statistical significance tests
+ * should extend [[StreamingTestMethod]] and introduce a new entry in
+ * [[StreamingTestMethod.TEST_NAME_TO_OBJECT]]
+ */
+private[stat] sealed trait StreamingTestMethod extends Serializable {
+
+ val MethodName: String
+ val NullHypothesis: String
+
+ protected type SummaryPairStream =
+ DStream[(StatCounter, StatCounter)]
+
+ /**
+ * Perform streaming 2-sample statistical significance testing.
+ *
+ * @param sampleSummaries stream pairs of summary statistics for the 2 samples
+ * @return stream of rest results
+ */
+ def doTest(sampleSummaries: SummaryPairStream): DStream[StreamingTestResult]
+
+ /**
+ * Implicit adapter to convert between streaming summary statistics type and the type required by
+ * the t-testing libraries.
+ */
+ protected implicit def toApacheCommonsStats(
+ summaryStats: StatCounter): StatisticalSummaryValues = {
+ new StatisticalSummaryValues(
+ summaryStats.mean,
+ summaryStats.variance,
+ summaryStats.count,
+ summaryStats.max,
+ summaryStats.min,
+ summaryStats.mean * summaryStats.count
+ )
+ }
+}
+
+/**
+ * Performs Welch's 2-sample t-test. The null hypothesis is that the two data sets have equal mean.
+ * This test does not assume equal variance between the two samples and does not assume equal
+ * sample size.
+ *
+ * More information: http://en.wikipedia.org/wiki/Welch%27s_t_test
+ */
+private[stat] object WelchTTest extends StreamingTestMethod with Logging {
+
+ final val MethodName = "Welch's 2-sample T-test"
+ final val NullHypothesis = "Both groups have same mean"
+
+ private final val TTester = MeatLocker(new TTest())
--- End diff --
OK
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3147][MLLib][Streaming] Streaming 2-sam...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/4716#discussion_r39824449
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/stat/test/StreamingTestMethod.scala ---
@@ -0,0 +1,165 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.stat.test
+
+import java.io.Serializable
+
+import scala.language.implicitConversions
+import scala.math.pow
+
+import com.twitter.chill.MeatLocker
+import org.apache.commons.math3.stat.descriptive.StatisticalSummaryValues
+import org.apache.commons.math3.stat.inference.TTest
+
+import org.apache.spark.Logging
+import org.apache.spark.streaming.dstream.DStream
+import org.apache.spark.util.StatCounter
+
+/**
+ * Significance testing methods for [[StreamingTest]]. New 2-sample statistical significance tests
+ * should extend [[StreamingTestMethod]] and introduce a new entry in
+ * [[StreamingTestMethod.TEST_NAME_TO_OBJECT]]
+ */
+private[stat] sealed trait StreamingTestMethod extends Serializable {
+
+ val MethodName: String
+ val NullHypothesis: String
+
+ protected type SummaryPairStream =
+ DStream[(StatCounter, StatCounter)]
+
+ /**
+ * Perform streaming 2-sample statistical significance testing.
+ *
+ * @param sampleSummaries stream pairs of summary statistics for the 2 samples
+ * @return stream of rest results
+ */
+ def doTest(sampleSummaries: SummaryPairStream): DStream[StreamingTestResult]
+
+ /**
+ * Implicit adapter to convert between streaming summary statistics type and the type required by
+ * the t-testing libraries.
+ */
+ protected implicit def toApacheCommonsStats(
+ summaryStats: StatCounter): StatisticalSummaryValues = {
+ new StatisticalSummaryValues(
+ summaryStats.mean,
+ summaryStats.variance,
+ summaryStats.count,
+ summaryStats.max,
+ summaryStats.min,
+ summaryStats.mean * summaryStats.count
+ )
+ }
+}
+
+/**
+ * Performs Welch's 2-sample t-test. The null hypothesis is that the two data sets have equal mean.
+ * This test does not assume equal variance between the two samples and does not assume equal
+ * sample size.
+ *
+ * More information: http://en.wikipedia.org/wiki/Welch%27s_t_test
+ */
+private[stat] object WelchTTest extends StreamingTestMethod with Logging {
+
+ final val MethodName = "Welch's 2-sample T-test"
--- End diff --
`T-test` -> `t-test`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org