You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by yinxusen <gi...@git.apache.org> on 2016/02/25 21:37:02 UTC

[GitHub] spark pull request: [SPARK-12042] Python API for mllib.stat.test.S...

GitHub user yinxusen opened a pull request:

    https://github.com/apache/spark/pull/11374

    [SPARK-12042] Python API for mllib.stat.test.StreamingTest

    ## What changes were proposed in this pull request?
    
    The patch adds python API for mllib.stat.test.StreamingTest under JIRA https://issues.apache.org/jira/browse/SPARK-12042.
    
    Note that for `StreamingTestResult`, unlike other test results in Python, I define it as a normal Python class which doesn't extend from `TestResult` with a `_java_obj` in it.
    
    ## How was this patch tested?
    
    The patch is tested with Python unit test.


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/yinxusen/spark SPARK-12042

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/11374.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #11374
    
----
commit 6867a8953d5b94cfa3b6520c2a52ef8f04ed72fb
Author: Xusen Yin <yi...@gmail.com>
Date:   2016-02-25T06:42:38Z

    A draft and runnable version

commit 079a87369e57a0454aaa0ff5a1ebefce97abcfc0
Author: Xusen Yin <yi...@gmail.com>
Date:   2016-02-25T08:09:52Z

    treat StreamingTestResult as an independent class

commit f70d7aa52a2006d2a86de2e4591839f34dabc50a
Author: Xusen Yin <yi...@gmail.com>
Date:   2016-02-25T20:02:24Z

    add test for streamingtest

commit 770703b341fd4f900718cf2fd8a1985170435b65
Author: Xusen Yin <yi...@gmail.com>
Date:   2016-02-25T20:22:38Z

    refine test

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12042] Python API for mllib.stat.test.S...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/11374#issuecomment-192768223
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #11374: [SPARK-12042] Python API for mllib.stat.test.Stre...

Posted by feynmanliang <gi...@git.apache.org>.
Github user feynmanliang commented on a diff in the pull request:

    https://github.com/apache/spark/pull/11374#discussion_r85647843
  
    --- Diff: mllib/src/main/scala/org/apache/spark/mllib/api/python/PythonMLLibAPI.scala ---
    @@ -1576,17 +1616,49 @@ private[spark] object SerDe extends SerDeBase with Serializable {
         }
       }
     
    +  private[python] class BinarySamplePickler extends BasePickler[BinarySample] {
    +    def saveState(obj: Object, out: OutputStream, pickler: Pickler): Unit = {
    +      val binarySample = obj.asInstanceOf[BinarySample]
    +      saveObjects(out, pickler, binarySample.isExperiment, binarySample.value)
    +    }
    +
    +    def construct(args: Array[AnyRef]): AnyRef = {
    +      if (args.length != 2) {
    +        throw new PickleException("should be 2")
    +      }
    +      BinarySample(args(0).asInstanceOf[Boolean], args(1).asInstanceOf[Double])
    +    }
    +  }
    +
    +  private[python] class StreamingTestResultPickler extends BasePickler[StreamingTestResult] {
    --- End diff --
    
    Do we need to test these in `PythonMLLibAPISuite`?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #11374: [SPARK-12042] Python API for mllib.stat.test.Stre...

Posted by feynmanliang <gi...@git.apache.org>.
Github user feynmanliang commented on a diff in the pull request:

    https://github.com/apache/spark/pull/11374#discussion_r85647607
  
    --- Diff: python/pyspark/mllib/stat/test.py ---
    @@ -80,3 +85,118 @@ class KolmogorovSmirnovTestResult(TestResult):
         """
         Contains test results for the Kolmogorov-Smirnov test.
         """
    +
    +
    +class BinarySample(namedtuple("BinarySample", ["isExperiment", "value"])):
    +    """
    +    Represents a (isExperiment, value) tuple.
    +
    +    .. versionadded:: 2.0.0
    +    """
    +
    +    def __reduce__(self):
    +        return BinarySample, (bool(self.isExperiment), float(self.value))
    +
    +
    +class StreamingTestResult(namedtuple("StreamingTestResult",
    +                                     ["pValue", "degreesOfFreedom", "statistic", "method",
    +                                      "nullHypothesis"])):
    +    """
    +    Contains test results for StreamingTest.
    +
    +    .. versionadded:: 2.0.0
    +    """
    +
    +    def __reduce__(self):
    +        return StreamingTestResult, (float(self.pValue),
    +                                     float(self.degreesOfFreedom), float(self.statistic),
    +                                     str(self.method), str(self.nullHypothesis))
    +
    +
    +class StreamingTest(object):
    +    """
    +    .. note:: Experimental
    +
    +    Online 2-sample significance testing for a stream of (Boolean, Double) pairs. The Boolean
    +    identifies which sample each observation comes from, and the Double is the numeric value of the
    +    observation.
    +
    +    To address novelty affects, the `peacePeriod` specifies a set number of initial RDD batches of
    +    the DStream to be dropped from significance testing.
    +
    +    The `windowSize` sets the number of batches each significance test is to be performed over. The
    +    window is sliding with a stride length of 1 batch. Setting windowSize to 0 will perform
    +    cumulative processing, using all batches seen so far.
    +
    +    Different tests may be used for assessing statistical significance depending on assumptions
    +    satisfied by data. For more details, see StreamingTestMethod. The `testMethod` specifies
    +    which test will be used.
    +
    +    .. versionadded:: 2.0.0
    +    """
    +
    +    def __init__(self):
    +        self._peacePeriod = 0
    +        self._windowSize = 0
    +        self._testMethod = "welch"
    +
    +    @since('2.0.0')
    +    def setPeacePeriod(self, peacePeriod):
    +        """
    +        Update peacePeriod
    +        :param peacePeriod:
    +          Set number of initial RDD batches of the DStream to be dropped from significance testing.
    +        """
    +        self._peacePeriod = peacePeriod
    +
    +    @since('2.0.0')
    +    def setWindowSize(self, windowSize):
    +        """
    +        Update windowSize
    +        :param windowSize:
    +          Set the number of batches each significance test is to be performed over.
    +        """
    +        self._windowSize = windowSize
    +
    +    @since('2.0.0')
    +    def setTestMethod(self, testMethod):
    +        """
    +        Update test method
    +        :param testMethod:
    +          Currently supported tests: `welch`, `student`.
    +        """
    +        assert(testMethod in ("welch", "student"),
    +               "Currently supported tests: \"welch\", \"student\"")
    +        self._testMethod = testMethod
    +
    +    @since('2.0.0')
    +    def registerStream(self, data):
    +        """
    +        Register a data stream to get its test result.
    +
    +        :param data:
    +          The input data stream, each element is a BinarySample instance.
    +        """
    +        self._validate(data)
    +        sc = SparkContext._active_spark_context
    +
    +        streamingTest = sc._jvm.org.apache.spark.mllib.stat.test.StreamingTest()
    --- End diff --
    
    Why did we not define a `StreamingTestModel` and assign an instance to `_model` (like in streaming K means)?



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #11374: [SPARK-12042] Python API for mllib.stat.test.Stre...

Posted by feynmanliang <gi...@git.apache.org>.
Github user feynmanliang commented on a diff in the pull request:

    https://github.com/apache/spark/pull/11374#discussion_r85647656
  
    --- Diff: python/pyspark/mllib/tests.py ---
    @@ -1688,6 +1689,44 @@ def test_binary_term_freqs(self):
                                        ": expected " + str(expected[i]) + ", got " + str(output[i]))
     
     
    +class StreamingTestTest(PySparkStreamingTestCase):
    +    def test_streaming_test_result_and_model(self):
    +        """
    +        Assert the StreamingTest return valid result, and the set method of it.
    +        """
    +
    +        checkpoint_path = tempfile.mkdtemp()
    +        self.ssc.checkpoint(checkpoint_path)
    --- End diff --
    
    Is this necessary?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12042] Python API for mllib.stat.test.S...

Posted by yinxusen <gi...@git.apache.org>.
Github user yinxusen commented on the pull request:

    https://github.com/apache/spark/pull/11374#issuecomment-192769022
  
    cc @feynmanliang 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12042] Python API for mllib.stat.test.S...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/11374#issuecomment-219192289
  
    **[Test build #58598 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/58598/consoleFull)** for PR 11374 at commit [`e4e8d5e`](https://github.com/apache/spark/commit/e4e8d5ec5d467d7efebb917c6c7e60b14f519e74).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #11374: [SPARK-12042] Python API for mllib.stat.test.Stre...

Posted by feynmanliang <gi...@git.apache.org>.
Github user feynmanliang commented on a diff in the pull request:

    https://github.com/apache/spark/pull/11374#discussion_r85647664
  
    --- Diff: python/pyspark/mllib/tests.py ---
    @@ -1688,6 +1689,44 @@ def test_binary_term_freqs(self):
                                        ": expected " + str(expected[i]) + ", got " + str(output[i]))
     
     
    +class StreamingTestTest(PySparkStreamingTestCase):
    +    def test_streaming_test_result_and_model(self):
    +        """
    +        Assert the StreamingTest return valid result, and the set method of it.
    +        """
    +
    +        checkpoint_path = tempfile.mkdtemp()
    +        self.ssc.checkpoint(checkpoint_path)
    +
    +        # Create the queue through which RDDs can be pushed to a QueueInputDStream.
    +        rdd_queue = []
    +        for i in range(5):
    +            rdd_queue += [self.ssc.sparkContext.parallelize(
    +                [BinarySample(True, j) for j in range(1, 1001)], 10)]
    +
    +        # Create the QueueInputDStream and use it do some processing.
    +        input_stream = self.ssc.queueStream(rdd_queue)
    +
    +        model = StreamingTest()
    +        model.setPeacePeriod(1)
    --- End diff --
    
    Can we break this into another test just for model params like https://github.com/apache/spark/blob/39e2bad6a866d27c3ca594d15e574a1da3ee84cc/python/pyspark/mllib/tests.py#L1165?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #11374: [SPARK-12042] Python API for mllib.stat.test.Stre...

Posted by feynmanliang <gi...@git.apache.org>.
Github user feynmanliang commented on a diff in the pull request:

    https://github.com/apache/spark/pull/11374#discussion_r85647632
  
    --- Diff: python/pyspark/mllib/tests.py ---
    @@ -1688,6 +1689,44 @@ def test_binary_term_freqs(self):
                                        ": expected " + str(expected[i]) + ", got " + str(output[i]))
     
     
    +class StreamingTestTest(PySparkStreamingTestCase):
    --- End diff --
    
    Why not `MLLibStreamingTestCase`?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #11374: [SPARK-12042] Python API for mllib.stat.test.StreamingTe...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/11374
  
    **[Test build #67696 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67696/consoleFull)** for PR 11374 at commit [`615fbbb`](https://github.com/apache/spark/commit/615fbbb4aaddd4c5a45f9fff4c58c395f1873340).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #11374: [SPARK-12042] Python API for mllib.stat.test.Stre...

Posted by feynmanliang <gi...@git.apache.org>.
Github user feynmanliang commented on a diff in the pull request:

    https://github.com/apache/spark/pull/11374#discussion_r85647272
  
    --- Diff: examples/src/main/python/mllib/streaming_test_example.py ---
    @@ -0,0 +1,57 @@
    +#
    +# Licensed to the Apache Software Foundation (ASF) under one or more
    +# contributor license agreements.  See the NOTICE file distributed with
    +# this work for additional information regarding copyright ownership.
    +# The ASF licenses this file to You under the Apache License, Version 2.0
    +# (the "License"); you may not use this file except in compliance with
    +# the License.  You may obtain a copy of the License at
    +#
    +#    http://www.apache.org/licenses/LICENSE-2.0
    +#
    +# Unless required by applicable law or agreed to in writing, software
    +# distributed under the License is distributed on an "AS IS" BASIS,
    +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    +# See the License for the specific language governing permissions and
    +# limitations under the License.
    +#
    +
    +"""
    +Create a DStream that contains several RDDs to show the StreamingTest of PySpark.
    +"""
    +import time
    +import tempfile
    +from shutil import rmtree
    +
    +from pyspark import SparkContext
    +from pyspark.streaming import StreamingContext
    +from pyspark.mllib.stat.test import BinarySample, StreamingTest
    --- End diff --
    
    `$example on$` and `$example off` appear to be used in other examples, though I'm not sure why myself


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #11374: [SPARK-12042] Python API for mllib.stat.test.Stre...

Posted by feynmanliang <gi...@git.apache.org>.
Github user feynmanliang commented on a diff in the pull request:

    https://github.com/apache/spark/pull/11374#discussion_r85647232
  
    --- Diff: examples/src/main/python/mllib/streaming_test_example.py ---
    @@ -0,0 +1,57 @@
    +#
    +# Licensed to the Apache Software Foundation (ASF) under one or more
    +# contributor license agreements.  See the NOTICE file distributed with
    +# this work for additional information regarding copyright ownership.
    +# The ASF licenses this file to You under the Apache License, Version 2.0
    +# (the "License"); you may not use this file except in compliance with
    +# the License.  You may obtain a copy of the License at
    +#
    +#    http://www.apache.org/licenses/LICENSE-2.0
    +#
    +# Unless required by applicable law or agreed to in writing, software
    +# distributed under the License is distributed on an "AS IS" BASIS,
    +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    +# See the License for the specific language governing permissions and
    +# limitations under the License.
    +#
    +
    +"""
    +Create a DStream that contains several RDDs to show the StreamingTest of PySpark.
    +"""
    +import time
    +import tempfile
    +from shutil import rmtree
    +
    +from pyspark import SparkContext
    +from pyspark.streaming import StreamingContext
    +from pyspark.mllib.stat.test import BinarySample, StreamingTest
    +
    +if __name__ == "__main__":
    +
    --- End diff --
    
    nit: don't include newline here


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #11374: [SPARK-12042] Python API for mllib.stat.test.StreamingTe...

Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:

    https://github.com/apache/spark/pull/11374
  
    Hi @yinxusen, are you able to proceed this further? If not, it seems it might be better closed for now.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #11374: [SPARK-12042] Python API for mllib.stat.test.Stre...

Posted by feynmanliang <gi...@git.apache.org>.
Github user feynmanliang commented on a diff in the pull request:

    https://github.com/apache/spark/pull/11374#discussion_r85647470
  
    --- Diff: mllib/src/main/scala/org/apache/spark/mllib/api/python/PythonMLLibAPI.scala ---
    @@ -1358,6 +1363,41 @@ private[spark] abstract class SerDeBase {
           }
         }.toJavaRDD()
       }
    +
    +  /**
    +   * Convert a DStream of Java objects to a DStream of serialized Python objects, that is usable by
    +   * PySpark.
    +   */
    +  def javaToPython(jDStream: JavaDStream[Any]): JavaDStream[Array[Byte]] = {
    +    val dStream = jDStream.dstream.mapPartitions { iter =>
    +      initialize()  // let it called in executor
    +      new SerDeUtil.AutoBatchedPickler(iter)
    +    }
    +    new JavaDStream[Array[Byte]](dStream)
    +  }
    +
    +  /**
    +   * Convert a DStream of serialized Python objects to a DStream of objects, that is usable by
    +   * PySpark.
    +   */
    +  def pythonToJava(pyDStream: JavaDStream[Array[Byte]], batched: Boolean): JavaDStream[Any] = {
    +    val dStream = pyDStream.dstream.mapPartitions { iter =>
    +      initialize()  // let it called in executor
    +      val unpickle = new Unpickler
    +      iter.flatMap { row =>
    +        val obj = unpickle.loads(row)
    +        if (batched) {
    +          obj match {
    +            case list: JArrayList[_] => list.asScala
    +            case arr: Array[_] => arr
    +          }
    +        } else {
    +        Seq(obj)
    +        }
    +      }
    +    }
    +    new JavaDStream[Any](dStream)
    --- End diff --
    
    No need for this constructor, just return `dStream` (in fact, don't even need the `dStream` variable)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #11374: [SPARK-12042] Python API for mllib.stat.test.Stre...

Posted by feynmanliang <gi...@git.apache.org>.
Github user feynmanliang commented on a diff in the pull request:

    https://github.com/apache/spark/pull/11374#discussion_r85647320
  
    --- Diff: mllib/src/main/scala/org/apache/spark/mllib/api/python/PythonMLLibAPI.scala ---
    @@ -44,7 +44,7 @@ import org.apache.spark.mllib.regression._
     import org.apache.spark.mllib.stat.{KernelDensity, MultivariateStatisticalSummary, Statistics}
     import org.apache.spark.mllib.stat.correlation.CorrelationNames
     import org.apache.spark.mllib.stat.distribution.MultivariateGaussian
    -import org.apache.spark.mllib.stat.test.{ChiSqTestResult, KolmogorovSmirnovTestResult}
    +import org.apache.spark.mllib.stat.test._
    --- End diff --
    
    Why are we using a wildcard import here?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #11374: [SPARK-12042] Python API for mllib.stat.test.StreamingTe...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/11374
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67696/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #11374: [SPARK-12042] Python API for mllib.stat.test.Stre...

Posted by feynmanliang <gi...@git.apache.org>.
Github user feynmanliang commented on a diff in the pull request:

    https://github.com/apache/spark/pull/11374#discussion_r85647567
  
    --- Diff: python/pyspark/mllib/stat/test.py ---
    @@ -80,3 +85,118 @@ class KolmogorovSmirnovTestResult(TestResult):
         """
         Contains test results for the Kolmogorov-Smirnov test.
         """
    +
    +
    +class BinarySample(namedtuple("BinarySample", ["isExperiment", "value"])):
    +    """
    +    Represents a (isExperiment, value) tuple.
    +
    +    .. versionadded:: 2.0.0
    +    """
    +
    +    def __reduce__(self):
    +        return BinarySample, (bool(self.isExperiment), float(self.value))
    +
    +
    +class StreamingTestResult(namedtuple("StreamingTestResult",
    +                                     ["pValue", "degreesOfFreedom", "statistic", "method",
    +                                      "nullHypothesis"])):
    +    """
    +    Contains test results for StreamingTest.
    +
    +    .. versionadded:: 2.0.0
    +    """
    +
    +    def __reduce__(self):
    +        return StreamingTestResult, (float(self.pValue),
    +                                     float(self.degreesOfFreedom), float(self.statistic),
    +                                     str(self.method), str(self.nullHypothesis))
    +
    +
    +class StreamingTest(object):
    +    """
    +    .. note:: Experimental
    +
    +    Online 2-sample significance testing for a stream of (Boolean, Double) pairs. The Boolean
    +    identifies which sample each observation comes from, and the Double is the numeric value of the
    +    observation.
    +
    +    To address novelty affects, the `peacePeriod` specifies a set number of initial RDD batches of
    +    the DStream to be dropped from significance testing.
    +
    +    The `windowSize` sets the number of batches each significance test is to be performed over. The
    +    window is sliding with a stride length of 1 batch. Setting windowSize to 0 will perform
    +    cumulative processing, using all batches seen so far.
    +
    +    Different tests may be used for assessing statistical significance depending on assumptions
    +    satisfied by data. For more details, see StreamingTestMethod. The `testMethod` specifies
    +    which test will be used.
    +
    +    .. versionadded:: 2.0.0
    +    """
    +
    +    def __init__(self):
    +        self._peacePeriod = 0
    +        self._windowSize = 0
    +        self._testMethod = "welch"
    +
    +    @since('2.0.0')
    +    def setPeacePeriod(self, peacePeriod):
    +        """
    +        Update peacePeriod
    +        :param peacePeriod:
    +          Set number of initial RDD batches of the DStream to be dropped from significance testing.
    +        """
    +        self._peacePeriod = peacePeriod
    +
    +    @since('2.0.0')
    +    def setWindowSize(self, windowSize):
    +        """
    +        Update windowSize
    +        :param windowSize:
    +          Set the number of batches each significance test is to be performed over.
    +        """
    +        self._windowSize = windowSize
    +
    +    @since('2.0.0')
    +    def setTestMethod(self, testMethod):
    +        """
    +        Update test method
    +        :param testMethod:
    +          Currently supported tests: `welch`, `student`.
    +        """
    +        assert(testMethod in ("welch", "student"),
    +               "Currently supported tests: \"welch\", \"student\"")
    +        self._testMethod = testMethod
    +
    +    @since('2.0.0')
    +    def registerStream(self, data):
    --- End diff --
    
    nit: data -> dstream


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12042] Python API for mllib.stat.test.S...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/11374#issuecomment-188993182
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/51989/
    Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12042] Python API for mllib.stat.test.S...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/11374#issuecomment-219192556
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/58598/
    Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #11374: [SPARK-12042] Python API for mllib.stat.test.Stre...

Posted by feynmanliang <gi...@git.apache.org>.
Github user feynmanliang commented on a diff in the pull request:

    https://github.com/apache/spark/pull/11374#discussion_r85647915
  
    --- Diff: mllib/src/main/scala/org/apache/spark/mllib/stat/test/TestResult.scala ---
    @@ -114,7 +114,7 @@ class KolmogorovSmirnovTestResult private[stat] (
      * Object containing the test results for streaming testing.
      */
     @Since("1.6.0")
    -private[stat] class StreamingTestResult @Since("1.6.0") (
    +class StreamingTestResult @Since("1.6.0") (
    --- End diff --
    
    Does this need to be public? Java API doesn't seem to require it


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #11374: [SPARK-12042] Python API for mllib.stat.test.StreamingTe...

Posted by yinxusen <gi...@git.apache.org>.
Github user yinxusen commented on the issue:

    https://github.com/apache/spark/pull/11374
  
    Ping @mengxr @feynmanliang @yanboliang 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12042] Python API for mllib.stat.test.S...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/11374#issuecomment-192758781
  
    **[Test build #52523 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52523/consoleFull)** for PR 11374 at commit [`e4e8d5e`](https://github.com/apache/spark/commit/e4e8d5ec5d467d7efebb917c6c7e60b14f519e74).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12042] Python API for mllib.stat.test.S...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/11374#issuecomment-188993093
  
    **[Test build #51989 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51989/consoleFull)** for PR 11374 at commit [`770703b`](https://github.com/apache/spark/commit/770703b341fd4f900718cf2fd8a1985170435b65).
     * This patch **fails PySpark unit tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #11374: [SPARK-12042] Python API for mllib.stat.test.Stre...

Posted by feynmanliang <gi...@git.apache.org>.
Github user feynmanliang commented on a diff in the pull request:

    https://github.com/apache/spark/pull/11374#discussion_r85647221
  
    --- Diff: examples/src/main/python/mllib/streaming_test_example.py ---
    @@ -0,0 +1,57 @@
    +#
    +# Licensed to the Apache Software Foundation (ASF) under one or more
    +# contributor license agreements.  See the NOTICE file distributed with
    +# this work for additional information regarding copyright ownership.
    +# The ASF licenses this file to You under the Apache License, Version 2.0
    +# (the "License"); you may not use this file except in compliance with
    +# the License.  You may obtain a copy of the License at
    +#
    +#    http://www.apache.org/licenses/LICENSE-2.0
    +#
    +# Unless required by applicable law or agreed to in writing, software
    +# distributed under the License is distributed on an "AS IS" BASIS,
    +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    +# See the License for the specific language governing permissions and
    +# limitations under the License.
    +#
    +
    +"""
    +Create a DStream that contains several RDDs to show the StreamingTest of PySpark.
    +"""
    +import time
    +import tempfile
    +from shutil import rmtree
    +
    +from pyspark import SparkContext
    +from pyspark.streaming import StreamingContext
    +from pyspark.mllib.stat.test import BinarySample, StreamingTest
    +
    +if __name__ == "__main__":
    +
    +    sc = SparkContext(appName="PythonStreamingTestExample")
    +    ssc = StreamingContext(sc, 1)
    +
    +    checkpoint_path = tempfile.mkdtemp()
    --- End diff --
    
    Is this necessary?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #11374: [SPARK-12042] Python API for mllib.stat.test.Stre...

Posted by feynmanliang <gi...@git.apache.org>.
Github user feynmanliang commented on a diff in the pull request:

    https://github.com/apache/spark/pull/11374#discussion_r85647207
  
    --- Diff: examples/src/main/python/mllib/streaming_test_example.py ---
    @@ -0,0 +1,57 @@
    +#
    +# Licensed to the Apache Software Foundation (ASF) under one or more
    +# contributor license agreements.  See the NOTICE file distributed with
    +# this work for additional information regarding copyright ownership.
    +# The ASF licenses this file to You under the Apache License, Version 2.0
    +# (the "License"); you may not use this file except in compliance with
    +# the License.  You may obtain a copy of the License at
    +#
    +#    http://www.apache.org/licenses/LICENSE-2.0
    +#
    +# Unless required by applicable law or agreed to in writing, software
    +# distributed under the License is distributed on an "AS IS" BASIS,
    +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    +# See the License for the specific language governing permissions and
    +# limitations under the License.
    +#
    +
    +"""
    +Create a DStream that contains several RDDs to show the StreamingTest of PySpark.
    +"""
    --- End diff --
    
    Seems like other examples are including a `from __future__ import print_function` here


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #11374: [SPARK-12042] Python API for mllib.stat.test.StreamingTe...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/11374
  
    **[Test build #67696 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67696/consoleFull)** for PR 11374 at commit [`615fbbb`](https://github.com/apache/spark/commit/615fbbb4aaddd4c5a45f9fff4c58c395f1873340).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #11374: [SPARK-12042] Python API for mllib.stat.test.StreamingTe...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/11374
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #11374: [SPARK-12042] Python API for mllib.stat.test.Stre...

Posted by feynmanliang <gi...@git.apache.org>.
Github user feynmanliang commented on a diff in the pull request:

    https://github.com/apache/spark/pull/11374#discussion_r85647534
  
    --- Diff: python/pyspark/mllib/stat/test.py ---
    @@ -15,10 +15,15 @@
     # limitations under the License.
     #
     
    +from collections import namedtuple
    +
    +from pyspark import SparkContext, since
     from pyspark.mllib.common import inherit_doc, JavaModelWrapper
    +from pyspark.streaming.dstream import DStream
     
     
    -__all__ = ["ChiSqTestResult", "KolmogorovSmirnovTestResult"]
    +__all__ = ["ChiSqTestResult", "KolmogorovSmirnovTestResult", "BinarySample", "StreamingTest",
    --- End diff --
    
    nit: alphabetize


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #11374: [SPARK-12042] Python API for mllib.stat.test.Stre...

Posted by feynmanliang <gi...@git.apache.org>.
Github user feynmanliang commented on a diff in the pull request:

    https://github.com/apache/spark/pull/11374#discussion_r85647905
  
    --- Diff: python/pyspark/mllib/stat/test.py ---
    @@ -80,3 +85,118 @@ class KolmogorovSmirnovTestResult(TestResult):
         """
         Contains test results for the Kolmogorov-Smirnov test.
         """
    +
    +
    +class BinarySample(namedtuple("BinarySample", ["isExperiment", "value"])):
    +    """
    +    Represents a (isExperiment, value) tuple.
    +
    +    .. versionadded:: 2.0.0
    +    """
    +
    +    def __reduce__(self):
    +        return BinarySample, (bool(self.isExperiment), float(self.value))
    +
    +
    +class StreamingTestResult(namedtuple("StreamingTestResult",
    +                                     ["pValue", "degreesOfFreedom", "statistic", "method",
    +                                      "nullHypothesis"])):
    +    """
    +    Contains test results for StreamingTest.
    +
    +    .. versionadded:: 2.0.0
    +    """
    +
    +    def __reduce__(self):
    +        return StreamingTestResult, (float(self.pValue),
    +                                     float(self.degreesOfFreedom), float(self.statistic),
    +                                     str(self.method), str(self.nullHypothesis))
    +
    +
    +class StreamingTest(object):
    +    """
    +    .. note:: Experimental
    +
    +    Online 2-sample significance testing for a stream of (Boolean, Double) pairs. The Boolean
    +    identifies which sample each observation comes from, and the Double is the numeric value of the
    +    observation.
    +
    +    To address novelty affects, the `peacePeriod` specifies a set number of initial RDD batches of
    +    the DStream to be dropped from significance testing.
    +
    +    The `windowSize` sets the number of batches each significance test is to be performed over. The
    +    window is sliding with a stride length of 1 batch. Setting windowSize to 0 will perform
    +    cumulative processing, using all batches seen so far.
    +
    +    Different tests may be used for assessing statistical significance depending on assumptions
    +    satisfied by data. For more details, see StreamingTestMethod. The `testMethod` specifies
    +    which test will be used.
    +
    +    .. versionadded:: 2.0.0
    +    """
    +
    +    def __init__(self):
    +        self._peacePeriod = 0
    +        self._windowSize = 0
    +        self._testMethod = "welch"
    +
    +    @since('2.0.0')
    +    def setPeacePeriod(self, peacePeriod):
    +        """
    +        Update peacePeriod
    +        :param peacePeriod:
    +          Set number of initial RDD batches of the DStream to be dropped from significance testing.
    +        """
    +        self._peacePeriod = peacePeriod
    +
    +    @since('2.0.0')
    +    def setWindowSize(self, windowSize):
    +        """
    +        Update windowSize
    +        :param windowSize:
    +          Set the number of batches each significance test is to be performed over.
    +        """
    +        self._windowSize = windowSize
    +
    +    @since('2.0.0')
    +    def setTestMethod(self, testMethod):
    +        """
    +        Update test method
    +        :param testMethod:
    +          Currently supported tests: `welch`, `student`.
    +        """
    +        assert(testMethod in ("welch", "student"),
    +               "Currently supported tests: \"welch\", \"student\"")
    +        self._testMethod = testMethod
    +
    +    @since('2.0.0')
    +    def registerStream(self, data):
    +        """
    +        Register a data stream to get its test result.
    +
    +        :param data:
    +          The input data stream, each element is a BinarySample instance.
    +        """
    +        self._validate(data)
    +        sc = SparkContext._active_spark_context
    +
    +        streamingTest = sc._jvm.org.apache.spark.mllib.stat.test.StreamingTest()
    +        streamingTest.setPeacePeriod(self._peacePeriod)
    +        streamingTest.setWindowSize(self._windowSize)
    +        streamingTest.setTestMethod(self._testMethod)
    +
    +        javaDStream = sc._jvm.SerDe.pythonToJava(data._jdstream, True)
    +        testResult = streamingTest.registerStream(javaDStream)
    --- End diff --
    
    Why do we need `pythonToJava` and `javaToPython`; its not used for streaming K means https://github.com/apache/spark/blob/39e2bad6a866d27c3ca594d15e574a1da3ee84cc/python/pyspark/mllib/clustering.py#L773


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12042] Python API for mllib.stat.test.S...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/11374#issuecomment-188972433
  
    **[Test build #51989 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51989/consoleFull)** for PR 11374 at commit [`770703b`](https://github.com/apache/spark/commit/770703b341fd4f900718cf2fd8a1985170435b65).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12042] Python API for mllib.stat.test.S...

Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the pull request:

    https://github.com/apache/spark/pull/11374#issuecomment-188992736
  
    cc: @feynmanliang 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #11374: [SPARK-12042] Python API for mllib.stat.test.Stre...

Posted by feynmanliang <gi...@git.apache.org>.
Github user feynmanliang commented on a diff in the pull request:

    https://github.com/apache/spark/pull/11374#discussion_r85647311
  
    --- Diff: examples/src/main/python/mllib/streaming_test_example.py ---
    @@ -0,0 +1,57 @@
    +#
    +# Licensed to the Apache Software Foundation (ASF) under one or more
    +# contributor license agreements.  See the NOTICE file distributed with
    +# this work for additional information regarding copyright ownership.
    +# The ASF licenses this file to You under the Apache License, Version 2.0
    +# (the "License"); you may not use this file except in compliance with
    +# the License.  You may obtain a copy of the License at
    +#
    +#    http://www.apache.org/licenses/LICENSE-2.0
    +#
    +# Unless required by applicable law or agreed to in writing, software
    +# distributed under the License is distributed on an "AS IS" BASIS,
    +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    +# See the License for the specific language governing permissions and
    +# limitations under the License.
    +#
    +
    +"""
    +Create a DStream that contains several RDDs to show the StreamingTest of PySpark.
    +"""
    +import time
    +import tempfile
    +from shutil import rmtree
    +
    +from pyspark import SparkContext
    +from pyspark.streaming import StreamingContext
    +from pyspark.mllib.stat.test import BinarySample, StreamingTest
    +
    +if __name__ == "__main__":
    +
    +    sc = SparkContext(appName="PythonStreamingTestExample")
    +    ssc = StreamingContext(sc, 1)
    +
    +    checkpoint_path = tempfile.mkdtemp()
    +    ssc.checkpoint(checkpoint_path)
    +
    +    # Create the queue through which RDDs can be pushed to a QueueInputDStream.
    +    rdd_queue = []
    +    for i in range(5):
    +        rdd_queue += [ssc.sparkContext.parallelize(
    +            [BinarySample(True, j) for j in range(1, 1001)], 10)]
    +
    +    # Create the QueueInputDStream and use it do some processing.
    +    input_stream = ssc.queueStream(rdd_queue)
    +
    +    model = StreamingTest()
    +    test_result = model.registerStream(input_stream)
    +
    +    test_result.pprint()
    +
    +    ssc.start()
    +    time.sleep(12)
    --- End diff --
    
    Is this necessary? Doesn't seem to be required for `streaming_kmeans` example


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12042] Python API for mllib.stat.test.S...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/11374#issuecomment-219192553
  
    **[Test build #58598 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/58598/consoleFull)** for PR 11374 at commit [`e4e8d5e`](https://github.com/apache/spark/commit/e4e8d5ec5d467d7efebb917c6c7e60b14f519e74).
     * This patch **fails R style tests**.
     * This patch **does not merge cleanly**.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12042] Python API for mllib.stat.test.S...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/11374#issuecomment-192767956
  
    **[Test build #52523 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52523/consoleFull)** for PR 11374 at commit [`e4e8d5e`](https://github.com/apache/spark/commit/e4e8d5ec5d467d7efebb917c6c7e60b14f519e74).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12042] Python API for mllib.stat.test.S...

Posted by yinxusen <gi...@git.apache.org>.
Github user yinxusen commented on the pull request:

    https://github.com/apache/spark/pull/11374#issuecomment-192758511
  
    test it please



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12042] Python API for mllib.stat.test.S...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/11374#issuecomment-192768226
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52523/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #11374: [SPARK-12042] Python API for mllib.stat.test.StreamingTe...

Posted by feynmanliang <gi...@git.apache.org>.
Github user feynmanliang commented on the issue:

    https://github.com/apache/spark/pull/11374
  
    I'll review this tonight 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #11374: [SPARK-12042] Python API for mllib.stat.test.Stre...

Posted by feynmanliang <gi...@git.apache.org>.
Github user feynmanliang commented on a diff in the pull request:

    https://github.com/apache/spark/pull/11374#discussion_r85647289
  
    --- Diff: examples/src/main/python/mllib/streaming_test_example.py ---
    @@ -0,0 +1,57 @@
    +#
    +# Licensed to the Apache Software Foundation (ASF) under one or more
    +# contributor license agreements.  See the NOTICE file distributed with
    +# this work for additional information regarding copyright ownership.
    +# The ASF licenses this file to You under the Apache License, Version 2.0
    +# (the "License"); you may not use this file except in compliance with
    +# the License.  You may obtain a copy of the License at
    +#
    +#    http://www.apache.org/licenses/LICENSE-2.0
    +#
    +# Unless required by applicable law or agreed to in writing, software
    +# distributed under the License is distributed on an "AS IS" BASIS,
    +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    +# See the License for the specific language governing permissions and
    +# limitations under the License.
    +#
    +
    +"""
    +Create a DStream that contains several RDDs to show the StreamingTest of PySpark.
    +"""
    +import time
    +import tempfile
    +from shutil import rmtree
    +
    +from pyspark import SparkContext
    +from pyspark.streaming import StreamingContext
    +from pyspark.mllib.stat.test import BinarySample, StreamingTest
    +
    +if __name__ == "__main__":
    +
    +    sc = SparkContext(appName="PythonStreamingTestExample")
    +    ssc = StreamingContext(sc, 1)
    +
    +    checkpoint_path = tempfile.mkdtemp()
    +    ssc.checkpoint(checkpoint_path)
    +
    +    # Create the queue through which RDDs can be pushed to a QueueInputDStream.
    +    rdd_queue = []
    --- End diff --
    
    use `camelCase` to be consistent with other examples


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #11374: [SPARK-12042] Python API for mllib.stat.test.Stre...

Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:

    https://github.com/apache/spark/pull/11374


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #11374: [SPARK-12042] Python API for mllib.stat.test.StreamingTe...

Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on the issue:

    https://github.com/apache/spark/pull/11374
  
    Any updates to this PR?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #11374: [SPARK-12042] Python API for mllib.stat.test.Stre...

Posted by feynmanliang <gi...@git.apache.org>.
Github user feynmanliang commented on a diff in the pull request:

    https://github.com/apache/spark/pull/11374#discussion_r85647454
  
    --- Diff: mllib/src/main/scala/org/apache/spark/mllib/api/python/PythonMLLibAPI.scala ---
    @@ -1358,6 +1363,41 @@ private[spark] abstract class SerDeBase {
           }
         }.toJavaRDD()
       }
    +
    +  /**
    +   * Convert a DStream of Java objects to a DStream of serialized Python objects, that is usable by
    +   * PySpark.
    +   */
    +  def javaToPython(jDStream: JavaDStream[Any]): JavaDStream[Array[Byte]] = {
    +    val dStream = jDStream.dstream.mapPartitions { iter =>
    +      initialize()  // let it called in executor
    +      new SerDeUtil.AutoBatchedPickler(iter)
    --- End diff --
    
    This should be an `Array[Byte]`, so you don't need to assign to `dStream` and then call the constructor on L1376


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #11374: [SPARK-12042] Python API for mllib.stat.test.StreamingTe...

Posted by feynmanliang <gi...@git.apache.org>.
Github user feynmanliang commented on the issue:

    https://github.com/apache/spark/pull/11374
  
    Apologies for the delay, I am traveling but I'll get this done this weekend. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12042] Python API for mllib.stat.test.S...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/11374#issuecomment-219192555
  
    Build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12042] Python API for mllib.stat.test.S...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/11374#issuecomment-188993181
  
    Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org