You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by dorx <gi...@git.apache.org> on 2014/06/09 23:06:49 UTC

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

GitHub user dorx opened a pull request:

    https://github.com/apache/spark/pull/1025

    [SPARK-2082] stratified sampling in PairRDDFunctions that guarantees exact sample size

    Implemented stratified sampling that guarantees exact sample size using ScaRSR with two passes over the RDD for sampling without replacement and three passes for sampling with replacement.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/dorx/spark stratified

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/1025.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #1025
    
----
commit 14419775202e6eef1f0e1f0c74c7be9030aca73d
Author: Doris Xin <do...@gmail.com>
Date:   2014-05-29T22:22:14Z

    SPARK-1939 Refactor takeSample method in RDD to use ScaSRS

commit ffea61a67d228edb476d29ca13a84bb3f9a22887
Author: Doris Xin <do...@gmail.com>
Date:   2014-05-30T00:55:54Z

    SPARK-1939: Refactor takeSample method in RDD
    
    Reviewer comments addressed:
    - commons-math3 is now a test-only dependency. bumped up to v3.3
    - comments added to explain what computeFraction is doing
    - fixed the unit for computeFraction to use BinomialDitro for without
    replacement sampling
    - stylistic fixes

commit 7cab53a3926f4351432e5e3600b0796b9a4146e4
Author: Doris Xin <do...@gmail.com>
Date:   2014-06-02T19:00:38Z

    fixed import bug in rdd.py

commit e3fd6a628317d559a08a7a20421e9c0618180902
Author: Doris Xin <do...@gmail.com>
Date:   2014-06-02T19:06:18Z

    Merge branch 'master' into takeSample

commit 9ee94ee3c28e8d808063fef4e5d39f06ab738e0b
Author: Doris Xin <do...@gmail.com>
Date:   2014-06-09T20:15:23Z

    [SPARK-2082] stratified sampling in PairRDDFunctions that guarantees exact sample size

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the pull request:

    https://github.com/apache/spark/pull/1025#issuecomment-50290028
  
    @dorx I removed commons-math3 from dependencies, separated `sampleByKey` and `sampleByKeyExact`, and corrected the math in waitlisting in sampling with replacement.
    
    Other than those changes, the refactoring was completely based on your implementation. I did some renaming but I'm not taking any credit for the implementation. Some of the refactoring was unnecessary, which was partially because I did this around 5am and my head was not clear. I'm really sorry for the unnecessary code changes.
    
    Since this is apparently your contribution, do you mind re-opening the PR? Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by dorx <gi...@git.apache.org>.
Github user dorx closed the pull request at:

    https://github.com/apache/spark/pull/1025


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1025#issuecomment-46518417
  
     Merged build triggered. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1025#issuecomment-47984385
  
     Merged build triggered. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by dorx <gi...@git.apache.org>.
Github user dorx commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1025#discussion_r14577393
  
    --- Diff: core/src/main/scala/org/apache/spark/util/random/SamplingUtils.scala ---
    @@ -45,11 +50,74 @@ private[spark] object SamplingUtils {
         val fraction = sampleSizeLowerBound.toDouble / total
         if (withReplacement) {
           val numStDev = if (sampleSizeLowerBound < 12) 9 else 5
    -      fraction + numStDev * math.sqrt(fraction / total)
    +      math.max(1e-10, fraction + numStDev * math.sqrt(fraction / total))
         } else {
           val delta = 1e-4
           val gamma = - math.log(delta) / total
    -      math.min(1, fraction + gamma + math.sqrt(gamma * gamma + 2 * gamma * fraction))
    +      math.min(1,
    +        math.max(1e-10, fraction + gamma + math.sqrt(gamma * gamma + 2 * gamma * fraction)))
    +    }
    +  }
    +}
    +
    +/**
    + * Utility functions that help us determine bounds on adjusted sampling rate to guarantee exact
    + * sample sizes with high confidence when sampling with replacement.
    + *
    + * The algorithm for guaranteeing sample size instantly accepts items whose associated value drawn
    + * from Pois(s) is less than the lower bound and puts items whose value is between the lower and
    + * upper bound in a waitlist. The final sample is consisted of all items accepted on the fly and a
    + * portion of the waitlist needed to make the exact sample size.
    + */
    +private[spark] object PoissonBounds {
    +
    +  val delta = 1e-4 / 3.0
    +
    +  /**
    +   * Compute the threshold for accepting items on the fly. The threshold value is a fairly small
    +   * number, which means if the item has an associated value < threshold, it is highly likely to
    +   * be in the final sample. Hence we accept items with values less than the returned value of this
    +   * function instantly.
    +   *
    +   * @param s sample size
    +   * @return threshold for accepting items on the fly
    +   */
    +  def getLowerBound(s: Double): Double = {
    --- End diff --
    
    Actually on a second thought, the implementation becomes unnecessarily muddled if we combine the two functions. I'd prefer to leave them as is for clarity. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by falaki <gi...@git.apache.org>.
Github user falaki commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1025#discussion_r14544802
  
    --- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
    @@ -0,0 +1,296 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.util.random
    +
    +import scala.collection.{Map, mutable}
    +import scala.collection.mutable.ArrayBuffer
    +import scala.reflect.ClassTag
    +
    +import org.apache.commons.math3.random.RandomDataGenerator
    +import org.apache.spark.{Logging, SparkContext, TaskContext}
    +import org.apache.spark.rdd.RDD
    +import org.apache.spark.util.Utils
    +import org.apache.spark.util.random.{PoissonBounds => PB}
    +
    +private[spark] object StratifiedSampler extends Logging {
    +
    +  /**
    +   * A version of {@link #aggregate()} that passes the TaskContext to the function that does
    +   * aggregation for each partition. This function avoids creating an extra depth in the RDD
    +   * lineage, as opposed to using mapPartitionsWithId, which results in slightly improved run time.
    +   */
    +  def aggregateWithContext[U: ClassTag, T: ClassTag](zeroValue: U)
    +      (rdd: RDD[T],
    +       seqOp: ((TaskContext, U), T) => U,
    +       combOp: (U, U) => U): U = {
    +    val sc: SparkContext = rdd.sparkContext
    +    // Clone the zero value since we will also be serializing it as part of tasks
    +    var jobResult = Utils.clone(zeroValue, sc.env.closureSerializer.newInstance())
    +    // pad seqOp and combOp with taskContext to conform to aggregate's signature in TraversableOnce
    +    val paddedSeqOp = (arg1: (TaskContext, U), item: T) => (arg1._1, seqOp(arg1, item))
    +    val paddedcombOp = (arg1: (TaskContext, U), arg2: (TaskContext, U)) =>
    +      (arg1._1, combOp(arg1._2, arg1._2))
    +    val cleanSeqOp = sc.clean(paddedSeqOp)
    +    val cleanCombOp = sc.clean(paddedcombOp)
    +    val aggregatePartition = (tc: TaskContext, it: Iterator[T]) =>
    +      (it.aggregate(tc, zeroValue)(cleanSeqOp, cleanCombOp))._2
    +    val mergeResult = (index: Int, taskResult: U) => jobResult = combOp(jobResult, taskResult)
    +    sc.runJob(rdd, aggregatePartition, mergeResult)
    +    jobResult
    +  }
    +
    +  /**
    +   * Returns the function used by aggregate to collect sampling statistics for each partition.
    +   */
    +  def getSeqOp[K, V](withReplacement: Boolean,
    +      fractionByKey: (K => Double),
    +      counts: Option[Map[K, Long]]): ((TaskContext, Result[K]),(K, V)) => Result[K] = {
    +    val delta = 5e-5
    +    (U: (TaskContext, Result[K]), item: (K, V)) => {
    +      val result = U._2
    +      val tc = U._1
    +      val rng = result.getRand(tc.partitionId)
    +      val fraction = fractionByKey(item._1)
    +      val stratum = result.getEntry(item._1)
    +      if (withReplacement) {
    +        // compute q1 and q2 only if they haven't been computed already
    +        // since they don't change from iteration to iteration.
    +        // TODO change this to the streaming version
    +        if (stratum.q1.isEmpty || stratum.q2.isEmpty) {
    +          val n = counts.get(item._1)
    +          val s = math.ceil(n * fraction).toLong
    +          val lmbd1 = PB.getLowerBound(s)
    +          val minCount = PB.getMinCount(lmbd1)
    +          val lmbd2 = if (lmbd1 == 0) PB.getUpperBound(s) else PB.getUpperBound(s - minCount)
    +          val q1 = lmbd1 / n
    +          val q2 = lmbd2 / n
    +          stratum.q1 = Some(q1)
    +          stratum.q2 = Some(q2)
    +        }
    +        val x1 = if (stratum.q1.get == 0) 0L else rng.nextPoisson(stratum.q1.get)
    +        if (x1 > 0) {
    +          stratum.incrNumAccepted(x1)
    +        }
    +        val x2 = rng.nextPoisson(stratum.q2.get).toInt
    +        if (x2 > 0) {
    +          stratum.addToWaitList(ArrayBuffer.fill(x2)(rng.nextUniform(0.0, 1.0)))
    +        }
    +      } else {
    +        // We use the streaming version of the algorithm for sampling without replacement.
    +        // Hence, q1 and q2 change on every iteration.
    +        val g1 = - math.log(delta) / stratum.numItems
    +        val g2 = (2.0 / 3.0) * g1
    +        val q1 = math.max(0, fraction + g2 - math.sqrt((g2 * g2 + 3 * g2 * fraction)))
    +        val q2 = math.min(1, fraction + g1 + math.sqrt(g1 * g1 + 2 * g1 * fraction))
    +
    +        val x = rng.nextUniform(0.0, 1.0)
    +        if (x < q1) {
    +          stratum.incrNumAccepted()
    +        } else if (x < q2) {
    +          stratum.addToWaitList(x)
    +        }
    +        stratum.q1 = Some(q1)
    +        stratum.q2 = Some(q2)
    +      }
    +      stratum.incrNumItems()
    +      result
    +    }
    +  }
    +
    +  /**
    +   * Returns the function used by aggregate to combine results from different partitions, as
    +   * returned by seqOp.
    +   */
    +  def getCombOp[K](): (Result[K], Result[K]) => Result[K] = {
    +    (r1: Result[K], r2: Result[K]) => {
    +      // take union of both key sets in case one partition doesn't contain all keys
    +      val keyUnion = r1.resultMap.keys.toSet.union(r2.resultMap.keys.toSet)
    +
    +      // Use r2 to keep the combined result since r1 is usual empty
    +      for (key <- keyUnion) {
    +        val entry1 = r1.resultMap.get(key)
    +        val entry2 = r2.resultMap.get(key)
    +        if (entry2.isEmpty && entry1.isDefined) {
    +          r2.resultMap += (key -> entry1.get)
    +        } else if (entry1.isDefined && entry2.isDefined) {
    +          entry2.get.addToWaitList(entry1.get.waitList)
    +          entry2.get.incrNumAccepted(entry1.get.numAccepted)
    +          entry2.get.incrNumItems(entry1.get.numItems)
    +        }
    +      }
    +      r2
    +    }
    +  }
    +
    +  /**
    +   * Given the result returned by the aggregate function, we need to determine the threshold used
    +   * to accept items to generate the exact sample size.
    +   */
    +  def computeThresholdByKey[K](finalResult: Map[K, Stratum], fractionByKey: (K => Double)):
    +    (K => Double) = {
    +    val thresholdByKey = new mutable.HashMap[K, Double]()
    +    for ((key, stratum) <- finalResult) {
    +      val fraction = fractionByKey(key)
    +      val s = math.ceil(stratum.numItems * fraction).toLong
    +      if (stratum.numAccepted > s) {
    +        logWarning("Pre-accepted too many")
    +        thresholdByKey += (key -> stratum.q1.get)
    +      } else {
    +        val numWaitListAccepted = (s - stratum.numAccepted).toInt
    +        if (numWaitListAccepted >= stratum.waitList.size) {
    +          logWarning("WaitList too short")
    +          thresholdByKey += (key -> stratum.q2.get)
    +        } else {
    +          thresholdByKey += (key -> stratum.waitList.sorted.apply(numWaitListAccepted))
    +        }
    +      }
    +    }
    +    thresholdByKey
    +  }
    +
    +  /**
    +   * Return the per partition sampling function used for sampling without replacement.
    +   *
    +   * When exact sample size is required, we make an additional pass over the RDD to determine the
    +   * exact sampling rate that guarantees sample size with high confidence.
    +   *
    +   * The sampling function has a unique seed per partition.
    +   */
    +  def getBernoulliSamplingFunction[K, V](rdd: RDD[(K,  V)],
    +      fractionByKey: K => Double,
    +      exact: Boolean,
    +      seed: Long): (Int, Iterator[(K, V)]) => Iterator[(K, V)] = {
    +    var samplingRateByKey = fractionByKey
    +    if (exact) {
    +      // determine threshold for each stratum and resample
    +      val seqOp = StratifiedSampler.getSeqOp[K,V](false, fractionByKey, None)
    +      val combOp = StratifiedSampler.getCombOp[K]()
    +      val zeroU = new Result[K](Map[K, Stratum](), seed = seed)
    +      val finalResult = aggregateWithContext(zeroU)(rdd, seqOp, combOp).resultMap
    +      samplingRateByKey = StratifiedSampler.computeThresholdByKey(finalResult, fractionByKey)
    +    }
    +    (idx: Int, iter: Iterator[(K, V)]) => {
    +      val random = new RandomDataGenerator
    +      random.reSeed(seed + idx)
    +      iter.filter(t => random.nextUniform(0.0, 1.0) < samplingRateByKey(t._1))
    +    }
    +  }
    +
    +  /**
    +   * Return the per partition sampling function used for sampling with replacement.
    +   *
    +   * When exact sample size is required, we make two additional passed over the RDD to determine
    +   * the exact sampling rate that guarantees sample size with high confidence. The first pass
    +   * counts the number of items in each stratum (group of items with the same key) in the RDD, and
    +   * the second pass uses the counts to determine exact sampling rates.
    +   *
    +   * The sampling function has a unique seed per partition.
    +   */
    +  def getPoissonSamplingFunction[K, V](rdd:RDD[(K,  V)],
    +      fractionByKey: K => Double,
    +      exact: Boolean,
    +      counts: Option[Map[K, Long]],
    +      seed: Long): (Int, Iterator[(K, V)]) => Iterator[(K, V)] = {
    +    // TODO implement the streaming version of sampling w/ replacement that doesn't require counts
    +    if (exact) {
    +      val seqOp = StratifiedSampler.getSeqOp[K,V](true, fractionByKey, counts)
    +      val combOp = StratifiedSampler.getCombOp[K]()
    +      val zeroU = new Result[K](Map[K, Stratum](), seed = seed)
    +      val finalResult = aggregateWithContext(zeroU)(rdd, seqOp, combOp).resultMap
    +      val thresholdByKey = StratifiedSampler.computeThresholdByKey(finalResult, fractionByKey)
    +      (idx: Int, iter: Iterator[(K, V)]) => {
    +        val random = new RandomDataGenerator()
    +        random.reSeed(seed + idx)
    +        iter.flatMap { t =>
    +          val q1 = finalResult.get(t._1).get.q1.getOrElse(0.0)
    --- End diff --
    
    Similarly, better names instead of q1, q2, g1, and g2.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by dorx <gi...@git.apache.org>.
Github user dorx commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1025#discussion_r14691577
  
    --- Diff: pom.xml ---
    @@ -257,6 +257,11 @@
     	    <version>1.5</version>
           </dependency>
           <dependency>
    +        <groupId>org.apache.commons</groupId>
    +        <artifactId>commons-math3</artifactId>
    +        <version>3.3</version>
    --- End diff --
    
    3.1.1 is good for me


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by dorx <gi...@git.apache.org>.
Github user dorx commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1025#discussion_r14906754
  
    --- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
    @@ -0,0 +1,311 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.util.random
    +
    +import scala.collection.Map
    +import scala.collection.mutable.{ArrayBuffer, HashMap, Map => MMap}
    +
    +import org.apache.commons.math3.random.RandomDataGenerator
    +import org.apache.spark.Logging
    +import org.apache.spark.rdd.RDD
    +
    +/**
    + * Auxiliary functions and data structures for the sampleByKey method in PairRDDFunctions.
    + *
    + * Essentially, when exact sample size is necessary, we make additional passes over the RDD to
    + * compute the exact threshold value to use for each stratum to guarantee exact sample size with
    + * high probability. This is achieved by maintaining a waitlist of size O(log(s)), where s is the
    + * desired sample size for each stratum.
    + *
    + * Like in simple random sampling, we generate a random value for each item from the
    + * uniform  distribution [0.0, 1.0]. All items with values <= min(values of items in the waitlist)
    + * are accepted into the sample instantly. The threshold for instant accept is designed so that
    + * s - numAccepted = O(log(s)), where s is again the desired sample size. Thus, by maintaining a
    + * waitlist size = O(log(s)), we will be able to create a sample of the exact size s by adding
    + * a portion of the waitlist to the set of items that are instantly accepted. The exact threshold
    + * is computed by sorting the values in the waitlist and picking the value at (s - numAccepted).
    + *
    + * Note that since we use the same seed for the RNG when computing the thresholds and the actual
    + * sample, our computed thresholds are guaranteed to produce the desired sample size.
    + *
    + * For more theoretical background on the sampling techniques used here, please refer to
    + * http://jmlr.org/proceedings/papers/v28/meng13a.html
    + */
    +
    +private[spark] object StratifiedSampler extends Logging {
    --- End diff --
    
    I'd prefer renaming the class to `StratifiedSamplingUtils` instead of adding too many things into `SamplingUtils`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1025#issuecomment-45957666
  
    
    Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15725/


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by dorx <gi...@git.apache.org>.
Github user dorx commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1025#discussion_r13889473
  
    --- Diff: core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala ---
    @@ -156,6 +161,182 @@ class PairRDDFunctions[K, V](self: RDD[(K, V)])
       }
     
       /**
    +   * Return a subset of this RDD sampled by key (via stratified sampling).
    +   * We guarantee a sample size = math.ceil(fraction * S_i), where S_i is the size of the ith
    +   * stratum.
    +   *
    +   * @param withReplacement whether to sample with or without replacement
    +   * @param fraction sampling rate
    +   * @param seed seed for the random number generator
    +   * @return RDD containing the sampled subset
    +   */
    +  def sampleByKey(withReplacement: Boolean,
    +      fraction: Double,
    +      seed: Long = Utils.random.nextLong): RDD[(K, V)]= {
    +
    +    class Stratum(var numItems: Long = 0L, var numAccepted: Long = 0L) extends Serializable {
    +      var waitList: ArrayBuffer[Double] = new ArrayBuffer[Double]
    +      var q1: Option[Double] = None
    +      var q2: Option[Double] = None
    +
    +      def incrNumItems(by: Long = 1L) = numItems += by
    +
    +      def incrNumAccepted(by: Long = 1L) = numAccepted += by
    +
    +      def addToWaitList(elem: Double) = waitList += elem
    +
    +      def addToWaitList(elems: ArrayBuffer[Double]) = waitList ++= elems
    +
    +      override def toString() = {
    +        "numItems: " + numItems + " numAccepted: " + numAccepted + " q1: " + q1 + " q2: " + q2 +
    +          " waitListSize:" + waitList.size
    +      }
    +    }
    +
    +    class Result(var resultMap: Map[K, Stratum], var cachedPartitionId: Option[Int] = None)
    +      extends Serializable {
    +      var rand: RandomDataGenerator = new RandomDataGenerator
    +
    +      def getEntry(key: K, numItems: Long = 0L): Stratum = {
    +        if (resultMap.get(key).isEmpty) {
    --- End diff --
    
    actually  
    if (!resultMap.contains(key)) {


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1025#issuecomment-46643449
  
     Merged build triggered. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by falaki <gi...@git.apache.org>.
Github user falaki commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1025#discussion_r14688145
  
    --- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
    @@ -0,0 +1,310 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.util.random
    +
    +import scala.collection.Map
    +import scala.collection.mutable.{ArrayBuffer, HashMap, Map => MMap}
    +
    +import org.apache.commons.math3.random.RandomDataGenerator
    +import org.apache.spark.Logging
    +import org.apache.spark.rdd.RDD
    +
    +/**
    + * Auxiliary functions and data structures for the sampleByKey method in PairRDDFunctions.
    + *
    + * Essentially, when exact sample size is necessary, we make additional passes over the RDD to
    + * compute the exact threshold value to use for each stratum to guarantee exact sample size with
    + * high probability. This is achieved by maintaining a waitlist of size O(log(s)), where s is the
    + * desired sample size for each stratum.
    + *
    + * Like in simple random sampling, we generate a random value for each item from the
    + * uniform  distribution [0.0, 1.0]. All items with values <= min(values of items in the waitlist)
    + * are accepted into the sample instantly. The threshold for instant accept is designed so that
    + * s - numAccepted = O(log(s)), where s is again the desired sample size. Thus, by maintaining a
    + * waitlist size = O(log(s)), we will be able to create a sample of the exact size s by adding
    + * a portion of the waitlist to the set of items that are instantly accepted. The exact threshold
    + * is computed by sorting the values in the waitlist and picking the value at (s - numAccepted).
    + *
    + * Note that since we use the same seed for the RNG when computing the thresholds and the actual
    + * sample, our computed thresholds are guaranteed to produce the desired sample size.
    + *
    + * For more theoretical background on the sampling techniques used here, please refer to
    + * http://jmlr.org/proceedings/papers/v28/meng13a.html
    + */
    +
    +private[spark] object StratifiedSampler extends Logging {
    +
    +  /**
    +   * Count the number of items instantly accepted and generate the waitlist for each stratum.
    +   *
    +   * This is only invoked when exact sample size is required.
    +   */
    +  def getCounts[K, V](rdd: RDD[(K, V)],
    +      withReplacement: Boolean,
    +      fractions: Map[K, Double],
    +      counts: Option[Map[K, Long]],
    +      seed: Long): MMap[K, Stratum] = {
    +    val combOp = getCombOp[K]
    +    val mappedPartitionRDD = rdd.mapPartitionsWithIndex({ case (partition, iter) =>
    --- End diff --
    
    Parenthesis is superfluous (the bracket is enough).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/1025#issuecomment-50393886
  
    QA results for PR 1025:<br>- This patch FAILED unit tests.<br><br>For more information see test ouptut:<br>https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17293/consoleFull


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1025#issuecomment-45553455
  
    Build started. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1025#issuecomment-45968209
  
    
    Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15741/


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1025#issuecomment-46608074
  
     Merged build triggered. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by mateiz <gi...@git.apache.org>.
Github user mateiz commented on the pull request:

    https://github.com/apache/spark/pull/1025#issuecomment-50101767
  
    Sorry, how was the API changed, was it making `sampleByKeyExact` a separate method and making it experimental? That actually seems okay to me, the algorithm there is quite a bit more involved.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1025#discussion_r14694248
  
    --- Diff: core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala ---
    @@ -195,6 +193,37 @@ class PairRDDFunctions[K, V](self: RDD[(K, V)])
       }
     
       /**
    +   * Return a subset of this RDD sampled by key (via stratified sampling).
    +   *
    +   * If exact set to true, we guarantee, with high probability, a sample size =
    +   * math.ceil(fraction * S_i), where S_i is the size of the ith stratum (collection of entries
    +   * that share the same key). When sampling without replacement, we need one additional pass over
    +   * the RDD to guarantee sample size with a 99.99% confidence; when sampling with replacement, we
    +   * need two additional passes.
    +   *
    +   * @param withReplacement whether to sample with or without replacement
    +   * @param fractions map of specific keys to sampling rates
    +   * @param seed seed for the random number generator
    +   * @param exact whether sample size needs to be exactly math.ceil(fraction * size) per stratum
    +   * @return RDD containing the sampled subset
    +   */
    +  def sampleByKey(withReplacement: Boolean,
    +      fractions: Map[K, Double],
    +      exact: Boolean = true,
    --- End diff --
    
    I would suggest using `false` as the default value, to match the behavior of `RDD.sample`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by dorx <gi...@git.apache.org>.
Github user dorx commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1025#discussion_r14577966
  
    --- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
    @@ -0,0 +1,296 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.util.random
    +
    +import scala.collection.{Map, mutable}
    +import scala.collection.mutable.ArrayBuffer
    +import scala.reflect.ClassTag
    +
    +import org.apache.commons.math3.random.RandomDataGenerator
    +import org.apache.spark.{Logging, SparkContext, TaskContext}
    +import org.apache.spark.rdd.RDD
    +import org.apache.spark.util.Utils
    +import org.apache.spark.util.random.{PoissonBounds => PB}
    +
    +private[spark] object StratifiedSampler extends Logging {
    +
    +  /**
    +   * A version of {@link #aggregate()} that passes the TaskContext to the function that does
    +   * aggregation for each partition. This function avoids creating an extra depth in the RDD
    +   * lineage, as opposed to using mapPartitionsWithId, which results in slightly improved run time.
    +   */
    +  def aggregateWithContext[U: ClassTag, T: ClassTag](zeroValue: U)
    +      (rdd: RDD[T],
    +       seqOp: ((TaskContext, U), T) => U,
    +       combOp: (U, U) => U): U = {
    +    val sc: SparkContext = rdd.sparkContext
    +    // Clone the zero value since we will also be serializing it as part of tasks
    +    var jobResult = Utils.clone(zeroValue, sc.env.closureSerializer.newInstance())
    +    // pad seqOp and combOp with taskContext to conform to aggregate's signature in TraversableOnce
    +    val paddedSeqOp = (arg1: (TaskContext, U), item: T) => (arg1._1, seqOp(arg1, item))
    +    val paddedcombOp = (arg1: (TaskContext, U), arg2: (TaskContext, U)) =>
    +      (arg1._1, combOp(arg1._2, arg1._2))
    +    val cleanSeqOp = sc.clean(paddedSeqOp)
    +    val cleanCombOp = sc.clean(paddedcombOp)
    +    val aggregatePartition = (tc: TaskContext, it: Iterator[T]) =>
    +      (it.aggregate(tc, zeroValue)(cleanSeqOp, cleanCombOp))._2
    +    val mergeResult = (index: Int, taskResult: U) => jobResult = combOp(jobResult, taskResult)
    +    sc.runJob(rdd, aggregatePartition, mergeResult)
    +    jobResult
    +  }
    +
    +  /**
    +   * Returns the function used by aggregate to collect sampling statistics for each partition.
    +   */
    +  def getSeqOp[K, V](withReplacement: Boolean,
    +      fractionByKey: (K => Double),
    +      counts: Option[Map[K, Long]]): ((TaskContext, Result[K]),(K, V)) => Result[K] = {
    +    val delta = 5e-5
    +    (U: (TaskContext, Result[K]), item: (K, V)) => {
    +      val result = U._2
    +      val tc = U._1
    +      val rng = result.getRand(tc.partitionId)
    +      val fraction = fractionByKey(item._1)
    +      val stratum = result.getEntry(item._1)
    +      if (withReplacement) {
    +        // compute q1 and q2 only if they haven't been computed already
    +        // since they don't change from iteration to iteration.
    +        // TODO change this to the streaming version
    +        if (stratum.q1.isEmpty || stratum.q2.isEmpty) {
    +          val n = counts.get(item._1)
    +          val s = math.ceil(n * fraction).toLong
    +          val lmbd1 = PB.getLowerBound(s)
    +          val minCount = PB.getMinCount(lmbd1)
    +          val lmbd2 = if (lmbd1 == 0) PB.getUpperBound(s) else PB.getUpperBound(s - minCount)
    +          val q1 = lmbd1 / n
    +          val q2 = lmbd2 / n
    +          stratum.q1 = Some(q1)
    +          stratum.q2 = Some(q2)
    +        }
    +        val x1 = if (stratum.q1.get == 0) 0L else rng.nextPoisson(stratum.q1.get)
    +        if (x1 > 0) {
    +          stratum.incrNumAccepted(x1)
    +        }
    +        val x2 = rng.nextPoisson(stratum.q2.get).toInt
    +        if (x2 > 0) {
    +          stratum.addToWaitList(ArrayBuffer.fill(x2)(rng.nextUniform(0.0, 1.0)))
    +        }
    +      } else {
    +        // We use the streaming version of the algorithm for sampling without replacement.
    +        // Hence, q1 and q2 change on every iteration.
    +        val g1 = - math.log(delta) / stratum.numItems
    +        val g2 = (2.0 / 3.0) * g1
    +        val q1 = math.max(0, fraction + g2 - math.sqrt((g2 * g2 + 3 * g2 * fraction)))
    +        val q2 = math.min(1, fraction + g1 + math.sqrt(g1 * g1 + 2 * g1 * fraction))
    +
    +        val x = rng.nextUniform(0.0, 1.0)
    +        if (x < q1) {
    +          stratum.incrNumAccepted()
    +        } else if (x < q2) {
    +          stratum.addToWaitList(x)
    +        }
    +        stratum.q1 = Some(q1)
    +        stratum.q2 = Some(q2)
    +      }
    +      stratum.incrNumItems()
    +      result
    +    }
    +  }
    +
    +  /**
    +   * Returns the function used by aggregate to combine results from different partitions, as
    +   * returned by seqOp.
    +   */
    +  def getCombOp[K](): (Result[K], Result[K]) => Result[K] = {
    +    (r1: Result[K], r2: Result[K]) => {
    +      // take union of both key sets in case one partition doesn't contain all keys
    +      val keyUnion = r1.resultMap.keys.toSet.union(r2.resultMap.keys.toSet)
    +
    +      // Use r2 to keep the combined result since r1 is usual empty
    +      for (key <- keyUnion) {
    +        val entry1 = r1.resultMap.get(key)
    --- End diff --
    
    r1.resultMap(key) is equivalent to r1.resultMap.get(key).get, which removed the ability to check for null entries. So the gets here aren't superfluous. But I'll move some of the logic into helper functions in Result instead.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1025#issuecomment-48386790
  
     Merged build triggered. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by falaki <gi...@git.apache.org>.
Github user falaki commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1025#discussion_r14544806
  
    --- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
    @@ -0,0 +1,296 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.util.random
    +
    +import scala.collection.{Map, mutable}
    +import scala.collection.mutable.ArrayBuffer
    +import scala.reflect.ClassTag
    +
    +import org.apache.commons.math3.random.RandomDataGenerator
    +import org.apache.spark.{Logging, SparkContext, TaskContext}
    +import org.apache.spark.rdd.RDD
    +import org.apache.spark.util.Utils
    +import org.apache.spark.util.random.{PoissonBounds => PB}
    +
    +private[spark] object StratifiedSampler extends Logging {
    +
    +  /**
    +   * A version of {@link #aggregate()} that passes the TaskContext to the function that does
    +   * aggregation for each partition. This function avoids creating an extra depth in the RDD
    +   * lineage, as opposed to using mapPartitionsWithId, which results in slightly improved run time.
    +   */
    +  def aggregateWithContext[U: ClassTag, T: ClassTag](zeroValue: U)
    +      (rdd: RDD[T],
    +       seqOp: ((TaskContext, U), T) => U,
    +       combOp: (U, U) => U): U = {
    +    val sc: SparkContext = rdd.sparkContext
    +    // Clone the zero value since we will also be serializing it as part of tasks
    +    var jobResult = Utils.clone(zeroValue, sc.env.closureSerializer.newInstance())
    +    // pad seqOp and combOp with taskContext to conform to aggregate's signature in TraversableOnce
    +    val paddedSeqOp = (arg1: (TaskContext, U), item: T) => (arg1._1, seqOp(arg1, item))
    +    val paddedcombOp = (arg1: (TaskContext, U), arg2: (TaskContext, U)) =>
    +      (arg1._1, combOp(arg1._2, arg1._2))
    +    val cleanSeqOp = sc.clean(paddedSeqOp)
    +    val cleanCombOp = sc.clean(paddedcombOp)
    +    val aggregatePartition = (tc: TaskContext, it: Iterator[T]) =>
    +      (it.aggregate(tc, zeroValue)(cleanSeqOp, cleanCombOp))._2
    +    val mergeResult = (index: Int, taskResult: U) => jobResult = combOp(jobResult, taskResult)
    +    sc.runJob(rdd, aggregatePartition, mergeResult)
    +    jobResult
    +  }
    +
    +  /**
    +   * Returns the function used by aggregate to collect sampling statistics for each partition.
    +   */
    +  def getSeqOp[K, V](withReplacement: Boolean,
    +      fractionByKey: (K => Double),
    +      counts: Option[Map[K, Long]]): ((TaskContext, Result[K]),(K, V)) => Result[K] = {
    +    val delta = 5e-5
    +    (U: (TaskContext, Result[K]), item: (K, V)) => {
    +      val result = U._2
    +      val tc = U._1
    +      val rng = result.getRand(tc.partitionId)
    +      val fraction = fractionByKey(item._1)
    +      val stratum = result.getEntry(item._1)
    +      if (withReplacement) {
    +        // compute q1 and q2 only if they haven't been computed already
    +        // since they don't change from iteration to iteration.
    +        // TODO change this to the streaming version
    +        if (stratum.q1.isEmpty || stratum.q2.isEmpty) {
    +          val n = counts.get(item._1)
    +          val s = math.ceil(n * fraction).toLong
    +          val lmbd1 = PB.getLowerBound(s)
    +          val minCount = PB.getMinCount(lmbd1)
    +          val lmbd2 = if (lmbd1 == 0) PB.getUpperBound(s) else PB.getUpperBound(s - minCount)
    +          val q1 = lmbd1 / n
    +          val q2 = lmbd2 / n
    +          stratum.q1 = Some(q1)
    +          stratum.q2 = Some(q2)
    +        }
    +        val x1 = if (stratum.q1.get == 0) 0L else rng.nextPoisson(stratum.q1.get)
    +        if (x1 > 0) {
    +          stratum.incrNumAccepted(x1)
    +        }
    +        val x2 = rng.nextPoisson(stratum.q2.get).toInt
    +        if (x2 > 0) {
    +          stratum.addToWaitList(ArrayBuffer.fill(x2)(rng.nextUniform(0.0, 1.0)))
    +        }
    +      } else {
    +        // We use the streaming version of the algorithm for sampling without replacement.
    +        // Hence, q1 and q2 change on every iteration.
    +        val g1 = - math.log(delta) / stratum.numItems
    +        val g2 = (2.0 / 3.0) * g1
    +        val q1 = math.max(0, fraction + g2 - math.sqrt((g2 * g2 + 3 * g2 * fraction)))
    +        val q2 = math.min(1, fraction + g1 + math.sqrt(g1 * g1 + 2 * g1 * fraction))
    +
    +        val x = rng.nextUniform(0.0, 1.0)
    +        if (x < q1) {
    +          stratum.incrNumAccepted()
    +        } else if (x < q2) {
    +          stratum.addToWaitList(x)
    +        }
    +        stratum.q1 = Some(q1)
    +        stratum.q2 = Some(q2)
    +      }
    +      stratum.incrNumItems()
    +      result
    +    }
    +  }
    +
    +  /**
    +   * Returns the function used by aggregate to combine results from different partitions, as
    +   * returned by seqOp.
    +   */
    +  def getCombOp[K](): (Result[K], Result[K]) => Result[K] = {
    +    (r1: Result[K], r2: Result[K]) => {
    +      // take union of both key sets in case one partition doesn't contain all keys
    +      val keyUnion = r1.resultMap.keys.toSet.union(r2.resultMap.keys.toSet)
    +
    +      // Use r2 to keep the combined result since r1 is usual empty
    +      for (key <- keyUnion) {
    +        val entry1 = r1.resultMap.get(key)
    +        val entry2 = r2.resultMap.get(key)
    +        if (entry2.isEmpty && entry1.isDefined) {
    +          r2.resultMap += (key -> entry1.get)
    +        } else if (entry1.isDefined && entry2.isDefined) {
    +          entry2.get.addToWaitList(entry1.get.waitList)
    +          entry2.get.incrNumAccepted(entry1.get.numAccepted)
    +          entry2.get.incrNumItems(entry1.get.numItems)
    +        }
    +      }
    +      r2
    +    }
    +  }
    +
    +  /**
    +   * Given the result returned by the aggregate function, we need to determine the threshold used
    +   * to accept items to generate the exact sample size.
    +   */
    +  def computeThresholdByKey[K](finalResult: Map[K, Stratum], fractionByKey: (K => Double)):
    +    (K => Double) = {
    +    val thresholdByKey = new mutable.HashMap[K, Double]()
    +    for ((key, stratum) <- finalResult) {
    +      val fraction = fractionByKey(key)
    +      val s = math.ceil(stratum.numItems * fraction).toLong
    +      if (stratum.numAccepted > s) {
    +        logWarning("Pre-accepted too many")
    +        thresholdByKey += (key -> stratum.q1.get)
    +      } else {
    +        val numWaitListAccepted = (s - stratum.numAccepted).toInt
    +        if (numWaitListAccepted >= stratum.waitList.size) {
    +          logWarning("WaitList too short")
    +          thresholdByKey += (key -> stratum.q2.get)
    +        } else {
    +          thresholdByKey += (key -> stratum.waitList.sorted.apply(numWaitListAccepted))
    +        }
    +      }
    +    }
    +    thresholdByKey
    +  }
    +
    +  /**
    +   * Return the per partition sampling function used for sampling without replacement.
    +   *
    +   * When exact sample size is required, we make an additional pass over the RDD to determine the
    +   * exact sampling rate that guarantees sample size with high confidence.
    +   *
    +   * The sampling function has a unique seed per partition.
    +   */
    +  def getBernoulliSamplingFunction[K, V](rdd: RDD[(K,  V)],
    +      fractionByKey: K => Double,
    +      exact: Boolean,
    +      seed: Long): (Int, Iterator[(K, V)]) => Iterator[(K, V)] = {
    +    var samplingRateByKey = fractionByKey
    +    if (exact) {
    +      // determine threshold for each stratum and resample
    +      val seqOp = StratifiedSampler.getSeqOp[K,V](false, fractionByKey, None)
    +      val combOp = StratifiedSampler.getCombOp[K]()
    +      val zeroU = new Result[K](Map[K, Stratum](), seed = seed)
    +      val finalResult = aggregateWithContext(zeroU)(rdd, seqOp, combOp).resultMap
    +      samplingRateByKey = StratifiedSampler.computeThresholdByKey(finalResult, fractionByKey)
    +    }
    +    (idx: Int, iter: Iterator[(K, V)]) => {
    +      val random = new RandomDataGenerator
    +      random.reSeed(seed + idx)
    +      iter.filter(t => random.nextUniform(0.0, 1.0) < samplingRateByKey(t._1))
    +    }
    +  }
    +
    +  /**
    +   * Return the per partition sampling function used for sampling with replacement.
    +   *
    +   * When exact sample size is required, we make two additional passed over the RDD to determine
    +   * the exact sampling rate that guarantees sample size with high confidence. The first pass
    +   * counts the number of items in each stratum (group of items with the same key) in the RDD, and
    +   * the second pass uses the counts to determine exact sampling rates.
    +   *
    +   * The sampling function has a unique seed per partition.
    +   */
    +  def getPoissonSamplingFunction[K, V](rdd:RDD[(K,  V)],
    +      fractionByKey: K => Double,
    +      exact: Boolean,
    +      counts: Option[Map[K, Long]],
    +      seed: Long): (Int, Iterator[(K, V)]) => Iterator[(K, V)] = {
    +    // TODO implement the streaming version of sampling w/ replacement that doesn't require counts
    +    if (exact) {
    +      val seqOp = StratifiedSampler.getSeqOp[K,V](true, fractionByKey, counts)
    +      val combOp = StratifiedSampler.getCombOp[K]()
    +      val zeroU = new Result[K](Map[K, Stratum](), seed = seed)
    +      val finalResult = aggregateWithContext(zeroU)(rdd, seqOp, combOp).resultMap
    +      val thresholdByKey = StratifiedSampler.computeThresholdByKey(finalResult, fractionByKey)
    +      (idx: Int, iter: Iterator[(K, V)]) => {
    +        val random = new RandomDataGenerator()
    +        random.reSeed(seed + idx)
    +        iter.flatMap { t =>
    +          val q1 = finalResult.get(t._1).get.q1.getOrElse(0.0)
    +          val q2 = finalResult.get(t._1).get.q2.getOrElse(0.0)
    --- End diff --
    
    No need to call get on a Map.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1025#discussion_r14691480
  
    --- Diff: pom.xml ---
    @@ -257,6 +257,11 @@
     	    <version>1.5</version>
           </dependency>
           <dependency>
    +        <groupId>org.apache.commons</groupId>
    +        <artifactId>commons-math3</artifactId>
    +        <version>3.3</version>
    --- End diff --
    
    Just found that hadoop-common depends on commons-math3-3.1.1 since v2.3.0 (HADOOP-9594). 3.3 is not compatible with 3.1.1. So there is (or will be) a problem no matter which version we include. Do we have alternative solutions?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by dorx <gi...@git.apache.org>.
Github user dorx commented on the pull request:

    https://github.com/apache/spark/pull/1025#issuecomment-48419578
  
    Jenkins, retest this please.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1025#issuecomment-46519999
  
    Merged build finished. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by dorx <gi...@git.apache.org>.
Github user dorx commented on the pull request:

    https://github.com/apache/spark/pull/1025#issuecomment-50073867
  
    Also, seems like there wasn't a single line of code preserved from before the updates. We should probably close this PR and let Xiangrui submit his version in a separate PR at this point.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by falaki <gi...@git.apache.org>.
Github user falaki commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1025#discussion_r14544795
  
    --- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
    @@ -0,0 +1,296 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.util.random
    +
    +import scala.collection.{Map, mutable}
    +import scala.collection.mutable.ArrayBuffer
    +import scala.reflect.ClassTag
    +
    +import org.apache.commons.math3.random.RandomDataGenerator
    +import org.apache.spark.{Logging, SparkContext, TaskContext}
    +import org.apache.spark.rdd.RDD
    +import org.apache.spark.util.Utils
    +import org.apache.spark.util.random.{PoissonBounds => PB}
    +
    +private[spark] object StratifiedSampler extends Logging {
    +
    +  /**
    +   * A version of {@link #aggregate()} that passes the TaskContext to the function that does
    +   * aggregation for each partition. This function avoids creating an extra depth in the RDD
    +   * lineage, as opposed to using mapPartitionsWithId, which results in slightly improved run time.
    +   */
    +  def aggregateWithContext[U: ClassTag, T: ClassTag](zeroValue: U)
    +      (rdd: RDD[T],
    +       seqOp: ((TaskContext, U), T) => U,
    +       combOp: (U, U) => U): U = {
    +    val sc: SparkContext = rdd.sparkContext
    +    // Clone the zero value since we will also be serializing it as part of tasks
    +    var jobResult = Utils.clone(zeroValue, sc.env.closureSerializer.newInstance())
    +    // pad seqOp and combOp with taskContext to conform to aggregate's signature in TraversableOnce
    +    val paddedSeqOp = (arg1: (TaskContext, U), item: T) => (arg1._1, seqOp(arg1, item))
    +    val paddedcombOp = (arg1: (TaskContext, U), arg2: (TaskContext, U)) =>
    +      (arg1._1, combOp(arg1._2, arg1._2))
    +    val cleanSeqOp = sc.clean(paddedSeqOp)
    +    val cleanCombOp = sc.clean(paddedcombOp)
    +    val aggregatePartition = (tc: TaskContext, it: Iterator[T]) =>
    +      (it.aggregate(tc, zeroValue)(cleanSeqOp, cleanCombOp))._2
    +    val mergeResult = (index: Int, taskResult: U) => jobResult = combOp(jobResult, taskResult)
    +    sc.runJob(rdd, aggregatePartition, mergeResult)
    +    jobResult
    +  }
    +
    +  /**
    +   * Returns the function used by aggregate to collect sampling statistics for each partition.
    +   */
    +  def getSeqOp[K, V](withReplacement: Boolean,
    +      fractionByKey: (K => Double),
    +      counts: Option[Map[K, Long]]): ((TaskContext, Result[K]),(K, V)) => Result[K] = {
    +    val delta = 5e-5
    +    (U: (TaskContext, Result[K]), item: (K, V)) => {
    +      val result = U._2
    +      val tc = U._1
    +      val rng = result.getRand(tc.partitionId)
    +      val fraction = fractionByKey(item._1)
    +      val stratum = result.getEntry(item._1)
    +      if (withReplacement) {
    +        // compute q1 and q2 only if they haven't been computed already
    +        // since they don't change from iteration to iteration.
    +        // TODO change this to the streaming version
    +        if (stratum.q1.isEmpty || stratum.q2.isEmpty) {
    +          val n = counts.get(item._1)
    +          val s = math.ceil(n * fraction).toLong
    +          val lmbd1 = PB.getLowerBound(s)
    +          val minCount = PB.getMinCount(lmbd1)
    +          val lmbd2 = if (lmbd1 == 0) PB.getUpperBound(s) else PB.getUpperBound(s - minCount)
    +          val q1 = lmbd1 / n
    +          val q2 = lmbd2 / n
    +          stratum.q1 = Some(q1)
    +          stratum.q2 = Some(q2)
    +        }
    +        val x1 = if (stratum.q1.get == 0) 0L else rng.nextPoisson(stratum.q1.get)
    +        if (x1 > 0) {
    +          stratum.incrNumAccepted(x1)
    +        }
    +        val x2 = rng.nextPoisson(stratum.q2.get).toInt
    +        if (x2 > 0) {
    +          stratum.addToWaitList(ArrayBuffer.fill(x2)(rng.nextUniform(0.0, 1.0)))
    +        }
    +      } else {
    +        // We use the streaming version of the algorithm for sampling without replacement.
    +        // Hence, q1 and q2 change on every iteration.
    +        val g1 = - math.log(delta) / stratum.numItems
    +        val g2 = (2.0 / 3.0) * g1
    +        val q1 = math.max(0, fraction + g2 - math.sqrt((g2 * g2 + 3 * g2 * fraction)))
    +        val q2 = math.min(1, fraction + g1 + math.sqrt(g1 * g1 + 2 * g1 * fraction))
    +
    +        val x = rng.nextUniform(0.0, 1.0)
    +        if (x < q1) {
    +          stratum.incrNumAccepted()
    +        } else if (x < q2) {
    +          stratum.addToWaitList(x)
    +        }
    +        stratum.q1 = Some(q1)
    +        stratum.q2 = Some(q2)
    +      }
    +      stratum.incrNumItems()
    +      result
    +    }
    +  }
    +
    +  /**
    +   * Returns the function used by aggregate to combine results from different partitions, as
    +   * returned by seqOp.
    +   */
    +  def getCombOp[K](): (Result[K], Result[K]) => Result[K] = {
    +    (r1: Result[K], r2: Result[K]) => {
    +      // take union of both key sets in case one partition doesn't contain all keys
    +      val keyUnion = r1.resultMap.keys.toSet.union(r2.resultMap.keys.toSet)
    +
    +      // Use r2 to keep the combined result since r1 is usual empty
    +      for (key <- keyUnion) {
    +        val entry1 = r1.resultMap.get(key)
    +        val entry2 = r2.resultMap.get(key)
    +        if (entry2.isEmpty && entry1.isDefined) {
    +          r2.resultMap += (key -> entry1.get)
    +        } else if (entry1.isDefined && entry2.isDefined) {
    +          entry2.get.addToWaitList(entry1.get.waitList)
    +          entry2.get.incrNumAccepted(entry1.get.numAccepted)
    +          entry2.get.incrNumItems(entry1.get.numItems)
    +        }
    +      }
    +      r2
    +    }
    +  }
    +
    +  /**
    +   * Given the result returned by the aggregate function, we need to determine the threshold used
    +   * to accept items to generate the exact sample size.
    +   */
    +  def computeThresholdByKey[K](finalResult: Map[K, Stratum], fractionByKey: (K => Double)):
    +    (K => Double) = {
    +    val thresholdByKey = new mutable.HashMap[K, Double]()
    +    for ((key, stratum) <- finalResult) {
    +      val fraction = fractionByKey(key)
    +      val s = math.ceil(stratum.numItems * fraction).toLong
    +      if (stratum.numAccepted > s) {
    +        logWarning("Pre-accepted too many")
    +        thresholdByKey += (key -> stratum.q1.get)
    +      } else {
    +        val numWaitListAccepted = (s - stratum.numAccepted).toInt
    +        if (numWaitListAccepted >= stratum.waitList.size) {
    +          logWarning("WaitList too short")
    +          thresholdByKey += (key -> stratum.q2.get)
    +        } else {
    +          thresholdByKey += (key -> stratum.waitList.sorted.apply(numWaitListAccepted))
    +        }
    +      }
    +    }
    +    thresholdByKey
    +  }
    +
    +  /**
    +   * Return the per partition sampling function used for sampling without replacement.
    +   *
    +   * When exact sample size is required, we make an additional pass over the RDD to determine the
    +   * exact sampling rate that guarantees sample size with high confidence.
    +   *
    +   * The sampling function has a unique seed per partition.
    +   */
    +  def getBernoulliSamplingFunction[K, V](rdd: RDD[(K,  V)],
    +      fractionByKey: K => Double,
    +      exact: Boolean,
    +      seed: Long): (Int, Iterator[(K, V)]) => Iterator[(K, V)] = {
    +    var samplingRateByKey = fractionByKey
    +    if (exact) {
    +      // determine threshold for each stratum and resample
    +      val seqOp = StratifiedSampler.getSeqOp[K,V](false, fractionByKey, None)
    +      val combOp = StratifiedSampler.getCombOp[K]()
    +      val zeroU = new Result[K](Map[K, Stratum](), seed = seed)
    +      val finalResult = aggregateWithContext(zeroU)(rdd, seqOp, combOp).resultMap
    +      samplingRateByKey = StratifiedSampler.computeThresholdByKey(finalResult, fractionByKey)
    +    }
    +    (idx: Int, iter: Iterator[(K, V)]) => {
    +      val random = new RandomDataGenerator
    +      random.reSeed(seed + idx)
    +      iter.filter(t => random.nextUniform(0.0, 1.0) < samplingRateByKey(t._1))
    +    }
    +  }
    +
    +  /**
    +   * Return the per partition sampling function used for sampling with replacement.
    +   *
    +   * When exact sample size is required, we make two additional passed over the RDD to determine
    +   * the exact sampling rate that guarantees sample size with high confidence. The first pass
    +   * counts the number of items in each stratum (group of items with the same key) in the RDD, and
    +   * the second pass uses the counts to determine exact sampling rates.
    +   *
    +   * The sampling function has a unique seed per partition.
    +   */
    +  def getPoissonSamplingFunction[K, V](rdd:RDD[(K,  V)],
    +      fractionByKey: K => Double,
    +      exact: Boolean,
    +      counts: Option[Map[K, Long]],
    +      seed: Long): (Int, Iterator[(K, V)]) => Iterator[(K, V)] = {
    +    // TODO implement the streaming version of sampling w/ replacement that doesn't require counts
    +    if (exact) {
    +      val seqOp = StratifiedSampler.getSeqOp[K,V](true, fractionByKey, counts)
    --- End diff --
    
    Similarly remove ```StratifiedSampler``` object name from these lines


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by falaki <gi...@git.apache.org>.
Github user falaki commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1025#discussion_r14544074
  
    --- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
    @@ -0,0 +1,296 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.util.random
    +
    +import scala.collection.{Map, mutable}
    +import scala.collection.mutable.ArrayBuffer
    +import scala.reflect.ClassTag
    +
    +import org.apache.commons.math3.random.RandomDataGenerator
    +import org.apache.spark.{Logging, SparkContext, TaskContext}
    +import org.apache.spark.rdd.RDD
    +import org.apache.spark.util.Utils
    +import org.apache.spark.util.random.{PoissonBounds => PB}
    +
    +private[spark] object StratifiedSampler extends Logging {
    +
    +  /**
    +   * A version of {@link #aggregate()} that passes the TaskContext to the function that does
    +   * aggregation for each partition. This function avoids creating an extra depth in the RDD
    +   * lineage, as opposed to using mapPartitionsWithId, which results in slightly improved run time.
    +   */
    +  def aggregateWithContext[U: ClassTag, T: ClassTag](zeroValue: U)
    +      (rdd: RDD[T],
    +       seqOp: ((TaskContext, U), T) => U,
    +       combOp: (U, U) => U): U = {
    +    val sc: SparkContext = rdd.sparkContext
    +    // Clone the zero value since we will also be serializing it as part of tasks
    +    var jobResult = Utils.clone(zeroValue, sc.env.closureSerializer.newInstance())
    +    // pad seqOp and combOp with taskContext to conform to aggregate's signature in TraversableOnce
    +    val paddedSeqOp = (arg1: (TaskContext, U), item: T) => (arg1._1, seqOp(arg1, item))
    +    val paddedcombOp = (arg1: (TaskContext, U), arg2: (TaskContext, U)) =>
    +      (arg1._1, combOp(arg1._2, arg1._2))
    +    val cleanSeqOp = sc.clean(paddedSeqOp)
    +    val cleanCombOp = sc.clean(paddedcombOp)
    +    val aggregatePartition = (tc: TaskContext, it: Iterator[T]) =>
    +      (it.aggregate(tc, zeroValue)(cleanSeqOp, cleanCombOp))._2
    +    val mergeResult = (index: Int, taskResult: U) => jobResult = combOp(jobResult, taskResult)
    +    sc.runJob(rdd, aggregatePartition, mergeResult)
    +    jobResult
    +  }
    +
    +  /**
    +   * Returns the function used by aggregate to collect sampling statistics for each partition.
    +   */
    +  def getSeqOp[K, V](withReplacement: Boolean,
    +      fractionByKey: (K => Double),
    +      counts: Option[Map[K, Long]]): ((TaskContext, Result[K]),(K, V)) => Result[K] = {
    --- End diff --
    
    Space after comma


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1025#discussion_r14694292
  
    --- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
    @@ -0,0 +1,311 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.util.random
    +
    +import scala.collection.Map
    +import scala.collection.mutable.{ArrayBuffer, HashMap, Map => MMap}
    +
    +import org.apache.commons.math3.random.RandomDataGenerator
    +import org.apache.spark.Logging
    +import org.apache.spark.rdd.RDD
    +
    +/**
    + * Auxiliary functions and data structures for the sampleByKey method in PairRDDFunctions.
    + *
    + * Essentially, when exact sample size is necessary, we make additional passes over the RDD to
    + * compute the exact threshold value to use for each stratum to guarantee exact sample size with
    + * high probability. This is achieved by maintaining a waitlist of size O(log(s)), where s is the
    + * desired sample size for each stratum.
    + *
    + * Like in simple random sampling, we generate a random value for each item from the
    + * uniform  distribution [0.0, 1.0]. All items with values <= min(values of items in the waitlist)
    + * are accepted into the sample instantly. The threshold for instant accept is designed so that
    + * s - numAccepted = O(log(s)), where s is again the desired sample size. Thus, by maintaining a
    + * waitlist size = O(log(s)), we will be able to create a sample of the exact size s by adding
    + * a portion of the waitlist to the set of items that are instantly accepted. The exact threshold
    + * is computed by sorting the values in the waitlist and picking the value at (s - numAccepted).
    + *
    + * Note that since we use the same seed for the RNG when computing the thresholds and the actual
    + * sample, our computed thresholds are guaranteed to produce the desired sample size.
    + *
    + * For more theoretical background on the sampling techniques used here, please refer to
    + * http://jmlr.org/proceedings/papers/v28/meng13a.html
    + */
    +
    +private[spark] object StratifiedSampler extends Logging {
    +
    +  /**
    +   * Count the number of items instantly accepted and generate the waitlist for each stratum.
    +   *
    +   * This is only invoked when exact sample size is required.
    +   */
    +  def getCounts[K, V](rdd: RDD[(K, V)],
    +      withReplacement: Boolean,
    +      fractions: Map[K, Double],
    +      counts: Option[Map[K, Long]],
    +      seed: Long): MMap[K, Stratum] = {
    +    val combOp = getCombOp[K]
    +    val mappedPartitionRDD = rdd.mapPartitionsWithIndex({ case (partition, iter) =>
    +      val zeroU: MMap[K, Stratum] = new HashMap[K, Stratum]()
    +      val rng = new RandomDataGenerator()
    +      rng.reSeed(seed + partition)
    +      val seqOp = getSeqOp(withReplacement, fractions, rng, counts)
    +      Iterator(iter.aggregate(zeroU)(seqOp, combOp))
    +    }, preservesPartitioning=true)
    +    mappedPartitionRDD.reduce(combOp)
    +  }
    +
    +  /**
    +   * Returns the function used by aggregate to collect sampling statistics for each partition.
    +   */
    +  def getSeqOp[K, V](withReplacement: Boolean,
    +      fractions: Map[K, Double],
    +      rng: RandomDataGenerator,
    +      counts: Option[Map[K, Long]]): (MMap[K, Stratum], (K, V)) => MMap[K, Stratum] = {
    +    val delta = 5e-5
    +    (result: MMap[K, Stratum], item: (K, V)) => {
    +      val key = item._1
    +      val fraction = fractions(key)
    +      if (!result.contains(key)) {
    +        result += (key -> new Stratum())
    +      }
    +      val stratum = result(key)
    +
    +      if (withReplacement) {
    +        // compute acceptBound and waitListBound only if they haven't been computed already
    +        // since they don't change from iteration to iteration.
    +        // TODO change this to the streaming version
    +        if (stratum.areBoundsEmpty) {
    +          val n = counts.get(key)
    +          val sampleSize = math.ceil(n * fraction).toLong
    +          val lmbd1 = PoissonBounds.getLowerBound(sampleSize)
    +          val minCount = PoissonBounds.getMinCount(lmbd1)
    +          val lmbd2 = if (lmbd1 == 0) {
    +            PoissonBounds.getUpperBound(sampleSize)
    +          } else {
    +            PoissonBounds.getUpperBound(sampleSize - minCount)
    +          }
    +          stratum.acceptBound = lmbd1 / n
    +          stratum.waitListBound = lmbd2 / n
    +        }
    +        val acceptBound = stratum.acceptBound
    +        val copiesAccepted = if (acceptBound == 0.0) 0L else rng.nextPoisson(acceptBound)
    +        if (copiesAccepted > 0) {
    +          stratum.incrNumAccepted(copiesAccepted)
    +        }
    +        val copiesWaitlisted = rng.nextPoisson(stratum.waitListBound).toInt
    +        if (copiesWaitlisted > 0) {
    +          stratum.addToWaitList(ArrayBuffer.fill(copiesWaitlisted)(rng.nextUniform(0.0, 1.0)))
    +        }
    +      } else {
    +        // We use the streaming version of the algorithm for sampling without replacement to avoid
    +        // using an extra pass over the RDD for computing the count.
    +        // Hence, acceptBound and waitListBound change on every iteration.
    +        val gamma1 = - math.log(delta) / stratum.numItems
    +        val gamma2 = (2.0 / 3.0) * gamma1
    +        stratum.acceptBound = math.max(0,
    +          fraction + gamma2 - math.sqrt(gamma2 * gamma2 + 3 * gamma2 * fraction))
    +        stratum.waitListBound = math.min(1,
    +          fraction + gamma1 + math.sqrt(gamma1 * gamma1 + 2 * gamma1 * fraction))
    +
    +        val x = rng.nextUniform(0.0, 1.0)
    +        if (x < stratum.acceptBound) {
    +          stratum.incrNumAccepted()
    +        } else if (x < stratum.waitListBound) {
    +          stratum.addToWaitList(x)
    +        }
    +      }
    +      stratum.incrNumItems()
    +      result
    +    }
    +  }
    +
    +  /**
    +   * Returns the function used combine results returned by seqOp from different partitions.
    +   */
    +  def getCombOp[K]: (MMap[K, Stratum], MMap[K, Stratum]) => MMap[K, Stratum] = {
    +    (result1: MMap[K, Stratum], result2: MMap[K, Stratum]) => {
    +      // take union of both key sets in case one partition doesn't contain all keys
    +      for (key <- result1.keySet.union(result2.keySet)) {
    +        // Use result2 to keep the combined result since r1 is usual empty
    +        val entry1 = result1.get(key)
    +        if (result2.contains(key)) {
    +          result2(key).merge(entry1)
    +        } else {
    +          if (entry1.isDefined) {
    +            result2 += (key -> entry1.get)
    +          }
    +        }
    +      }
    +      result2
    +    }
    +  }
    +
    +  /**
    +   * Given the result returned by getCounts, determine the threshold for accepting items to
    +   * generate exact sample size.
    +   *
    +   * To do so, we compute sampleSize = math.ceil(size * samplingRate) for each stratum and compare
    +   * it to the number of items that were accepted instantly and the number of items in the waitlist
    +   * for that stratum. Most of the time, numAccepted <= sampleSize <= (numAccepted + numWaitlisted),
    +   * which means we need to sort the elements in the waitlist by their associated values in order
    +   * to find the value T s.t. |{elements in the stratum whose associated values <= T}| = sampleSize.
    +   * Note that all elements in the waitlist have values >= bound for instant accept, so a T value
    +   * in the waitlist range would allow all elements that were instantly accepted on the first pass
    +   * to be included in the sample.
    +   */
    +  def computeThresholdByKey[K](finalResult: Map[K, Stratum],
    +      fractions: Map[K, Double]): Map[K, Double] = {
    +    val thresholdByKey = new HashMap[K, Double]()
    +    for ((key, stratum) <- finalResult) {
    +      val sampleSize = math.ceil(stratum.numItems * fractions(key)).toLong
    +      if (stratum.numAccepted > sampleSize) {
    +        logWarning("Pre-accepted too many")
    +        thresholdByKey += (key -> stratum.acceptBound)
    +      } else {
    +        val numWaitListAccepted = (sampleSize - stratum.numAccepted).toInt
    +        if (numWaitListAccepted >= stratum.waitList.size) {
    +          logWarning("WaitList too short")
    +          thresholdByKey += (key -> stratum.waitListBound)
    +        } else {
    +          thresholdByKey += (key -> stratum.waitList.sorted.apply(numWaitListAccepted))
    +        }
    +      }
    +    }
    +    thresholdByKey
    +  }
    +
    +  /**
    +   * Return the per partition sampling function used for sampling without replacement.
    +   *
    +   * When exact sample size is required, we make an additional pass over the RDD to determine the
    +   * exact sampling rate that guarantees sample size with high confidence.
    +   *
    +   * The sampling function has a unique seed per partition.
    +   */
    +  def getBernoulliSamplingFunction[K, V](rdd: RDD[(K,  V)],
    +      fractions: Map[K, Double],
    +      exact: Boolean,
    +      seed: Long): (Int, Iterator[(K, V)]) => Iterator[(K, V)] = {
    +    var samplingRateByKey = fractions
    +    if (exact) {
    +      // determine threshold for each stratum and resample
    +      val finalResult = getCounts(rdd, false, fractions, None, seed)
    +      samplingRateByKey = computeThresholdByKey(finalResult, fractions)
    +    }
    +    (idx: Int, iter: Iterator[(K, V)]) => {
    +      val rng = new RandomDataGenerator
    +      rng.reSeed(seed + idx)
    +      iter.filter(t => rng.nextUniform(0.0, 1.0) < samplingRateByKey(t._1))
    +    }
    +  }
    +
    +  /**
    +   * Return the per partition sampling function used for sampling with replacement.
    +   *
    +   * When exact sample size is required, we make two additional passed over the RDD to determine
    +   * the exact sampling rate that guarantees sample size with high confidence. The first pass
    +   * counts the number of items in each stratum (group of items with the same key) in the RDD, and
    +   * the second pass uses the counts to determine exact sampling rates.
    +   *
    +   * The sampling function has a unique seed per partition.
    +   */
    +  def getPoissonSamplingFunction[K, V](rdd: RDD[(K, V)],
    +      fractions: Map[K, Double],
    +      exact: Boolean,
    +      counts: Option[Map[K, Long]],
    +      seed: Long): (Int, Iterator[(K, V)]) => Iterator[(K, V)] = {
    +    // TODO implement the streaming version of sampling w/ replacement that doesn't require counts
    +    if (exact) {
    +      val finalResult = getCounts(rdd, true, fractions, counts, seed)
    +      val thresholdByKey = computeThresholdByKey(finalResult, fractions)
    +      (idx: Int, iter: Iterator[(K, V)]) => {
    +        val rng = new RandomDataGenerator()
    +        rng.reSeed(seed + idx)
    +        iter.flatMap { item =>
    +          val key = item._1
    +          val acceptBound = finalResult(key).acceptBound
    +          val copiesAccepted = if (acceptBound == 0) 0L else rng.nextPoisson(acceptBound)
    +          val copiesWailisted = rng.nextPoisson(finalResult(key).waitListBound).toInt
    +          val copiesInSample = copiesAccepted +
    +            (0 until copiesWailisted).count(i => rng.nextUniform(0.0, 1.0) < thresholdByKey(key))
    --- End diff --
    
    key doesn't change here. So we don't need to recompute it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by dorx <gi...@git.apache.org>.
Github user dorx commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1025#discussion_r14906412
  
    --- Diff: core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala ---
    @@ -195,6 +193,37 @@ class PairRDDFunctions[K, V](self: RDD[(K, V)])
       }
     
       /**
    +   * Return a subset of this RDD sampled by key (via stratified sampling).
    +   *
    +   * If exact set to true, we guarantee, with high probability, a sample size =
    +   * math.ceil(fraction * S_i), where S_i is the size of the ith stratum (collection of entries
    +   * that share the same key). When sampling without replacement, we need one additional pass over
    +   * the RDD to guarantee sample size with a 99.99% confidence; when sampling with replacement, we
    +   * need two additional passes.
    +   *
    +   * @param withReplacement whether to sample with or without replacement
    +   * @param fractions map of specific keys to sampling rates
    +   * @param seed seed for the random number generator
    +   * @param exact whether sample size needs to be exactly math.ceil(fraction * size) per stratum
    +   * @return RDD containing the sampled subset
    +   */
    +  def sampleByKey(withReplacement: Boolean,
    +      fractions: Map[K, Double],
    +      exact: Boolean = true,
    +      seed: Long = Utils.random.nextLong): RDD[(K, V)]= {
    +
    +    require(fractions.forall {case(k, v) => v >= 0.0}, "Invalid sampling rates.")
    --- End diff --
    
    This is done to be consistent with sample. Would you like `Invalid` in sample to be changed as well?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1025#discussion_r14907668
  
    --- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
    @@ -0,0 +1,311 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.util.random
    +
    +import scala.collection.Map
    +import scala.collection.mutable.{ArrayBuffer, HashMap, Map => MMap}
    +
    +import org.apache.commons.math3.random.RandomDataGenerator
    +import org.apache.spark.Logging
    +import org.apache.spark.rdd.RDD
    +
    +/**
    + * Auxiliary functions and data structures for the sampleByKey method in PairRDDFunctions.
    + *
    + * Essentially, when exact sample size is necessary, we make additional passes over the RDD to
    + * compute the exact threshold value to use for each stratum to guarantee exact sample size with
    + * high probability. This is achieved by maintaining a waitlist of size O(log(s)), where s is the
    + * desired sample size for each stratum.
    + *
    + * Like in simple random sampling, we generate a random value for each item from the
    + * uniform  distribution [0.0, 1.0]. All items with values <= min(values of items in the waitlist)
    + * are accepted into the sample instantly. The threshold for instant accept is designed so that
    + * s - numAccepted = O(log(s)), where s is again the desired sample size. Thus, by maintaining a
    + * waitlist size = O(log(s)), we will be able to create a sample of the exact size s by adding
    + * a portion of the waitlist to the set of items that are instantly accepted. The exact threshold
    + * is computed by sorting the values in the waitlist and picking the value at (s - numAccepted).
    + *
    + * Note that since we use the same seed for the RNG when computing the thresholds and the actual
    + * sample, our computed thresholds are guaranteed to produce the desired sample size.
    + *
    + * For more theoretical background on the sampling techniques used here, please refer to
    + * http://jmlr.org/proceedings/papers/v28/meng13a.html
    + */
    +
    +private[spark] object StratifiedSampler extends Logging {
    +
    +  /**
    +   * Count the number of items instantly accepted and generate the waitlist for each stratum.
    +   *
    +   * This is only invoked when exact sample size is required.
    +   */
    +  def getCounts[K, V](rdd: RDD[(K, V)],
    +      withReplacement: Boolean,
    +      fractions: Map[K, Double],
    +      counts: Option[Map[K, Long]],
    +      seed: Long): MMap[K, Stratum] = {
    +    val combOp = getCombOp[K]
    +    val mappedPartitionRDD = rdd.mapPartitionsWithIndex({ case (partition, iter) =>
    +      val zeroU: MMap[K, Stratum] = new HashMap[K, Stratum]()
    +      val rng = new RandomDataGenerator()
    +      rng.reSeed(seed + partition)
    +      val seqOp = getSeqOp(withReplacement, fractions, rng, counts)
    +      Iterator(iter.aggregate(zeroU)(seqOp, combOp))
    +    }, preservesPartitioning=true)
    +    mappedPartitionRDD.reduce(combOp)
    +  }
    +
    +  /**
    +   * Returns the function used by aggregate to collect sampling statistics for each partition.
    +   */
    +  def getSeqOp[K, V](withReplacement: Boolean,
    +      fractions: Map[K, Double],
    +      rng: RandomDataGenerator,
    +      counts: Option[Map[K, Long]]): (MMap[K, Stratum], (K, V)) => MMap[K, Stratum] = {
    +    val delta = 5e-5
    +    (result: MMap[K, Stratum], item: (K, V)) => {
    +      val key = item._1
    +      val fraction = fractions(key)
    +      if (!result.contains(key)) {
    +        result += (key -> new Stratum())
    +      }
    +      val stratum = result(key)
    +
    +      if (withReplacement) {
    +        // compute acceptBound and waitListBound only if they haven't been computed already
    +        // since they don't change from iteration to iteration.
    +        // TODO change this to the streaming version
    +        if (stratum.areBoundsEmpty) {
    +          val n = counts.get(key)
    +          val sampleSize = math.ceil(n * fraction).toLong
    +          val lmbd1 = PoissonBounds.getLowerBound(sampleSize)
    +          val minCount = PoissonBounds.getMinCount(lmbd1)
    +          val lmbd2 = if (lmbd1 == 0) {
    +            PoissonBounds.getUpperBound(sampleSize)
    +          } else {
    +            PoissonBounds.getUpperBound(sampleSize - minCount)
    +          }
    +          stratum.acceptBound = lmbd1 / n
    +          stratum.waitListBound = lmbd2 / n
    +        }
    +        val acceptBound = stratum.acceptBound
    +        val copiesAccepted = if (acceptBound == 0.0) 0L else rng.nextPoisson(acceptBound)
    +        if (copiesAccepted > 0) {
    +          stratum.incrNumAccepted(copiesAccepted)
    +        }
    +        val copiesWaitlisted = rng.nextPoisson(stratum.waitListBound).toInt
    +        if (copiesWaitlisted > 0) {
    +          stratum.addToWaitList(ArrayBuffer.fill(copiesWaitlisted)(rng.nextUniform(0.0, 1.0)))
    +        }
    +      } else {
    +        // We use the streaming version of the algorithm for sampling without replacement to avoid
    +        // using an extra pass over the RDD for computing the count.
    +        // Hence, acceptBound and waitListBound change on every iteration.
    +        val gamma1 = - math.log(delta) / stratum.numItems
    +        val gamma2 = (2.0 / 3.0) * gamma1
    +        stratum.acceptBound = math.max(0,
    +          fraction + gamma2 - math.sqrt(gamma2 * gamma2 + 3 * gamma2 * fraction))
    +        stratum.waitListBound = math.min(1,
    +          fraction + gamma1 + math.sqrt(gamma1 * gamma1 + 2 * gamma1 * fraction))
    +
    +        val x = rng.nextUniform(0.0, 1.0)
    +        if (x < stratum.acceptBound) {
    +          stratum.incrNumAccepted()
    +        } else if (x < stratum.waitListBound) {
    +          stratum.addToWaitList(x)
    +        }
    +      }
    +      stratum.incrNumItems()
    +      result
    +    }
    +  }
    +
    +  /**
    +   * Returns the function used combine results returned by seqOp from different partitions.
    +   */
    +  def getCombOp[K]: (MMap[K, Stratum], MMap[K, Stratum]) => MMap[K, Stratum] = {
    +    (result1: MMap[K, Stratum], result2: MMap[K, Stratum]) => {
    +      // take union of both key sets in case one partition doesn't contain all keys
    +      for (key <- result1.keySet.union(result2.keySet)) {
    +        // Use result2 to keep the combined result since r1 is usual empty
    --- End diff --
    
    Check Spark's RDD.aggregate doc.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/1025#issuecomment-50381620
  
    QA results for PR 1025:<br>- This patch FAILED unit tests.<br>- This patch merges cleanly<br>- This patch adds no public classes<br><br>For more information see test ouptut:<br>https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17296/consoleFull


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1025#issuecomment-45562050
  
     Build triggered. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1025#discussion_r14694252
  
    --- Diff: core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala ---
    @@ -195,6 +193,37 @@ class PairRDDFunctions[K, V](self: RDD[(K, V)])
       }
     
       /**
    +   * Return a subset of this RDD sampled by key (via stratified sampling).
    +   *
    +   * If exact set to true, we guarantee, with high probability, a sample size =
    +   * math.ceil(fraction * S_i), where S_i is the size of the ith stratum (collection of entries
    +   * that share the same key). When sampling without replacement, we need one additional pass over
    +   * the RDD to guarantee sample size with a 99.99% confidence; when sampling with replacement, we
    +   * need two additional passes.
    +   *
    +   * @param withReplacement whether to sample with or without replacement
    +   * @param fractions map of specific keys to sampling rates
    +   * @param seed seed for the random number generator
    +   * @param exact whether sample size needs to be exactly math.ceil(fraction * size) per stratum
    +   * @return RDD containing the sampled subset
    +   */
    +  def sampleByKey(withReplacement: Boolean,
    +      fractions: Map[K, Double],
    +      exact: Boolean = true,
    +      seed: Long = Utils.random.nextLong): RDD[(K, V)]= {
    +
    +    require(fractions.forall {case(k, v) => v >= 0.0}, "Invalid sampling rates.")
    +
    +    val samplingFunc = if (withReplacement) {
    +      val counts = if (exact) Some(this.countByKey()) else None
    +      StratifiedSampler.getPoissonSamplingFunction(self, fractions, exact, counts, seed)
    +    } else {
    +      StratifiedSampler.getBernoulliSamplingFunction(self, fractions, exact, seed)
    +    }
    +    self.mapPartitionsWithIndex(samplingFunc, preservesPartitioning=true)
    --- End diff --
    
    spaces around `=`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1025#issuecomment-48388111
  
    
    Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16416/


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1025#discussion_r13578453
  
    --- Diff: core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala ---
    @@ -156,6 +161,182 @@ class PairRDDFunctions[K, V](self: RDD[(K, V)])
       }
     
       /**
    +   * Return a subset of this RDD sampled by key (via stratified sampling).
    +   * We guarantee a sample size = math.ceil(fraction * S_i), where S_i is the size of the ith
    --- End diff --
    
    + "with high probability"


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1025#issuecomment-45568512
  
    Build started. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by dorx <gi...@git.apache.org>.
Github user dorx commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1025#discussion_r13617209
  
    --- Diff: core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala ---
    @@ -27,8 +27,12 @@ import scala.collection.Map
     import scala.collection.mutable
     import scala.collection.mutable.ArrayBuffer
     import scala.reflect.ClassTag
    +import scala.util.control.Breaks._
    --- End diff --
    
    any reason why this is disallowed? It's actually used inside a for loop for "continue"


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by dorx <gi...@git.apache.org>.
Github user dorx commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1025#discussion_r14906349
  
    --- Diff: core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala ---
    @@ -195,6 +193,37 @@ class PairRDDFunctions[K, V](self: RDD[(K, V)])
       }
     
       /**
    +   * Return a subset of this RDD sampled by key (via stratified sampling).
    +   *
    +   * If exact set to true, we guarantee, with high probability, a sample size =
    --- End diff --
    
    So I was told that a rule of thumb is summarize what the function does in one line first and then dive into the details. I can add the explanation in a second paragraph (with a line skip) for elaboration on sampleByKey. For future reference, do you prefer a big chunk of text up front?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1025#discussion_r14694281
  
    --- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
    @@ -0,0 +1,311 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.util.random
    +
    +import scala.collection.Map
    +import scala.collection.mutable.{ArrayBuffer, HashMap, Map => MMap}
    +
    +import org.apache.commons.math3.random.RandomDataGenerator
    +import org.apache.spark.Logging
    +import org.apache.spark.rdd.RDD
    +
    +/**
    + * Auxiliary functions and data structures for the sampleByKey method in PairRDDFunctions.
    + *
    + * Essentially, when exact sample size is necessary, we make additional passes over the RDD to
    + * compute the exact threshold value to use for each stratum to guarantee exact sample size with
    + * high probability. This is achieved by maintaining a waitlist of size O(log(s)), where s is the
    + * desired sample size for each stratum.
    + *
    + * Like in simple random sampling, we generate a random value for each item from the
    + * uniform  distribution [0.0, 1.0]. All items with values <= min(values of items in the waitlist)
    + * are accepted into the sample instantly. The threshold for instant accept is designed so that
    + * s - numAccepted = O(log(s)), where s is again the desired sample size. Thus, by maintaining a
    + * waitlist size = O(log(s)), we will be able to create a sample of the exact size s by adding
    + * a portion of the waitlist to the set of items that are instantly accepted. The exact threshold
    + * is computed by sorting the values in the waitlist and picking the value at (s - numAccepted).
    + *
    + * Note that since we use the same seed for the RNG when computing the thresholds and the actual
    + * sample, our computed thresholds are guaranteed to produce the desired sample size.
    + *
    + * For more theoretical background on the sampling techniques used here, please refer to
    + * http://jmlr.org/proceedings/papers/v28/meng13a.html
    + */
    +
    +private[spark] object StratifiedSampler extends Logging {
    +
    +  /**
    +   * Count the number of items instantly accepted and generate the waitlist for each stratum.
    +   *
    +   * This is only invoked when exact sample size is required.
    +   */
    +  def getCounts[K, V](rdd: RDD[(K, V)],
    +      withReplacement: Boolean,
    +      fractions: Map[K, Double],
    +      counts: Option[Map[K, Long]],
    +      seed: Long): MMap[K, Stratum] = {
    +    val combOp = getCombOp[K]
    +    val mappedPartitionRDD = rdd.mapPartitionsWithIndex({ case (partition, iter) =>
    +      val zeroU: MMap[K, Stratum] = new HashMap[K, Stratum]()
    +      val rng = new RandomDataGenerator()
    +      rng.reSeed(seed + partition)
    +      val seqOp = getSeqOp(withReplacement, fractions, rng, counts)
    +      Iterator(iter.aggregate(zeroU)(seqOp, combOp))
    +    }, preservesPartitioning=true)
    +    mappedPartitionRDD.reduce(combOp)
    +  }
    +
    +  /**
    +   * Returns the function used by aggregate to collect sampling statistics for each partition.
    +   */
    +  def getSeqOp[K, V](withReplacement: Boolean,
    +      fractions: Map[K, Double],
    +      rng: RandomDataGenerator,
    +      counts: Option[Map[K, Long]]): (MMap[K, Stratum], (K, V)) => MMap[K, Stratum] = {
    +    val delta = 5e-5
    +    (result: MMap[K, Stratum], item: (K, V)) => {
    +      val key = item._1
    +      val fraction = fractions(key)
    +      if (!result.contains(key)) {
    +        result += (key -> new Stratum())
    +      }
    +      val stratum = result(key)
    +
    +      if (withReplacement) {
    +        // compute acceptBound and waitListBound only if they haven't been computed already
    +        // since they don't change from iteration to iteration.
    +        // TODO change this to the streaming version
    +        if (stratum.areBoundsEmpty) {
    +          val n = counts.get(key)
    +          val sampleSize = math.ceil(n * fraction).toLong
    +          val lmbd1 = PoissonBounds.getLowerBound(sampleSize)
    +          val minCount = PoissonBounds.getMinCount(lmbd1)
    +          val lmbd2 = if (lmbd1 == 0) {
    +            PoissonBounds.getUpperBound(sampleSize)
    +          } else {
    +            PoissonBounds.getUpperBound(sampleSize - minCount)
    +          }
    +          stratum.acceptBound = lmbd1 / n
    +          stratum.waitListBound = lmbd2 / n
    +        }
    +        val acceptBound = stratum.acceptBound
    +        val copiesAccepted = if (acceptBound == 0.0) 0L else rng.nextPoisson(acceptBound)
    +        if (copiesAccepted > 0) {
    +          stratum.incrNumAccepted(copiesAccepted)
    +        }
    +        val copiesWaitlisted = rng.nextPoisson(stratum.waitListBound).toInt
    +        if (copiesWaitlisted > 0) {
    +          stratum.addToWaitList(ArrayBuffer.fill(copiesWaitlisted)(rng.nextUniform(0.0, 1.0)))
    +        }
    +      } else {
    +        // We use the streaming version of the algorithm for sampling without replacement to avoid
    +        // using an extra pass over the RDD for computing the count.
    +        // Hence, acceptBound and waitListBound change on every iteration.
    +        val gamma1 = - math.log(delta) / stratum.numItems
    +        val gamma2 = (2.0 / 3.0) * gamma1
    +        stratum.acceptBound = math.max(0,
    --- End diff --
    
    Those bounds are also used in `takeSample`. Should we have BernoulliBounds in SamplingUtils, similar to PoissonBounds?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1025#discussion_r14694250
  
    --- Diff: core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala ---
    @@ -195,6 +193,37 @@ class PairRDDFunctions[K, V](self: RDD[(K, V)])
       }
     
       /**
    +   * Return a subset of this RDD sampled by key (via stratified sampling).
    +   *
    +   * If exact set to true, we guarantee, with high probability, a sample size =
    +   * math.ceil(fraction * S_i), where S_i is the size of the ith stratum (collection of entries
    +   * that share the same key). When sampling without replacement, we need one additional pass over
    +   * the RDD to guarantee sample size with a 99.99% confidence; when sampling with replacement, we
    +   * need two additional passes.
    +   *
    +   * @param withReplacement whether to sample with or without replacement
    +   * @param fractions map of specific keys to sampling rates
    +   * @param seed seed for the random number generator
    +   * @param exact whether sample size needs to be exactly math.ceil(fraction * size) per stratum
    +   * @return RDD containing the sampled subset
    +   */
    +  def sampleByKey(withReplacement: Boolean,
    +      fractions: Map[K, Double],
    +      exact: Boolean = true,
    +      seed: Long = Utils.random.nextLong): RDD[(K, V)]= {
    +
    +    require(fractions.forall {case(k, v) => v >= 0.0}, "Invalid sampling rates.")
    --- End diff --
    
    `fractions.values.forall(v => v >= 0.0)`
    
    Change `Invalid` to `Negative` to be more specific.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by dorx <gi...@git.apache.org>.
Github user dorx commented on the pull request:

    https://github.com/apache/spark/pull/1025#issuecomment-46376803
  
    working on migrating from commons-math3 to cern.jet.random


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1025#discussion_r13578415
  
    --- Diff: core/src/main/scala/org/apache/spark/util/random/SamplingUtils.scala ---
    @@ -0,0 +1,56 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.util.random
    +
    +import org.apache.commons.math3.distribution.{PoissonDistribution, NormalDistribution}
    +
    +private[spark] object PoissonBounds {
    +
    +  val delta = 1e-4 / 3.0
    +  val phi = new NormalDistribution().cumulativeProbability(1.0 - delta)
    --- End diff --
    
    `phi` is not used.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/1025#issuecomment-50070185
  
    QA results for PR 1025:<br>- This patch FAILED unit tests.<br>- This patch merges cleanly<br>- This patch adds the following public classes (experimental):<br>case class Task[K](stratum: K, count: Long, sampleSize: Long)<br>trait Reviewer[K] extends Serializable with Logging {<br>class BernoulliReviewer[K](@transient tasks: Iterable[Task[K]]) extends Reviewer[K] {<br>case class Policy(preselect: Double, waitlist: Double)<br>class PoissonReviewer[K](@transient tasks: Iterable[Task[K]]) extends Reviewer[K] {<br>case class Policy(preselect: Double, waitlist: Double, var threshold: Double = Double.NaN)<br>class SelectionResultMap[K] extends Serializable {<br>class SelectionResult extends Serializable {<br><br>For more information see test ouptut:<br>https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17130/consoleFull


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1025#issuecomment-47983319
  
    Merged build started. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1025#issuecomment-48130352
  
     Merged build triggered. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by falaki <gi...@git.apache.org>.
Github user falaki commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1025#discussion_r13630894
  
    --- Diff: core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala ---
    @@ -156,6 +161,182 @@ class PairRDDFunctions[K, V](self: RDD[(K, V)])
       }
     
       /**
    +   * Return a subset of this RDD sampled by key (via stratified sampling).
    +   * We guarantee a sample size = math.ceil(fraction * S_i), where S_i is the size of the ith
    +   * stratum.
    +   *
    +   * @param withReplacement whether to sample with or without replacement
    +   * @param fraction sampling rate
    +   * @param seed seed for the random number generator
    +   * @return RDD containing the sampled subset
    +   */
    +  def sampleByKey(withReplacement: Boolean,
    +      fraction: Double,
    +      seed: Long = Utils.random.nextLong): RDD[(K, V)]= {
    +
    +    class Stratum(var numItems: Long = 0L, var numAccepted: Long = 0L) extends Serializable {
    +      var waitList: ArrayBuffer[Double] = new ArrayBuffer[Double]
    +      var q1: Option[Double] = None
    +      var q2: Option[Double] = None
    +
    +      def incrNumItems(by: Long = 1L) = numItems += by
    +
    +      def incrNumAccepted(by: Long = 1L) = numAccepted += by
    +
    +      def addToWaitList(elem: Double) = waitList += elem
    +
    +      def addToWaitList(elems: ArrayBuffer[Double]) = waitList ++= elems
    +
    +      override def toString() = {
    +        "numItems: " + numItems + " numAccepted: " + numAccepted + " q1: " + q1 + " q2: " + q2 +
    +          " waitListSize:" + waitList.size
    +      }
    +    }
    +
    +    class Result(var resultMap: Map[K, Stratum], var cachedPartitionId: Option[Int] = None)
    +      extends Serializable {
    +      var rand: RandomDataGenerator = new RandomDataGenerator
    +
    +      def getEntry(key: K, numItems: Long = 0L): Stratum = {
    +        if (resultMap.get(key).isEmpty) {
    +          resultMap += (key -> new Stratum(numItems))
    +        }
    +        resultMap.get(key).get
    +      }
    +
    +      def getRand(partitionId: Int): RandomDataGenerator = {
    +        if (cachedPartitionId.isEmpty || cachedPartitionId.get != partitionId) {
    +          cachedPartitionId = Some(partitionId)
    +          rand.reSeed(seed + partitionId)
    +        }
    +        rand
    +      }
    +    }
    +
    +    // TODO implement the streaming version of sampling w/ replacement that doesn't require counts
    +    // in order to save one pass over the RDD
    +    val counts = if (withReplacement) Some(this.countByKey()) else None
    +
    +    val seqOp = (U: (TaskContext, Result), item: (K, V)) => {
    +      val delta = 5e-5
    +      val result = U._2
    +      val tc = U._1
    +      val rng = result.getRand(tc.partitionId)
    +      val stratum = result.getEntry(item._1)
    +      if (withReplacement) {
    +        // compute q1 and q2 only if they haven't been computed already
    +        // since they don't change from iteration to iteration.
    +        // TODO change this to the streaming version
    +        if (stratum.q1.isEmpty || stratum.q2.isEmpty) {
    +          val n = counts.get(item._1)
    +          val s = math.ceil(n * fraction).toLong
    +          val lmbd1 = PB.getLambda1(s)
    +          val minCount = PB.getMinCount(lmbd1)
    +          val lmbd2 = if (lmbd1 == 0) PB.getLambda2(s) else PB.getLambda2(s - minCount)
    +          val q1 = lmbd1 / n
    +          val q2 = lmbd2 / n
    +          stratum.q1 = Some(q1)
    +          stratum.q2 = Some(q2)
    +        }
    +        val x1 = if (stratum.q1.get == 0) 0L else rng.nextPoisson(stratum.q1.get)
    +        if (x1 > 0) {
    +          stratum.incrNumAccepted(x1)
    +        }
    +        val x2 = rng.nextPoisson(stratum.q2.get).toInt
    +        if (x2 > 0) {
    +          stratum.addToWaitList(ArrayBuffer.fill(x2)(rng.nextUniform(0.0, 1.0)))
    +        }
    +      } else {
    +        val g1 = - math.log(delta) / stratum.numItems
    +        val g2 = (2.0 / 3.0) * g1
    +        val q1 = math.max(0, fraction + g2 - math.sqrt((g2 * g2 + 3 * g2 * fraction)))
    +        val q2 = math.min(1, fraction + g1 + math.sqrt(g1 * g1 + 2 * g1 * fraction))
    +
    +        val x = rng.nextUniform(0.0, 1.0)
    +        if (x < q1) {
    +          stratum.incrNumAccepted()
    +        } else if ( x < q2) {
    +          stratum.addToWaitList(x)
    +        }
    +        stratum.q1 = Some(q1)
    +        stratum.q2 = Some(q2)
    +      }
    +      stratum.incrNumItems()
    +      result
    +    }
    +
    +    val combOp = (r1: Result, r2: Result) => {
    +      // take union of both key sets in case one partion doesn't contain all keys
    +      val keyUnion = r1.resultMap.keys.toSet.union(r2.resultMap.keys.toSet)
    +
    +      // Use r2 to keep the combined result since r1 is usual empty
    +      for (key <- keyUnion) {
    +        val entry1 = r1.resultMap.get(key)
    +        val entry2 = r2.resultMap.get(key)
    +        if (entry2.isEmpty && entry1.isDefined) {
    +          r2.resultMap += (key -> entry1.get)
    +        } else if (entry1.isDefined && entry2.isDefined) {
    +          entry2.get.addToWaitList(entry1.get.waitList)
    +          entry2.get.incrNumAccepted(entry1.get.numAccepted)
    +          entry2.get.incrNumItems(entry1.get.numItems)
    +        }
    +      }
    +      r2
    +    }
    +
    +    val zeroU = new Result(Map[K, Stratum]())
    +
    +    // determine threshold for each stratum and resample
    +    val finalResult = self.aggregateWithContext(zeroU)(seqOp, combOp).resultMap
    +    val thresholdByKey = new mutable.HashMap[K, Double]()
    +    for ((key, stratum) <- finalResult) {
    +      val s = math.ceil(stratum.numItems * fraction).toLong
    +      breakable {
    +        if (stratum.numAccepted > s) {
    +          logWarning("Pre-accepted too many")
    +          thresholdByKey += (key -> stratum.q1.get)
    +          break
    +        }
    +        val numWaitListAccepted = (s - stratum.numAccepted).toInt
    +        if (numWaitListAccepted >= stratum.waitList.size) {
    +          logWarning("WaitList too short")
    +          thresholdByKey += (key -> stratum.q2.get)
    +        } else {
    +          thresholdByKey += (key -> stratum.waitList.sorted.apply(numWaitListAccepted))
    +        }
    +      }
    +    }
    +
    +    if (withReplacement) {
    +      // Poisson sampler
    +      self.mapPartitionsWithIndex((idx: Int, iter: Iterator[(K, V)]) => {
    +        val random = new RandomDataGenerator()
    +        random.reSeed(seed + idx)
    +        iter.flatMap { t =>
    +          val q1 = finalResult.get(t._1).get.q1.get
    +          val q2 = finalResult.get(t._1).get.q2.get
    +          val x1 = if (q1 == 0) 0L else random.nextPoisson(q1)
    +          val x2 = random.nextPoisson(q2).toInt
    +          val x = x1 + (0 until x2).filter(i => random.nextUniform(0.0, 1.0) <
    +            thresholdByKey.get(t._1).get).size
    --- End diff --
    
    You can make it simpler by using count:
    ```scala
    (0 until x2).count(i => random.nextUniform(0.0, 1.0) < thresholdByKey.get(t._1).get)
    ```



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/1025#issuecomment-50379743
  
    QA tests have started for PR 1025. This patch DID NOT merge cleanly! <br>View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17293/consoleFull


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by falaki <gi...@git.apache.org>.
Github user falaki commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1025#discussion_r14544689
  
    --- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
    @@ -0,0 +1,296 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.util.random
    +
    +import scala.collection.{Map, mutable}
    +import scala.collection.mutable.ArrayBuffer
    +import scala.reflect.ClassTag
    +
    +import org.apache.commons.math3.random.RandomDataGenerator
    +import org.apache.spark.{Logging, SparkContext, TaskContext}
    +import org.apache.spark.rdd.RDD
    +import org.apache.spark.util.Utils
    +import org.apache.spark.util.random.{PoissonBounds => PB}
    +
    +private[spark] object StratifiedSampler extends Logging {
    +
    +  /**
    +   * A version of {@link #aggregate()} that passes the TaskContext to the function that does
    +   * aggregation for each partition. This function avoids creating an extra depth in the RDD
    +   * lineage, as opposed to using mapPartitionsWithId, which results in slightly improved run time.
    +   */
    +  def aggregateWithContext[U: ClassTag, T: ClassTag](zeroValue: U)
    +      (rdd: RDD[T],
    +       seqOp: ((TaskContext, U), T) => U,
    +       combOp: (U, U) => U): U = {
    +    val sc: SparkContext = rdd.sparkContext
    +    // Clone the zero value since we will also be serializing it as part of tasks
    +    var jobResult = Utils.clone(zeroValue, sc.env.closureSerializer.newInstance())
    +    // pad seqOp and combOp with taskContext to conform to aggregate's signature in TraversableOnce
    +    val paddedSeqOp = (arg1: (TaskContext, U), item: T) => (arg1._1, seqOp(arg1, item))
    +    val paddedcombOp = (arg1: (TaskContext, U), arg2: (TaskContext, U)) =>
    +      (arg1._1, combOp(arg1._2, arg1._2))
    +    val cleanSeqOp = sc.clean(paddedSeqOp)
    +    val cleanCombOp = sc.clean(paddedcombOp)
    +    val aggregatePartition = (tc: TaskContext, it: Iterator[T]) =>
    +      (it.aggregate(tc, zeroValue)(cleanSeqOp, cleanCombOp))._2
    +    val mergeResult = (index: Int, taskResult: U) => jobResult = combOp(jobResult, taskResult)
    +    sc.runJob(rdd, aggregatePartition, mergeResult)
    +    jobResult
    +  }
    +
    +  /**
    +   * Returns the function used by aggregate to collect sampling statistics for each partition.
    +   */
    +  def getSeqOp[K, V](withReplacement: Boolean,
    +      fractionByKey: (K => Double),
    +      counts: Option[Map[K, Long]]): ((TaskContext, Result[K]),(K, V)) => Result[K] = {
    +    val delta = 5e-5
    +    (U: (TaskContext, Result[K]), item: (K, V)) => {
    +      val result = U._2
    +      val tc = U._1
    +      val rng = result.getRand(tc.partitionId)
    +      val fraction = fractionByKey(item._1)
    +      val stratum = result.getEntry(item._1)
    +      if (withReplacement) {
    +        // compute q1 and q2 only if they haven't been computed already
    +        // since they don't change from iteration to iteration.
    +        // TODO change this to the streaming version
    +        if (stratum.q1.isEmpty || stratum.q2.isEmpty) {
    +          val n = counts.get(item._1)
    +          val s = math.ceil(n * fraction).toLong
    +          val lmbd1 = PB.getLowerBound(s)
    +          val minCount = PB.getMinCount(lmbd1)
    +          val lmbd2 = if (lmbd1 == 0) PB.getUpperBound(s) else PB.getUpperBound(s - minCount)
    +          val q1 = lmbd1 / n
    +          val q2 = lmbd2 / n
    +          stratum.q1 = Some(q1)
    +          stratum.q2 = Some(q2)
    +        }
    +        val x1 = if (stratum.q1.get == 0) 0L else rng.nextPoisson(stratum.q1.get)
    +        if (x1 > 0) {
    +          stratum.incrNumAccepted(x1)
    +        }
    +        val x2 = rng.nextPoisson(stratum.q2.get).toInt
    +        if (x2 > 0) {
    +          stratum.addToWaitList(ArrayBuffer.fill(x2)(rng.nextUniform(0.0, 1.0)))
    +        }
    +      } else {
    +        // We use the streaming version of the algorithm for sampling without replacement.
    +        // Hence, q1 and q2 change on every iteration.
    +        val g1 = - math.log(delta) / stratum.numItems
    +        val g2 = (2.0 / 3.0) * g1
    +        val q1 = math.max(0, fraction + g2 - math.sqrt((g2 * g2 + 3 * g2 * fraction)))
    +        val q2 = math.min(1, fraction + g1 + math.sqrt(g1 * g1 + 2 * g1 * fraction))
    +
    +        val x = rng.nextUniform(0.0, 1.0)
    +        if (x < q1) {
    +          stratum.incrNumAccepted()
    +        } else if (x < q2) {
    +          stratum.addToWaitList(x)
    +        }
    +        stratum.q1 = Some(q1)
    +        stratum.q2 = Some(q2)
    +      }
    +      stratum.incrNumItems()
    +      result
    +    }
    +  }
    +
    +  /**
    +   * Returns the function used by aggregate to combine results from different partitions, as
    +   * returned by seqOp.
    +   */
    +  def getCombOp[K](): (Result[K], Result[K]) => Result[K] = {
    +    (r1: Result[K], r2: Result[K]) => {
    +      // take union of both key sets in case one partition doesn't contain all keys
    +      val keyUnion = r1.resultMap.keys.toSet.union(r2.resultMap.keys.toSet)
    +
    +      // Use r2 to keep the combined result since r1 is usual empty
    +      for (key <- keyUnion) {
    +        val entry1 = r1.resultMap.get(key)
    --- End diff --
    
    get() is superfluous, this can just be ```r1.resultMap(key)```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the pull request:

    https://github.com/apache/spark/pull/1025#issuecomment-50528615
  
    LGTM. Merged into master. Thanks!!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by falaki <gi...@git.apache.org>.
Github user falaki commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1025#discussion_r14544520
  
    --- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
    @@ -0,0 +1,296 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.util.random
    +
    +import scala.collection.{Map, mutable}
    +import scala.collection.mutable.ArrayBuffer
    +import scala.reflect.ClassTag
    +
    +import org.apache.commons.math3.random.RandomDataGenerator
    +import org.apache.spark.{Logging, SparkContext, TaskContext}
    +import org.apache.spark.rdd.RDD
    +import org.apache.spark.util.Utils
    +import org.apache.spark.util.random.{PoissonBounds => PB}
    --- End diff --
    
    Let's not rename this. The original name is more descriptive.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1025#discussion_r14694285
  
    --- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
    @@ -0,0 +1,311 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.util.random
    +
    +import scala.collection.Map
    +import scala.collection.mutable.{ArrayBuffer, HashMap, Map => MMap}
    +
    +import org.apache.commons.math3.random.RandomDataGenerator
    +import org.apache.spark.Logging
    +import org.apache.spark.rdd.RDD
    +
    +/**
    + * Auxiliary functions and data structures for the sampleByKey method in PairRDDFunctions.
    + *
    + * Essentially, when exact sample size is necessary, we make additional passes over the RDD to
    + * compute the exact threshold value to use for each stratum to guarantee exact sample size with
    + * high probability. This is achieved by maintaining a waitlist of size O(log(s)), where s is the
    + * desired sample size for each stratum.
    + *
    + * Like in simple random sampling, we generate a random value for each item from the
    + * uniform  distribution [0.0, 1.0]. All items with values <= min(values of items in the waitlist)
    + * are accepted into the sample instantly. The threshold for instant accept is designed so that
    + * s - numAccepted = O(log(s)), where s is again the desired sample size. Thus, by maintaining a
    + * waitlist size = O(log(s)), we will be able to create a sample of the exact size s by adding
    + * a portion of the waitlist to the set of items that are instantly accepted. The exact threshold
    + * is computed by sorting the values in the waitlist and picking the value at (s - numAccepted).
    + *
    + * Note that since we use the same seed for the RNG when computing the thresholds and the actual
    + * sample, our computed thresholds are guaranteed to produce the desired sample size.
    + *
    + * For more theoretical background on the sampling techniques used here, please refer to
    + * http://jmlr.org/proceedings/papers/v28/meng13a.html
    + */
    +
    +private[spark] object StratifiedSampler extends Logging {
    +
    +  /**
    +   * Count the number of items instantly accepted and generate the waitlist for each stratum.
    +   *
    +   * This is only invoked when exact sample size is required.
    +   */
    +  def getCounts[K, V](rdd: RDD[(K, V)],
    +      withReplacement: Boolean,
    +      fractions: Map[K, Double],
    +      counts: Option[Map[K, Long]],
    +      seed: Long): MMap[K, Stratum] = {
    +    val combOp = getCombOp[K]
    +    val mappedPartitionRDD = rdd.mapPartitionsWithIndex({ case (partition, iter) =>
    +      val zeroU: MMap[K, Stratum] = new HashMap[K, Stratum]()
    +      val rng = new RandomDataGenerator()
    +      rng.reSeed(seed + partition)
    +      val seqOp = getSeqOp(withReplacement, fractions, rng, counts)
    +      Iterator(iter.aggregate(zeroU)(seqOp, combOp))
    +    }, preservesPartitioning=true)
    +    mappedPartitionRDD.reduce(combOp)
    +  }
    +
    +  /**
    +   * Returns the function used by aggregate to collect sampling statistics for each partition.
    +   */
    +  def getSeqOp[K, V](withReplacement: Boolean,
    +      fractions: Map[K, Double],
    +      rng: RandomDataGenerator,
    +      counts: Option[Map[K, Long]]): (MMap[K, Stratum], (K, V)) => MMap[K, Stratum] = {
    +    val delta = 5e-5
    +    (result: MMap[K, Stratum], item: (K, V)) => {
    +      val key = item._1
    +      val fraction = fractions(key)
    +      if (!result.contains(key)) {
    +        result += (key -> new Stratum())
    +      }
    +      val stratum = result(key)
    +
    +      if (withReplacement) {
    +        // compute acceptBound and waitListBound only if they haven't been computed already
    +        // since they don't change from iteration to iteration.
    +        // TODO change this to the streaming version
    +        if (stratum.areBoundsEmpty) {
    +          val n = counts.get(key)
    +          val sampleSize = math.ceil(n * fraction).toLong
    +          val lmbd1 = PoissonBounds.getLowerBound(sampleSize)
    +          val minCount = PoissonBounds.getMinCount(lmbd1)
    +          val lmbd2 = if (lmbd1 == 0) {
    +            PoissonBounds.getUpperBound(sampleSize)
    +          } else {
    +            PoissonBounds.getUpperBound(sampleSize - minCount)
    +          }
    +          stratum.acceptBound = lmbd1 / n
    +          stratum.waitListBound = lmbd2 / n
    +        }
    +        val acceptBound = stratum.acceptBound
    +        val copiesAccepted = if (acceptBound == 0.0) 0L else rng.nextPoisson(acceptBound)
    +        if (copiesAccepted > 0) {
    +          stratum.incrNumAccepted(copiesAccepted)
    +        }
    +        val copiesWaitlisted = rng.nextPoisson(stratum.waitListBound).toInt
    +        if (copiesWaitlisted > 0) {
    +          stratum.addToWaitList(ArrayBuffer.fill(copiesWaitlisted)(rng.nextUniform(0.0, 1.0)))
    +        }
    +      } else {
    +        // We use the streaming version of the algorithm for sampling without replacement to avoid
    +        // using an extra pass over the RDD for computing the count.
    +        // Hence, acceptBound and waitListBound change on every iteration.
    +        val gamma1 = - math.log(delta) / stratum.numItems
    +        val gamma2 = (2.0 / 3.0) * gamma1
    +        stratum.acceptBound = math.max(0,
    +          fraction + gamma2 - math.sqrt(gamma2 * gamma2 + 3 * gamma2 * fraction))
    +        stratum.waitListBound = math.min(1,
    +          fraction + gamma1 + math.sqrt(gamma1 * gamma1 + 2 * gamma1 * fraction))
    +
    +        val x = rng.nextUniform(0.0, 1.0)
    +        if (x < stratum.acceptBound) {
    +          stratum.incrNumAccepted()
    +        } else if (x < stratum.waitListBound) {
    +          stratum.addToWaitList(x)
    +        }
    +      }
    +      stratum.incrNumItems()
    +      result
    +    }
    +  }
    +
    +  /**
    +   * Returns the function used combine results returned by seqOp from different partitions.
    +   */
    +  def getCombOp[K]: (MMap[K, Stratum], MMap[K, Stratum]) => MMap[K, Stratum] = {
    +    (result1: MMap[K, Stratum], result2: MMap[K, Stratum]) => {
    +      // take union of both key sets in case one partition doesn't contain all keys
    +      for (key <- result1.keySet.union(result2.keySet)) {
    +        // Use result2 to keep the combined result since r1 is usual empty
    --- End diff --
    
    The contract of aggregate only allows modification to the first argument.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1025#issuecomment-46644693
  
    
    Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15938/


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1025#issuecomment-48416874
  
    All automated tests passed.
    Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16431/


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1025#issuecomment-46377060
  
    
    Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15859/


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1025#discussion_r14694267
  
    --- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
    @@ -0,0 +1,311 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.util.random
    +
    +import scala.collection.Map
    +import scala.collection.mutable.{ArrayBuffer, HashMap, Map => MMap}
    +
    +import org.apache.commons.math3.random.RandomDataGenerator
    +import org.apache.spark.Logging
    +import org.apache.spark.rdd.RDD
    +
    +/**
    + * Auxiliary functions and data structures for the sampleByKey method in PairRDDFunctions.
    + *
    + * Essentially, when exact sample size is necessary, we make additional passes over the RDD to
    + * compute the exact threshold value to use for each stratum to guarantee exact sample size with
    + * high probability. This is achieved by maintaining a waitlist of size O(log(s)), where s is the
    + * desired sample size for each stratum.
    + *
    + * Like in simple random sampling, we generate a random value for each item from the
    + * uniform  distribution [0.0, 1.0]. All items with values <= min(values of items in the waitlist)
    + * are accepted into the sample instantly. The threshold for instant accept is designed so that
    + * s - numAccepted = O(log(s)), where s is again the desired sample size. Thus, by maintaining a
    + * waitlist size = O(log(s)), we will be able to create a sample of the exact size s by adding
    + * a portion of the waitlist to the set of items that are instantly accepted. The exact threshold
    + * is computed by sorting the values in the waitlist and picking the value at (s - numAccepted).
    + *
    + * Note that since we use the same seed for the RNG when computing the thresholds and the actual
    + * sample, our computed thresholds are guaranteed to produce the desired sample size.
    + *
    + * For more theoretical background on the sampling techniques used here, please refer to
    + * http://jmlr.org/proceedings/papers/v28/meng13a.html
    + */
    +
    +private[spark] object StratifiedSampler extends Logging {
    --- End diff --
    
    As mentioned in the doc, this is not a sampler but something like `StratifiedSamplingUtils`. I think it is okay to merge the functions into `SamplingUtils`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by dorx <gi...@git.apache.org>.
GitHub user dorx reopened a pull request:

    https://github.com/apache/spark/pull/1025

    [SPARK-2082] stratified sampling in PairRDDFunctions that guarantees exact sample size

    Implemented stratified sampling that guarantees exact sample size using ScaRSR with two passes over the RDD for sampling without replacement and three passes for sampling with replacement.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/dorx/spark stratified

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/1025.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #1025
    
----
commit 14419775202e6eef1f0e1f0c74c7be9030aca73d
Author: Doris Xin <do...@gmail.com>
Date:   2014-05-29T22:22:14Z

    SPARK-1939 Refactor takeSample method in RDD to use ScaSRS

commit ffea61a67d228edb476d29ca13a84bb3f9a22887
Author: Doris Xin <do...@gmail.com>
Date:   2014-05-30T00:55:54Z

    SPARK-1939: Refactor takeSample method in RDD
    
    Reviewer comments addressed:
    - commons-math3 is now a test-only dependency. bumped up to v3.3
    - comments added to explain what computeFraction is doing
    - fixed the unit for computeFraction to use BinomialDitro for without
    replacement sampling
    - stylistic fixes

commit 7cab53a3926f4351432e5e3600b0796b9a4146e4
Author: Doris Xin <do...@gmail.com>
Date:   2014-06-02T19:00:38Z

    fixed import bug in rdd.py

commit e3fd6a628317d559a08a7a20421e9c0618180902
Author: Doris Xin <do...@gmail.com>
Date:   2014-06-02T19:06:18Z

    Merge branch 'master' into takeSample

commit 9ee94ee3c28e8d808063fef4e5d39f06ab738e0b
Author: Doris Xin <do...@gmail.com>
Date:   2014-06-09T20:15:23Z

    [SPARK-2082] stratified sampling in PairRDDFunctions that guarantees exact sample size

commit 1d413ce877a67379a0a74afefba071c018b0ca70
Author: Doris Xin <do...@gmail.com>
Date:   2014-06-09T21:14:26Z

    fixed checkstyle issues

commit 7e1a48182ebec54cd3a6a290b1dc27b928f57dba
Author: Doris Xin <do...@gmail.com>
Date:   2014-06-09T21:29:30Z

    changed the permission on SamplingUtil

commit 46f6c8c86f3e1fdaf49b63796f1cd5bd6db79ec7
Author: Doris Xin <do...@gmail.com>
Date:   2014-06-10T00:39:50Z

    fixed the NPE caused by closures being cleaned before being passed into the aggregate function

commit 50581fc8b08bd5f18cdf2288772c22f2549af0a5
Author: Doris Xin <do...@gmail.com>
Date:   2014-06-12T21:36:37Z

    added a TODO for logging in python

commit 73276111a2c4a9354bfbf6414afceabf70fb21e5
Author: Doris Xin <do...@gmail.com>
Date:   2014-06-13T00:28:14Z

    merge master

commit 9e74ab505e5441eedfed5dbfbeac37566d3de1f0
Author: Doris Xin <do...@gmail.com>
Date:   2014-06-17T22:24:22Z

    Separated out most of the logic in sampleByKey
    
    into StratifiedSampler in util.random

commit 90d94c0d4f4d909fb99b8e72f9a09ca5329e070c
Author: Doris Xin <do...@gmail.com>
Date:   2014-06-17T22:57:11Z

    merge master

commit 0214a7659c62e4ff0f68f6e09cd7846547cd3bcb
Author: Doris Xin <do...@gmail.com>
Date:   2014-06-18T02:22:32Z

    cleanUp
    
    Addressed reviewer comments and added better documentation of code.
    Added commons-math3 as a dependency of spark (okay’ed by Matei). “mvm
    clean install” compiled. Recovered files that were reverted by accident
    in the merge.
    TODOs: figure out API for sampleByKeyExact and update Java, Python, and
    the markdown file accordingly.

commit 944a10cff3c218bafcb8b43e7e3f309cc644633e
Author: Doris Xin <do...@gmail.com>
Date:   2014-06-19T02:48:22Z

    [SPARK-2145] Add lower bound on sampling rate
    
    to guarantee sampling performance

commit 1fe1cff58d99f336406f67f89f53fe6ab3bfde5e
Author: Doris Xin <do...@gmail.com>
Date:   2014-06-19T19:46:59Z

    Changed fractionByKey to a map to enable arg check

commit bd9dc6e08444ec83c5d6adc4c452b49d1ac2b154
Author: Doris Xin <do...@gmail.com>
Date:   2014-06-19T20:00:47Z

    unit bug and style violation fixed

commit 4ad516b14559f06a32d65ef0ce2fa2d7526610bc
Author: Doris Xin <do...@gmail.com>
Date:   2014-06-20T18:38:09Z

    remove unused imports from PairRDDFunctions

commit 254e03c96e1f2aaa5baa9c3d384adeb117e0b7ab
Author: Doris Xin <do...@gmail.com>
Date:   2014-07-03T20:49:46Z

    minor fixes and Java API.
    
    punting on python for now. moved aggregateWithContext out of RDD

commit 6b5b10b71669590f862ec2b7d004ba976d59676c
Author: Doris Xin <do...@gmail.com>
Date:   2014-07-03T20:58:29Z

    Merge branch 'master' into stratified

commit ee9d260e5eb6ee2d6912e57491daeded1704248c
Author: Doris Xin <do...@gmail.com>
Date:   2014-07-06T23:20:39Z

    addressed reviewer comments

commit bbfb8c91a68deae076db7d9fdc518b6cee8a99f1
Author: Doris Xin <do...@gmail.com>
Date:   2014-07-06T23:21:20Z

    Merge branch 'master' into stratified

commit 9884a9f03b18f3170d90d168c246d78fc463028e
Author: Doris Xin <do...@gmail.com>
Date:   2014-07-08T18:58:39Z

    style fix

commit 680b677bc5276e1499c59c7e24abfae7d85e5c7d
Author: Doris Xin <do...@gmail.com>
Date:   2014-07-08T23:49:25Z

    use mapPartitionWithIndex instead
    
    also better readability and lots more comments.

commit a2bf756454b2ae6b71ddc67457c0f69f165b937d
Author: Doris Xin <do...@gmail.com>
Date:   2014-07-08T23:50:17Z

    Merge branch 'master' into stratified

commit a10e68dd9c23b0fc1f25effd9ccd0ac3e7299206
Author: Doris Xin <do...@gmail.com>
Date:   2014-07-09T01:09:14Z

    style fix

commit f4c21f324075bc5e7b8cba07a5c47d23fade542f
Author: Doris Xin <do...@gmail.com>
Date:   2014-07-15T01:34:10Z

    Reviewer comments
    
    Added BernoulliBounds

commit eecee5fb31aa678db872eccbd7385461ea3a6a96
Author: Doris Xin <do...@gmail.com>
Date:   2014-07-15T02:56:58Z

    Merge branch 'master' into stratified
    
    Conflicts:
    	project/SparkBuild.scala

commit 1f9e2654921194baba29d58b862f21bd53c29470
Author: Xiangrui Meng <me...@databricks.com>
Date:   2014-07-24T04:15:05Z

    merge master

commit bd6652d8e69c18dc8ee60ae00272fb99958bfae2
Author: Xiangrui Meng <me...@databricks.com>
Date:   2014-07-24T04:15:54Z

    change commons-math3 scope to test

commit 3f6f0b784a0456f98619e40dd97a04352efd2192
Author: Xiangrui Meng <me...@databricks.com>
Date:   2014-07-24T12:41:58Z

    refractor stratified sampling impl

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1025#issuecomment-46608301
  
    
    Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15919/


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1025#issuecomment-48131246
  
    All automated tests passed.
    Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16359/


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by dorx <gi...@git.apache.org>.
Github user dorx commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1025#discussion_r14542699
  
    --- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
    @@ -0,0 +1,296 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.util.random
    +
    +import scala.collection.{Map, mutable}
    +import scala.collection.mutable.ArrayBuffer
    +import scala.reflect.ClassTag
    +
    +import org.apache.commons.math3.random.RandomDataGenerator
    +import org.apache.spark.{Logging, SparkContext, TaskContext}
    +import org.apache.spark.rdd.RDD
    +import org.apache.spark.util.Utils
    +import org.apache.spark.util.random.{PoissonBounds => PB}
    +
    +private[spark] object StratifiedSampler extends Logging {
    +
    +  /**
    +   * A version of {@link #aggregate()} that passes the TaskContext to the function that does
    +   * aggregation for each partition. This function avoids creating an extra depth in the RDD
    +   * lineage, as opposed to using mapPartitionsWithId, which results in slightly improved run time.
    +   */
    +  def aggregateWithContext[U: ClassTag, T: ClassTag](zeroValue: U)
    +      (rdd: RDD[T],
    +       seqOp: ((TaskContext, U), T) => U,
    +       combOp: (U, U) => U): U = {
    +    val sc: SparkContext = rdd.sparkContext
    +    // Clone the zero value since we will also be serializing it as part of tasks
    +    var jobResult = Utils.clone(zeroValue, sc.env.closureSerializer.newInstance())
    +    // pad seqOp and combOp with taskContext to conform to aggregate's signature in TraversableOnce
    +    val paddedSeqOp = (arg1: (TaskContext, U), item: T) => (arg1._1, seqOp(arg1, item))
    +    val paddedcombOp = (arg1: (TaskContext, U), arg2: (TaskContext, U)) =>
    +      (arg1._1, combOp(arg1._2, arg1._2))
    +    val cleanSeqOp = sc.clean(paddedSeqOp)
    +    val cleanCombOp = sc.clean(paddedcombOp)
    +    val aggregatePartition = (tc: TaskContext, it: Iterator[T]) =>
    +      (it.aggregate(tc, zeroValue)(cleanSeqOp, cleanCombOp))._2
    +    val mergeResult = (index: Int, taskResult: U) => jobResult = combOp(jobResult, taskResult)
    +    sc.runJob(rdd, aggregatePartition, mergeResult)
    +    jobResult
    +  }
    +
    +  /**
    +   * Returns the function used by aggregate to collect sampling statistics for each partition.
    +   */
    +  def getSeqOp[K, V](withReplacement: Boolean,
    +      fractionByKey: (K => Double),
    +      counts: Option[Map[K, Long]]): ((TaskContext, Result[K]),(K, V)) => Result[K] = {
    +    val delta = 5e-5
    +    (U: (TaskContext, Result[K]), item: (K, V)) => {
    +      val result = U._2
    +      val tc = U._1
    +      val rng = result.getRand(tc.partitionId)
    +      val fraction = fractionByKey(item._1)
    +      val stratum = result.getEntry(item._1)
    +      if (withReplacement) {
    +        // compute q1 and q2 only if they haven't been computed already
    +        // since they don't change from iteration to iteration.
    +        // TODO change this to the streaming version
    +        if (stratum.q1.isEmpty || stratum.q2.isEmpty) {
    +          val n = counts.get(item._1)
    +          val s = math.ceil(n * fraction).toLong
    +          val lmbd1 = PB.getLowerBound(s)
    +          val minCount = PB.getMinCount(lmbd1)
    +          val lmbd2 = if (lmbd1 == 0) PB.getUpperBound(s) else PB.getUpperBound(s - minCount)
    +          val q1 = lmbd1 / n
    +          val q2 = lmbd2 / n
    +          stratum.q1 = Some(q1)
    +          stratum.q2 = Some(q2)
    +        }
    +        val x1 = if (stratum.q1.get == 0) 0L else rng.nextPoisson(stratum.q1.get)
    +        if (x1 > 0) {
    +          stratum.incrNumAccepted(x1)
    +        }
    +        val x2 = rng.nextPoisson(stratum.q2.get).toInt
    +        if (x2 > 0) {
    +          stratum.addToWaitList(ArrayBuffer.fill(x2)(rng.nextUniform(0.0, 1.0)))
    +        }
    +      } else {
    +        // We use the streaming version of the algorithm for sampling without replacement.
    +        // Hence, q1 and q2 change on every iteration.
    +        val g1 = - math.log(delta) / stratum.numItems
    +        val g2 = (2.0 / 3.0) * g1
    +        val q1 = math.max(0, fraction + g2 - math.sqrt((g2 * g2 + 3 * g2 * fraction)))
    +        val q2 = math.min(1, fraction + g1 + math.sqrt(g1 * g1 + 2 * g1 * fraction))
    +
    +        val x = rng.nextUniform(0.0, 1.0)
    +        if (x < q1) {
    +          stratum.incrNumAccepted()
    +        } else if (x < q2) {
    +          stratum.addToWaitList(x)
    +        }
    +        stratum.q1 = Some(q1)
    +        stratum.q2 = Some(q2)
    +      }
    +      stratum.incrNumItems()
    +      result
    +    }
    +  }
    +
    +  /**
    +   * Returns the function used by aggregate to combine results from different partitions, as
    +   * returned by seqOp.
    +   */
    +  def getCombOp[K](): (Result[K], Result[K]) => Result[K] = {
    +    (r1: Result[K], r2: Result[K]) => {
    +      // take union of both key sets in case one partition doesn't contain all keys
    +      val keyUnion = r1.resultMap.keys.toSet.union(r2.resultMap.keys.toSet)
    +
    +      // Use r2 to keep the combined result since r1 is usual empty
    +      for (key <- keyUnion) {
    +        val entry1 = r1.resultMap.get(key)
    +        val entry2 = r2.resultMap.get(key)
    +        if (entry2.isEmpty && entry1.isDefined) {
    +          r2.resultMap += (key -> entry1.get)
    +        } else if (entry1.isDefined && entry2.isDefined) {
    +          entry2.get.addToWaitList(entry1.get.waitList)
    +          entry2.get.incrNumAccepted(entry1.get.numAccepted)
    +          entry2.get.incrNumItems(entry1.get.numItems)
    +        }
    +      }
    +      r2
    +    }
    +  }
    +
    +  /**
    +   * Given the result returned by the aggregate function, we need to determine the threshold used
    +   * to accept items to generate the exact sample size.
    +   */
    +  def computeThresholdByKey[K](finalResult: Map[K, Stratum], fractionByKey: (K => Double)):
    +    (K => Double) = {
    +    val thresholdByKey = new mutable.HashMap[K, Double]()
    +    for ((key, stratum) <- finalResult) {
    +      val fraction = fractionByKey(key)
    +      val s = math.ceil(stratum.numItems * fraction).toLong
    +      if (stratum.numAccepted > s) {
    +        logWarning("Pre-accepted too many")
    +        thresholdByKey += (key -> stratum.q1.get)
    +      } else {
    +        val numWaitListAccepted = (s - stratum.numAccepted).toInt
    +        if (numWaitListAccepted >= stratum.waitList.size) {
    +          logWarning("WaitList too short")
    +          thresholdByKey += (key -> stratum.q2.get)
    +        } else {
    +          thresholdByKey += (key -> stratum.waitList.sorted.apply(numWaitListAccepted))
    +        }
    +      }
    +    }
    +    thresholdByKey
    +  }
    +
    +  /**
    +   * Return the per partition sampling function used for sampling without replacement.
    +   *
    +   * When exact sample size is required, we make an additional pass over the RDD to determine the
    +   * exact sampling rate that guarantees sample size with high confidence.
    +   *
    +   * The sampling function has a unique seed per partition.
    +   */
    +  def getBernoulliSamplingFunction[K, V](rdd: RDD[(K,  V)],
    +      fractionByKey: K => Double,
    +      exact: Boolean,
    +      seed: Long): (Int, Iterator[(K, V)]) => Iterator[(K, V)] = {
    +    var samplingRateByKey = fractionByKey
    +    if (exact) {
    +      // determine threshold for each stratum and resample
    +      val seqOp = StratifiedSampler.getSeqOp[K,V](false, fractionByKey, None)
    +      val combOp = StratifiedSampler.getCombOp[K]()
    +      val zeroU = new Result[K](Map[K, Stratum](), seed = seed)
    +      val finalResult = aggregateWithContext(zeroU)(rdd, seqOp, combOp).resultMap
    +      samplingRateByKey = StratifiedSampler.computeThresholdByKey(finalResult, fractionByKey)
    +    }
    +    (idx: Int, iter: Iterator[(K, V)]) => {
    +      val random = new RandomDataGenerator
    +      random.reSeed(seed + idx)
    +      iter.filter(t => random.nextUniform(0.0, 1.0) < samplingRateByKey(t._1))
    +    }
    +  }
    +
    +  /**
    +   * Return the per partition sampling function used for sampling with replacement.
    +   *
    +   * When exact sample size is required, we make two additional passed over the RDD to determine
    +   * the exact sampling rate that guarantees sample size with high confidence. The first pass
    +   * counts the number of items in each stratum (group of items with the same key) in the RDD, and
    +   * the second pass uses the counts to determine exact sampling rates.
    +   *
    +   * The sampling function has a unique seed per partition.
    +   */
    +  def getPoissonSamplingFunction[K, V](rdd:RDD[(K,  V)],
    +      fractionByKey: K => Double,
    +      exact: Boolean,
    +      counts: Option[Map[K, Long]],
    +      seed: Long): (Int, Iterator[(K, V)]) => Iterator[(K, V)] = {
    +    // TODO implement the streaming version of sampling w/ replacement that doesn't require counts
    +    if (exact) {
    +      val seqOp = StratifiedSampler.getSeqOp[K,V](true, fractionByKey, counts)
    +      val combOp = StratifiedSampler.getCombOp[K]()
    +      val zeroU = new Result[K](Map[K, Stratum](), seed = seed)
    +      val finalResult = aggregateWithContext(zeroU)(rdd, seqOp, combOp).resultMap
    +      val thresholdByKey = StratifiedSampler.computeThresholdByKey(finalResult, fractionByKey)
    +      (idx: Int, iter: Iterator[(K, V)]) => {
    +        val random = new RandomDataGenerator()
    +        random.reSeed(seed + idx)
    +        iter.flatMap { t =>
    +          val q1 = finalResult.get(t._1).get.q1.getOrElse(0.0)
    +          val q2 = finalResult.get(t._1).get.q2.getOrElse(0.0)
    +          val x1 = if (q1 == 0) 0L else random.nextPoisson(q1)
    +          val x2 = random.nextPoisson(q2).toInt
    +          val x = x1 + (0 until x2).count(i => random.nextUniform(0.0, 1.0) < thresholdByKey(t._1))
    +          if (x > 0) {
    +            Iterator.fill(x.toInt)(t)
    +          } else {
    +            Iterator.empty
    +          }
    +        }
    +      }
    +    } else {
    +      (idx: Int, iter: Iterator[(K, V)]) => {
    +        val random = new RandomDataGenerator()
    +        random.reSeed(seed + idx)
    +        iter.flatMap { t =>
    +          val count = random.nextPoisson(fractionByKey(t._1)).toInt
    +          if (count > 0) {
    +            Iterator.fill(count)(t)
    +          } else {
    +            Iterator.empty
    +          }
    +        }
    +      }
    +    }
    +  }
    +}
    +
    +/**
    + * Object used by seqOp to keep track of the number of items accepted and items waitlisted per
    + * stratum, as well as the bounds for accepting and waitlisting items.
    + */
    +private[random] class Stratum(var numItems: Long = 0L, var numAccepted: Long = 0L)
    --- End diff --
    
    "[random]" can be removed. it's only used in this class


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1025#issuecomment-47983304
  
     Merged build triggered. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by falaki <gi...@git.apache.org>.
Github user falaki commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1025#discussion_r14544595
  
    --- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
    @@ -0,0 +1,296 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.util.random
    +
    +import scala.collection.{Map, mutable}
    +import scala.collection.mutable.ArrayBuffer
    +import scala.reflect.ClassTag
    +
    +import org.apache.commons.math3.random.RandomDataGenerator
    +import org.apache.spark.{Logging, SparkContext, TaskContext}
    +import org.apache.spark.rdd.RDD
    +import org.apache.spark.util.Utils
    +import org.apache.spark.util.random.{PoissonBounds => PB}
    +
    +private[spark] object StratifiedSampler extends Logging {
    +
    +  /**
    +   * A version of {@link #aggregate()} that passes the TaskContext to the function that does
    +   * aggregation for each partition. This function avoids creating an extra depth in the RDD
    +   * lineage, as opposed to using mapPartitionsWithId, which results in slightly improved run time.
    +   */
    +  def aggregateWithContext[U: ClassTag, T: ClassTag](zeroValue: U)
    +      (rdd: RDD[T],
    +       seqOp: ((TaskContext, U), T) => U,
    +       combOp: (U, U) => U): U = {
    +    val sc: SparkContext = rdd.sparkContext
    +    // Clone the zero value since we will also be serializing it as part of tasks
    +    var jobResult = Utils.clone(zeroValue, sc.env.closureSerializer.newInstance())
    +    // pad seqOp and combOp with taskContext to conform to aggregate's signature in TraversableOnce
    +    val paddedSeqOp = (arg1: (TaskContext, U), item: T) => (arg1._1, seqOp(arg1, item))
    +    val paddedcombOp = (arg1: (TaskContext, U), arg2: (TaskContext, U)) =>
    +      (arg1._1, combOp(arg1._2, arg1._2))
    +    val cleanSeqOp = sc.clean(paddedSeqOp)
    +    val cleanCombOp = sc.clean(paddedcombOp)
    +    val aggregatePartition = (tc: TaskContext, it: Iterator[T]) =>
    +      (it.aggregate(tc, zeroValue)(cleanSeqOp, cleanCombOp))._2
    +    val mergeResult = (index: Int, taskResult: U) => jobResult = combOp(jobResult, taskResult)
    +    sc.runJob(rdd, aggregatePartition, mergeResult)
    +    jobResult
    +  }
    +
    +  /**
    +   * Returns the function used by aggregate to collect sampling statistics for each partition.
    +   */
    +  def getSeqOp[K, V](withReplacement: Boolean,
    +      fractionByKey: (K => Double),
    +      counts: Option[Map[K, Long]]): ((TaskContext, Result[K]),(K, V)) => Result[K] = {
    +    val delta = 5e-5
    +    (U: (TaskContext, Result[K]), item: (K, V)) => {
    +      val result = U._2
    +      val tc = U._1
    +      val rng = result.getRand(tc.partitionId)
    +      val fraction = fractionByKey(item._1)
    +      val stratum = result.getEntry(item._1)
    +      if (withReplacement) {
    +        // compute q1 and q2 only if they haven't been computed already
    +        // since they don't change from iteration to iteration.
    +        // TODO change this to the streaming version
    +        if (stratum.q1.isEmpty || stratum.q2.isEmpty) {
    +          val n = counts.get(item._1)
    +          val s = math.ceil(n * fraction).toLong
    +          val lmbd1 = PB.getLowerBound(s)
    +          val minCount = PB.getMinCount(lmbd1)
    +          val lmbd2 = if (lmbd1 == 0) PB.getUpperBound(s) else PB.getUpperBound(s - minCount)
    +          val q1 = lmbd1 / n
    +          val q2 = lmbd2 / n
    +          stratum.q1 = Some(q1)
    +          stratum.q2 = Some(q2)
    +        }
    +        val x1 = if (stratum.q1.get == 0) 0L else rng.nextPoisson(stratum.q1.get)
    +        if (x1 > 0) {
    +          stratum.incrNumAccepted(x1)
    +        }
    +        val x2 = rng.nextPoisson(stratum.q2.get).toInt
    +        if (x2 > 0) {
    +          stratum.addToWaitList(ArrayBuffer.fill(x2)(rng.nextUniform(0.0, 1.0)))
    +        }
    +      } else {
    +        // We use the streaming version of the algorithm for sampling without replacement.
    +        // Hence, q1 and q2 change on every iteration.
    --- End diff --
    
    We are implementing a non-trivial algorithm here. Lets use more descriptive names instead of g1, q2, q1, and q2. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1025#discussion_r14500775
  
    --- Diff: core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala ---
    @@ -27,8 +27,12 @@ import scala.collection.Map
     import scala.collection.mutable
     import scala.collection.mutable.ArrayBuffer
     import scala.reflect.ClassTag
    +import scala.util.control.Breaks._
    --- End diff --
    
    This is not a native feature of Scala, neither a for loop. Try to use while with a flag that terminates the loop.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by falaki <gi...@git.apache.org>.
Github user falaki commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1025#discussion_r14544621
  
    --- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
    @@ -0,0 +1,296 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.util.random
    +
    +import scala.collection.{Map, mutable}
    +import scala.collection.mutable.ArrayBuffer
    +import scala.reflect.ClassTag
    +
    +import org.apache.commons.math3.random.RandomDataGenerator
    +import org.apache.spark.{Logging, SparkContext, TaskContext}
    +import org.apache.spark.rdd.RDD
    +import org.apache.spark.util.Utils
    +import org.apache.spark.util.random.{PoissonBounds => PB}
    +
    +private[spark] object StratifiedSampler extends Logging {
    +
    +  /**
    +   * A version of {@link #aggregate()} that passes the TaskContext to the function that does
    +   * aggregation for each partition. This function avoids creating an extra depth in the RDD
    +   * lineage, as opposed to using mapPartitionsWithId, which results in slightly improved run time.
    +   */
    +  def aggregateWithContext[U: ClassTag, T: ClassTag](zeroValue: U)
    +      (rdd: RDD[T],
    +       seqOp: ((TaskContext, U), T) => U,
    +       combOp: (U, U) => U): U = {
    +    val sc: SparkContext = rdd.sparkContext
    +    // Clone the zero value since we will also be serializing it as part of tasks
    +    var jobResult = Utils.clone(zeroValue, sc.env.closureSerializer.newInstance())
    +    // pad seqOp and combOp with taskContext to conform to aggregate's signature in TraversableOnce
    +    val paddedSeqOp = (arg1: (TaskContext, U), item: T) => (arg1._1, seqOp(arg1, item))
    +    val paddedcombOp = (arg1: (TaskContext, U), arg2: (TaskContext, U)) =>
    +      (arg1._1, combOp(arg1._2, arg1._2))
    +    val cleanSeqOp = sc.clean(paddedSeqOp)
    +    val cleanCombOp = sc.clean(paddedcombOp)
    +    val aggregatePartition = (tc: TaskContext, it: Iterator[T]) =>
    +      (it.aggregate(tc, zeroValue)(cleanSeqOp, cleanCombOp))._2
    +    val mergeResult = (index: Int, taskResult: U) => jobResult = combOp(jobResult, taskResult)
    +    sc.runJob(rdd, aggregatePartition, mergeResult)
    +    jobResult
    +  }
    +
    +  /**
    +   * Returns the function used by aggregate to collect sampling statistics for each partition.
    +   */
    +  def getSeqOp[K, V](withReplacement: Boolean,
    +      fractionByKey: (K => Double),
    +      counts: Option[Map[K, Long]]): ((TaskContext, Result[K]),(K, V)) => Result[K] = {
    +    val delta = 5e-5
    +    (U: (TaskContext, Result[K]), item: (K, V)) => {
    +      val result = U._2
    +      val tc = U._1
    +      val rng = result.getRand(tc.partitionId)
    +      val fraction = fractionByKey(item._1)
    +      val stratum = result.getEntry(item._1)
    +      if (withReplacement) {
    +        // compute q1 and q2 only if they haven't been computed already
    +        // since they don't change from iteration to iteration.
    +        // TODO change this to the streaming version
    +        if (stratum.q1.isEmpty || stratum.q2.isEmpty) {
    +          val n = counts.get(item._1)
    +          val s = math.ceil(n * fraction).toLong
    +          val lmbd1 = PB.getLowerBound(s)
    +          val minCount = PB.getMinCount(lmbd1)
    +          val lmbd2 = if (lmbd1 == 0) PB.getUpperBound(s) else PB.getUpperBound(s - minCount)
    +          val q1 = lmbd1 / n
    +          val q2 = lmbd2 / n
    +          stratum.q1 = Some(q1)
    +          stratum.q2 = Some(q2)
    +        }
    +        val x1 = if (stratum.q1.get == 0) 0L else rng.nextPoisson(stratum.q1.get)
    +        if (x1 > 0) {
    +          stratum.incrNumAccepted(x1)
    +        }
    +        val x2 = rng.nextPoisson(stratum.q2.get).toInt
    +        if (x2 > 0) {
    +          stratum.addToWaitList(ArrayBuffer.fill(x2)(rng.nextUniform(0.0, 1.0)))
    +        }
    +      } else {
    +        // We use the streaming version of the algorithm for sampling without replacement.
    +        // Hence, q1 and q2 change on every iteration.
    +        val g1 = - math.log(delta) / stratum.numItems
    +        val g2 = (2.0 / 3.0) * g1
    +        val q1 = math.max(0, fraction + g2 - math.sqrt((g2 * g2 + 3 * g2 * fraction)))
    +        val q2 = math.min(1, fraction + g1 + math.sqrt(g1 * g1 + 2 * g1 * fraction))
    +
    +        val x = rng.nextUniform(0.0, 1.0)
    +        if (x < q1) {
    +          stratum.incrNumAccepted()
    +        } else if (x < q2) {
    +          stratum.addToWaitList(x)
    +        }
    +        stratum.q1 = Some(q1)
    +        stratum.q2 = Some(q2)
    +      }
    +      stratum.incrNumItems()
    +      result
    +    }
    +  }
    +
    +  /**
    +   * Returns the function used by aggregate to combine results from different partitions, as
    +   * returned by seqOp.
    +   */
    +  def getCombOp[K](): (Result[K], Result[K]) => Result[K] = {
    +    (r1: Result[K], r2: Result[K]) => {
    +      // take union of both key sets in case one partition doesn't contain all keys
    +      val keyUnion = r1.resultMap.keys.toSet.union(r2.resultMap.keys.toSet)
    +
    +      // Use r2 to keep the combined result since r1 is usual empty
    +      for (key <- keyUnion) {
    +        val entry1 = r1.resultMap.get(key)
    +        val entry2 = r2.resultMap.get(key)
    +        if (entry2.isEmpty && entry1.isDefined) {
    +          r2.resultMap += (key -> entry1.get)
    +        } else if (entry1.isDefined && entry2.isDefined) {
    +          entry2.get.addToWaitList(entry1.get.waitList)
    +          entry2.get.incrNumAccepted(entry1.get.numAccepted)
    +          entry2.get.incrNumItems(entry1.get.numItems)
    +        }
    +      }
    +      r2
    +    }
    +  }
    +
    +  /**
    +   * Given the result returned by the aggregate function, we need to determine the threshold used
    +   * to accept items to generate the exact sample size.
    +   */
    +  def computeThresholdByKey[K](finalResult: Map[K, Stratum], fractionByKey: (K => Double)):
    +    (K => Double) = {
    +    val thresholdByKey = new mutable.HashMap[K, Double]()
    +    for ((key, stratum) <- finalResult) {
    +      val fraction = fractionByKey(key)
    +      val s = math.ceil(stratum.numItems * fraction).toLong
    +      if (stratum.numAccepted > s) {
    +        logWarning("Pre-accepted too many")
    +        thresholdByKey += (key -> stratum.q1.get)
    +      } else {
    +        val numWaitListAccepted = (s - stratum.numAccepted).toInt
    +        if (numWaitListAccepted >= stratum.waitList.size) {
    +          logWarning("WaitList too short")
    +          thresholdByKey += (key -> stratum.q2.get)
    +        } else {
    +          thresholdByKey += (key -> stratum.waitList.sorted.apply(numWaitListAccepted))
    +        }
    +      }
    +    }
    +    thresholdByKey
    +  }
    +
    +  /**
    +   * Return the per partition sampling function used for sampling without replacement.
    +   *
    +   * When exact sample size is required, we make an additional pass over the RDD to determine the
    +   * exact sampling rate that guarantees sample size with high confidence.
    +   *
    +   * The sampling function has a unique seed per partition.
    +   */
    +  def getBernoulliSamplingFunction[K, V](rdd: RDD[(K,  V)],
    +      fractionByKey: K => Double,
    +      exact: Boolean,
    +      seed: Long): (Int, Iterator[(K, V)]) => Iterator[(K, V)] = {
    +    var samplingRateByKey = fractionByKey
    +    if (exact) {
    +      // determine threshold for each stratum and resample
    +      val seqOp = StratifiedSampler.getSeqOp[K,V](false, fractionByKey, None)
    +      val combOp = StratifiedSampler.getCombOp[K]()
    +      val zeroU = new Result[K](Map[K, Stratum](), seed = seed)
    +      val finalResult = aggregateWithContext(zeroU)(rdd, seqOp, combOp).resultMap
    +      samplingRateByKey = StratifiedSampler.computeThresholdByKey(finalResult, fractionByKey)
    +    }
    +    (idx: Int, iter: Iterator[(K, V)]) => {
    +      val random = new RandomDataGenerator
    +      random.reSeed(seed + idx)
    +      iter.filter(t => random.nextUniform(0.0, 1.0) < samplingRateByKey(t._1))
    +    }
    +  }
    +
    +  /**
    +   * Return the per partition sampling function used for sampling with replacement.
    +   *
    +   * When exact sample size is required, we make two additional passed over the RDD to determine
    +   * the exact sampling rate that guarantees sample size with high confidence. The first pass
    +   * counts the number of items in each stratum (group of items with the same key) in the RDD, and
    +   * the second pass uses the counts to determine exact sampling rates.
    +   *
    +   * The sampling function has a unique seed per partition.
    +   */
    +  def getPoissonSamplingFunction[K, V](rdd:RDD[(K,  V)],
    +      fractionByKey: K => Double,
    +      exact: Boolean,
    +      counts: Option[Map[K, Long]],
    +      seed: Long): (Int, Iterator[(K, V)]) => Iterator[(K, V)] = {
    +    // TODO implement the streaming version of sampling w/ replacement that doesn't require counts
    +    if (exact) {
    +      val seqOp = StratifiedSampler.getSeqOp[K,V](true, fractionByKey, counts)
    +      val combOp = StratifiedSampler.getCombOp[K]()
    +      val zeroU = new Result[K](Map[K, Stratum](), seed = seed)
    +      val finalResult = aggregateWithContext(zeroU)(rdd, seqOp, combOp).resultMap
    +      val thresholdByKey = StratifiedSampler.computeThresholdByKey(finalResult, fractionByKey)
    +      (idx: Int, iter: Iterator[(K, V)]) => {
    +        val random = new RandomDataGenerator()
    +        random.reSeed(seed + idx)
    +        iter.flatMap { t =>
    +          val q1 = finalResult.get(t._1).get.q1.getOrElse(0.0)
    +          val q2 = finalResult.get(t._1).get.q2.getOrElse(0.0)
    +          val x1 = if (q1 == 0) 0L else random.nextPoisson(q1)
    +          val x2 = random.nextPoisson(q2).toInt
    +          val x = x1 + (0 until x2).count(i => random.nextUniform(0.0, 1.0) < thresholdByKey(t._1))
    +          if (x > 0) {
    +            Iterator.fill(x.toInt)(t)
    +          } else {
    +            Iterator.empty
    +          }
    +        }
    +      }
    +    } else {
    +      (idx: Int, iter: Iterator[(K, V)]) => {
    +        val random = new RandomDataGenerator()
    +        random.reSeed(seed + idx)
    +        iter.flatMap { t =>
    +          val count = random.nextPoisson(fractionByKey(t._1)).toInt
    +          if (count > 0) {
    +            Iterator.fill(count)(t)
    +          } else {
    +            Iterator.empty
    +          }
    +        }
    +      }
    +    }
    +  }
    +}
    +
    +/**
    + * Object used by seqOp to keep track of the number of items accepted and items waitlisted per
    + * stratum, as well as the bounds for accepting and waitlisting items.
    + */
    +private[random] class Stratum(var numItems: Long = 0L, var numAccepted: Long = 0L)
    +  extends Serializable {
    +
    +  var waitList: ArrayBuffer[Double] = new ArrayBuffer[Double]
    +  var q1: Option[Double] = None // upper bound for accepting item instantly
    +  var q2: Option[Double] = None // upper bound for adding item to waitlist
    +
    +  def incrNumItems(by: Long = 1L) = numItems += by
    +
    +  def incrNumAccepted(by: Long = 1L) = numAccepted += by
    +
    +  def addToWaitList(elem: Double) = waitList += elem
    +
    +  def addToWaitList(elems: ArrayBuffer[Double]) = waitList ++= elems
    +}
    +
    +/**
    + * Object used by seqOp and combOp to keep track of the sampling statistics for all strata.
    + *
    + * When used by seqOp for each partition, we also keep track of the partition ID in this object
    + * to make sure a single random number generator with a unique seed is used for each partition.
    + */
    +private[random] class Result[K](var resultMap: Map[K, Stratum],
    --- End diff --
    
    resultMap can be a value, of type mutable Map.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1025#issuecomment-45553454
  
    
    Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15579/


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by falaki <gi...@git.apache.org>.
Github user falaki commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1025#discussion_r13630674
  
    --- Diff: core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala ---
    @@ -156,6 +161,182 @@ class PairRDDFunctions[K, V](self: RDD[(K, V)])
       }
     
       /**
    +   * Return a subset of this RDD sampled by key (via stratified sampling).
    +   * We guarantee a sample size = math.ceil(fraction * S_i), where S_i is the size of the ith
    +   * stratum.
    +   *
    +   * @param withReplacement whether to sample with or without replacement
    +   * @param fraction sampling rate
    +   * @param seed seed for the random number generator
    +   * @return RDD containing the sampled subset
    +   */
    +  def sampleByKey(withReplacement: Boolean,
    +      fraction: Double,
    +      seed: Long = Utils.random.nextLong): RDD[(K, V)]= {
    +
    +    class Stratum(var numItems: Long = 0L, var numAccepted: Long = 0L) extends Serializable {
    +      var waitList: ArrayBuffer[Double] = new ArrayBuffer[Double]
    +      var q1: Option[Double] = None
    +      var q2: Option[Double] = None
    --- End diff --
    
    Either pick more clear names (for q1 and q2) or add a line comment for what they represent.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1025#discussion_r13578427
  
    --- Diff: core/src/main/scala/org/apache/spark/util/random/SamplingUtils.scala ---
    @@ -0,0 +1,56 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.util.random
    +
    +import org.apache.commons.math3.distribution.{PoissonDistribution, NormalDistribution}
    +
    +private[spark] object PoissonBounds {
    +
    +  val delta = 1e-4 / 3.0
    +  val phi = new NormalDistribution().cumulativeProbability(1.0 - delta)
    +
    +  def getLambda1(s: Double): Double = {
    +    var lb = math.max(0.0, s - math.sqrt(s / delta)) // Chebyshev's inequality
    +    var ub = s
    +    while (lb < ub - 1.0) {
    +      val m = (lb + ub) / 2.0
    +      val poisson = new PoissonDistribution(m, 1e-15)
    +      val y = poisson.inverseCumulativeProbability(1 - delta)
    +      if (y > s) ub = m else lb = m
    +    }
    +    lb
    +  }
    +
    +  def getMinCount(lmbd: Double): Double = {
    +    if(lmbd == 0) return 0
    +    val poisson = new PoissonDistribution(lmbd, 1e-15)
    +    poisson.inverseCumulativeProbability(delta)
    +  }
    +
    +  def getLambda2(s: Double): Double = {
    --- End diff --
    
    `getUpperBound`?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by dorx <gi...@git.apache.org>.
Github user dorx commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1025#discussion_r14907670
  
    --- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
    @@ -0,0 +1,311 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.util.random
    +
    +import scala.collection.Map
    +import scala.collection.mutable.{ArrayBuffer, HashMap, Map => MMap}
    +
    +import org.apache.commons.math3.random.RandomDataGenerator
    +import org.apache.spark.Logging
    +import org.apache.spark.rdd.RDD
    +
    +/**
    + * Auxiliary functions and data structures for the sampleByKey method in PairRDDFunctions.
    + *
    + * Essentially, when exact sample size is necessary, we make additional passes over the RDD to
    + * compute the exact threshold value to use for each stratum to guarantee exact sample size with
    + * high probability. This is achieved by maintaining a waitlist of size O(log(s)), where s is the
    + * desired sample size for each stratum.
    + *
    + * Like in simple random sampling, we generate a random value for each item from the
    + * uniform  distribution [0.0, 1.0]. All items with values <= min(values of items in the waitlist)
    + * are accepted into the sample instantly. The threshold for instant accept is designed so that
    + * s - numAccepted = O(log(s)), where s is again the desired sample size. Thus, by maintaining a
    + * waitlist size = O(log(s)), we will be able to create a sample of the exact size s by adding
    + * a portion of the waitlist to the set of items that are instantly accepted. The exact threshold
    + * is computed by sorting the values in the waitlist and picking the value at (s - numAccepted).
    + *
    + * Note that since we use the same seed for the RNG when computing the thresholds and the actual
    + * sample, our computed thresholds are guaranteed to produce the desired sample size.
    + *
    + * For more theoretical background on the sampling techniques used here, please refer to
    + * http://jmlr.org/proceedings/papers/v28/meng13a.html
    + */
    +
    +private[spark] object StratifiedSampler extends Logging {
    +
    +  /**
    +   * Count the number of items instantly accepted and generate the waitlist for each stratum.
    +   *
    +   * This is only invoked when exact sample size is required.
    +   */
    +  def getCounts[K, V](rdd: RDD[(K, V)],
    +      withReplacement: Boolean,
    +      fractions: Map[K, Double],
    +      counts: Option[Map[K, Long]],
    +      seed: Long): MMap[K, Stratum] = {
    +    val combOp = getCombOp[K]
    +    val mappedPartitionRDD = rdd.mapPartitionsWithIndex({ case (partition, iter) =>
    +      val zeroU: MMap[K, Stratum] = new HashMap[K, Stratum]()
    +      val rng = new RandomDataGenerator()
    +      rng.reSeed(seed + partition)
    +      val seqOp = getSeqOp(withReplacement, fractions, rng, counts)
    +      Iterator(iter.aggregate(zeroU)(seqOp, combOp))
    +    }, preservesPartitioning=true)
    +    mappedPartitionRDD.reduce(combOp)
    +  }
    +
    +  /**
    +   * Returns the function used by aggregate to collect sampling statistics for each partition.
    +   */
    +  def getSeqOp[K, V](withReplacement: Boolean,
    +      fractions: Map[K, Double],
    +      rng: RandomDataGenerator,
    +      counts: Option[Map[K, Long]]): (MMap[K, Stratum], (K, V)) => MMap[K, Stratum] = {
    +    val delta = 5e-5
    +    (result: MMap[K, Stratum], item: (K, V)) => {
    +      val key = item._1
    +      val fraction = fractions(key)
    +      if (!result.contains(key)) {
    +        result += (key -> new Stratum())
    +      }
    +      val stratum = result(key)
    +
    +      if (withReplacement) {
    +        // compute acceptBound and waitListBound only if they haven't been computed already
    +        // since they don't change from iteration to iteration.
    +        // TODO change this to the streaming version
    +        if (stratum.areBoundsEmpty) {
    +          val n = counts.get(key)
    +          val sampleSize = math.ceil(n * fraction).toLong
    +          val lmbd1 = PoissonBounds.getLowerBound(sampleSize)
    +          val minCount = PoissonBounds.getMinCount(lmbd1)
    +          val lmbd2 = if (lmbd1 == 0) {
    +            PoissonBounds.getUpperBound(sampleSize)
    +          } else {
    +            PoissonBounds.getUpperBound(sampleSize - minCount)
    +          }
    +          stratum.acceptBound = lmbd1 / n
    +          stratum.waitListBound = lmbd2 / n
    +        }
    +        val acceptBound = stratum.acceptBound
    +        val copiesAccepted = if (acceptBound == 0.0) 0L else rng.nextPoisson(acceptBound)
    +        if (copiesAccepted > 0) {
    +          stratum.incrNumAccepted(copiesAccepted)
    +        }
    +        val copiesWaitlisted = rng.nextPoisson(stratum.waitListBound).toInt
    +        if (copiesWaitlisted > 0) {
    +          stratum.addToWaitList(ArrayBuffer.fill(copiesWaitlisted)(rng.nextUniform(0.0, 1.0)))
    +        }
    +      } else {
    +        // We use the streaming version of the algorithm for sampling without replacement to avoid
    +        // using an extra pass over the RDD for computing the count.
    +        // Hence, acceptBound and waitListBound change on every iteration.
    +        val gamma1 = - math.log(delta) / stratum.numItems
    +        val gamma2 = (2.0 / 3.0) * gamma1
    +        stratum.acceptBound = math.max(0,
    +          fraction + gamma2 - math.sqrt(gamma2 * gamma2 + 3 * gamma2 * fraction))
    +        stratum.waitListBound = math.min(1,
    +          fraction + gamma1 + math.sqrt(gamma1 * gamma1 + 2 * gamma1 * fraction))
    +
    +        val x = rng.nextUniform(0.0, 1.0)
    +        if (x < stratum.acceptBound) {
    +          stratum.incrNumAccepted()
    +        } else if (x < stratum.waitListBound) {
    +          stratum.addToWaitList(x)
    +        }
    +      }
    +      stratum.incrNumItems()
    +      result
    +    }
    +  }
    +
    +  /**
    +   * Returns the function used combine results returned by seqOp from different partitions.
    +   */
    +  def getCombOp[K]: (MMap[K, Stratum], MMap[K, Stratum]) => MMap[K, Stratum] = {
    +    (result1: MMap[K, Stratum], result2: MMap[K, Stratum]) => {
    +      // take union of both key sets in case one partition doesn't contain all keys
    +      for (key <- result1.keySet.union(result2.keySet)) {
    +        // Use result2 to keep the combined result since r1 is usual empty
    +        val entry1 = result1.get(key)
    +        if (result2.contains(key)) {
    +          result2(key).merge(entry1)
    +        } else {
    +          if (entry1.isDefined) {
    +            result2 += (key -> entry1.get)
    +          }
    +        }
    +      }
    +      result2
    +    }
    +  }
    +
    +  /**
    +   * Given the result returned by getCounts, determine the threshold for accepting items to
    +   * generate exact sample size.
    +   *
    +   * To do so, we compute sampleSize = math.ceil(size * samplingRate) for each stratum and compare
    +   * it to the number of items that were accepted instantly and the number of items in the waitlist
    +   * for that stratum. Most of the time, numAccepted <= sampleSize <= (numAccepted + numWaitlisted),
    +   * which means we need to sort the elements in the waitlist by their associated values in order
    +   * to find the value T s.t. |{elements in the stratum whose associated values <= T}| = sampleSize.
    +   * Note that all elements in the waitlist have values >= bound for instant accept, so a T value
    +   * in the waitlist range would allow all elements that were instantly accepted on the first pass
    +   * to be included in the sample.
    +   */
    +  def computeThresholdByKey[K](finalResult: Map[K, Stratum],
    +      fractions: Map[K, Double]): Map[K, Double] = {
    +    val thresholdByKey = new HashMap[K, Double]()
    +    for ((key, stratum) <- finalResult) {
    +      val sampleSize = math.ceil(stratum.numItems * fractions(key)).toLong
    +      if (stratum.numAccepted > sampleSize) {
    +        logWarning("Pre-accepted too many")
    +        thresholdByKey += (key -> stratum.acceptBound)
    +      } else {
    +        val numWaitListAccepted = (sampleSize - stratum.numAccepted).toInt
    +        if (numWaitListAccepted >= stratum.waitList.size) {
    +          logWarning("WaitList too short")
    +          thresholdByKey += (key -> stratum.waitListBound)
    +        } else {
    +          thresholdByKey += (key -> stratum.waitList.sorted.apply(numWaitListAccepted))
    +        }
    +      }
    +    }
    +    thresholdByKey
    +  }
    +
    +  /**
    +   * Return the per partition sampling function used for sampling without replacement.
    +   *
    +   * When exact sample size is required, we make an additional pass over the RDD to determine the
    +   * exact sampling rate that guarantees sample size with high confidence.
    +   *
    +   * The sampling function has a unique seed per partition.
    +   */
    +  def getBernoulliSamplingFunction[K, V](rdd: RDD[(K,  V)],
    +      fractions: Map[K, Double],
    +      exact: Boolean,
    +      seed: Long): (Int, Iterator[(K, V)]) => Iterator[(K, V)] = {
    +    var samplingRateByKey = fractions
    +    if (exact) {
    +      // determine threshold for each stratum and resample
    +      val finalResult = getCounts(rdd, false, fractions, None, seed)
    +      samplingRateByKey = computeThresholdByKey(finalResult, fractions)
    +    }
    +    (idx: Int, iter: Iterator[(K, V)]) => {
    +      val rng = new RandomDataGenerator
    +      rng.reSeed(seed + idx)
    +      iter.filter(t => rng.nextUniform(0.0, 1.0) < samplingRateByKey(t._1))
    +    }
    +  }
    +
    +  /**
    +   * Return the per partition sampling function used for sampling with replacement.
    +   *
    +   * When exact sample size is required, we make two additional passed over the RDD to determine
    +   * the exact sampling rate that guarantees sample size with high confidence. The first pass
    +   * counts the number of items in each stratum (group of items with the same key) in the RDD, and
    +   * the second pass uses the counts to determine exact sampling rates.
    +   *
    +   * The sampling function has a unique seed per partition.
    +   */
    +  def getPoissonSamplingFunction[K, V](rdd: RDD[(K, V)],
    +      fractions: Map[K, Double],
    +      exact: Boolean,
    +      counts: Option[Map[K, Long]],
    +      seed: Long): (Int, Iterator[(K, V)]) => Iterator[(K, V)] = {
    +    // TODO implement the streaming version of sampling w/ replacement that doesn't require counts
    +    if (exact) {
    +      val finalResult = getCounts(rdd, true, fractions, counts, seed)
    +      val thresholdByKey = computeThresholdByKey(finalResult, fractions)
    +      (idx: Int, iter: Iterator[(K, V)]) => {
    +        val rng = new RandomDataGenerator()
    +        rng.reSeed(seed + idx)
    +        iter.flatMap { item =>
    +          val key = item._1
    +          val acceptBound = finalResult(key).acceptBound
    +          val copiesAccepted = if (acceptBound == 0) 0L else rng.nextPoisson(acceptBound)
    +          val copiesWailisted = rng.nextPoisson(finalResult(key).waitListBound).toInt
    +          val copiesInSample = copiesAccepted +
    +            (0 until copiesWailisted).count(i => rng.nextUniform(0.0, 1.0) < thresholdByKey(key))
    --- End diff --
    
    Where is the key recomputed?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by falaki <gi...@git.apache.org>.
Github user falaki commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1025#discussion_r14688338
  
    --- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
    @@ -0,0 +1,310 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.util.random
    +
    +import scala.collection.Map
    +import scala.collection.mutable.{ArrayBuffer, HashMap, Map => MMap}
    +
    +import org.apache.commons.math3.random.RandomDataGenerator
    +import org.apache.spark.Logging
    +import org.apache.spark.rdd.RDD
    +
    +/**
    + * Auxiliary functions and data structures for the sampleByKey method in PairRDDFunctions.
    + *
    + * Essentially, when exact sample size is necessary, we make additional passes over the RDD to
    + * compute the exact threshold value to use for each stratum to guarantee exact sample size with
    + * high probability. This is achieved by maintaining a waitlist of size O(log(s)), where s is the
    + * desired sample size for each stratum.
    + *
    + * Like in simple random sampling, we generate a random value for each item from the
    + * uniform  distribution [0.0, 1.0]. All items with values <= min(values of items in the waitlist)
    + * are accepted into the sample instantly. The threshold for instant accept is designed so that
    + * s - numAccepted = O(log(s)), where s is again the desired sample size. Thus, by maintaining a
    + * waitlist size = O(log(s)), we will be able to create a sample of the exact size s by adding
    + * a portion of the waitlist to the set of items that are instantly accepted. The exact threshold
    + * is computed by sorting the values in the waitlist and picking the value at (s - numAccepted).
    + *
    + * Note that since we use the same seed for the RNG when computing the thresholds and the actual
    + * sample, our computed thresholds are guaranteed to produce the desired sample size.
    + *
    + * For more theoretical background on the sampling techniques used here, please refer to
    + * http://jmlr.org/proceedings/papers/v28/meng13a.html
    + */
    +
    +private[spark] object StratifiedSampler extends Logging {
    +
    +  /**
    +   * Count the number of items instantly accepted and generate the waitlist for each stratum.
    +   *
    +   * This is only invoked when exact sample size is required.
    +   */
    +  def getCounts[K, V](rdd: RDD[(K, V)],
    +      withReplacement: Boolean,
    +      fractions: Map[K, Double],
    +      counts: Option[Map[K, Long]],
    +      seed: Long): MMap[K, Stratum] = {
    +    val combOp = getCombOp[K]
    +    val mappedPartitionRDD = rdd.mapPartitionsWithIndex({ case (partition, iter) =>
    +      val zeroU: MMap[K, Stratum] = new HashMap[K, Stratum]()
    +      val rng = new RandomDataGenerator()
    +      rng.reSeed(seed + partition)
    +      val seqOp = getSeqOp(withReplacement, fractions, rng, counts)
    +      Iterator(iter.aggregate(zeroU)(seqOp, combOp))
    +    }, preservesPartitioning=true)
    +    mappedPartitionRDD.reduce(combOp)
    +  }
    +
    +  /**
    +   * Returns the function used by aggregate to collect sampling statistics for each partition.
    +   */
    +  def getSeqOp[K, V](withReplacement: Boolean,
    +      fractions: Map[K, Double],
    +      rng: RandomDataGenerator,
    +      counts: Option[Map[K, Long]]): (MMap[K, Stratum], (K, V)) => MMap[K, Stratum] = {
    +    val delta = 5e-5
    +    (result: MMap[K, Stratum], item: (K, V)) => {
    +      val key = item._1
    +      val fraction = fractions(key)
    +      if (!result.contains(key)) {
    +        result += (key -> new Stratum())
    +      }
    +      val stratum = result(key)
    +
    +      if (withReplacement) {
    +        // compute acceptBound and waitListBound only if they haven't been computed already
    +        // since they don't change from iteration to iteration.
    +        // TODO change this to the streaming version
    +        if (stratum.areBoundsEmpty) {
    +          val n = counts.get(key)
    +          val sampleSize = math.ceil(n * fraction).toLong
    +          val lmbd1 = PoissonBounds.getLowerBound(sampleSize)
    +          val minCount = PoissonBounds.getMinCount(lmbd1)
    +          val lmbd2 = if (lmbd1 == 0) {
    +            PoissonBounds.getUpperBound(sampleSize)
    +          } else {
    +            PoissonBounds.getUpperBound(sampleSize - minCount)
    +          }
    +          stratum.acceptBound = lmbd1 / n
    +          stratum.waitListBound = lmbd2 / n
    +        }
    +        val acceptBound = stratum.acceptBound
    +        val copiesAccepted = if (acceptBound == 0.0) 0L else rng.nextPoisson(acceptBound)
    +        if (copiesAccepted > 0) {
    +          stratum.incrNumAccepted(copiesAccepted)
    +        }
    +        val copiesWaitlisted = rng.nextPoisson(stratum.waitListBound).toInt
    +        if (copiesWaitlisted > 0) {
    +          stratum.addToWaitList(ArrayBuffer.fill(copiesWaitlisted)(rng.nextUniform(0.0, 1.0)))
    +        }
    +      } else {
    +        // We use the streaming version of the algorithm for sampling without replacement to avoid
    +        // using an extra pass over the RDD for computing the count.
    +        // Hence, acceptBound and waitListBound change on every iteration.
    +        val g1 = - math.log(delta) / stratum.numItems // gamma1
    +        val g2 = (2.0 / 3.0) * g1                     // gamma 2
    --- End diff --
    
    Minor: I still think it is nicer to just rename ```g1``` to ```gamma1```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by falaki <gi...@git.apache.org>.
Github user falaki commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1025#discussion_r14544684
  
    --- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
    @@ -0,0 +1,296 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.util.random
    +
    +import scala.collection.{Map, mutable}
    +import scala.collection.mutable.ArrayBuffer
    +import scala.reflect.ClassTag
    +
    +import org.apache.commons.math3.random.RandomDataGenerator
    +import org.apache.spark.{Logging, SparkContext, TaskContext}
    +import org.apache.spark.rdd.RDD
    +import org.apache.spark.util.Utils
    +import org.apache.spark.util.random.{PoissonBounds => PB}
    +
    +private[spark] object StratifiedSampler extends Logging {
    +
    +  /**
    +   * A version of {@link #aggregate()} that passes the TaskContext to the function that does
    +   * aggregation for each partition. This function avoids creating an extra depth in the RDD
    +   * lineage, as opposed to using mapPartitionsWithId, which results in slightly improved run time.
    +   */
    +  def aggregateWithContext[U: ClassTag, T: ClassTag](zeroValue: U)
    +      (rdd: RDD[T],
    +       seqOp: ((TaskContext, U), T) => U,
    +       combOp: (U, U) => U): U = {
    +    val sc: SparkContext = rdd.sparkContext
    +    // Clone the zero value since we will also be serializing it as part of tasks
    +    var jobResult = Utils.clone(zeroValue, sc.env.closureSerializer.newInstance())
    +    // pad seqOp and combOp with taskContext to conform to aggregate's signature in TraversableOnce
    +    val paddedSeqOp = (arg1: (TaskContext, U), item: T) => (arg1._1, seqOp(arg1, item))
    +    val paddedcombOp = (arg1: (TaskContext, U), arg2: (TaskContext, U)) =>
    +      (arg1._1, combOp(arg1._2, arg1._2))
    +    val cleanSeqOp = sc.clean(paddedSeqOp)
    +    val cleanCombOp = sc.clean(paddedcombOp)
    +    val aggregatePartition = (tc: TaskContext, it: Iterator[T]) =>
    +      (it.aggregate(tc, zeroValue)(cleanSeqOp, cleanCombOp))._2
    +    val mergeResult = (index: Int, taskResult: U) => jobResult = combOp(jobResult, taskResult)
    +    sc.runJob(rdd, aggregatePartition, mergeResult)
    +    jobResult
    +  }
    +
    +  /**
    +   * Returns the function used by aggregate to collect sampling statistics for each partition.
    +   */
    +  def getSeqOp[K, V](withReplacement: Boolean,
    +      fractionByKey: (K => Double),
    +      counts: Option[Map[K, Long]]): ((TaskContext, Result[K]),(K, V)) => Result[K] = {
    +    val delta = 5e-5
    +    (U: (TaskContext, Result[K]), item: (K, V)) => {
    +      val result = U._2
    +      val tc = U._1
    +      val rng = result.getRand(tc.partitionId)
    +      val fraction = fractionByKey(item._1)
    +      val stratum = result.getEntry(item._1)
    +      if (withReplacement) {
    +        // compute q1 and q2 only if they haven't been computed already
    +        // since they don't change from iteration to iteration.
    +        // TODO change this to the streaming version
    +        if (stratum.q1.isEmpty || stratum.q2.isEmpty) {
    +          val n = counts.get(item._1)
    +          val s = math.ceil(n * fraction).toLong
    +          val lmbd1 = PB.getLowerBound(s)
    +          val minCount = PB.getMinCount(lmbd1)
    +          val lmbd2 = if (lmbd1 == 0) PB.getUpperBound(s) else PB.getUpperBound(s - minCount)
    +          val q1 = lmbd1 / n
    +          val q2 = lmbd2 / n
    +          stratum.q1 = Some(q1)
    +          stratum.q2 = Some(q2)
    +        }
    +        val x1 = if (stratum.q1.get == 0) 0L else rng.nextPoisson(stratum.q1.get)
    +        if (x1 > 0) {
    +          stratum.incrNumAccepted(x1)
    +        }
    +        val x2 = rng.nextPoisson(stratum.q2.get).toInt
    +        if (x2 > 0) {
    +          stratum.addToWaitList(ArrayBuffer.fill(x2)(rng.nextUniform(0.0, 1.0)))
    +        }
    +      } else {
    +        // We use the streaming version of the algorithm for sampling without replacement.
    +        // Hence, q1 and q2 change on every iteration.
    +        val g1 = - math.log(delta) / stratum.numItems
    +        val g2 = (2.0 / 3.0) * g1
    +        val q1 = math.max(0, fraction + g2 - math.sqrt((g2 * g2 + 3 * g2 * fraction)))
    +        val q2 = math.min(1, fraction + g1 + math.sqrt(g1 * g1 + 2 * g1 * fraction))
    +
    +        val x = rng.nextUniform(0.0, 1.0)
    +        if (x < q1) {
    +          stratum.incrNumAccepted()
    +        } else if (x < q2) {
    +          stratum.addToWaitList(x)
    +        }
    +        stratum.q1 = Some(q1)
    +        stratum.q2 = Some(q2)
    +      }
    +      stratum.incrNumItems()
    +      result
    +    }
    +  }
    +
    +  /**
    +   * Returns the function used by aggregate to combine results from different partitions, as
    +   * returned by seqOp.
    +   */
    +  def getCombOp[K](): (Result[K], Result[K]) => Result[K] = {
    +    (r1: Result[K], r2: Result[K]) => {
    +      // take union of both key sets in case one partition doesn't contain all keys
    +      val keyUnion = r1.resultMap.keys.toSet.union(r2.resultMap.keys.toSet)
    +
    +      // Use r2 to keep the combined result since r1 is usual empty
    +      for (key <- keyUnion) {
    +        val entry1 = r1.resultMap.get(key)
    +        val entry2 = r2.resultMap.get(key)
    +        if (entry2.isEmpty && entry1.isDefined) {
    +          r2.resultMap += (key -> entry1.get)
    --- End diff --
    
    Since we already have a helper class for Result, we can make these lines simpler by adding some helper methods to Result. For example this could become something like:
    ```scala
    r2.addMapping(key, entry1)
    ```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by falaki <gi...@git.apache.org>.
Github user falaki commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1025#discussion_r14544754
  
    --- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
    @@ -0,0 +1,296 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.util.random
    +
    +import scala.collection.{Map, mutable}
    +import scala.collection.mutable.ArrayBuffer
    +import scala.reflect.ClassTag
    +
    +import org.apache.commons.math3.random.RandomDataGenerator
    +import org.apache.spark.{Logging, SparkContext, TaskContext}
    +import org.apache.spark.rdd.RDD
    +import org.apache.spark.util.Utils
    +import org.apache.spark.util.random.{PoissonBounds => PB}
    +
    +private[spark] object StratifiedSampler extends Logging {
    +
    +  /**
    +   * A version of {@link #aggregate()} that passes the TaskContext to the function that does
    +   * aggregation for each partition. This function avoids creating an extra depth in the RDD
    +   * lineage, as opposed to using mapPartitionsWithId, which results in slightly improved run time.
    +   */
    +  def aggregateWithContext[U: ClassTag, T: ClassTag](zeroValue: U)
    +      (rdd: RDD[T],
    +       seqOp: ((TaskContext, U), T) => U,
    +       combOp: (U, U) => U): U = {
    +    val sc: SparkContext = rdd.sparkContext
    +    // Clone the zero value since we will also be serializing it as part of tasks
    +    var jobResult = Utils.clone(zeroValue, sc.env.closureSerializer.newInstance())
    +    // pad seqOp and combOp with taskContext to conform to aggregate's signature in TraversableOnce
    +    val paddedSeqOp = (arg1: (TaskContext, U), item: T) => (arg1._1, seqOp(arg1, item))
    +    val paddedcombOp = (arg1: (TaskContext, U), arg2: (TaskContext, U)) =>
    +      (arg1._1, combOp(arg1._2, arg1._2))
    +    val cleanSeqOp = sc.clean(paddedSeqOp)
    +    val cleanCombOp = sc.clean(paddedcombOp)
    +    val aggregatePartition = (tc: TaskContext, it: Iterator[T]) =>
    +      (it.aggregate(tc, zeroValue)(cleanSeqOp, cleanCombOp))._2
    +    val mergeResult = (index: Int, taskResult: U) => jobResult = combOp(jobResult, taskResult)
    +    sc.runJob(rdd, aggregatePartition, mergeResult)
    +    jobResult
    +  }
    +
    +  /**
    +   * Returns the function used by aggregate to collect sampling statistics for each partition.
    +   */
    +  def getSeqOp[K, V](withReplacement: Boolean,
    +      fractionByKey: (K => Double),
    +      counts: Option[Map[K, Long]]): ((TaskContext, Result[K]),(K, V)) => Result[K] = {
    +    val delta = 5e-5
    +    (U: (TaskContext, Result[K]), item: (K, V)) => {
    +      val result = U._2
    +      val tc = U._1
    +      val rng = result.getRand(tc.partitionId)
    +      val fraction = fractionByKey(item._1)
    +      val stratum = result.getEntry(item._1)
    +      if (withReplacement) {
    +        // compute q1 and q2 only if they haven't been computed already
    +        // since they don't change from iteration to iteration.
    +        // TODO change this to the streaming version
    +        if (stratum.q1.isEmpty || stratum.q2.isEmpty) {
    +          val n = counts.get(item._1)
    +          val s = math.ceil(n * fraction).toLong
    +          val lmbd1 = PB.getLowerBound(s)
    +          val minCount = PB.getMinCount(lmbd1)
    +          val lmbd2 = if (lmbd1 == 0) PB.getUpperBound(s) else PB.getUpperBound(s - minCount)
    +          val q1 = lmbd1 / n
    +          val q2 = lmbd2 / n
    +          stratum.q1 = Some(q1)
    +          stratum.q2 = Some(q2)
    +        }
    +        val x1 = if (stratum.q1.get == 0) 0L else rng.nextPoisson(stratum.q1.get)
    +        if (x1 > 0) {
    +          stratum.incrNumAccepted(x1)
    +        }
    +        val x2 = rng.nextPoisson(stratum.q2.get).toInt
    +        if (x2 > 0) {
    +          stratum.addToWaitList(ArrayBuffer.fill(x2)(rng.nextUniform(0.0, 1.0)))
    +        }
    +      } else {
    +        // We use the streaming version of the algorithm for sampling without replacement.
    +        // Hence, q1 and q2 change on every iteration.
    +        val g1 = - math.log(delta) / stratum.numItems
    +        val g2 = (2.0 / 3.0) * g1
    +        val q1 = math.max(0, fraction + g2 - math.sqrt((g2 * g2 + 3 * g2 * fraction)))
    +        val q2 = math.min(1, fraction + g1 + math.sqrt(g1 * g1 + 2 * g1 * fraction))
    +
    +        val x = rng.nextUniform(0.0, 1.0)
    +        if (x < q1) {
    +          stratum.incrNumAccepted()
    +        } else if (x < q2) {
    +          stratum.addToWaitList(x)
    +        }
    +        stratum.q1 = Some(q1)
    +        stratum.q2 = Some(q2)
    +      }
    +      stratum.incrNumItems()
    +      result
    +    }
    +  }
    +
    +  /**
    +   * Returns the function used by aggregate to combine results from different partitions, as
    +   * returned by seqOp.
    +   */
    +  def getCombOp[K](): (Result[K], Result[K]) => Result[K] = {
    +    (r1: Result[K], r2: Result[K]) => {
    +      // take union of both key sets in case one partition doesn't contain all keys
    +      val keyUnion = r1.resultMap.keys.toSet.union(r2.resultMap.keys.toSet)
    +
    +      // Use r2 to keep the combined result since r1 is usual empty
    +      for (key <- keyUnion) {
    +        val entry1 = r1.resultMap.get(key)
    +        val entry2 = r2.resultMap.get(key)
    +        if (entry2.isEmpty && entry1.isDefined) {
    +          r2.resultMap += (key -> entry1.get)
    +        } else if (entry1.isDefined && entry2.isDefined) {
    +          entry2.get.addToWaitList(entry1.get.waitList)
    +          entry2.get.incrNumAccepted(entry1.get.numAccepted)
    +          entry2.get.incrNumItems(entry1.get.numItems)
    +        }
    +      }
    +      r2
    +    }
    +  }
    +
    +  /**
    +   * Given the result returned by the aggregate function, we need to determine the threshold used
    +   * to accept items to generate the exact sample size.
    +   */
    +  def computeThresholdByKey[K](finalResult: Map[K, Stratum], fractionByKey: (K => Double)):
    +    (K => Double) = {
    +    val thresholdByKey = new mutable.HashMap[K, Double]()
    +    for ((key, stratum) <- finalResult) {
    --- End diff --
    
    Some comments that explain the logic here would be great.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1025#discussion_r14694381
  
    --- Diff: core/src/test/scala/org/apache/spark/rdd/PairRDDFunctionsSuite.scala ---
    @@ -83,6 +83,125 @@ class PairRDDFunctionsSuite extends FunSuite with SharedSparkContext {
         assert(valuesFor2.toList.sorted === List(1))
       }
     
    +  test("sampleByKey") {
    +    def stratifier (fractionPositive: Double) = {
    +      (x: Int) => if (x % 10 < (10 * fractionPositive).toInt) "1" else "0"
    +    }
    +
    +    def checkSize(exact: Boolean,
    +        withReplacement: Boolean,
    +        expected: Long,
    +        actual: Long,
    +        p: Double): Boolean = {
    +      if (exact) {
    +        return expected == actual
    +      }
    +      val stdev = if (withReplacement) math.sqrt(expected) else math.sqrt(expected * p * (1 - p))
    +      // Very forgiving margin since we're dealing with very small sample sizes most of the time
    +      math.abs(actual - expected) <= 6 * stdev
    +    }
    +
    +    // Without replacement validation
    +    def takeSampleAndValidateBernoulli(stratifiedData: RDD[(String, Int)],
    +        exact: Boolean,
    +        samplingRate: Double,
    +        seed: Long,
    +        n: Long) = {
    +      val expectedSampleSize = stratifiedData.countByKey().mapValues(count =>
    +        math.ceil(count * samplingRate).toInt)
    +      val fractions = Map("1" -> samplingRate, "0" -> samplingRate)
    +      val sample = stratifiedData.sampleByKey(false, fractions, exact, seed)
    +      val sampleCounts = sample.countByKey()
    +      val takeSample = sample.collect()
    +      assert(sampleCounts.forall {case(k,v) =>
    +        checkSize(exact, false, expectedSampleSize(k), v, samplingRate)})
    +      assert(takeSample.size === takeSample.toSet.size)
    +      assert(takeSample.forall(x => 1 <= x._2 && x._2 <= n), s"elements not in [1, $n]")
    +    }
    +
    +    // With replacement validation
    +    def takeSampleAndValidatePoisson(stratifiedData: RDD[(String, Int)],
    +        exact: Boolean,
    +        samplingRate: Double,
    +        seed: Long,
    +        n: Long) = {
    +      val expectedSampleSize = stratifiedData.countByKey().mapValues(count =>
    +        math.ceil(count * samplingRate).toInt)
    +      val fractions = Map("1" -> samplingRate, "0" -> samplingRate)
    +      val sample = stratifiedData.sampleByKey(true, fractions, exact, seed)
    +      val sampleCounts = sample.countByKey()
    +      val takeSample = sample.collect()
    +      assert(sampleCounts.forall {case(k,v) =>
    --- End diff --
    
    same style issus


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by dorx <gi...@git.apache.org>.
Github user dorx commented on the pull request:

    https://github.com/apache/spark/pull/1025#issuecomment-48419179
  
    Holding out on updating the docs until the python version is supported.
    For the python version, any objections to using _jrdd to invoke the java version of sampleByKey?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1025#discussion_r14694287
  
    --- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
    @@ -0,0 +1,311 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.util.random
    +
    +import scala.collection.Map
    +import scala.collection.mutable.{ArrayBuffer, HashMap, Map => MMap}
    +
    +import org.apache.commons.math3.random.RandomDataGenerator
    +import org.apache.spark.Logging
    +import org.apache.spark.rdd.RDD
    +
    +/**
    + * Auxiliary functions and data structures for the sampleByKey method in PairRDDFunctions.
    + *
    + * Essentially, when exact sample size is necessary, we make additional passes over the RDD to
    + * compute the exact threshold value to use for each stratum to guarantee exact sample size with
    + * high probability. This is achieved by maintaining a waitlist of size O(log(s)), where s is the
    + * desired sample size for each stratum.
    + *
    + * Like in simple random sampling, we generate a random value for each item from the
    + * uniform  distribution [0.0, 1.0]. All items with values <= min(values of items in the waitlist)
    + * are accepted into the sample instantly. The threshold for instant accept is designed so that
    + * s - numAccepted = O(log(s)), where s is again the desired sample size. Thus, by maintaining a
    + * waitlist size = O(log(s)), we will be able to create a sample of the exact size s by adding
    + * a portion of the waitlist to the set of items that are instantly accepted. The exact threshold
    + * is computed by sorting the values in the waitlist and picking the value at (s - numAccepted).
    + *
    + * Note that since we use the same seed for the RNG when computing the thresholds and the actual
    + * sample, our computed thresholds are guaranteed to produce the desired sample size.
    + *
    + * For more theoretical background on the sampling techniques used here, please refer to
    + * http://jmlr.org/proceedings/papers/v28/meng13a.html
    + */
    +
    +private[spark] object StratifiedSampler extends Logging {
    +
    +  /**
    +   * Count the number of items instantly accepted and generate the waitlist for each stratum.
    +   *
    +   * This is only invoked when exact sample size is required.
    +   */
    +  def getCounts[K, V](rdd: RDD[(K, V)],
    +      withReplacement: Boolean,
    +      fractions: Map[K, Double],
    +      counts: Option[Map[K, Long]],
    +      seed: Long): MMap[K, Stratum] = {
    +    val combOp = getCombOp[K]
    +    val mappedPartitionRDD = rdd.mapPartitionsWithIndex({ case (partition, iter) =>
    +      val zeroU: MMap[K, Stratum] = new HashMap[K, Stratum]()
    +      val rng = new RandomDataGenerator()
    +      rng.reSeed(seed + partition)
    +      val seqOp = getSeqOp(withReplacement, fractions, rng, counts)
    +      Iterator(iter.aggregate(zeroU)(seqOp, combOp))
    +    }, preservesPartitioning=true)
    +    mappedPartitionRDD.reduce(combOp)
    +  }
    +
    +  /**
    +   * Returns the function used by aggregate to collect sampling statistics for each partition.
    +   */
    +  def getSeqOp[K, V](withReplacement: Boolean,
    +      fractions: Map[K, Double],
    +      rng: RandomDataGenerator,
    +      counts: Option[Map[K, Long]]): (MMap[K, Stratum], (K, V)) => MMap[K, Stratum] = {
    +    val delta = 5e-5
    +    (result: MMap[K, Stratum], item: (K, V)) => {
    +      val key = item._1
    +      val fraction = fractions(key)
    +      if (!result.contains(key)) {
    +        result += (key -> new Stratum())
    +      }
    +      val stratum = result(key)
    +
    +      if (withReplacement) {
    +        // compute acceptBound and waitListBound only if they haven't been computed already
    +        // since they don't change from iteration to iteration.
    +        // TODO change this to the streaming version
    +        if (stratum.areBoundsEmpty) {
    +          val n = counts.get(key)
    +          val sampleSize = math.ceil(n * fraction).toLong
    +          val lmbd1 = PoissonBounds.getLowerBound(sampleSize)
    +          val minCount = PoissonBounds.getMinCount(lmbd1)
    +          val lmbd2 = if (lmbd1 == 0) {
    +            PoissonBounds.getUpperBound(sampleSize)
    +          } else {
    +            PoissonBounds.getUpperBound(sampleSize - minCount)
    +          }
    +          stratum.acceptBound = lmbd1 / n
    +          stratum.waitListBound = lmbd2 / n
    +        }
    +        val acceptBound = stratum.acceptBound
    +        val copiesAccepted = if (acceptBound == 0.0) 0L else rng.nextPoisson(acceptBound)
    +        if (copiesAccepted > 0) {
    +          stratum.incrNumAccepted(copiesAccepted)
    +        }
    +        val copiesWaitlisted = rng.nextPoisson(stratum.waitListBound).toInt
    +        if (copiesWaitlisted > 0) {
    +          stratum.addToWaitList(ArrayBuffer.fill(copiesWaitlisted)(rng.nextUniform(0.0, 1.0)))
    +        }
    +      } else {
    +        // We use the streaming version of the algorithm for sampling without replacement to avoid
    +        // using an extra pass over the RDD for computing the count.
    +        // Hence, acceptBound and waitListBound change on every iteration.
    +        val gamma1 = - math.log(delta) / stratum.numItems
    +        val gamma2 = (2.0 / 3.0) * gamma1
    +        stratum.acceptBound = math.max(0,
    +          fraction + gamma2 - math.sqrt(gamma2 * gamma2 + 3 * gamma2 * fraction))
    +        stratum.waitListBound = math.min(1,
    +          fraction + gamma1 + math.sqrt(gamma1 * gamma1 + 2 * gamma1 * fraction))
    +
    +        val x = rng.nextUniform(0.0, 1.0)
    +        if (x < stratum.acceptBound) {
    +          stratum.incrNumAccepted()
    +        } else if (x < stratum.waitListBound) {
    +          stratum.addToWaitList(x)
    +        }
    +      }
    +      stratum.incrNumItems()
    +      result
    +    }
    +  }
    +
    +  /**
    +   * Returns the function used combine results returned by seqOp from different partitions.
    +   */
    +  def getCombOp[K]: (MMap[K, Stratum], MMap[K, Stratum]) => MMap[K, Stratum] = {
    +    (result1: MMap[K, Stratum], result2: MMap[K, Stratum]) => {
    +      // take union of both key sets in case one partition doesn't contain all keys
    +      for (key <- result1.keySet.union(result2.keySet)) {
    +        // Use result2 to keep the combined result since r1 is usual empty
    +        val entry1 = result1.get(key)
    +        if (result2.contains(key)) {
    +          result2(key).merge(entry1)
    +        } else {
    +          if (entry1.isDefined) {
    +            result2 += (key -> entry1.get)
    +          }
    +        }
    +      }
    +      result2
    +    }
    +  }
    +
    +  /**
    +   * Given the result returned by getCounts, determine the threshold for accepting items to
    +   * generate exact sample size.
    +   *
    +   * To do so, we compute sampleSize = math.ceil(size * samplingRate) for each stratum and compare
    +   * it to the number of items that were accepted instantly and the number of items in the waitlist
    +   * for that stratum. Most of the time, numAccepted <= sampleSize <= (numAccepted + numWaitlisted),
    +   * which means we need to sort the elements in the waitlist by their associated values in order
    +   * to find the value T s.t. |{elements in the stratum whose associated values <= T}| = sampleSize.
    +   * Note that all elements in the waitlist have values >= bound for instant accept, so a T value
    +   * in the waitlist range would allow all elements that were instantly accepted on the first pass
    +   * to be included in the sample.
    +   */
    +  def computeThresholdByKey[K](finalResult: Map[K, Stratum],
    +      fractions: Map[K, Double]): Map[K, Double] = {
    +    val thresholdByKey = new HashMap[K, Double]()
    +    for ((key, stratum) <- finalResult) {
    +      val sampleSize = math.ceil(stratum.numItems * fractions(key)).toLong
    +      if (stratum.numAccepted > sampleSize) {
    +        logWarning("Pre-accepted too many")
    +        thresholdByKey += (key -> stratum.acceptBound)
    +      } else {
    +        val numWaitListAccepted = (sampleSize - stratum.numAccepted).toInt
    +        if (numWaitListAccepted >= stratum.waitList.size) {
    +          logWarning("WaitList too short")
    +          thresholdByKey += (key -> stratum.waitListBound)
    +        } else {
    +          thresholdByKey += (key -> stratum.waitList.sorted.apply(numWaitListAccepted))
    +        }
    +      }
    +    }
    +    thresholdByKey
    +  }
    +
    +  /**
    +   * Return the per partition sampling function used for sampling without replacement.
    +   *
    +   * When exact sample size is required, we make an additional pass over the RDD to determine the
    +   * exact sampling rate that guarantees sample size with high confidence.
    +   *
    +   * The sampling function has a unique seed per partition.
    +   */
    +  def getBernoulliSamplingFunction[K, V](rdd: RDD[(K,  V)],
    +      fractions: Map[K, Double],
    +      exact: Boolean,
    +      seed: Long): (Int, Iterator[(K, V)]) => Iterator[(K, V)] = {
    +    var samplingRateByKey = fractions
    +    if (exact) {
    +      // determine threshold for each stratum and resample
    +      val finalResult = getCounts(rdd, false, fractions, None, seed)
    +      samplingRateByKey = computeThresholdByKey(finalResult, fractions)
    +    }
    +    (idx: Int, iter: Iterator[(K, V)]) => {
    +      val rng = new RandomDataGenerator
    +      rng.reSeed(seed + idx)
    +      iter.filter(t => rng.nextUniform(0.0, 1.0) < samplingRateByKey(t._1))
    --- End diff --
    
    Please add a comment here saying the way we call rng must be the same as in `getCounts`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by falaki <gi...@git.apache.org>.
Github user falaki commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1025#discussion_r14622797
  
    --- Diff: core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala ---
    @@ -195,6 +193,37 @@ class PairRDDFunctions[K, V](self: RDD[(K, V)])
       }
     
       /**
    +   * Return a subset of this RDD sampled by key (via stratified sampling).
    +   *
    +   * If exact set to true, we guarantee, with high probability, a sample size =
    +   * math.ceil(fraction * S_i), where S_i is the size of the ith stratum (collection of entries
    +   * that share the same key). When sampling without replacement, we need one additional pass over
    +   * the RDD to guarantee sample size with a 99.99% confidence; when sampling with replacement, we
    +   * need two additional passes.
    +   *
    +   * @param withReplacement whether to sample with or without replacement
    +   * @param fractions map of specific keys to sampling rates
    +   * @param seed seed for the random number generator
    +   * @param exact whether sample size needs to be exactly math.ceil(fraction * size) per stratum
    +   * @return RDD containing the sampled subset
    +   */
    +  def sampleByKey(withReplacement: Boolean,
    +      fractions: Map[K, Double],
    +      exact: Boolean = true,
    +      seed: Long = Utils.random.nextLong): RDD[(K, V)]= {
    +
    +    require(fractions.forall {case(k, v) => v >= 0.0}, "Invalid sampling rates.")
    +
    +    val samplingFunc = if (withReplacement) {
    +      val counts = if (exact) Some(this.countByKey()) else None
    +        StratifiedSampler.getPoissonSamplingFunction(self, fractions, exact, counts, seed)
    --- End diff --
    
    Fix indent.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by dorx <gi...@git.apache.org>.
Github user dorx commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1025#discussion_r14906065
  
    --- Diff: pom.xml ---
    @@ -257,6 +257,11 @@
     	    <version>1.5</version>
           </dependency>
           <dependency>
    +        <groupId>org.apache.commons</groupId>
    +        <artifactId>commons-math3</artifactId>
    +        <version>3.3</version>
    --- End diff --
    
    The final decision was to remove dependency on commons-math3 and use approximation for Poisson inverseCDF instead. @srowen 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1025#issuecomment-48422320
  
    All automated tests passed.
    Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16441/


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1025#issuecomment-48386183
  
    Merged build finished. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1025#issuecomment-45545540
  
     Build triggered. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1025#issuecomment-48130356
  
    Merged build started. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1025#issuecomment-48414132
  
    Merged build started. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by dorx <gi...@git.apache.org>.
Github user dorx commented on the pull request:

    https://github.com/apache/spark/pull/1025#issuecomment-46402513
  
    @srowen Hey Sean, turns out Colt(or breeze for that matter) doesn't have inverseCDF for the distributions, which I need for the implementation. The original thought was to avoid increasing the number of dependencies for Spark, as we need to support multiple build frameworks (and it's a lot of compatibility testing work for the person merging the PR). As it turns out, there isn't really a viable substitute in the current dependencies, which was why I went and got the approval for adding commons-math3 as a dependency.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1025#discussion_r14694234
  
    --- Diff: core/src/main/scala/org/apache/spark/api/java/JavaPairRDD.scala ---
    @@ -130,6 +130,38 @@ class JavaPairRDD[K, V](val rdd: RDD[(K, V)])
         new JavaPairRDD[K, V](rdd.sample(withReplacement, fraction, seed))
     
       /**
    +   * Return a subset of this RDD sampled by key (via stratified sampling).
    +   */
    +  def sampleByKey(withReplacement: Boolean,
    +      fractions: JMap[K, Double],
    +      exact: Boolean,
    +      seed: Long): JavaPairRDD[K, V] =
    +    new JavaPairRDD[K, V](rdd.sampleByKey(withReplacement, fractions, exact, seed))
    +
    +
    --- End diff --
    
    remove empty line


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1025#discussion_r14907687
  
    --- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
    @@ -0,0 +1,311 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.util.random
    +
    +import scala.collection.Map
    +import scala.collection.mutable.{ArrayBuffer, HashMap, Map => MMap}
    +
    +import org.apache.commons.math3.random.RandomDataGenerator
    +import org.apache.spark.Logging
    +import org.apache.spark.rdd.RDD
    +
    +/**
    + * Auxiliary functions and data structures for the sampleByKey method in PairRDDFunctions.
    + *
    + * Essentially, when exact sample size is necessary, we make additional passes over the RDD to
    + * compute the exact threshold value to use for each stratum to guarantee exact sample size with
    + * high probability. This is achieved by maintaining a waitlist of size O(log(s)), where s is the
    + * desired sample size for each stratum.
    + *
    + * Like in simple random sampling, we generate a random value for each item from the
    + * uniform  distribution [0.0, 1.0]. All items with values <= min(values of items in the waitlist)
    + * are accepted into the sample instantly. The threshold for instant accept is designed so that
    + * s - numAccepted = O(log(s)), where s is again the desired sample size. Thus, by maintaining a
    + * waitlist size = O(log(s)), we will be able to create a sample of the exact size s by adding
    + * a portion of the waitlist to the set of items that are instantly accepted. The exact threshold
    + * is computed by sorting the values in the waitlist and picking the value at (s - numAccepted).
    + *
    + * Note that since we use the same seed for the RNG when computing the thresholds and the actual
    + * sample, our computed thresholds are guaranteed to produce the desired sample size.
    + *
    + * For more theoretical background on the sampling techniques used here, please refer to
    + * http://jmlr.org/proceedings/papers/v28/meng13a.html
    + */
    +
    +private[spark] object StratifiedSampler extends Logging {
    +
    +  /**
    +   * Count the number of items instantly accepted and generate the waitlist for each stratum.
    +   *
    +   * This is only invoked when exact sample size is required.
    +   */
    +  def getCounts[K, V](rdd: RDD[(K, V)],
    +      withReplacement: Boolean,
    +      fractions: Map[K, Double],
    +      counts: Option[Map[K, Long]],
    +      seed: Long): MMap[K, Stratum] = {
    +    val combOp = getCombOp[K]
    +    val mappedPartitionRDD = rdd.mapPartitionsWithIndex({ case (partition, iter) =>
    +      val zeroU: MMap[K, Stratum] = new HashMap[K, Stratum]()
    +      val rng = new RandomDataGenerator()
    +      rng.reSeed(seed + partition)
    +      val seqOp = getSeqOp(withReplacement, fractions, rng, counts)
    +      Iterator(iter.aggregate(zeroU)(seqOp, combOp))
    +    }, preservesPartitioning=true)
    +    mappedPartitionRDD.reduce(combOp)
    +  }
    +
    +  /**
    +   * Returns the function used by aggregate to collect sampling statistics for each partition.
    +   */
    +  def getSeqOp[K, V](withReplacement: Boolean,
    +      fractions: Map[K, Double],
    +      rng: RandomDataGenerator,
    +      counts: Option[Map[K, Long]]): (MMap[K, Stratum], (K, V)) => MMap[K, Stratum] = {
    +    val delta = 5e-5
    +    (result: MMap[K, Stratum], item: (K, V)) => {
    +      val key = item._1
    +      val fraction = fractions(key)
    +      if (!result.contains(key)) {
    +        result += (key -> new Stratum())
    +      }
    +      val stratum = result(key)
    +
    +      if (withReplacement) {
    +        // compute acceptBound and waitListBound only if they haven't been computed already
    +        // since they don't change from iteration to iteration.
    +        // TODO change this to the streaming version
    +        if (stratum.areBoundsEmpty) {
    +          val n = counts.get(key)
    +          val sampleSize = math.ceil(n * fraction).toLong
    +          val lmbd1 = PoissonBounds.getLowerBound(sampleSize)
    +          val minCount = PoissonBounds.getMinCount(lmbd1)
    +          val lmbd2 = if (lmbd1 == 0) {
    +            PoissonBounds.getUpperBound(sampleSize)
    +          } else {
    +            PoissonBounds.getUpperBound(sampleSize - minCount)
    +          }
    +          stratum.acceptBound = lmbd1 / n
    +          stratum.waitListBound = lmbd2 / n
    +        }
    +        val acceptBound = stratum.acceptBound
    +        val copiesAccepted = if (acceptBound == 0.0) 0L else rng.nextPoisson(acceptBound)
    +        if (copiesAccepted > 0) {
    +          stratum.incrNumAccepted(copiesAccepted)
    +        }
    +        val copiesWaitlisted = rng.nextPoisson(stratum.waitListBound).toInt
    +        if (copiesWaitlisted > 0) {
    +          stratum.addToWaitList(ArrayBuffer.fill(copiesWaitlisted)(rng.nextUniform(0.0, 1.0)))
    +        }
    +      } else {
    +        // We use the streaming version of the algorithm for sampling without replacement to avoid
    +        // using an extra pass over the RDD for computing the count.
    +        // Hence, acceptBound and waitListBound change on every iteration.
    +        val gamma1 = - math.log(delta) / stratum.numItems
    +        val gamma2 = (2.0 / 3.0) * gamma1
    +        stratum.acceptBound = math.max(0,
    +          fraction + gamma2 - math.sqrt(gamma2 * gamma2 + 3 * gamma2 * fraction))
    +        stratum.waitListBound = math.min(1,
    +          fraction + gamma1 + math.sqrt(gamma1 * gamma1 + 2 * gamma1 * fraction))
    +
    +        val x = rng.nextUniform(0.0, 1.0)
    +        if (x < stratum.acceptBound) {
    +          stratum.incrNumAccepted()
    +        } else if (x < stratum.waitListBound) {
    +          stratum.addToWaitList(x)
    +        }
    +      }
    +      stratum.incrNumItems()
    +      result
    +    }
    +  }
    +
    +  /**
    +   * Returns the function used combine results returned by seqOp from different partitions.
    +   */
    +  def getCombOp[K]: (MMap[K, Stratum], MMap[K, Stratum]) => MMap[K, Stratum] = {
    +    (result1: MMap[K, Stratum], result2: MMap[K, Stratum]) => {
    +      // take union of both key sets in case one partition doesn't contain all keys
    +      for (key <- result1.keySet.union(result2.keySet)) {
    +        // Use result2 to keep the combined result since r1 is usual empty
    +        val entry1 = result1.get(key)
    +        if (result2.contains(key)) {
    +          result2(key).merge(entry1)
    +        } else {
    +          if (entry1.isDefined) {
    +            result2 += (key -> entry1.get)
    +          }
    +        }
    +      }
    +      result2
    +    }
    +  }
    +
    +  /**
    +   * Given the result returned by getCounts, determine the threshold for accepting items to
    +   * generate exact sample size.
    +   *
    +   * To do so, we compute sampleSize = math.ceil(size * samplingRate) for each stratum and compare
    +   * it to the number of items that were accepted instantly and the number of items in the waitlist
    +   * for that stratum. Most of the time, numAccepted <= sampleSize <= (numAccepted + numWaitlisted),
    +   * which means we need to sort the elements in the waitlist by their associated values in order
    +   * to find the value T s.t. |{elements in the stratum whose associated values <= T}| = sampleSize.
    +   * Note that all elements in the waitlist have values >= bound for instant accept, so a T value
    +   * in the waitlist range would allow all elements that were instantly accepted on the first pass
    +   * to be included in the sample.
    +   */
    +  def computeThresholdByKey[K](finalResult: Map[K, Stratum],
    +      fractions: Map[K, Double]): Map[K, Double] = {
    +    val thresholdByKey = new HashMap[K, Double]()
    +    for ((key, stratum) <- finalResult) {
    +      val sampleSize = math.ceil(stratum.numItems * fractions(key)).toLong
    +      if (stratum.numAccepted > sampleSize) {
    +        logWarning("Pre-accepted too many")
    +        thresholdByKey += (key -> stratum.acceptBound)
    +      } else {
    +        val numWaitListAccepted = (sampleSize - stratum.numAccepted).toInt
    +        if (numWaitListAccepted >= stratum.waitList.size) {
    +          logWarning("WaitList too short")
    +          thresholdByKey += (key -> stratum.waitListBound)
    +        } else {
    +          thresholdByKey += (key -> stratum.waitList.sorted.apply(numWaitListAccepted))
    +        }
    +      }
    +    }
    +    thresholdByKey
    +  }
    +
    +  /**
    +   * Return the per partition sampling function used for sampling without replacement.
    +   *
    +   * When exact sample size is required, we make an additional pass over the RDD to determine the
    +   * exact sampling rate that guarantees sample size with high confidence.
    +   *
    +   * The sampling function has a unique seed per partition.
    +   */
    +  def getBernoulliSamplingFunction[K, V](rdd: RDD[(K,  V)],
    +      fractions: Map[K, Double],
    +      exact: Boolean,
    +      seed: Long): (Int, Iterator[(K, V)]) => Iterator[(K, V)] = {
    +    var samplingRateByKey = fractions
    +    if (exact) {
    +      // determine threshold for each stratum and resample
    +      val finalResult = getCounts(rdd, false, fractions, None, seed)
    +      samplingRateByKey = computeThresholdByKey(finalResult, fractions)
    +    }
    +    (idx: Int, iter: Iterator[(K, V)]) => {
    +      val rng = new RandomDataGenerator
    +      rng.reSeed(seed + idx)
    +      iter.filter(t => rng.nextUniform(0.0, 1.0) < samplingRateByKey(t._1))
    +    }
    +  }
    +
    +  /**
    +   * Return the per partition sampling function used for sampling with replacement.
    +   *
    +   * When exact sample size is required, we make two additional passed over the RDD to determine
    +   * the exact sampling rate that guarantees sample size with high confidence. The first pass
    +   * counts the number of items in each stratum (group of items with the same key) in the RDD, and
    +   * the second pass uses the counts to determine exact sampling rates.
    +   *
    +   * The sampling function has a unique seed per partition.
    +   */
    +  def getPoissonSamplingFunction[K, V](rdd: RDD[(K, V)],
    +      fractions: Map[K, Double],
    +      exact: Boolean,
    +      counts: Option[Map[K, Long]],
    +      seed: Long): (Int, Iterator[(K, V)]) => Iterator[(K, V)] = {
    +    // TODO implement the streaming version of sampling w/ replacement that doesn't require counts
    +    if (exact) {
    +      val finalResult = getCounts(rdd, true, fractions, counts, seed)
    +      val thresholdByKey = computeThresholdByKey(finalResult, fractions)
    +      (idx: Int, iter: Iterator[(K, V)]) => {
    +        val rng = new RandomDataGenerator()
    +        rng.reSeed(seed + idx)
    +        iter.flatMap { item =>
    +          val key = item._1
    +          val acceptBound = finalResult(key).acceptBound
    +          val copiesAccepted = if (acceptBound == 0) 0L else rng.nextPoisson(acceptBound)
    +          val copiesWailisted = rng.nextPoisson(finalResult(key).waitListBound).toInt
    +          val copiesInSample = copiesAccepted +
    +            (0 until copiesWailisted).count(i => rng.nextUniform(0.0, 1.0) < thresholdByKey(key))
    +          if (copiesInSample > 0) {
    +            Iterator.fill(copiesInSample.toInt)(item)
    +          } else {
    +            Iterator.empty
    +          }
    +        }
    +      }
    +    } else {
    +      (idx: Int, iter: Iterator[(K, V)]) => {
    +        val rng = new RandomDataGenerator()
    +        rng.reSeed(seed + idx)
    +        iter.flatMap { item =>
    +          val count = rng.nextPoisson(fractions(item._1)).toInt
    +          if (count > 0) {
    +            Iterator.fill(count)(item)
    +          } else {
    +            Iterator.empty
    +          }
    +        }
    +      }
    +    }
    +  }
    +}
    +
    +/**
    + * Object used by seqOp to keep track of the number of items accepted and items waitlisted per
    + * stratum, as well as the bounds for accepting and waitlisting items.
    + *
    + * `[random]` here is necessary since it's in the return type signature of seqOp defined above
    --- End diff --
    
    I meant the comment is not necessary.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1025#discussion_r14694251
  
    --- Diff: core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala ---
    @@ -195,6 +193,37 @@ class PairRDDFunctions[K, V](self: RDD[(K, V)])
       }
     
       /**
    +   * Return a subset of this RDD sampled by key (via stratified sampling).
    +   *
    +   * If exact set to true, we guarantee, with high probability, a sample size =
    +   * math.ceil(fraction * S_i), where S_i is the size of the ith stratum (collection of entries
    +   * that share the same key). When sampling without replacement, we need one additional pass over
    +   * the RDD to guarantee sample size with a 99.99% confidence; when sampling with replacement, we
    +   * need two additional passes.
    +   *
    +   * @param withReplacement whether to sample with or without replacement
    +   * @param fractions map of specific keys to sampling rates
    +   * @param seed seed for the random number generator
    +   * @param exact whether sample size needs to be exactly math.ceil(fraction * size) per stratum
    +   * @return RDD containing the sampled subset
    +   */
    +  def sampleByKey(withReplacement: Boolean,
    +      fractions: Map[K, Double],
    +      exact: Boolean = true,
    +      seed: Long = Utils.random.nextLong): RDD[(K, V)]= {
    +
    +    require(fractions.forall {case(k, v) => v >= 0.0}, "Invalid sampling rates.")
    +
    +    val samplingFunc = if (withReplacement) {
    +      val counts = if (exact) Some(this.countByKey()) else None
    --- End diff --
    
    `counts` is not used outside `getPoissionSamplingFunction`. Shall we move it into `getPoissonSamplingFunction`?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by falaki <gi...@git.apache.org>.
Github user falaki commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1025#discussion_r13659499
  
    --- Diff: core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala ---
    @@ -156,6 +161,182 @@ class PairRDDFunctions[K, V](self: RDD[(K, V)])
       }
     
       /**
    +   * Return a subset of this RDD sampled by key (via stratified sampling).
    +   * We guarantee a sample size = math.ceil(fraction * S_i), where S_i is the size of the ith
    +   * stratum.
    +   *
    +   * @param withReplacement whether to sample with or without replacement
    +   * @param fraction sampling rate
    +   * @param seed seed for the random number generator
    +   * @return RDD containing the sampled subset
    +   */
    +  def sampleByKey(withReplacement: Boolean,
    +      fraction: Double,
    +      seed: Long = Utils.random.nextLong): RDD[(K, V)]= {
    +
    +    class Stratum(var numItems: Long = 0L, var numAccepted: Long = 0L) extends Serializable {
    +      var waitList: ArrayBuffer[Double] = new ArrayBuffer[Double]
    +      var q1: Option[Double] = None
    +      var q2: Option[Double] = None
    +
    +      def incrNumItems(by: Long = 1L) = numItems += by
    +
    +      def incrNumAccepted(by: Long = 1L) = numAccepted += by
    +
    +      def addToWaitList(elem: Double) = waitList += elem
    +
    +      def addToWaitList(elems: ArrayBuffer[Double]) = waitList ++= elems
    +
    +      override def toString() = {
    +        "numItems: " + numItems + " numAccepted: " + numAccepted + " q1: " + q1 + " q2: " + q2 +
    +          " waitListSize:" + waitList.size
    +      }
    +    }
    +
    +    class Result(var resultMap: Map[K, Stratum], var cachedPartitionId: Option[Int] = None)
    +      extends Serializable {
    +      var rand: RandomDataGenerator = new RandomDataGenerator
    +
    +      def getEntry(key: K, numItems: Long = 0L): Stratum = {
    +        if (resultMap.get(key).isEmpty) {
    +          resultMap += (key -> new Stratum(numItems))
    +        }
    +        resultMap.get(key).get
    --- End diff --
    
    ```resultMap(key)```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1025#discussion_r14907544
  
    --- Diff: core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala ---
    @@ -195,6 +193,37 @@ class PairRDDFunctions[K, V](self: RDD[(K, V)])
       }
     
       /**
    +   * Return a subset of this RDD sampled by key (via stratified sampling).
    +   *
    +   * If exact set to true, we guarantee, with high probability, a sample size =
    --- End diff --
    
    The first sentence should provide a summary and the following paragraph should expand this first sentence if some terms there are not well-defined, e.g., `sample by key` in this case.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1025#issuecomment-48416873
  
    Merged build finished. All automated tests passed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1025#discussion_r14694380
  
    --- Diff: core/src/test/scala/org/apache/spark/rdd/PairRDDFunctionsSuite.scala ---
    @@ -83,6 +83,125 @@ class PairRDDFunctionsSuite extends FunSuite with SharedSparkContext {
         assert(valuesFor2.toList.sorted === List(1))
       }
     
    +  test("sampleByKey") {
    +    def stratifier (fractionPositive: Double) = {
    +      (x: Int) => if (x % 10 < (10 * fractionPositive).toInt) "1" else "0"
    +    }
    +
    +    def checkSize(exact: Boolean,
    +        withReplacement: Boolean,
    +        expected: Long,
    +        actual: Long,
    +        p: Double): Boolean = {
    +      if (exact) {
    +        return expected == actual
    +      }
    +      val stdev = if (withReplacement) math.sqrt(expected) else math.sqrt(expected * p * (1 - p))
    +      // Very forgiving margin since we're dealing with very small sample sizes most of the time
    +      math.abs(actual - expected) <= 6 * stdev
    +    }
    +
    +    // Without replacement validation
    +    def takeSampleAndValidateBernoulli(stratifiedData: RDD[(String, Int)],
    +        exact: Boolean,
    +        samplingRate: Double,
    +        seed: Long,
    +        n: Long) = {
    +      val expectedSampleSize = stratifiedData.countByKey().mapValues(count =>
    +        math.ceil(count * samplingRate).toInt)
    +      val fractions = Map("1" -> samplingRate, "0" -> samplingRate)
    +      val sample = stratifiedData.sampleByKey(false, fractions, exact, seed)
    +      val sampleCounts = sample.countByKey()
    +      val takeSample = sample.collect()
    +      assert(sampleCounts.forall {case(k,v) =>
    --- End diff --
    
    Put spaces after `{` and `,`.
    
    Better put assert inside foreach so we can output more info if something goes wrong.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1025#discussion_r14694253
  
    --- Diff: core/src/main/scala/org/apache/spark/util/random/SamplingUtils.scala ---
    @@ -45,11 +50,78 @@ private[spark] object SamplingUtils {
         val fraction = sampleSizeLowerBound.toDouble / total
         if (withReplacement) {
           val numStDev = if (sampleSizeLowerBound < 12) 9 else 5
    -      fraction + numStDev * math.sqrt(fraction / total)
    +      math.max(1e-10, fraction + numStDev * math.sqrt(fraction / total))
         } else {
           val delta = 1e-4
           val gamma = - math.log(delta) / total
    -      math.min(1, fraction + gamma + math.sqrt(gamma * gamma + 2 * gamma * fraction))
    +      math.min(1,
    +        math.max(1e-10, fraction + gamma + math.sqrt(gamma * gamma + 2 * gamma * fraction)))
    +    }
    +  }
    +}
    +
    +/**
    + * Utility functions that help us determine bounds on adjusted sampling rate to guarantee exact
    + * sample sizes with high confidence when sampling with replacement.
    + *
    + * The algorithm for guaranteeing sample size instantly accepts items whose associated value drawn
    --- End diff --
    
    I think the documentation is wrong. The bounds are not used as thresholds in sampling with replacement. The lower bound q1 guarantees that if sampling according to Pois(q1/n) the sample size is smaller than s, while the upper bound q2 guarantees that if sampling according to Pois(q2/n) the sample size is greater than s.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1025#issuecomment-48419772
  
     Merged build triggered. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by falaki <gi...@git.apache.org>.
Github user falaki commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1025#discussion_r14688613
  
    --- Diff: core/src/test/scala/org/apache/spark/rdd/PairRDDFunctionsSuite.scala ---
    @@ -83,6 +83,120 @@ class PairRDDFunctionsSuite extends FunSuite with SharedSparkContext {
         assert(valuesFor2.toList.sorted === List(1))
       }
     
    +  test("sampleByKey") {
    +    def stratifier (fractionPositive: Double) = {
    +      (x: Int) => if (x % 10 < (10 * fractionPositive).toInt) "1" else "0"
    +    }
    +
    +    def checkSize(exact: Boolean, withReplacement: Boolean,
    +        expected: Long, actual: Long, p: Double): Boolean = {
    +      if (exact) {
    +        return expected == actual
    +      }
    +      val stdev = if (withReplacement) math.sqrt(expected) else math.sqrt(expected * p * (1 - p))
    +      // Very forgiving margin since we're dealing with very small sample sizes most of the time
    +      math.abs(actual - expected) <= 6 * stdev
    +    }
    +
    +    // Without replacement validation
    +    def takeSampleAndValidateBernoulli(stratifiedData: RDD[(String, Int)],
    +        exact: Boolean,
    +        samplingRate: Double,
    +        seed: Long,
    +        n: Long) = {
    +      val expectedSampleSize = stratifiedData.countByKey().mapValues(count =>
    +        math.ceil(count * samplingRate).toInt)
    +      val fractions = Map("1" -> samplingRate, "0" -> samplingRate)
    +      val sample = stratifiedData.sampleByKey(false, fractions, exact, seed)
    +      val sampleCounts = sample.countByKey()
    +      val takeSample = sample.collect()
    +      assert(sampleCounts.forall({case(k,v) =>
    +        checkSize(exact, false, expectedSampleSize(k), v, samplingRate)}))
    +      assert(takeSample.size === takeSample.toSet.size)
    +      assert(takeSample.forall(x => 1 <= x._2 && x._2 <= n), s"elements not in [1, $n]")
    +    }
    +
    +    // With replacement validation
    +    def takeSampleAndValidatePoisson(stratifiedData: RDD[(String, Int)],
    +        exact: Boolean,
    +        samplingRate: Double,
    +        seed: Long,
    +        n: Long) = {
    +      val expectedSampleSize = stratifiedData.countByKey().mapValues(count =>
    +        math.ceil(count * samplingRate).toInt)
    +      val fractions = Map("1" -> samplingRate, "0" -> samplingRate)
    +      val sample = stratifiedData.sampleByKey(true, fractions, exact, seed)
    +      val sampleCounts = sample.countByKey()
    +      val takeSample = sample.collect()
    +      assert(sampleCounts.forall({case(k,v) =>
    --- End diff --
    
    here as well.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by falaki <gi...@git.apache.org>.
Github user falaki commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1025#discussion_r13630611
  
    --- Diff: core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala ---
    @@ -156,6 +161,182 @@ class PairRDDFunctions[K, V](self: RDD[(K, V)])
       }
     
       /**
    +   * Return a subset of this RDD sampled by key (via stratified sampling).
    +   * We guarantee a sample size = math.ceil(fraction * S_i), where S_i is the size of the ith
    +   * stratum.
    +   *
    +   * @param withReplacement whether to sample with or without replacement
    +   * @param fraction sampling rate
    +   * @param seed seed for the random number generator
    +   * @return RDD containing the sampled subset
    +   */
    +  def sampleByKey(withReplacement: Boolean,
    +      fraction: Double,
    +      seed: Long = Utils.random.nextLong): RDD[(K, V)]= {
    +
    +    class Stratum(var numItems: Long = 0L, var numAccepted: Long = 0L) extends Serializable {
    +      var waitList: ArrayBuffer[Double] = new ArrayBuffer[Double]
    +      var q1: Option[Double] = None
    +      var q2: Option[Double] = None
    +
    +      def incrNumItems(by: Long = 1L) = numItems += by
    +
    +      def incrNumAccepted(by: Long = 1L) = numAccepted += by
    +
    +      def addToWaitList(elem: Double) = waitList += elem
    +
    +      def addToWaitList(elems: ArrayBuffer[Double]) = waitList ++= elems
    +
    +      override def toString() = {
    +        "numItems: " + numItems + " numAccepted: " + numAccepted + " q1: " + q1 + " q2: " + q2 +
    +          " waitListSize:" + waitList.size
    +      }
    +    }
    +
    +    class Result(var resultMap: Map[K, Stratum], var cachedPartitionId: Option[Int] = None)
    +      extends Serializable {
    +      var rand: RandomDataGenerator = new RandomDataGenerator
    +
    +      def getEntry(key: K, numItems: Long = 0L): Stratum = {
    +        if (resultMap.get(key).isEmpty) {
    --- End diff --
    
    This would be more readable:
    ```scala
    resultMap.contains(key)
    ```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1025#issuecomment-46376522
  
    Build started. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1025#issuecomment-46376884
  
     Merged build triggered. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1025#issuecomment-45553588
  
    Build started. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1025#discussion_r13566493
  
    --- Diff: core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala ---
    @@ -46,7 +50,8 @@ import org.apache.spark.Partitioner.defaultPartitioner
     import org.apache.spark.SparkContext._
     import org.apache.spark.partial.{BoundedDouble, PartialResult}
     import org.apache.spark.serializer.Serializer
    -import org.apache.spark.util.SerializableHyperLogLog
    +import org.apache.spark.util.{Utils, SerializableHyperLogLog}
    --- End diff --
    
    SerializableHyperLogLog no longer exists


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1025#issuecomment-45571806
  
    All automated tests passed.
    Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15589/


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1025#issuecomment-45968207
  
    Merged build finished. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:

    https://github.com/apache/spark/pull/1025#issuecomment-46402006
  
    @dorx you said above you are trying to use the RNG from Jet? why, if Commons math3 is usable now? I prefer the latter FWIW. I probably misunderstand your comment.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/1025#issuecomment-50388808
  
    QA results for PR 1025:<br>- This patch FAILED unit tests.<br>- This patch merges cleanly<br>- This patch adds no public classes<br><br>For more information see test ouptut:<br>https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17300/consoleFull


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1025#issuecomment-46608300
  
    Merged build finished. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1025#issuecomment-46376511
  
     Build triggered. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by falaki <gi...@git.apache.org>.
Github user falaki commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1025#discussion_r14544785
  
    --- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
    @@ -0,0 +1,296 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.util.random
    +
    +import scala.collection.{Map, mutable}
    +import scala.collection.mutable.ArrayBuffer
    +import scala.reflect.ClassTag
    +
    +import org.apache.commons.math3.random.RandomDataGenerator
    +import org.apache.spark.{Logging, SparkContext, TaskContext}
    +import org.apache.spark.rdd.RDD
    +import org.apache.spark.util.Utils
    +import org.apache.spark.util.random.{PoissonBounds => PB}
    +
    +private[spark] object StratifiedSampler extends Logging {
    +
    +  /**
    +   * A version of {@link #aggregate()} that passes the TaskContext to the function that does
    +   * aggregation for each partition. This function avoids creating an extra depth in the RDD
    +   * lineage, as opposed to using mapPartitionsWithId, which results in slightly improved run time.
    +   */
    +  def aggregateWithContext[U: ClassTag, T: ClassTag](zeroValue: U)
    +      (rdd: RDD[T],
    +       seqOp: ((TaskContext, U), T) => U,
    +       combOp: (U, U) => U): U = {
    +    val sc: SparkContext = rdd.sparkContext
    +    // Clone the zero value since we will also be serializing it as part of tasks
    +    var jobResult = Utils.clone(zeroValue, sc.env.closureSerializer.newInstance())
    +    // pad seqOp and combOp with taskContext to conform to aggregate's signature in TraversableOnce
    +    val paddedSeqOp = (arg1: (TaskContext, U), item: T) => (arg1._1, seqOp(arg1, item))
    +    val paddedcombOp = (arg1: (TaskContext, U), arg2: (TaskContext, U)) =>
    +      (arg1._1, combOp(arg1._2, arg1._2))
    +    val cleanSeqOp = sc.clean(paddedSeqOp)
    +    val cleanCombOp = sc.clean(paddedcombOp)
    +    val aggregatePartition = (tc: TaskContext, it: Iterator[T]) =>
    +      (it.aggregate(tc, zeroValue)(cleanSeqOp, cleanCombOp))._2
    +    val mergeResult = (index: Int, taskResult: U) => jobResult = combOp(jobResult, taskResult)
    +    sc.runJob(rdd, aggregatePartition, mergeResult)
    +    jobResult
    +  }
    +
    +  /**
    +   * Returns the function used by aggregate to collect sampling statistics for each partition.
    +   */
    +  def getSeqOp[K, V](withReplacement: Boolean,
    +      fractionByKey: (K => Double),
    +      counts: Option[Map[K, Long]]): ((TaskContext, Result[K]),(K, V)) => Result[K] = {
    +    val delta = 5e-5
    +    (U: (TaskContext, Result[K]), item: (K, V)) => {
    +      val result = U._2
    +      val tc = U._1
    +      val rng = result.getRand(tc.partitionId)
    +      val fraction = fractionByKey(item._1)
    +      val stratum = result.getEntry(item._1)
    +      if (withReplacement) {
    +        // compute q1 and q2 only if they haven't been computed already
    +        // since they don't change from iteration to iteration.
    +        // TODO change this to the streaming version
    +        if (stratum.q1.isEmpty || stratum.q2.isEmpty) {
    +          val n = counts.get(item._1)
    +          val s = math.ceil(n * fraction).toLong
    +          val lmbd1 = PB.getLowerBound(s)
    +          val minCount = PB.getMinCount(lmbd1)
    +          val lmbd2 = if (lmbd1 == 0) PB.getUpperBound(s) else PB.getUpperBound(s - minCount)
    +          val q1 = lmbd1 / n
    +          val q2 = lmbd2 / n
    +          stratum.q1 = Some(q1)
    +          stratum.q2 = Some(q2)
    +        }
    +        val x1 = if (stratum.q1.get == 0) 0L else rng.nextPoisson(stratum.q1.get)
    +        if (x1 > 0) {
    +          stratum.incrNumAccepted(x1)
    +        }
    +        val x2 = rng.nextPoisson(stratum.q2.get).toInt
    +        if (x2 > 0) {
    +          stratum.addToWaitList(ArrayBuffer.fill(x2)(rng.nextUniform(0.0, 1.0)))
    +        }
    +      } else {
    +        // We use the streaming version of the algorithm for sampling without replacement.
    +        // Hence, q1 and q2 change on every iteration.
    +        val g1 = - math.log(delta) / stratum.numItems
    +        val g2 = (2.0 / 3.0) * g1
    +        val q1 = math.max(0, fraction + g2 - math.sqrt((g2 * g2 + 3 * g2 * fraction)))
    +        val q2 = math.min(1, fraction + g1 + math.sqrt(g1 * g1 + 2 * g1 * fraction))
    +
    +        val x = rng.nextUniform(0.0, 1.0)
    +        if (x < q1) {
    +          stratum.incrNumAccepted()
    +        } else if (x < q2) {
    +          stratum.addToWaitList(x)
    +        }
    +        stratum.q1 = Some(q1)
    +        stratum.q2 = Some(q2)
    +      }
    +      stratum.incrNumItems()
    +      result
    +    }
    +  }
    +
    +  /**
    +   * Returns the function used by aggregate to combine results from different partitions, as
    +   * returned by seqOp.
    +   */
    +  def getCombOp[K](): (Result[K], Result[K]) => Result[K] = {
    +    (r1: Result[K], r2: Result[K]) => {
    +      // take union of both key sets in case one partition doesn't contain all keys
    +      val keyUnion = r1.resultMap.keys.toSet.union(r2.resultMap.keys.toSet)
    +
    +      // Use r2 to keep the combined result since r1 is usual empty
    +      for (key <- keyUnion) {
    +        val entry1 = r1.resultMap.get(key)
    +        val entry2 = r2.resultMap.get(key)
    +        if (entry2.isEmpty && entry1.isDefined) {
    +          r2.resultMap += (key -> entry1.get)
    +        } else if (entry1.isDefined && entry2.isDefined) {
    +          entry2.get.addToWaitList(entry1.get.waitList)
    +          entry2.get.incrNumAccepted(entry1.get.numAccepted)
    +          entry2.get.incrNumItems(entry1.get.numItems)
    +        }
    +      }
    +      r2
    +    }
    +  }
    +
    +  /**
    +   * Given the result returned by the aggregate function, we need to determine the threshold used
    +   * to accept items to generate the exact sample size.
    +   */
    +  def computeThresholdByKey[K](finalResult: Map[K, Stratum], fractionByKey: (K => Double)):
    +    (K => Double) = {
    +    val thresholdByKey = new mutable.HashMap[K, Double]()
    +    for ((key, stratum) <- finalResult) {
    +      val fraction = fractionByKey(key)
    +      val s = math.ceil(stratum.numItems * fraction).toLong
    +      if (stratum.numAccepted > s) {
    +        logWarning("Pre-accepted too many")
    +        thresholdByKey += (key -> stratum.q1.get)
    +      } else {
    +        val numWaitListAccepted = (s - stratum.numAccepted).toInt
    +        if (numWaitListAccepted >= stratum.waitList.size) {
    +          logWarning("WaitList too short")
    +          thresholdByKey += (key -> stratum.q2.get)
    +        } else {
    +          thresholdByKey += (key -> stratum.waitList.sorted.apply(numWaitListAccepted))
    +        }
    +      }
    +    }
    +    thresholdByKey
    +  }
    +
    +  /**
    +   * Return the per partition sampling function used for sampling without replacement.
    +   *
    +   * When exact sample size is required, we make an additional pass over the RDD to determine the
    +   * exact sampling rate that guarantees sample size with high confidence.
    +   *
    +   * The sampling function has a unique seed per partition.
    +   */
    +  def getBernoulliSamplingFunction[K, V](rdd: RDD[(K,  V)],
    +      fractionByKey: K => Double,
    +      exact: Boolean,
    +      seed: Long): (Int, Iterator[(K, V)]) => Iterator[(K, V)] = {
    +    var samplingRateByKey = fractionByKey
    +    if (exact) {
    +      // determine threshold for each stratum and resample
    +      val seqOp = StratifiedSampler.getSeqOp[K,V](false, fractionByKey, None)
    --- End diff --
    
    In these lines we can remove ```StratifiedSampler```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by falaki <gi...@git.apache.org>.
Github user falaki commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1025#discussion_r13630716
  
    --- Diff: core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala ---
    @@ -156,6 +161,182 @@ class PairRDDFunctions[K, V](self: RDD[(K, V)])
       }
     
       /**
    +   * Return a subset of this RDD sampled by key (via stratified sampling).
    +   * We guarantee a sample size = math.ceil(fraction * S_i), where S_i is the size of the ith
    +   * stratum.
    +   *
    +   * @param withReplacement whether to sample with or without replacement
    +   * @param fraction sampling rate
    +   * @param seed seed for the random number generator
    +   * @return RDD containing the sampled subset
    +   */
    +  def sampleByKey(withReplacement: Boolean,
    +      fraction: Double,
    +      seed: Long = Utils.random.nextLong): RDD[(K, V)]= {
    +
    +    class Stratum(var numItems: Long = 0L, var numAccepted: Long = 0L) extends Serializable {
    +      var waitList: ArrayBuffer[Double] = new ArrayBuffer[Double]
    +      var q1: Option[Double] = None
    +      var q2: Option[Double] = None
    +
    +      def incrNumItems(by: Long = 1L) = numItems += by
    +
    +      def incrNumAccepted(by: Long = 1L) = numAccepted += by
    +
    +      def addToWaitList(elem: Double) = waitList += elem
    +
    +      def addToWaitList(elems: ArrayBuffer[Double]) = waitList ++= elems
    +
    +      override def toString() = {
    +        "numItems: " + numItems + " numAccepted: " + numAccepted + " q1: " + q1 + " q2: " + q2 +
    +          " waitListSize:" + waitList.size
    +      }
    +    }
    +
    +    class Result(var resultMap: Map[K, Stratum], var cachedPartitionId: Option[Int] = None)
    +      extends Serializable {
    +      var rand: RandomDataGenerator = new RandomDataGenerator
    +
    +      def getEntry(key: K, numItems: Long = 0L): Stratum = {
    +        if (resultMap.get(key).isEmpty) {
    +          resultMap += (key -> new Stratum(numItems))
    +        }
    +        resultMap.get(key).get
    +      }
    +
    +      def getRand(partitionId: Int): RandomDataGenerator = {
    +        if (cachedPartitionId.isEmpty || cachedPartitionId.get != partitionId) {
    +          cachedPartitionId = Some(partitionId)
    +          rand.reSeed(seed + partitionId)
    +        }
    +        rand
    +      }
    +    }
    +
    +    // TODO implement the streaming version of sampling w/ replacement that doesn't require counts
    +    // in order to save one pass over the RDD
    +    val counts = if (withReplacement) Some(this.countByKey()) else None
    +
    +    val seqOp = (U: (TaskContext, Result), item: (K, V)) => {
    +      val delta = 5e-5
    +      val result = U._2
    +      val tc = U._1
    +      val rng = result.getRand(tc.partitionId)
    +      val stratum = result.getEntry(item._1)
    +      if (withReplacement) {
    +        // compute q1 and q2 only if they haven't been computed already
    +        // since they don't change from iteration to iteration.
    +        // TODO change this to the streaming version
    +        if (stratum.q1.isEmpty || stratum.q2.isEmpty) {
    +          val n = counts.get(item._1)
    +          val s = math.ceil(n * fraction).toLong
    +          val lmbd1 = PB.getLambda1(s)
    +          val minCount = PB.getMinCount(lmbd1)
    +          val lmbd2 = if (lmbd1 == 0) PB.getLambda2(s) else PB.getLambda2(s - minCount)
    +          val q1 = lmbd1 / n
    +          val q2 = lmbd2 / n
    +          stratum.q1 = Some(q1)
    +          stratum.q2 = Some(q2)
    +        }
    +        val x1 = if (stratum.q1.get == 0) 0L else rng.nextPoisson(stratum.q1.get)
    +        if (x1 > 0) {
    +          stratum.incrNumAccepted(x1)
    +        }
    +        val x2 = rng.nextPoisson(stratum.q2.get).toInt
    +        if (x2 > 0) {
    +          stratum.addToWaitList(ArrayBuffer.fill(x2)(rng.nextUniform(0.0, 1.0)))
    +        }
    +      } else {
    +        val g1 = - math.log(delta) / stratum.numItems
    +        val g2 = (2.0 / 3.0) * g1
    +        val q1 = math.max(0, fraction + g2 - math.sqrt((g2 * g2 + 3 * g2 * fraction)))
    +        val q2 = math.min(1, fraction + g1 + math.sqrt(g1 * g1 + 2 * g1 * fraction))
    +
    +        val x = rng.nextUniform(0.0, 1.0)
    +        if (x < q1) {
    +          stratum.incrNumAccepted()
    +        } else if ( x < q2) {
    +          stratum.addToWaitList(x)
    +        }
    +        stratum.q1 = Some(q1)
    +        stratum.q2 = Some(q2)
    +      }
    +      stratum.incrNumItems()
    +      result
    +    }
    +
    +    val combOp = (r1: Result, r2: Result) => {
    +      // take union of both key sets in case one partion doesn't contain all keys
    --- End diff --
    
    partition


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1025#discussion_r13578424
  
    --- Diff: core/src/main/scala/org/apache/spark/util/random/SamplingUtils.scala ---
    @@ -0,0 +1,56 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.util.random
    +
    +import org.apache.commons.math3.distribution.{PoissonDistribution, NormalDistribution}
    +
    +private[spark] object PoissonBounds {
    +
    +  val delta = 1e-4 / 3.0
    +  val phi = new NormalDistribution().cumulativeProbability(1.0 - delta)
    +
    +  def getLambda1(s: Double): Double = {
    --- End diff --
    
    `getLowerBound`?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/1025#issuecomment-50065418
  
    QA tests have started for PR 1025. This patch merges cleanly. <br>View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17130/consoleFull


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1025#discussion_r13566500
  
    --- Diff: core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala ---
    @@ -27,8 +27,12 @@ import scala.collection.Map
     import scala.collection.mutable
     import scala.collection.mutable.ArrayBuffer
     import scala.reflect.ClassTag
    +import scala.util.control.Breaks._
    --- End diff --
    
    remove this


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1025#issuecomment-48419505
  
    Merged build finished. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1025#discussion_r14694283
  
    --- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
    @@ -0,0 +1,311 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.util.random
    +
    +import scala.collection.Map
    +import scala.collection.mutable.{ArrayBuffer, HashMap, Map => MMap}
    +
    +import org.apache.commons.math3.random.RandomDataGenerator
    +import org.apache.spark.Logging
    +import org.apache.spark.rdd.RDD
    +
    +/**
    + * Auxiliary functions and data structures for the sampleByKey method in PairRDDFunctions.
    + *
    + * Essentially, when exact sample size is necessary, we make additional passes over the RDD to
    + * compute the exact threshold value to use for each stratum to guarantee exact sample size with
    + * high probability. This is achieved by maintaining a waitlist of size O(log(s)), where s is the
    + * desired sample size for each stratum.
    + *
    + * Like in simple random sampling, we generate a random value for each item from the
    + * uniform  distribution [0.0, 1.0]. All items with values <= min(values of items in the waitlist)
    + * are accepted into the sample instantly. The threshold for instant accept is designed so that
    + * s - numAccepted = O(log(s)), where s is again the desired sample size. Thus, by maintaining a
    + * waitlist size = O(log(s)), we will be able to create a sample of the exact size s by adding
    + * a portion of the waitlist to the set of items that are instantly accepted. The exact threshold
    + * is computed by sorting the values in the waitlist and picking the value at (s - numAccepted).
    + *
    + * Note that since we use the same seed for the RNG when computing the thresholds and the actual
    + * sample, our computed thresholds are guaranteed to produce the desired sample size.
    + *
    + * For more theoretical background on the sampling techniques used here, please refer to
    + * http://jmlr.org/proceedings/papers/v28/meng13a.html
    + */
    +
    +private[spark] object StratifiedSampler extends Logging {
    +
    +  /**
    +   * Count the number of items instantly accepted and generate the waitlist for each stratum.
    +   *
    +   * This is only invoked when exact sample size is required.
    +   */
    +  def getCounts[K, V](rdd: RDD[(K, V)],
    +      withReplacement: Boolean,
    +      fractions: Map[K, Double],
    +      counts: Option[Map[K, Long]],
    +      seed: Long): MMap[K, Stratum] = {
    +    val combOp = getCombOp[K]
    +    val mappedPartitionRDD = rdd.mapPartitionsWithIndex({ case (partition, iter) =>
    +      val zeroU: MMap[K, Stratum] = new HashMap[K, Stratum]()
    +      val rng = new RandomDataGenerator()
    +      rng.reSeed(seed + partition)
    +      val seqOp = getSeqOp(withReplacement, fractions, rng, counts)
    +      Iterator(iter.aggregate(zeroU)(seqOp, combOp))
    +    }, preservesPartitioning=true)
    +    mappedPartitionRDD.reduce(combOp)
    +  }
    +
    +  /**
    +   * Returns the function used by aggregate to collect sampling statistics for each partition.
    +   */
    +  def getSeqOp[K, V](withReplacement: Boolean,
    +      fractions: Map[K, Double],
    +      rng: RandomDataGenerator,
    +      counts: Option[Map[K, Long]]): (MMap[K, Stratum], (K, V)) => MMap[K, Stratum] = {
    +    val delta = 5e-5
    +    (result: MMap[K, Stratum], item: (K, V)) => {
    +      val key = item._1
    +      val fraction = fractions(key)
    +      if (!result.contains(key)) {
    +        result += (key -> new Stratum())
    +      }
    +      val stratum = result(key)
    +
    +      if (withReplacement) {
    +        // compute acceptBound and waitListBound only if they haven't been computed already
    +        // since they don't change from iteration to iteration.
    +        // TODO change this to the streaming version
    +        if (stratum.areBoundsEmpty) {
    +          val n = counts.get(key)
    +          val sampleSize = math.ceil(n * fraction).toLong
    +          val lmbd1 = PoissonBounds.getLowerBound(sampleSize)
    +          val minCount = PoissonBounds.getMinCount(lmbd1)
    +          val lmbd2 = if (lmbd1 == 0) {
    +            PoissonBounds.getUpperBound(sampleSize)
    +          } else {
    +            PoissonBounds.getUpperBound(sampleSize - minCount)
    +          }
    +          stratum.acceptBound = lmbd1 / n
    +          stratum.waitListBound = lmbd2 / n
    +        }
    +        val acceptBound = stratum.acceptBound
    +        val copiesAccepted = if (acceptBound == 0.0) 0L else rng.nextPoisson(acceptBound)
    +        if (copiesAccepted > 0) {
    +          stratum.incrNumAccepted(copiesAccepted)
    +        }
    +        val copiesWaitlisted = rng.nextPoisson(stratum.waitListBound).toInt
    +        if (copiesWaitlisted > 0) {
    +          stratum.addToWaitList(ArrayBuffer.fill(copiesWaitlisted)(rng.nextUniform(0.0, 1.0)))
    +        }
    +      } else {
    +        // We use the streaming version of the algorithm for sampling without replacement to avoid
    +        // using an extra pass over the RDD for computing the count.
    +        // Hence, acceptBound and waitListBound change on every iteration.
    +        val gamma1 = - math.log(delta) / stratum.numItems
    +        val gamma2 = (2.0 / 3.0) * gamma1
    +        stratum.acceptBound = math.max(0,
    +          fraction + gamma2 - math.sqrt(gamma2 * gamma2 + 3 * gamma2 * fraction))
    +        stratum.waitListBound = math.min(1,
    +          fraction + gamma1 + math.sqrt(gamma1 * gamma1 + 2 * gamma1 * fraction))
    +
    +        val x = rng.nextUniform(0.0, 1.0)
    +        if (x < stratum.acceptBound) {
    +          stratum.incrNumAccepted()
    +        } else if (x < stratum.waitListBound) {
    +          stratum.addToWaitList(x)
    +        }
    +      }
    +      stratum.incrNumItems()
    +      result
    +    }
    +  }
    +
    +  /**
    +   * Returns the function used combine results returned by seqOp from different partitions.
    +   */
    +  def getCombOp[K]: (MMap[K, Stratum], MMap[K, Stratum]) => MMap[K, Stratum] = {
    +    (result1: MMap[K, Stratum], result2: MMap[K, Stratum]) => {
    +      // take union of both key sets in case one partition doesn't contain all keys
    +      for (key <- result1.keySet.union(result2.keySet)) {
    --- End diff --
    
    foreach is more efficient than for:
    
    `result1.keySet.union(result2.keySet).foreach { key =>`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1025#discussion_r14694258
  
    --- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
    @@ -0,0 +1,311 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.util.random
    +
    +import scala.collection.Map
    +import scala.collection.mutable.{ArrayBuffer, HashMap, Map => MMap}
    --- End diff --
    
    Do not import a class under `mutable` if there is one with the same name under `immutable`. Instead, import `scala.collection.mutable` and reference `Map` and `HashMap` as `mutable.Map` and `mutable.HashMap` in the code. Importing `mutable.ArrayBuffer` is fine.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/1025#issuecomment-48985401
  
    QA tests have started for PR 1025. This patch merges cleanly. <br>View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16650/consoleFull


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by dorx <gi...@git.apache.org>.
Github user dorx commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1025#discussion_r14577949
  
    --- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
    @@ -0,0 +1,296 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.util.random
    +
    +import scala.collection.{Map, mutable}
    +import scala.collection.mutable.ArrayBuffer
    +import scala.reflect.ClassTag
    +
    +import org.apache.commons.math3.random.RandomDataGenerator
    +import org.apache.spark.{Logging, SparkContext, TaskContext}
    +import org.apache.spark.rdd.RDD
    +import org.apache.spark.util.Utils
    +import org.apache.spark.util.random.{PoissonBounds => PB}
    +
    +private[spark] object StratifiedSampler extends Logging {
    +
    +  /**
    +   * A version of {@link #aggregate()} that passes the TaskContext to the function that does
    +   * aggregation for each partition. This function avoids creating an extra depth in the RDD
    +   * lineage, as opposed to using mapPartitionsWithId, which results in slightly improved run time.
    +   */
    +  def aggregateWithContext[U: ClassTag, T: ClassTag](zeroValue: U)
    +      (rdd: RDD[T],
    +       seqOp: ((TaskContext, U), T) => U,
    +       combOp: (U, U) => U): U = {
    +    val sc: SparkContext = rdd.sparkContext
    +    // Clone the zero value since we will also be serializing it as part of tasks
    +    var jobResult = Utils.clone(zeroValue, sc.env.closureSerializer.newInstance())
    +    // pad seqOp and combOp with taskContext to conform to aggregate's signature in TraversableOnce
    +    val paddedSeqOp = (arg1: (TaskContext, U), item: T) => (arg1._1, seqOp(arg1, item))
    +    val paddedcombOp = (arg1: (TaskContext, U), arg2: (TaskContext, U)) =>
    +      (arg1._1, combOp(arg1._2, arg1._2))
    +    val cleanSeqOp = sc.clean(paddedSeqOp)
    +    val cleanCombOp = sc.clean(paddedcombOp)
    +    val aggregatePartition = (tc: TaskContext, it: Iterator[T]) =>
    +      (it.aggregate(tc, zeroValue)(cleanSeqOp, cleanCombOp))._2
    +    val mergeResult = (index: Int, taskResult: U) => jobResult = combOp(jobResult, taskResult)
    +    sc.runJob(rdd, aggregatePartition, mergeResult)
    +    jobResult
    +  }
    +
    +  /**
    +   * Returns the function used by aggregate to collect sampling statistics for each partition.
    +   */
    +  def getSeqOp[K, V](withReplacement: Boolean,
    +      fractionByKey: (K => Double),
    +      counts: Option[Map[K, Long]]): ((TaskContext, Result[K]),(K, V)) => Result[K] = {
    +    val delta = 5e-5
    +    (U: (TaskContext, Result[K]), item: (K, V)) => {
    +      val result = U._2
    +      val tc = U._1
    +      val rng = result.getRand(tc.partitionId)
    +      val fraction = fractionByKey(item._1)
    +      val stratum = result.getEntry(item._1)
    +      if (withReplacement) {
    +        // compute q1 and q2 only if they haven't been computed already
    +        // since they don't change from iteration to iteration.
    +        // TODO change this to the streaming version
    +        if (stratum.q1.isEmpty || stratum.q2.isEmpty) {
    +          val n = counts.get(item._1)
    +          val s = math.ceil(n * fraction).toLong
    +          val lmbd1 = PB.getLowerBound(s)
    +          val minCount = PB.getMinCount(lmbd1)
    +          val lmbd2 = if (lmbd1 == 0) PB.getUpperBound(s) else PB.getUpperBound(s - minCount)
    +          val q1 = lmbd1 / n
    +          val q2 = lmbd2 / n
    +          stratum.q1 = Some(q1)
    +          stratum.q2 = Some(q2)
    +        }
    +        val x1 = if (stratum.q1.get == 0) 0L else rng.nextPoisson(stratum.q1.get)
    +        if (x1 > 0) {
    +          stratum.incrNumAccepted(x1)
    +        }
    +        val x2 = rng.nextPoisson(stratum.q2.get).toInt
    +        if (x2 > 0) {
    +          stratum.addToWaitList(ArrayBuffer.fill(x2)(rng.nextUniform(0.0, 1.0)))
    +        }
    +      } else {
    +        // We use the streaming version of the algorithm for sampling without replacement.
    +        // Hence, q1 and q2 change on every iteration.
    --- End diff --
    
    keeping g1 and g2 (since they're just constants) but changed q1 to q2 to more descriptive names. The expressions for q1  and q2 contains multiple references to g1 and g2. Longer names will make those two expressions really long.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by dorx <gi...@git.apache.org>.
Github user dorx commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1025#discussion_r14907870
  
    --- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
    @@ -0,0 +1,311 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.util.random
    +
    +import scala.collection.Map
    +import scala.collection.mutable.{ArrayBuffer, HashMap, Map => MMap}
    +
    +import org.apache.commons.math3.random.RandomDataGenerator
    +import org.apache.spark.Logging
    +import org.apache.spark.rdd.RDD
    +
    +/**
    + * Auxiliary functions and data structures for the sampleByKey method in PairRDDFunctions.
    + *
    + * Essentially, when exact sample size is necessary, we make additional passes over the RDD to
    + * compute the exact threshold value to use for each stratum to guarantee exact sample size with
    + * high probability. This is achieved by maintaining a waitlist of size O(log(s)), where s is the
    + * desired sample size for each stratum.
    + *
    + * Like in simple random sampling, we generate a random value for each item from the
    + * uniform  distribution [0.0, 1.0]. All items with values <= min(values of items in the waitlist)
    + * are accepted into the sample instantly. The threshold for instant accept is designed so that
    + * s - numAccepted = O(log(s)), where s is again the desired sample size. Thus, by maintaining a
    + * waitlist size = O(log(s)), we will be able to create a sample of the exact size s by adding
    + * a portion of the waitlist to the set of items that are instantly accepted. The exact threshold
    + * is computed by sorting the values in the waitlist and picking the value at (s - numAccepted).
    + *
    + * Note that since we use the same seed for the RNG when computing the thresholds and the actual
    + * sample, our computed thresholds are guaranteed to produce the desired sample size.
    + *
    + * For more theoretical background on the sampling techniques used here, please refer to
    + * http://jmlr.org/proceedings/papers/v28/meng13a.html
    + */
    +
    +private[spark] object StratifiedSampler extends Logging {
    +
    +  /**
    +   * Count the number of items instantly accepted and generate the waitlist for each stratum.
    +   *
    +   * This is only invoked when exact sample size is required.
    +   */
    +  def getCounts[K, V](rdd: RDD[(K, V)],
    +      withReplacement: Boolean,
    +      fractions: Map[K, Double],
    +      counts: Option[Map[K, Long]],
    +      seed: Long): MMap[K, Stratum] = {
    +    val combOp = getCombOp[K]
    +    val mappedPartitionRDD = rdd.mapPartitionsWithIndex({ case (partition, iter) =>
    +      val zeroU: MMap[K, Stratum] = new HashMap[K, Stratum]()
    +      val rng = new RandomDataGenerator()
    +      rng.reSeed(seed + partition)
    +      val seqOp = getSeqOp(withReplacement, fractions, rng, counts)
    +      Iterator(iter.aggregate(zeroU)(seqOp, combOp))
    +    }, preservesPartitioning=true)
    +    mappedPartitionRDD.reduce(combOp)
    +  }
    +
    +  /**
    +   * Returns the function used by aggregate to collect sampling statistics for each partition.
    +   */
    +  def getSeqOp[K, V](withReplacement: Boolean,
    +      fractions: Map[K, Double],
    +      rng: RandomDataGenerator,
    +      counts: Option[Map[K, Long]]): (MMap[K, Stratum], (K, V)) => MMap[K, Stratum] = {
    +    val delta = 5e-5
    +    (result: MMap[K, Stratum], item: (K, V)) => {
    +      val key = item._1
    +      val fraction = fractions(key)
    +      if (!result.contains(key)) {
    +        result += (key -> new Stratum())
    +      }
    +      val stratum = result(key)
    +
    +      if (withReplacement) {
    +        // compute acceptBound and waitListBound only if they haven't been computed already
    +        // since they don't change from iteration to iteration.
    +        // TODO change this to the streaming version
    +        if (stratum.areBoundsEmpty) {
    +          val n = counts.get(key)
    +          val sampleSize = math.ceil(n * fraction).toLong
    +          val lmbd1 = PoissonBounds.getLowerBound(sampleSize)
    +          val minCount = PoissonBounds.getMinCount(lmbd1)
    +          val lmbd2 = if (lmbd1 == 0) {
    +            PoissonBounds.getUpperBound(sampleSize)
    +          } else {
    +            PoissonBounds.getUpperBound(sampleSize - minCount)
    +          }
    +          stratum.acceptBound = lmbd1 / n
    +          stratum.waitListBound = lmbd2 / n
    +        }
    +        val acceptBound = stratum.acceptBound
    +        val copiesAccepted = if (acceptBound == 0.0) 0L else rng.nextPoisson(acceptBound)
    +        if (copiesAccepted > 0) {
    +          stratum.incrNumAccepted(copiesAccepted)
    +        }
    +        val copiesWaitlisted = rng.nextPoisson(stratum.waitListBound).toInt
    +        if (copiesWaitlisted > 0) {
    +          stratum.addToWaitList(ArrayBuffer.fill(copiesWaitlisted)(rng.nextUniform(0.0, 1.0)))
    +        }
    +      } else {
    +        // We use the streaming version of the algorithm for sampling without replacement to avoid
    +        // using an extra pass over the RDD for computing the count.
    +        // Hence, acceptBound and waitListBound change on every iteration.
    +        val gamma1 = - math.log(delta) / stratum.numItems
    +        val gamma2 = (2.0 / 3.0) * gamma1
    +        stratum.acceptBound = math.max(0,
    +          fraction + gamma2 - math.sqrt(gamma2 * gamma2 + 3 * gamma2 * fraction))
    +        stratum.waitListBound = math.min(1,
    +          fraction + gamma1 + math.sqrt(gamma1 * gamma1 + 2 * gamma1 * fraction))
    +
    +        val x = rng.nextUniform(0.0, 1.0)
    +        if (x < stratum.acceptBound) {
    +          stratum.incrNumAccepted()
    +        } else if (x < stratum.waitListBound) {
    +          stratum.addToWaitList(x)
    +        }
    +      }
    +      stratum.incrNumItems()
    +      result
    +    }
    +  }
    +
    +  /**
    +   * Returns the function used combine results returned by seqOp from different partitions.
    +   */
    +  def getCombOp[K]: (MMap[K, Stratum], MMap[K, Stratum]) => MMap[K, Stratum] = {
    +    (result1: MMap[K, Stratum], result2: MMap[K, Stratum]) => {
    +      // take union of both key sets in case one partition doesn't contain all keys
    +      for (key <- result1.keySet.union(result2.keySet)) {
    +        // Use result2 to keep the combined result since r1 is usual empty
    +        val entry1 = result1.get(key)
    +        if (result2.contains(key)) {
    +          result2(key).merge(entry1)
    +        } else {
    +          if (entry1.isDefined) {
    +            result2 += (key -> entry1.get)
    +          }
    +        }
    +      }
    +      result2
    +    }
    +  }
    +
    +  /**
    +   * Given the result returned by getCounts, determine the threshold for accepting items to
    +   * generate exact sample size.
    +   *
    +   * To do so, we compute sampleSize = math.ceil(size * samplingRate) for each stratum and compare
    +   * it to the number of items that were accepted instantly and the number of items in the waitlist
    +   * for that stratum. Most of the time, numAccepted <= sampleSize <= (numAccepted + numWaitlisted),
    +   * which means we need to sort the elements in the waitlist by their associated values in order
    +   * to find the value T s.t. |{elements in the stratum whose associated values <= T}| = sampleSize.
    +   * Note that all elements in the waitlist have values >= bound for instant accept, so a T value
    +   * in the waitlist range would allow all elements that were instantly accepted on the first pass
    +   * to be included in the sample.
    +   */
    +  def computeThresholdByKey[K](finalResult: Map[K, Stratum],
    +      fractions: Map[K, Double]): Map[K, Double] = {
    +    val thresholdByKey = new HashMap[K, Double]()
    +    for ((key, stratum) <- finalResult) {
    +      val sampleSize = math.ceil(stratum.numItems * fractions(key)).toLong
    +      if (stratum.numAccepted > sampleSize) {
    +        logWarning("Pre-accepted too many")
    +        thresholdByKey += (key -> stratum.acceptBound)
    +      } else {
    +        val numWaitListAccepted = (sampleSize - stratum.numAccepted).toInt
    +        if (numWaitListAccepted >= stratum.waitList.size) {
    +          logWarning("WaitList too short")
    +          thresholdByKey += (key -> stratum.waitListBound)
    +        } else {
    +          thresholdByKey += (key -> stratum.waitList.sorted.apply(numWaitListAccepted))
    +        }
    +      }
    +    }
    +    thresholdByKey
    +  }
    +
    +  /**
    +   * Return the per partition sampling function used for sampling without replacement.
    +   *
    +   * When exact sample size is required, we make an additional pass over the RDD to determine the
    +   * exact sampling rate that guarantees sample size with high confidence.
    +   *
    +   * The sampling function has a unique seed per partition.
    +   */
    +  def getBernoulliSamplingFunction[K, V](rdd: RDD[(K,  V)],
    +      fractions: Map[K, Double],
    +      exact: Boolean,
    +      seed: Long): (Int, Iterator[(K, V)]) => Iterator[(K, V)] = {
    +    var samplingRateByKey = fractions
    +    if (exact) {
    +      // determine threshold for each stratum and resample
    +      val finalResult = getCounts(rdd, false, fractions, None, seed)
    +      samplingRateByKey = computeThresholdByKey(finalResult, fractions)
    +    }
    +    (idx: Int, iter: Iterator[(K, V)]) => {
    +      val rng = new RandomDataGenerator
    +      rng.reSeed(seed + idx)
    +      iter.filter(t => rng.nextUniform(0.0, 1.0) < samplingRateByKey(t._1))
    +    }
    +  }
    +
    +  /**
    +   * Return the per partition sampling function used for sampling with replacement.
    +   *
    +   * When exact sample size is required, we make two additional passed over the RDD to determine
    +   * the exact sampling rate that guarantees sample size with high confidence. The first pass
    +   * counts the number of items in each stratum (group of items with the same key) in the RDD, and
    +   * the second pass uses the counts to determine exact sampling rates.
    +   *
    +   * The sampling function has a unique seed per partition.
    +   */
    +  def getPoissonSamplingFunction[K, V](rdd: RDD[(K, V)],
    +      fractions: Map[K, Double],
    +      exact: Boolean,
    +      counts: Option[Map[K, Long]],
    +      seed: Long): (Int, Iterator[(K, V)]) => Iterator[(K, V)] = {
    +    // TODO implement the streaming version of sampling w/ replacement that doesn't require counts
    +    if (exact) {
    +      val finalResult = getCounts(rdd, true, fractions, counts, seed)
    +      val thresholdByKey = computeThresholdByKey(finalResult, fractions)
    +      (idx: Int, iter: Iterator[(K, V)]) => {
    +        val rng = new RandomDataGenerator()
    +        rng.reSeed(seed + idx)
    +        iter.flatMap { item =>
    +          val key = item._1
    +          val acceptBound = finalResult(key).acceptBound
    +          val copiesAccepted = if (acceptBound == 0) 0L else rng.nextPoisson(acceptBound)
    +          val copiesWailisted = rng.nextPoisson(finalResult(key).waitListBound).toInt
    +          val copiesInSample = copiesAccepted +
    +            (0 until copiesWailisted).count(i => rng.nextUniform(0.0, 1.0) < thresholdByKey(key))
    +          if (copiesInSample > 0) {
    +            Iterator.fill(copiesInSample.toInt)(item)
    +          } else {
    +            Iterator.empty
    +          }
    +        }
    +      }
    +    } else {
    +      (idx: Int, iter: Iterator[(K, V)]) => {
    +        val rng = new RandomDataGenerator()
    +        rng.reSeed(seed + idx)
    +        iter.flatMap { item =>
    +          val count = rng.nextPoisson(fractions(item._1)).toInt
    +          if (count > 0) {
    +            Iterator.fill(count)(item)
    +          } else {
    +            Iterator.empty
    +          }
    +        }
    +      }
    +    }
    +  }
    +}
    +
    +/**
    + * Object used by seqOp to keep track of the number of items accepted and items waitlisted per
    + * stratum, as well as the bounds for accepting and waitlisting items.
    + *
    + * `[random]` here is necessary since it's in the return type signature of seqOp defined above
    --- End diff --
    
    I don't know. I can see people down the road wanting to remove it thinking it's superfluous and break compile by accident.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1025#discussion_r14694296
  
    --- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
    @@ -0,0 +1,311 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.util.random
    +
    +import scala.collection.Map
    +import scala.collection.mutable.{ArrayBuffer, HashMap, Map => MMap}
    +
    +import org.apache.commons.math3.random.RandomDataGenerator
    +import org.apache.spark.Logging
    +import org.apache.spark.rdd.RDD
    +
    +/**
    + * Auxiliary functions and data structures for the sampleByKey method in PairRDDFunctions.
    + *
    + * Essentially, when exact sample size is necessary, we make additional passes over the RDD to
    + * compute the exact threshold value to use for each stratum to guarantee exact sample size with
    + * high probability. This is achieved by maintaining a waitlist of size O(log(s)), where s is the
    + * desired sample size for each stratum.
    + *
    + * Like in simple random sampling, we generate a random value for each item from the
    + * uniform  distribution [0.0, 1.0]. All items with values <= min(values of items in the waitlist)
    + * are accepted into the sample instantly. The threshold for instant accept is designed so that
    + * s - numAccepted = O(log(s)), where s is again the desired sample size. Thus, by maintaining a
    + * waitlist size = O(log(s)), we will be able to create a sample of the exact size s by adding
    + * a portion of the waitlist to the set of items that are instantly accepted. The exact threshold
    + * is computed by sorting the values in the waitlist and picking the value at (s - numAccepted).
    + *
    + * Note that since we use the same seed for the RNG when computing the thresholds and the actual
    + * sample, our computed thresholds are guaranteed to produce the desired sample size.
    + *
    + * For more theoretical background on the sampling techniques used here, please refer to
    + * http://jmlr.org/proceedings/papers/v28/meng13a.html
    + */
    +
    +private[spark] object StratifiedSampler extends Logging {
    +
    +  /**
    +   * Count the number of items instantly accepted and generate the waitlist for each stratum.
    +   *
    +   * This is only invoked when exact sample size is required.
    +   */
    +  def getCounts[K, V](rdd: RDD[(K, V)],
    +      withReplacement: Boolean,
    +      fractions: Map[K, Double],
    +      counts: Option[Map[K, Long]],
    +      seed: Long): MMap[K, Stratum] = {
    +    val combOp = getCombOp[K]
    +    val mappedPartitionRDD = rdd.mapPartitionsWithIndex({ case (partition, iter) =>
    +      val zeroU: MMap[K, Stratum] = new HashMap[K, Stratum]()
    +      val rng = new RandomDataGenerator()
    +      rng.reSeed(seed + partition)
    +      val seqOp = getSeqOp(withReplacement, fractions, rng, counts)
    +      Iterator(iter.aggregate(zeroU)(seqOp, combOp))
    +    }, preservesPartitioning=true)
    +    mappedPartitionRDD.reduce(combOp)
    +  }
    +
    +  /**
    +   * Returns the function used by aggregate to collect sampling statistics for each partition.
    +   */
    +  def getSeqOp[K, V](withReplacement: Boolean,
    +      fractions: Map[K, Double],
    +      rng: RandomDataGenerator,
    +      counts: Option[Map[K, Long]]): (MMap[K, Stratum], (K, V)) => MMap[K, Stratum] = {
    +    val delta = 5e-5
    +    (result: MMap[K, Stratum], item: (K, V)) => {
    +      val key = item._1
    +      val fraction = fractions(key)
    +      if (!result.contains(key)) {
    +        result += (key -> new Stratum())
    +      }
    +      val stratum = result(key)
    +
    +      if (withReplacement) {
    +        // compute acceptBound and waitListBound only if they haven't been computed already
    +        // since they don't change from iteration to iteration.
    +        // TODO change this to the streaming version
    +        if (stratum.areBoundsEmpty) {
    +          val n = counts.get(key)
    +          val sampleSize = math.ceil(n * fraction).toLong
    +          val lmbd1 = PoissonBounds.getLowerBound(sampleSize)
    +          val minCount = PoissonBounds.getMinCount(lmbd1)
    +          val lmbd2 = if (lmbd1 == 0) {
    +            PoissonBounds.getUpperBound(sampleSize)
    +          } else {
    +            PoissonBounds.getUpperBound(sampleSize - minCount)
    +          }
    +          stratum.acceptBound = lmbd1 / n
    +          stratum.waitListBound = lmbd2 / n
    +        }
    +        val acceptBound = stratum.acceptBound
    +        val copiesAccepted = if (acceptBound == 0.0) 0L else rng.nextPoisson(acceptBound)
    +        if (copiesAccepted > 0) {
    +          stratum.incrNumAccepted(copiesAccepted)
    +        }
    +        val copiesWaitlisted = rng.nextPoisson(stratum.waitListBound).toInt
    +        if (copiesWaitlisted > 0) {
    +          stratum.addToWaitList(ArrayBuffer.fill(copiesWaitlisted)(rng.nextUniform(0.0, 1.0)))
    +        }
    +      } else {
    +        // We use the streaming version of the algorithm for sampling without replacement to avoid
    +        // using an extra pass over the RDD for computing the count.
    +        // Hence, acceptBound and waitListBound change on every iteration.
    +        val gamma1 = - math.log(delta) / stratum.numItems
    +        val gamma2 = (2.0 / 3.0) * gamma1
    +        stratum.acceptBound = math.max(0,
    +          fraction + gamma2 - math.sqrt(gamma2 * gamma2 + 3 * gamma2 * fraction))
    +        stratum.waitListBound = math.min(1,
    +          fraction + gamma1 + math.sqrt(gamma1 * gamma1 + 2 * gamma1 * fraction))
    +
    +        val x = rng.nextUniform(0.0, 1.0)
    +        if (x < stratum.acceptBound) {
    +          stratum.incrNumAccepted()
    +        } else if (x < stratum.waitListBound) {
    +          stratum.addToWaitList(x)
    +        }
    +      }
    +      stratum.incrNumItems()
    +      result
    +    }
    +  }
    +
    +  /**
    +   * Returns the function used combine results returned by seqOp from different partitions.
    +   */
    +  def getCombOp[K]: (MMap[K, Stratum], MMap[K, Stratum]) => MMap[K, Stratum] = {
    +    (result1: MMap[K, Stratum], result2: MMap[K, Stratum]) => {
    +      // take union of both key sets in case one partition doesn't contain all keys
    +      for (key <- result1.keySet.union(result2.keySet)) {
    +        // Use result2 to keep the combined result since r1 is usual empty
    +        val entry1 = result1.get(key)
    +        if (result2.contains(key)) {
    +          result2(key).merge(entry1)
    +        } else {
    +          if (entry1.isDefined) {
    +            result2 += (key -> entry1.get)
    +          }
    +        }
    +      }
    +      result2
    +    }
    +  }
    +
    +  /**
    +   * Given the result returned by getCounts, determine the threshold for accepting items to
    +   * generate exact sample size.
    +   *
    +   * To do so, we compute sampleSize = math.ceil(size * samplingRate) for each stratum and compare
    +   * it to the number of items that were accepted instantly and the number of items in the waitlist
    +   * for that stratum. Most of the time, numAccepted <= sampleSize <= (numAccepted + numWaitlisted),
    +   * which means we need to sort the elements in the waitlist by their associated values in order
    +   * to find the value T s.t. |{elements in the stratum whose associated values <= T}| = sampleSize.
    +   * Note that all elements in the waitlist have values >= bound for instant accept, so a T value
    +   * in the waitlist range would allow all elements that were instantly accepted on the first pass
    +   * to be included in the sample.
    +   */
    +  def computeThresholdByKey[K](finalResult: Map[K, Stratum],
    +      fractions: Map[K, Double]): Map[K, Double] = {
    +    val thresholdByKey = new HashMap[K, Double]()
    +    for ((key, stratum) <- finalResult) {
    +      val sampleSize = math.ceil(stratum.numItems * fractions(key)).toLong
    +      if (stratum.numAccepted > sampleSize) {
    +        logWarning("Pre-accepted too many")
    +        thresholdByKey += (key -> stratum.acceptBound)
    +      } else {
    +        val numWaitListAccepted = (sampleSize - stratum.numAccepted).toInt
    +        if (numWaitListAccepted >= stratum.waitList.size) {
    +          logWarning("WaitList too short")
    +          thresholdByKey += (key -> stratum.waitListBound)
    +        } else {
    +          thresholdByKey += (key -> stratum.waitList.sorted.apply(numWaitListAccepted))
    +        }
    +      }
    +    }
    +    thresholdByKey
    +  }
    +
    +  /**
    +   * Return the per partition sampling function used for sampling without replacement.
    +   *
    +   * When exact sample size is required, we make an additional pass over the RDD to determine the
    +   * exact sampling rate that guarantees sample size with high confidence.
    +   *
    +   * The sampling function has a unique seed per partition.
    +   */
    +  def getBernoulliSamplingFunction[K, V](rdd: RDD[(K,  V)],
    +      fractions: Map[K, Double],
    +      exact: Boolean,
    +      seed: Long): (Int, Iterator[(K, V)]) => Iterator[(K, V)] = {
    +    var samplingRateByKey = fractions
    +    if (exact) {
    +      // determine threshold for each stratum and resample
    +      val finalResult = getCounts(rdd, false, fractions, None, seed)
    +      samplingRateByKey = computeThresholdByKey(finalResult, fractions)
    +    }
    +    (idx: Int, iter: Iterator[(K, V)]) => {
    +      val rng = new RandomDataGenerator
    +      rng.reSeed(seed + idx)
    +      iter.filter(t => rng.nextUniform(0.0, 1.0) < samplingRateByKey(t._1))
    +    }
    +  }
    +
    +  /**
    +   * Return the per partition sampling function used for sampling with replacement.
    +   *
    +   * When exact sample size is required, we make two additional passed over the RDD to determine
    +   * the exact sampling rate that guarantees sample size with high confidence. The first pass
    +   * counts the number of items in each stratum (group of items with the same key) in the RDD, and
    +   * the second pass uses the counts to determine exact sampling rates.
    +   *
    +   * The sampling function has a unique seed per partition.
    +   */
    +  def getPoissonSamplingFunction[K, V](rdd: RDD[(K, V)],
    +      fractions: Map[K, Double],
    +      exact: Boolean,
    +      counts: Option[Map[K, Long]],
    +      seed: Long): (Int, Iterator[(K, V)]) => Iterator[(K, V)] = {
    +    // TODO implement the streaming version of sampling w/ replacement that doesn't require counts
    +    if (exact) {
    +      val finalResult = getCounts(rdd, true, fractions, counts, seed)
    +      val thresholdByKey = computeThresholdByKey(finalResult, fractions)
    +      (idx: Int, iter: Iterator[(K, V)]) => {
    +        val rng = new RandomDataGenerator()
    +        rng.reSeed(seed + idx)
    +        iter.flatMap { item =>
    +          val key = item._1
    +          val acceptBound = finalResult(key).acceptBound
    +          val copiesAccepted = if (acceptBound == 0) 0L else rng.nextPoisson(acceptBound)
    +          val copiesWailisted = rng.nextPoisson(finalResult(key).waitListBound).toInt
    +          val copiesInSample = copiesAccepted +
    +            (0 until copiesWailisted).count(i => rng.nextUniform(0.0, 1.0) < thresholdByKey(key))
    +          if (copiesInSample > 0) {
    +            Iterator.fill(copiesInSample.toInt)(item)
    +          } else {
    +            Iterator.empty
    +          }
    +        }
    +      }
    +    } else {
    +      (idx: Int, iter: Iterator[(K, V)]) => {
    +        val rng = new RandomDataGenerator()
    +        rng.reSeed(seed + idx)
    +        iter.flatMap { item =>
    +          val count = rng.nextPoisson(fractions(item._1)).toInt
    +          if (count > 0) {
    +            Iterator.fill(count)(item)
    +          } else {
    +            Iterator.empty
    +          }
    +        }
    +      }
    +    }
    +  }
    +}
    +
    +/**
    + * Object used by seqOp to keep track of the number of items accepted and items waitlisted per
    + * stratum, as well as the bounds for accepting and waitlisting items.
    + *
    + * `[random]` here is necessary since it's in the return type signature of seqOp defined above
    + */
    +private[random] class Stratum(var numItems: Long = 0L, var numAccepted: Long = 0L)
    --- End diff --
    
    This is not a stratum but something like an `AcceptanceSummary` or `AcceptanceResult`.
    
    The implementation could be simplified:
    
    1) replace `private val _waitlist` and `def waitlist` by `val waitlist`
    2) define `var acceptBound` and initialize it to `Double.NaN` to avoid possibly boxing/unboxing. Same for `waitlistBound`.
    3) `this.incrNumItems(num)` can be replaced by `this.numItems += num`, which should be concise and sufficient. Same for `incrNumAccepted` and `addToWaitList`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by falaki <gi...@git.apache.org>.
Github user falaki commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1025#discussion_r14688121
  
    --- Diff: core/src/main/scala/org/apache/spark/util/random/SamplingUtils.scala ---
    @@ -45,11 +50,75 @@ private[spark] object SamplingUtils {
         val fraction = sampleSizeLowerBound.toDouble / total
         if (withReplacement) {
           val numStDev = if (sampleSizeLowerBound < 12) 9 else 5
    -      fraction + numStDev * math.sqrt(fraction / total)
    +      math.max(1e-10, fraction + numStDev * math.sqrt(fraction / total))
         } else {
           val delta = 1e-4
           val gamma = - math.log(delta) / total
    -      math.min(1, fraction + gamma + math.sqrt(gamma * gamma + 2 * gamma * fraction))
    +      math.min(1,
    +        math.max(1e-10, fraction + gamma + math.sqrt(gamma * gamma + 2 * gamma * fraction)))
    +    }
    +  }
    +}
    +
    +/**
    + * Utility functions that help us determine bounds on adjusted sampling rate to guarantee exact
    + * sample sizes with high confidence when sampling with replacement.
    + *
    + * The algorithm for guaranteeing sample size instantly accepts items whose associated value drawn
    + * from Pois(s) is less than the lower bound and puts items whose value is between the lower and
    + * upper bound in a waitlist. The final sample is consisted of all items accepted on the fly and a
    + * portion of the waitlist needed to make the exact sample size.
    + */
    +private[spark] object PoissonBounds {
    +
    +  val delta = 1e-4 / 3.0
    +  val epsilon = 1e-15
    +
    +  /**
    +   * Compute the threshold for accepting items on the fly. The threshold value is a fairly small
    +   * number, which means if the item has an associated value < threshold, it is highly likely to
    +   * be in the final sample. Hence we accept items with values less than the returned value of this
    +   * function instantly.
    +   *
    +   * @param s sample size
    +   * @return threshold for accepting items on the fly
    +   */
    +  def getLowerBound(s: Double): Double = {
    +    var lb = math.max(0.0, s - math.sqrt(s / delta)) // Chebyshev's inequality
    +    var ub = s
    +    while (lb < ub - 1.0) {
    +      val m = (lb + ub) / 2.0
    +      val poisson = new PoissonDistribution(m, epsilon)
    +      val y = poisson.inverseCumulativeProbability(1 - delta)
    +      if (y > s) ub = m else lb = m
    +    }
    +    lb
    +  }
    +
    +  def getMinCount(lmbd: Double): Double = {
    +    if (lmbd == 0) return 0
    --- End diff --
    
    Style nit: it would be nicer to avoid 'return' and write this as an expression.
    ```
    if (lmbd == 0) {
      0
    } else {
      val poisson = new PoissonDistribution(lmbd, epsilon)
      poisson.inverseCumulativeProbability(delta)
    }
    ```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by falaki <gi...@git.apache.org>.
Github user falaki commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1025#discussion_r14544709
  
    --- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
    @@ -0,0 +1,296 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.util.random
    +
    +import scala.collection.{Map, mutable}
    +import scala.collection.mutable.ArrayBuffer
    +import scala.reflect.ClassTag
    +
    +import org.apache.commons.math3.random.RandomDataGenerator
    +import org.apache.spark.{Logging, SparkContext, TaskContext}
    +import org.apache.spark.rdd.RDD
    +import org.apache.spark.util.Utils
    +import org.apache.spark.util.random.{PoissonBounds => PB}
    +
    +private[spark] object StratifiedSampler extends Logging {
    +
    +  /**
    +   * A version of {@link #aggregate()} that passes the TaskContext to the function that does
    +   * aggregation for each partition. This function avoids creating an extra depth in the RDD
    +   * lineage, as opposed to using mapPartitionsWithId, which results in slightly improved run time.
    +   */
    +  def aggregateWithContext[U: ClassTag, T: ClassTag](zeroValue: U)
    +      (rdd: RDD[T],
    +       seqOp: ((TaskContext, U), T) => U,
    +       combOp: (U, U) => U): U = {
    +    val sc: SparkContext = rdd.sparkContext
    +    // Clone the zero value since we will also be serializing it as part of tasks
    +    var jobResult = Utils.clone(zeroValue, sc.env.closureSerializer.newInstance())
    +    // pad seqOp and combOp with taskContext to conform to aggregate's signature in TraversableOnce
    +    val paddedSeqOp = (arg1: (TaskContext, U), item: T) => (arg1._1, seqOp(arg1, item))
    +    val paddedcombOp = (arg1: (TaskContext, U), arg2: (TaskContext, U)) =>
    +      (arg1._1, combOp(arg1._2, arg1._2))
    +    val cleanSeqOp = sc.clean(paddedSeqOp)
    +    val cleanCombOp = sc.clean(paddedcombOp)
    +    val aggregatePartition = (tc: TaskContext, it: Iterator[T]) =>
    +      (it.aggregate(tc, zeroValue)(cleanSeqOp, cleanCombOp))._2
    +    val mergeResult = (index: Int, taskResult: U) => jobResult = combOp(jobResult, taskResult)
    +    sc.runJob(rdd, aggregatePartition, mergeResult)
    +    jobResult
    +  }
    +
    +  /**
    +   * Returns the function used by aggregate to collect sampling statistics for each partition.
    +   */
    +  def getSeqOp[K, V](withReplacement: Boolean,
    +      fractionByKey: (K => Double),
    +      counts: Option[Map[K, Long]]): ((TaskContext, Result[K]),(K, V)) => Result[K] = {
    +    val delta = 5e-5
    +    (U: (TaskContext, Result[K]), item: (K, V)) => {
    +      val result = U._2
    +      val tc = U._1
    +      val rng = result.getRand(tc.partitionId)
    +      val fraction = fractionByKey(item._1)
    +      val stratum = result.getEntry(item._1)
    +      if (withReplacement) {
    +        // compute q1 and q2 only if they haven't been computed already
    +        // since they don't change from iteration to iteration.
    +        // TODO change this to the streaming version
    +        if (stratum.q1.isEmpty || stratum.q2.isEmpty) {
    +          val n = counts.get(item._1)
    +          val s = math.ceil(n * fraction).toLong
    +          val lmbd1 = PB.getLowerBound(s)
    +          val minCount = PB.getMinCount(lmbd1)
    +          val lmbd2 = if (lmbd1 == 0) PB.getUpperBound(s) else PB.getUpperBound(s - minCount)
    +          val q1 = lmbd1 / n
    +          val q2 = lmbd2 / n
    +          stratum.q1 = Some(q1)
    +          stratum.q2 = Some(q2)
    +        }
    +        val x1 = if (stratum.q1.get == 0) 0L else rng.nextPoisson(stratum.q1.get)
    +        if (x1 > 0) {
    +          stratum.incrNumAccepted(x1)
    +        }
    +        val x2 = rng.nextPoisson(stratum.q2.get).toInt
    +        if (x2 > 0) {
    +          stratum.addToWaitList(ArrayBuffer.fill(x2)(rng.nextUniform(0.0, 1.0)))
    +        }
    +      } else {
    +        // We use the streaming version of the algorithm for sampling without replacement.
    +        // Hence, q1 and q2 change on every iteration.
    +        val g1 = - math.log(delta) / stratum.numItems
    +        val g2 = (2.0 / 3.0) * g1
    +        val q1 = math.max(0, fraction + g2 - math.sqrt((g2 * g2 + 3 * g2 * fraction)))
    +        val q2 = math.min(1, fraction + g1 + math.sqrt(g1 * g1 + 2 * g1 * fraction))
    +
    +        val x = rng.nextUniform(0.0, 1.0)
    +        if (x < q1) {
    +          stratum.incrNumAccepted()
    +        } else if (x < q2) {
    +          stratum.addToWaitList(x)
    +        }
    +        stratum.q1 = Some(q1)
    +        stratum.q2 = Some(q2)
    +      }
    +      stratum.incrNumItems()
    +      result
    +    }
    +  }
    +
    +  /**
    +   * Returns the function used by aggregate to combine results from different partitions, as
    +   * returned by seqOp.
    +   */
    +  def getCombOp[K](): (Result[K], Result[K]) => Result[K] = {
    +    (r1: Result[K], r2: Result[K]) => {
    +      // take union of both key sets in case one partition doesn't contain all keys
    +      val keyUnion = r1.resultMap.keys.toSet.union(r2.resultMap.keys.toSet)
    +
    +      // Use r2 to keep the combined result since r1 is usual empty
    +      for (key <- keyUnion) {
    +        val entry1 = r1.resultMap.get(key)
    +        val entry2 = r2.resultMap.get(key)
    +        if (entry2.isEmpty && entry1.isDefined) {
    +          r2.resultMap += (key -> entry1.get)
    +        } else if (entry1.isDefined && entry2.isDefined) {
    +          entry2.get.addToWaitList(entry1.get.waitList)
    +          entry2.get.incrNumAccepted(entry1.get.numAccepted)
    +          entry2.get.incrNumItems(entry1.get.numItems)
    +        }
    +      }
    +      r2
    +    }
    +  }
    +
    +  /**
    +   * Given the result returned by the aggregate function, we need to determine the threshold used
    +   * to accept items to generate the exact sample size.
    +   */
    +  def computeThresholdByKey[K](finalResult: Map[K, Stratum], fractionByKey: (K => Double)):
    --- End diff --
    
    Style nit: Break the after the first argument, so the return type of the function is not on a new line by itself.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/1025#issuecomment-50383895
  
    QA tests have started for PR 1025. This patch merges cleanly. <br>View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17300/consoleFull


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/1025#issuecomment-50409833
  
    QA tests have started for PR 1025. This patch merges cleanly. <br>View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17307/consoleFull


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/1025#issuecomment-50521377
  
    QA tests have started for PR 1025. This patch merges cleanly. <br>View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17368/consoleFull


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by falaki <gi...@git.apache.org>.
Github user falaki commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1025#discussion_r14688398
  
    --- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
    @@ -0,0 +1,310 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.util.random
    +
    +import scala.collection.Map
    +import scala.collection.mutable.{ArrayBuffer, HashMap, Map => MMap}
    +
    +import org.apache.commons.math3.random.RandomDataGenerator
    +import org.apache.spark.Logging
    +import org.apache.spark.rdd.RDD
    +
    +/**
    + * Auxiliary functions and data structures for the sampleByKey method in PairRDDFunctions.
    + *
    + * Essentially, when exact sample size is necessary, we make additional passes over the RDD to
    + * compute the exact threshold value to use for each stratum to guarantee exact sample size with
    + * high probability. This is achieved by maintaining a waitlist of size O(log(s)), where s is the
    + * desired sample size for each stratum.
    + *
    + * Like in simple random sampling, we generate a random value for each item from the
    + * uniform  distribution [0.0, 1.0]. All items with values <= min(values of items in the waitlist)
    + * are accepted into the sample instantly. The threshold for instant accept is designed so that
    + * s - numAccepted = O(log(s)), where s is again the desired sample size. Thus, by maintaining a
    + * waitlist size = O(log(s)), we will be able to create a sample of the exact size s by adding
    + * a portion of the waitlist to the set of items that are instantly accepted. The exact threshold
    + * is computed by sorting the values in the waitlist and picking the value at (s - numAccepted).
    + *
    + * Note that since we use the same seed for the RNG when computing the thresholds and the actual
    + * sample, our computed thresholds are guaranteed to produce the desired sample size.
    + *
    + * For more theoretical background on the sampling techniques used here, please refer to
    + * http://jmlr.org/proceedings/papers/v28/meng13a.html
    + */
    +
    +private[spark] object StratifiedSampler extends Logging {
    +
    +  /**
    +   * Count the number of items instantly accepted and generate the waitlist for each stratum.
    +   *
    +   * This is only invoked when exact sample size is required.
    +   */
    +  def getCounts[K, V](rdd: RDD[(K, V)],
    +      withReplacement: Boolean,
    +      fractions: Map[K, Double],
    +      counts: Option[Map[K, Long]],
    +      seed: Long): MMap[K, Stratum] = {
    +    val combOp = getCombOp[K]
    +    val mappedPartitionRDD = rdd.mapPartitionsWithIndex({ case (partition, iter) =>
    +      val zeroU: MMap[K, Stratum] = new HashMap[K, Stratum]()
    +      val rng = new RandomDataGenerator()
    +      rng.reSeed(seed + partition)
    +      val seqOp = getSeqOp(withReplacement, fractions, rng, counts)
    +      Iterator(iter.aggregate(zeroU)(seqOp, combOp))
    +    }, preservesPartitioning=true)
    +    mappedPartitionRDD.reduce(combOp)
    +  }
    +
    +  /**
    +   * Returns the function used by aggregate to collect sampling statistics for each partition.
    +   */
    +  def getSeqOp[K, V](withReplacement: Boolean,
    +      fractions: Map[K, Double],
    +      rng: RandomDataGenerator,
    +      counts: Option[Map[K, Long]]): (MMap[K, Stratum], (K, V)) => MMap[K, Stratum] = {
    +    val delta = 5e-5
    +    (result: MMap[K, Stratum], item: (K, V)) => {
    +      val key = item._1
    +      val fraction = fractions(key)
    +      if (!result.contains(key)) {
    +        result += (key -> new Stratum())
    +      }
    +      val stratum = result(key)
    +
    +      if (withReplacement) {
    +        // compute acceptBound and waitListBound only if they haven't been computed already
    +        // since they don't change from iteration to iteration.
    +        // TODO change this to the streaming version
    +        if (stratum.areBoundsEmpty) {
    +          val n = counts.get(key)
    +          val sampleSize = math.ceil(n * fraction).toLong
    +          val lmbd1 = PoissonBounds.getLowerBound(sampleSize)
    +          val minCount = PoissonBounds.getMinCount(lmbd1)
    +          val lmbd2 = if (lmbd1 == 0) {
    +            PoissonBounds.getUpperBound(sampleSize)
    +          } else {
    +            PoissonBounds.getUpperBound(sampleSize - minCount)
    +          }
    +          stratum.acceptBound = lmbd1 / n
    +          stratum.waitListBound = lmbd2 / n
    +        }
    +        val acceptBound = stratum.acceptBound
    +        val copiesAccepted = if (acceptBound == 0.0) 0L else rng.nextPoisson(acceptBound)
    +        if (copiesAccepted > 0) {
    +          stratum.incrNumAccepted(copiesAccepted)
    +        }
    +        val copiesWaitlisted = rng.nextPoisson(stratum.waitListBound).toInt
    +        if (copiesWaitlisted > 0) {
    +          stratum.addToWaitList(ArrayBuffer.fill(copiesWaitlisted)(rng.nextUniform(0.0, 1.0)))
    +        }
    +      } else {
    +        // We use the streaming version of the algorithm for sampling without replacement to avoid
    +        // using an extra pass over the RDD for computing the count.
    +        // Hence, acceptBound and waitListBound change on every iteration.
    +        val g1 = - math.log(delta) / stratum.numItems // gamma1
    +        val g2 = (2.0 / 3.0) * g1                     // gamma 2
    +        stratum.acceptBound = math.max(0, fraction + g2 - math.sqrt(g2 * g2 + 3 * g2 * fraction))
    +        stratum.waitListBound = math.min(1, fraction + g1 + math.sqrt(g1 * g1 + 2 * g1 * fraction))
    +
    +        val x = rng.nextUniform(0.0, 1.0)
    +        if (x < stratum.acceptBound) {
    +          stratum.incrNumAccepted()
    +        } else if (x < stratum.waitListBound) {
    +          stratum.addToWaitList(x)
    +        }
    +      }
    +      stratum.incrNumItems()
    +      result
    +    }
    +  }
    +
    +  /**
    +   * Returns the function used combine results returned by seqOp from different partitions.
    +   */
    +  def getCombOp[K]: (MMap[K, Stratum], MMap[K, Stratum]) => MMap[K, Stratum] = {
    +    (r1: MMap[K, Stratum], r2: MMap[K, Stratum]) => {
    +      // take union of both key sets in case one partition doesn't contain all keys
    +      for (key <- r1.keySet.union(r2.keySet)) {
    +        // Use r2 to keep the combined result since r1 is usual empty
    +        val entry1 = r1.get(key)
    +        if (r2.contains(key)) {
    +          r2(key).merge(entry1)
    +        } else {
    +          if (entry1.isDefined) {
    +            r2 += (key -> entry1.get)
    +          }
    +        }
    +      }
    +      r2
    +    }
    +  }
    +
    +  /**
    +   * Given the result returned by getCounts, determine the threshold for accepting items to
    +   * generate exact sample size.
    +   *
    +   * To do so, we compute sampleSize = math.ceil(size * samplingRate) for each stratum and compare
    +   * it to the number of items that were accepted instantly and the number of items in the waitlist
    +   * for that stratum. Most of the time, numAccepted <= sampleSize <= (numAccepted + numWaitlisted),
    +   * which means we need to sort the elements in the waitlist by their associated values in order
    +   * to find the value T s.t. |{elements in the stratum whose associated values <= T}| = sampleSize.
    +   * Note that all elements in the waitlist have values >= bound for instant accept, so a T value
    +   * in the waitlist range would allow all elements that were instantly accepted on the first pass
    +   * to be included in the sample.
    +   */
    +  def computeThresholdByKey[K](finalResult: Map[K, Stratum],
    +      fractions: Map[K, Double]): Map[K, Double] = {
    +    val thresholdByKey = new HashMap[K, Double]()
    +    for ((key, stratum) <- finalResult) {
    +      val sampleSize = math.ceil(stratum.numItems * fractions(key)).toLong
    +      if (stratum.numAccepted > sampleSize) {
    +        logWarning("Pre-accepted too many")
    +        thresholdByKey += (key -> stratum.acceptBound)
    +      } else {
    +        val numWaitListAccepted = (sampleSize - stratum.numAccepted).toInt
    +        if (numWaitListAccepted >= stratum.waitList.size) {
    +          logWarning("WaitList too short")
    +          thresholdByKey += (key -> stratum.waitListBound)
    +        } else {
    +          thresholdByKey += (key -> stratum.waitList.sorted.apply(numWaitListAccepted))
    +        }
    +      }
    +    }
    +    thresholdByKey
    +  }
    +
    +  /**
    +   * Return the per partition sampling function used for sampling without replacement.
    +   *
    +   * When exact sample size is required, we make an additional pass over the RDD to determine the
    +   * exact sampling rate that guarantees sample size with high confidence.
    +   *
    +   * The sampling function has a unique seed per partition.
    +   */
    +  def getBernoulliSamplingFunction[K, V](rdd: RDD[(K,  V)],
    +      fractions: Map[K, Double],
    +      exact: Boolean,
    +      seed: Long): (Int, Iterator[(K, V)]) => Iterator[(K, V)] = {
    +    var samplingRateByKey = fractions
    +    if (exact) {
    +      // determine threshold for each stratum and resample
    +      val finalResult = getCounts(rdd, false, fractions, None, seed)
    +      samplingRateByKey = computeThresholdByKey(finalResult, fractions)
    +    }
    +    (idx: Int, iter: Iterator[(K, V)]) => {
    +      val rng = new RandomDataGenerator
    +      rng.reSeed(seed + idx)
    +      iter.filter(t => rng.nextUniform(0.0, 1.0) < samplingRateByKey(t._1))
    +    }
    +  }
    +
    +  /**
    +   * Return the per partition sampling function used for sampling with replacement.
    +   *
    +   * When exact sample size is required, we make two additional passed over the RDD to determine
    +   * the exact sampling rate that guarantees sample size with high confidence. The first pass
    +   * counts the number of items in each stratum (group of items with the same key) in the RDD, and
    +   * the second pass uses the counts to determine exact sampling rates.
    +   *
    +   * The sampling function has a unique seed per partition.
    +   */
    +  def getPoissonSamplingFunction[K, V](rdd: RDD[(K, V)],
    +      fractions: Map[K, Double],
    +      exact: Boolean,
    +      counts: Option[Map[K, Long]],
    +      seed: Long): (Int, Iterator[(K, V)]) => Iterator[(K, V)] = {
    +    // TODO implement the streaming version of sampling w/ replacement that doesn't require counts
    +    if (exact) {
    +      val finalResult = getCounts(rdd, true, fractions, counts, seed)
    +      val thresholdByKey = computeThresholdByKey(finalResult, fractions)
    +      (idx: Int, iter: Iterator[(K, V)]) => {
    +        val rng = new RandomDataGenerator()
    +        rng.reSeed(seed + idx)
    +        iter.flatMap { item =>
    +          val key = item._1
    +          val q1 = finalResult(key).acceptBound
    +          val q2 = finalResult(key).waitListBound
    --- End diff --
    
    Similarly for q1 and q2, more descriptive names can help.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by falaki <gi...@git.apache.org>.
Github user falaki commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1025#discussion_r14542829
  
    --- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
    @@ -0,0 +1,296 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.util.random
    +
    +import scala.collection.{Map, mutable}
    +import scala.collection.mutable.ArrayBuffer
    +import scala.reflect.ClassTag
    +
    +import org.apache.commons.math3.random.RandomDataGenerator
    +import org.apache.spark.{Logging, SparkContext, TaskContext}
    +import org.apache.spark.rdd.RDD
    +import org.apache.spark.util.Utils
    +import org.apache.spark.util.random.{PoissonBounds => PB}
    +
    +private[spark] object StratifiedSampler extends Logging {
    +
    +  /**
    +   * A version of {@link #aggregate()} that passes the TaskContext to the function that does
    +   * aggregation for each partition. This function avoids creating an extra depth in the RDD
    +   * lineage, as opposed to using mapPartitionsWithId, which results in slightly improved run time.
    --- End diff --
    
    Do you mean ```mapPartiionsWithIndex()```?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1025#issuecomment-47990765
  
    Merged build finished. All automated tests passed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1025#issuecomment-45553587
  
    
    Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15581/


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1025#discussion_r13578391
  
    --- Diff: core/src/main/scala/org/apache/spark/util/random/SamplingUtils.scala ---
    @@ -0,0 +1,56 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.util.random
    +
    +import org.apache.commons.math3.distribution.{PoissonDistribution, NormalDistribution}
    +
    +private[spark] object PoissonBounds {
    --- End diff --
    
    Though private, this object needs some documentation. I understand this is a little unfair ... :p


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by falaki <gi...@git.apache.org>.
Github user falaki commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1025#discussion_r14688363
  
    --- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
    @@ -0,0 +1,310 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.util.random
    +
    +import scala.collection.Map
    +import scala.collection.mutable.{ArrayBuffer, HashMap, Map => MMap}
    +
    +import org.apache.commons.math3.random.RandomDataGenerator
    +import org.apache.spark.Logging
    +import org.apache.spark.rdd.RDD
    +
    +/**
    + * Auxiliary functions and data structures for the sampleByKey method in PairRDDFunctions.
    + *
    + * Essentially, when exact sample size is necessary, we make additional passes over the RDD to
    + * compute the exact threshold value to use for each stratum to guarantee exact sample size with
    + * high probability. This is achieved by maintaining a waitlist of size O(log(s)), where s is the
    + * desired sample size for each stratum.
    + *
    + * Like in simple random sampling, we generate a random value for each item from the
    + * uniform  distribution [0.0, 1.0]. All items with values <= min(values of items in the waitlist)
    + * are accepted into the sample instantly. The threshold for instant accept is designed so that
    + * s - numAccepted = O(log(s)), where s is again the desired sample size. Thus, by maintaining a
    + * waitlist size = O(log(s)), we will be able to create a sample of the exact size s by adding
    + * a portion of the waitlist to the set of items that are instantly accepted. The exact threshold
    + * is computed by sorting the values in the waitlist and picking the value at (s - numAccepted).
    + *
    + * Note that since we use the same seed for the RNG when computing the thresholds and the actual
    + * sample, our computed thresholds are guaranteed to produce the desired sample size.
    + *
    + * For more theoretical background on the sampling techniques used here, please refer to
    + * http://jmlr.org/proceedings/papers/v28/meng13a.html
    + */
    +
    +private[spark] object StratifiedSampler extends Logging {
    +
    +  /**
    +   * Count the number of items instantly accepted and generate the waitlist for each stratum.
    +   *
    +   * This is only invoked when exact sample size is required.
    +   */
    +  def getCounts[K, V](rdd: RDD[(K, V)],
    +      withReplacement: Boolean,
    +      fractions: Map[K, Double],
    +      counts: Option[Map[K, Long]],
    +      seed: Long): MMap[K, Stratum] = {
    +    val combOp = getCombOp[K]
    +    val mappedPartitionRDD = rdd.mapPartitionsWithIndex({ case (partition, iter) =>
    +      val zeroU: MMap[K, Stratum] = new HashMap[K, Stratum]()
    +      val rng = new RandomDataGenerator()
    +      rng.reSeed(seed + partition)
    +      val seqOp = getSeqOp(withReplacement, fractions, rng, counts)
    +      Iterator(iter.aggregate(zeroU)(seqOp, combOp))
    +    }, preservesPartitioning=true)
    +    mappedPartitionRDD.reduce(combOp)
    +  }
    +
    +  /**
    +   * Returns the function used by aggregate to collect sampling statistics for each partition.
    +   */
    +  def getSeqOp[K, V](withReplacement: Boolean,
    +      fractions: Map[K, Double],
    +      rng: RandomDataGenerator,
    +      counts: Option[Map[K, Long]]): (MMap[K, Stratum], (K, V)) => MMap[K, Stratum] = {
    +    val delta = 5e-5
    +    (result: MMap[K, Stratum], item: (K, V)) => {
    +      val key = item._1
    +      val fraction = fractions(key)
    +      if (!result.contains(key)) {
    +        result += (key -> new Stratum())
    +      }
    +      val stratum = result(key)
    +
    +      if (withReplacement) {
    +        // compute acceptBound and waitListBound only if they haven't been computed already
    +        // since they don't change from iteration to iteration.
    +        // TODO change this to the streaming version
    +        if (stratum.areBoundsEmpty) {
    +          val n = counts.get(key)
    +          val sampleSize = math.ceil(n * fraction).toLong
    +          val lmbd1 = PoissonBounds.getLowerBound(sampleSize)
    +          val minCount = PoissonBounds.getMinCount(lmbd1)
    +          val lmbd2 = if (lmbd1 == 0) {
    +            PoissonBounds.getUpperBound(sampleSize)
    +          } else {
    +            PoissonBounds.getUpperBound(sampleSize - minCount)
    +          }
    +          stratum.acceptBound = lmbd1 / n
    +          stratum.waitListBound = lmbd2 / n
    +        }
    +        val acceptBound = stratum.acceptBound
    +        val copiesAccepted = if (acceptBound == 0.0) 0L else rng.nextPoisson(acceptBound)
    +        if (copiesAccepted > 0) {
    +          stratum.incrNumAccepted(copiesAccepted)
    +        }
    +        val copiesWaitlisted = rng.nextPoisson(stratum.waitListBound).toInt
    +        if (copiesWaitlisted > 0) {
    +          stratum.addToWaitList(ArrayBuffer.fill(copiesWaitlisted)(rng.nextUniform(0.0, 1.0)))
    +        }
    +      } else {
    +        // We use the streaming version of the algorithm for sampling without replacement to avoid
    +        // using an extra pass over the RDD for computing the count.
    +        // Hence, acceptBound and waitListBound change on every iteration.
    +        val g1 = - math.log(delta) / stratum.numItems // gamma1
    +        val g2 = (2.0 / 3.0) * g1                     // gamma 2
    +        stratum.acceptBound = math.max(0, fraction + g2 - math.sqrt(g2 * g2 + 3 * g2 * fraction))
    +        stratum.waitListBound = math.min(1, fraction + g1 + math.sqrt(g1 * g1 + 2 * g1 * fraction))
    +
    +        val x = rng.nextUniform(0.0, 1.0)
    +        if (x < stratum.acceptBound) {
    +          stratum.incrNumAccepted()
    +        } else if (x < stratum.waitListBound) {
    +          stratum.addToWaitList(x)
    +        }
    +      }
    +      stratum.incrNumItems()
    +      result
    +    }
    +  }
    +
    +  /**
    +   * Returns the function used combine results returned by seqOp from different partitions.
    +   */
    +  def getCombOp[K]: (MMap[K, Stratum], MMap[K, Stratum]) => MMap[K, Stratum] = {
    +    (r1: MMap[K, Stratum], r2: MMap[K, Stratum]) => {
    --- End diff --
    
    Minor: Similarly for r1 and r2


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1025#issuecomment-46608095
  
    Merged build started. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1025#issuecomment-48414125
  
     Merged build triggered. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by falaki <gi...@git.apache.org>.
Github user falaki commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1025#discussion_r13677098
  
    --- Diff: core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala ---
    @@ -156,6 +161,182 @@ class PairRDDFunctions[K, V](self: RDD[(K, V)])
       }
     
       /**
    +   * Return a subset of this RDD sampled by key (via stratified sampling).
    +   * We guarantee a sample size = math.ceil(fraction * S_i), where S_i is the size of the ith
    +   * stratum.
    +   *
    +   * @param withReplacement whether to sample with or without replacement
    +   * @param fraction sampling rate
    +   * @param seed seed for the random number generator
    +   * @return RDD containing the sampled subset
    +   */
    +  def sampleByKey(withReplacement: Boolean,
    +      fraction: Double,
    +      seed: Long = Utils.random.nextLong): RDD[(K, V)]= {
    +
    +    class Stratum(var numItems: Long = 0L, var numAccepted: Long = 0L) extends Serializable {
    +      var waitList: ArrayBuffer[Double] = new ArrayBuffer[Double]
    +      var q1: Option[Double] = None
    +      var q2: Option[Double] = None
    +
    +      def incrNumItems(by: Long = 1L) = numItems += by
    +
    +      def incrNumAccepted(by: Long = 1L) = numAccepted += by
    +
    +      def addToWaitList(elem: Double) = waitList += elem
    +
    +      def addToWaitList(elems: ArrayBuffer[Double]) = waitList ++= elems
    +
    +      override def toString() = {
    +        "numItems: " + numItems + " numAccepted: " + numAccepted + " q1: " + q1 + " q2: " + q2 +
    +          " waitListSize:" + waitList.size
    +      }
    +    }
    +
    +    class Result(var resultMap: Map[K, Stratum], var cachedPartitionId: Option[Int] = None)
    +      extends Serializable {
    +      var rand: RandomDataGenerator = new RandomDataGenerator
    +
    +      def getEntry(key: K, numItems: Long = 0L): Stratum = {
    +        if (resultMap.get(key).isEmpty) {
    +          resultMap += (key -> new Stratum(numItems))
    +        }
    +        resultMap.get(key).get
    +      }
    +
    +      def getRand(partitionId: Int): RandomDataGenerator = {
    +        if (cachedPartitionId.isEmpty || cachedPartitionId.get != partitionId) {
    +          cachedPartitionId = Some(partitionId)
    +          rand.reSeed(seed + partitionId)
    +        }
    +        rand
    +      }
    +    }
    +
    +    // TODO implement the streaming version of sampling w/ replacement that doesn't require counts
    +    // in order to save one pass over the RDD
    +    val counts = if (withReplacement) Some(this.countByKey()) else None
    +
    +    val seqOp = (U: (TaskContext, Result), item: (K, V)) => {
    +      val delta = 5e-5
    +      val result = U._2
    +      val tc = U._1
    +      val rng = result.getRand(tc.partitionId)
    +      val stratum = result.getEntry(item._1)
    +      if (withReplacement) {
    +        // compute q1 and q2 only if they haven't been computed already
    +        // since they don't change from iteration to iteration.
    +        // TODO change this to the streaming version
    +        if (stratum.q1.isEmpty || stratum.q2.isEmpty) {
    +          val n = counts.get(item._1)
    +          val s = math.ceil(n * fraction).toLong
    +          val lmbd1 = PB.getLambda1(s)
    +          val minCount = PB.getMinCount(lmbd1)
    +          val lmbd2 = if (lmbd1 == 0) PB.getLambda2(s) else PB.getLambda2(s - minCount)
    +          val q1 = lmbd1 / n
    +          val q2 = lmbd2 / n
    +          stratum.q1 = Some(q1)
    +          stratum.q2 = Some(q2)
    +        }
    +        val x1 = if (stratum.q1.get == 0) 0L else rng.nextPoisson(stratum.q1.get)
    +        if (x1 > 0) {
    +          stratum.incrNumAccepted(x1)
    +        }
    +        val x2 = rng.nextPoisson(stratum.q2.get).toInt
    +        if (x2 > 0) {
    +          stratum.addToWaitList(ArrayBuffer.fill(x2)(rng.nextUniform(0.0, 1.0)))
    +        }
    +      } else {
    +        val g1 = - math.log(delta) / stratum.numItems
    +        val g2 = (2.0 / 3.0) * g1
    +        val q1 = math.max(0, fraction + g2 - math.sqrt((g2 * g2 + 3 * g2 * fraction)))
    +        val q2 = math.min(1, fraction + g1 + math.sqrt(g1 * g1 + 2 * g1 * fraction))
    +
    +        val x = rng.nextUniform(0.0, 1.0)
    +        if (x < q1) {
    +          stratum.incrNumAccepted()
    +        } else if ( x < q2) {
    +          stratum.addToWaitList(x)
    +        }
    +        stratum.q1 = Some(q1)
    +        stratum.q2 = Some(q2)
    +      }
    +      stratum.incrNumItems()
    +      result
    +    }
    +
    +    val combOp = (r1: Result, r2: Result) => {
    +      // take union of both key sets in case one partion doesn't contain all keys
    +      val keyUnion = r1.resultMap.keys.toSet.union(r2.resultMap.keys.toSet)
    +
    +      // Use r2 to keep the combined result since r1 is usual empty
    +      for (key <- keyUnion) {
    +        val entry1 = r1.resultMap.get(key)
    +        val entry2 = r2.resultMap.get(key)
    +        if (entry2.isEmpty && entry1.isDefined) {
    +          r2.resultMap += (key -> entry1.get)
    +        } else if (entry1.isDefined && entry2.isDefined) {
    +          entry2.get.addToWaitList(entry1.get.waitList)
    +          entry2.get.incrNumAccepted(entry1.get.numAccepted)
    +          entry2.get.incrNumItems(entry1.get.numItems)
    +        }
    +      }
    +      r2
    +    }
    +
    +    val zeroU = new Result(Map[K, Stratum]())
    +
    +    // determine threshold for each stratum and resample
    +    val finalResult = self.aggregateWithContext(zeroU)(seqOp, combOp).resultMap
    +    val thresholdByKey = new mutable.HashMap[K, Double]()
    +    for ((key, stratum) <- finalResult) {
    +      val s = math.ceil(stratum.numItems * fraction).toLong
    +      breakable {
    +        if (stratum.numAccepted > s) {
    +          logWarning("Pre-accepted too many")
    +          thresholdByKey += (key -> stratum.q1.get)
    +          break
    +        }
    +        val numWaitListAccepted = (s - stratum.numAccepted).toInt
    +        if (numWaitListAccepted >= stratum.waitList.size) {
    +          logWarning("WaitList too short")
    +          thresholdByKey += (key -> stratum.q2.get)
    +        } else {
    +          thresholdByKey += (key -> stratum.waitList.sorted.apply(numWaitListAccepted))
    +        }
    +      }
    +    }
    +
    +    if (withReplacement) {
    +      // Poisson sampler
    +      self.mapPartitionsWithIndex((idx: Int, iter: Iterator[(K, V)]) => {
    +        val random = new RandomDataGenerator()
    +        random.reSeed(seed + idx)
    +        iter.flatMap { t =>
    +          val q1 = finalResult.get(t._1).get.q1.get
    +          val q2 = finalResult.get(t._1).get.q2.get
    --- End diff --
    
    Use getOrElse to be safer.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1025#issuecomment-45558761
  
    
    Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15582/


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1025#issuecomment-47990766
  
    All automated tests passed.
    Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16313/


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1025#issuecomment-45957663
  
    Build finished. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1025#issuecomment-48131244
  
    Merged build finished. All automated tests passed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1025#discussion_r14907335
  
    --- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
    @@ -0,0 +1,311 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.util.random
    +
    +import scala.collection.Map
    +import scala.collection.mutable.{ArrayBuffer, HashMap, Map => MMap}
    --- End diff --
    
    `immutable.HashMap`: http://www.scala-lang.org/api/2.11.1/index.html#scala.collection.immutable.HashMap
    
    `MMap` may be confusing because of `MultiMap`: http://www.scala-lang.org/api/2.11.1/index.html#scala.collection.immutable.MultiMap


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by falaki <gi...@git.apache.org>.
Github user falaki commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1025#discussion_r14542796
  
    --- Diff: core/src/main/scala/org/apache/spark/util/random/SamplingUtils.scala ---
    @@ -45,11 +50,74 @@ private[spark] object SamplingUtils {
         val fraction = sampleSizeLowerBound.toDouble / total
         if (withReplacement) {
           val numStDev = if (sampleSizeLowerBound < 12) 9 else 5
    -      fraction + numStDev * math.sqrt(fraction / total)
    +      math.max(1e-10, fraction + numStDev * math.sqrt(fraction / total))
         } else {
           val delta = 1e-4
           val gamma = - math.log(delta) / total
    -      math.min(1, fraction + gamma + math.sqrt(gamma * gamma + 2 * gamma * fraction))
    +      math.min(1,
    +        math.max(1e-10, fraction + gamma + math.sqrt(gamma * gamma + 2 * gamma * fraction)))
    +    }
    +  }
    +}
    +
    +/**
    + * Utility functions that help us determine bounds on adjusted sampling rate to guarantee exact
    + * sample sizes with high confidence when sampling with replacement.
    + *
    + * The algorithm for guaranteeing sample size instantly accepts items whose associated value drawn
    + * from Pois(s) is less than the lower bound and puts items whose value is between the lower and
    + * upper bound in a waitlist. The final sample is consisted of all items accepted on the fly and a
    + * portion of the waitlist needed to make the exact sample size.
    + */
    +private[spark] object PoissonBounds {
    +
    +  val delta = 1e-4 / 3.0
    +
    +  /**
    +   * Compute the threshold for accepting items on the fly. The threshold value is a fairly small
    +   * number, which means if the item has an associated value < threshold, it is highly likely to
    +   * be in the final sample. Hence we accept items with values less than the returned value of this
    +   * function instantly.
    +   *
    +   * @param s sample size
    +   * @return threshold for accepting items on the fly
    +   */
    +  def getLowerBound(s: Double): Double = {
    --- End diff --
    
    Let's combine this function with getUpperBound() into a single getPoissonBounds() function that returns a tuple. There is good overlap between the two functions and they are used in the same place.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:

    https://github.com/apache/spark/pull/1025


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by falaki <gi...@git.apache.org>.
Github user falaki commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1025#discussion_r14544234
  
    --- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
    @@ -0,0 +1,296 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.util.random
    +
    +import scala.collection.{Map, mutable}
    +import scala.collection.mutable.ArrayBuffer
    +import scala.reflect.ClassTag
    +
    +import org.apache.commons.math3.random.RandomDataGenerator
    +import org.apache.spark.{Logging, SparkContext, TaskContext}
    +import org.apache.spark.rdd.RDD
    +import org.apache.spark.util.Utils
    +import org.apache.spark.util.random.{PoissonBounds => PB}
    +
    +private[spark] object StratifiedSampler extends Logging {
    +
    +  /**
    +   * A version of {@link #aggregate()} that passes the TaskContext to the function that does
    +   * aggregation for each partition. This function avoids creating an extra depth in the RDD
    +   * lineage, as opposed to using mapPartitionsWithId, which results in slightly improved run time.
    +   */
    +  def aggregateWithContext[U: ClassTag, T: ClassTag](zeroValue: U)
    +      (rdd: RDD[T],
    +       seqOp: ((TaskContext, U), T) => U,
    +       combOp: (U, U) => U): U = {
    +    val sc: SparkContext = rdd.sparkContext
    +    // Clone the zero value since we will also be serializing it as part of tasks
    +    var jobResult = Utils.clone(zeroValue, sc.env.closureSerializer.newInstance())
    +    // pad seqOp and combOp with taskContext to conform to aggregate's signature in TraversableOnce
    +    val paddedSeqOp = (arg1: (TaskContext, U), item: T) => (arg1._1, seqOp(arg1, item))
    +    val paddedcombOp = (arg1: (TaskContext, U), arg2: (TaskContext, U)) =>
    +      (arg1._1, combOp(arg1._2, arg1._2))
    +    val cleanSeqOp = sc.clean(paddedSeqOp)
    +    val cleanCombOp = sc.clean(paddedcombOp)
    +    val aggregatePartition = (tc: TaskContext, it: Iterator[T]) =>
    +      (it.aggregate(tc, zeroValue)(cleanSeqOp, cleanCombOp))._2
    +    val mergeResult = (index: Int, taskResult: U) => jobResult = combOp(jobResult, taskResult)
    +    sc.runJob(rdd, aggregatePartition, mergeResult)
    +    jobResult
    +  }
    +
    +  /**
    +   * Returns the function used by aggregate to collect sampling statistics for each partition.
    +   */
    +  def getSeqOp[K, V](withReplacement: Boolean,
    +      fractionByKey: (K => Double),
    +      counts: Option[Map[K, Long]]): ((TaskContext, Result[K]),(K, V)) => Result[K] = {
    +    val delta = 5e-5
    +    (U: (TaskContext, Result[K]), item: (K, V)) => {
    +      val result = U._2
    +      val tc = U._1
    +      val rng = result.getRand(tc.partitionId)
    +      val fraction = fractionByKey(item._1)
    +      val stratum = result.getEntry(item._1)
    +      if (withReplacement) {
    +        // compute q1 and q2 only if they haven't been computed already
    +        // since they don't change from iteration to iteration.
    +        // TODO change this to the streaming version
    +        if (stratum.q1.isEmpty || stratum.q2.isEmpty) {
    +          val n = counts.get(item._1)
    +          val s = math.ceil(n * fraction).toLong
    +          val lmbd1 = PB.getLowerBound(s)
    +          val minCount = PB.getMinCount(lmbd1)
    +          val lmbd2 = if (lmbd1 == 0) PB.getUpperBound(s) else PB.getUpperBound(s - minCount)
    +          val q1 = lmbd1 / n
    +          val q2 = lmbd2 / n
    +          stratum.q1 = Some(q1)
    +          stratum.q2 = Some(q2)
    +        }
    +        val x1 = if (stratum.q1.get == 0) 0L else rng.nextPoisson(stratum.q1.get)
    +        if (x1 > 0) {
    +          stratum.incrNumAccepted(x1)
    +        }
    +        val x2 = rng.nextPoisson(stratum.q2.get).toInt
    +        if (x2 > 0) {
    +          stratum.addToWaitList(ArrayBuffer.fill(x2)(rng.nextUniform(0.0, 1.0)))
    +        }
    +      } else {
    +        // We use the streaming version of the algorithm for sampling without replacement.
    +        // Hence, q1 and q2 change on every iteration.
    +        val g1 = - math.log(delta) / stratum.numItems
    +        val g2 = (2.0 / 3.0) * g1
    +        val q1 = math.max(0, fraction + g2 - math.sqrt((g2 * g2 + 3 * g2 * fraction)))
    +        val q2 = math.min(1, fraction + g1 + math.sqrt(g1 * g1 + 2 * g1 * fraction))
    +
    +        val x = rng.nextUniform(0.0, 1.0)
    +        if (x < q1) {
    +          stratum.incrNumAccepted()
    +        } else if (x < q2) {
    +          stratum.addToWaitList(x)
    +        }
    +        stratum.q1 = Some(q1)
    +        stratum.q2 = Some(q2)
    +      }
    +      stratum.incrNumItems()
    +      result
    +    }
    +  }
    +
    +  /**
    +   * Returns the function used by aggregate to combine results from different partitions, as
    +   * returned by seqOp.
    +   */
    +  def getCombOp[K](): (Result[K], Result[K]) => Result[K] = {
    +    (r1: Result[K], r2: Result[K]) => {
    +      // take union of both key sets in case one partition doesn't contain all keys
    +      val keyUnion = r1.resultMap.keys.toSet.union(r2.resultMap.keys.toSet)
    +
    +      // Use r2 to keep the combined result since r1 is usual empty
    +      for (key <- keyUnion) {
    +        val entry1 = r1.resultMap.get(key)
    +        val entry2 = r2.resultMap.get(key)
    +        if (entry2.isEmpty && entry1.isDefined) {
    +          r2.resultMap += (key -> entry1.get)
    +        } else if (entry1.isDefined && entry2.isDefined) {
    +          entry2.get.addToWaitList(entry1.get.waitList)
    +          entry2.get.incrNumAccepted(entry1.get.numAccepted)
    +          entry2.get.incrNumItems(entry1.get.numItems)
    +        }
    +      }
    +      r2
    +    }
    +  }
    +
    +  /**
    +   * Given the result returned by the aggregate function, we need to determine the threshold used
    +   * to accept items to generate the exact sample size.
    +   */
    +  def computeThresholdByKey[K](finalResult: Map[K, Stratum], fractionByKey: (K => Double)):
    +    (K => Double) = {
    +    val thresholdByKey = new mutable.HashMap[K, Double]()
    +    for ((key, stratum) <- finalResult) {
    +      val fraction = fractionByKey(key)
    +      val s = math.ceil(stratum.numItems * fraction).toLong
    +      if (stratum.numAccepted > s) {
    +        logWarning("Pre-accepted too many")
    +        thresholdByKey += (key -> stratum.q1.get)
    +      } else {
    +        val numWaitListAccepted = (s - stratum.numAccepted).toInt
    +        if (numWaitListAccepted >= stratum.waitList.size) {
    +          logWarning("WaitList too short")
    +          thresholdByKey += (key -> stratum.q2.get)
    +        } else {
    +          thresholdByKey += (key -> stratum.waitList.sorted.apply(numWaitListAccepted))
    +        }
    +      }
    +    }
    +    thresholdByKey
    +  }
    +
    +  /**
    +   * Return the per partition sampling function used for sampling without replacement.
    +   *
    +   * When exact sample size is required, we make an additional pass over the RDD to determine the
    +   * exact sampling rate that guarantees sample size with high confidence.
    +   *
    +   * The sampling function has a unique seed per partition.
    +   */
    +  def getBernoulliSamplingFunction[K, V](rdd: RDD[(K,  V)],
    +      fractionByKey: K => Double,
    +      exact: Boolean,
    +      seed: Long): (Int, Iterator[(K, V)]) => Iterator[(K, V)] = {
    +    var samplingRateByKey = fractionByKey
    +    if (exact) {
    +      // determine threshold for each stratum and resample
    +      val seqOp = StratifiedSampler.getSeqOp[K,V](false, fractionByKey, None)
    +      val combOp = StratifiedSampler.getCombOp[K]()
    +      val zeroU = new Result[K](Map[K, Stratum](), seed = seed)
    +      val finalResult = aggregateWithContext(zeroU)(rdd, seqOp, combOp).resultMap
    +      samplingRateByKey = StratifiedSampler.computeThresholdByKey(finalResult, fractionByKey)
    +    }
    +    (idx: Int, iter: Iterator[(K, V)]) => {
    +      val random = new RandomDataGenerator
    +      random.reSeed(seed + idx)
    +      iter.filter(t => random.nextUniform(0.0, 1.0) < samplingRateByKey(t._1))
    +    }
    +  }
    +
    +  /**
    +   * Return the per partition sampling function used for sampling with replacement.
    +   *
    +   * When exact sample size is required, we make two additional passed over the RDD to determine
    +   * the exact sampling rate that guarantees sample size with high confidence. The first pass
    +   * counts the number of items in each stratum (group of items with the same key) in the RDD, and
    +   * the second pass uses the counts to determine exact sampling rates.
    +   *
    +   * The sampling function has a unique seed per partition.
    +   */
    +  def getPoissonSamplingFunction[K, V](rdd:RDD[(K,  V)],
    +      fractionByKey: K => Double,
    +      exact: Boolean,
    +      counts: Option[Map[K, Long]],
    +      seed: Long): (Int, Iterator[(K, V)]) => Iterator[(K, V)] = {
    +    // TODO implement the streaming version of sampling w/ replacement that doesn't require counts
    +    if (exact) {
    +      val seqOp = StratifiedSampler.getSeqOp[K,V](true, fractionByKey, counts)
    +      val combOp = StratifiedSampler.getCombOp[K]()
    +      val zeroU = new Result[K](Map[K, Stratum](), seed = seed)
    +      val finalResult = aggregateWithContext(zeroU)(rdd, seqOp, combOp).resultMap
    +      val thresholdByKey = StratifiedSampler.computeThresholdByKey(finalResult, fractionByKey)
    +      (idx: Int, iter: Iterator[(K, V)]) => {
    +        val random = new RandomDataGenerator()
    +        random.reSeed(seed + idx)
    +        iter.flatMap { t =>
    +          val q1 = finalResult.get(t._1).get.q1.getOrElse(0.0)
    +          val q2 = finalResult.get(t._1).get.q2.getOrElse(0.0)
    +          val x1 = if (q1 == 0) 0L else random.nextPoisson(q1)
    +          val x2 = random.nextPoisson(q2).toInt
    +          val x = x1 + (0 until x2).count(i => random.nextUniform(0.0, 1.0) < thresholdByKey(t._1))
    +          if (x > 0) {
    +            Iterator.fill(x.toInt)(t)
    +          } else {
    +            Iterator.empty
    +          }
    +        }
    +      }
    +    } else {
    +      (idx: Int, iter: Iterator[(K, V)]) => {
    +        val random = new RandomDataGenerator()
    +        random.reSeed(seed + idx)
    +        iter.flatMap { t =>
    +          val count = random.nextPoisson(fractionByKey(t._1)).toInt
    +          if (count > 0) {
    +            Iterator.fill(count)(t)
    +          } else {
    +            Iterator.empty
    +          }
    +        }
    +      }
    +    }
    +  }
    +}
    +
    +/**
    + * Object used by seqOp to keep track of the number of items accepted and items waitlisted per
    + * stratum, as well as the bounds for accepting and waitlisting items.
    + */
    +private[random] class Stratum(var numItems: Long = 0L, var numAccepted: Long = 0L)
    +  extends Serializable {
    +
    +  var waitList: ArrayBuffer[Double] = new ArrayBuffer[Double]
    +  var q1: Option[Double] = None // upper bound for accepting item instantly
    +  var q2: Option[Double] = None // upper bound for adding item to waitlist
    +
    +  def incrNumItems(by: Long = 1L) = numItems += by
    +
    +  def incrNumAccepted(by: Long = 1L) = numAccepted += by
    +
    +  def addToWaitList(elem: Double) = waitList += elem
    +
    +  def addToWaitList(elems: ArrayBuffer[Double]) = waitList ++= elems
    +}
    +
    +/**
    + * Object used by seqOp and combOp to keep track of the sampling statistics for all strata.
    + *
    + * When used by seqOp for each partition, we also keep track of the partition ID in this object
    + * to make sure a single random number generator with a unique seed is used for each partition.
    + */
    +private[random] class Result[K](var resultMap: Map[K, Stratum],
    --- End diff --
    
    Similarly, "[random]" can be removed here.
    Also please change the indent of arguments in next lines to two tabs (see the style guide at: https://cwiki.apache.org/confluence/display/SPARK/Spark+Code+Style+Guide#SparkCodeStyleGuide-Indentation)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1025#issuecomment-48418912
  
    Merged build started. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1025#issuecomment-45964261
  
    Merged build started. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1025#issuecomment-46613651
  
    
    Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15922/


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1025#discussion_r14694274
  
    --- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
    @@ -0,0 +1,311 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.util.random
    +
    +import scala.collection.Map
    +import scala.collection.mutable.{ArrayBuffer, HashMap, Map => MMap}
    +
    +import org.apache.commons.math3.random.RandomDataGenerator
    +import org.apache.spark.Logging
    +import org.apache.spark.rdd.RDD
    +
    +/**
    + * Auxiliary functions and data structures for the sampleByKey method in PairRDDFunctions.
    + *
    + * Essentially, when exact sample size is necessary, we make additional passes over the RDD to
    + * compute the exact threshold value to use for each stratum to guarantee exact sample size with
    + * high probability. This is achieved by maintaining a waitlist of size O(log(s)), where s is the
    + * desired sample size for each stratum.
    + *
    + * Like in simple random sampling, we generate a random value for each item from the
    + * uniform  distribution [0.0, 1.0]. All items with values <= min(values of items in the waitlist)
    + * are accepted into the sample instantly. The threshold for instant accept is designed so that
    + * s - numAccepted = O(log(s)), where s is again the desired sample size. Thus, by maintaining a
    + * waitlist size = O(log(s)), we will be able to create a sample of the exact size s by adding
    + * a portion of the waitlist to the set of items that are instantly accepted. The exact threshold
    + * is computed by sorting the values in the waitlist and picking the value at (s - numAccepted).
    + *
    + * Note that since we use the same seed for the RNG when computing the thresholds and the actual
    + * sample, our computed thresholds are guaranteed to produce the desired sample size.
    + *
    + * For more theoretical background on the sampling techniques used here, please refer to
    + * http://jmlr.org/proceedings/papers/v28/meng13a.html
    + */
    +
    +private[spark] object StratifiedSampler extends Logging {
    +
    +  /**
    +   * Count the number of items instantly accepted and generate the waitlist for each stratum.
    +   *
    +   * This is only invoked when exact sample size is required.
    +   */
    +  def getCounts[K, V](rdd: RDD[(K, V)],
    --- End diff --
    
    The name is a little confusing, especially there is an argument named `counts`. See the comments below on the name of `Stratum`. If we change `Stratum` to `AcceptanceResult`, this could be changed to `getAcceptanceResults`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1025#discussion_r14694237
  
    --- Diff: core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala ---
    @@ -195,6 +193,37 @@ class PairRDDFunctions[K, V](self: RDD[(K, V)])
       }
     
       /**
    +   * Return a subset of this RDD sampled by key (via stratified sampling).
    +   *
    +   * If exact set to true, we guarantee, with high probability, a sample size =
    --- End diff --
    
    Before explaining `exact`, please add more doc explaining what `sampleByKey` does.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by dorx <gi...@git.apache.org>.
Github user dorx commented on the pull request:

    https://github.com/apache/spark/pull/1025#issuecomment-50071767
  
    Looks like there's some API changes from Xiangrui's updates. @mateiz @pwendell


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by dorx <gi...@git.apache.org>.
Github user dorx commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1025#discussion_r14906825
  
    --- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
    @@ -0,0 +1,311 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.util.random
    +
    +import scala.collection.Map
    +import scala.collection.mutable.{ArrayBuffer, HashMap, Map => MMap}
    +
    +import org.apache.commons.math3.random.RandomDataGenerator
    +import org.apache.spark.Logging
    +import org.apache.spark.rdd.RDD
    +
    +/**
    + * Auxiliary functions and data structures for the sampleByKey method in PairRDDFunctions.
    + *
    + * Essentially, when exact sample size is necessary, we make additional passes over the RDD to
    + * compute the exact threshold value to use for each stratum to guarantee exact sample size with
    + * high probability. This is achieved by maintaining a waitlist of size O(log(s)), where s is the
    + * desired sample size for each stratum.
    + *
    + * Like in simple random sampling, we generate a random value for each item from the
    + * uniform  distribution [0.0, 1.0]. All items with values <= min(values of items in the waitlist)
    + * are accepted into the sample instantly. The threshold for instant accept is designed so that
    + * s - numAccepted = O(log(s)), where s is again the desired sample size. Thus, by maintaining a
    + * waitlist size = O(log(s)), we will be able to create a sample of the exact size s by adding
    + * a portion of the waitlist to the set of items that are instantly accepted. The exact threshold
    + * is computed by sorting the values in the waitlist and picking the value at (s - numAccepted).
    + *
    + * Note that since we use the same seed for the RNG when computing the thresholds and the actual
    + * sample, our computed thresholds are guaranteed to produce the desired sample size.
    + *
    + * For more theoretical background on the sampling techniques used here, please refer to
    + * http://jmlr.org/proceedings/papers/v28/meng13a.html
    + */
    +
    +private[spark] object StratifiedSampler extends Logging {
    +
    +  /**
    +   * Count the number of items instantly accepted and generate the waitlist for each stratum.
    +   *
    +   * This is only invoked when exact sample size is required.
    +   */
    +  def getCounts[K, V](rdd: RDD[(K, V)],
    +      withReplacement: Boolean,
    +      fractions: Map[K, Double],
    +      counts: Option[Map[K, Long]],
    +      seed: Long): MMap[K, Stratum] = {
    +    val combOp = getCombOp[K]
    +    val mappedPartitionRDD = rdd.mapPartitionsWithIndex({ case (partition, iter) =>
    +      val zeroU: MMap[K, Stratum] = new HashMap[K, Stratum]()
    +      val rng = new RandomDataGenerator()
    +      rng.reSeed(seed + partition)
    +      val seqOp = getSeqOp(withReplacement, fractions, rng, counts)
    +      Iterator(iter.aggregate(zeroU)(seqOp, combOp))
    +    }, preservesPartitioning=true)
    +    mappedPartitionRDD.reduce(combOp)
    +  }
    +
    +  /**
    +   * Returns the function used by aggregate to collect sampling statistics for each partition.
    +   */
    +  def getSeqOp[K, V](withReplacement: Boolean,
    +      fractions: Map[K, Double],
    +      rng: RandomDataGenerator,
    +      counts: Option[Map[K, Long]]): (MMap[K, Stratum], (K, V)) => MMap[K, Stratum] = {
    +    val delta = 5e-5
    +    (result: MMap[K, Stratum], item: (K, V)) => {
    +      val key = item._1
    +      val fraction = fractions(key)
    +      if (!result.contains(key)) {
    +        result += (key -> new Stratum())
    +      }
    +      val stratum = result(key)
    +
    +      if (withReplacement) {
    +        // compute acceptBound and waitListBound only if they haven't been computed already
    +        // since they don't change from iteration to iteration.
    +        // TODO change this to the streaming version
    +        if (stratum.areBoundsEmpty) {
    +          val n = counts.get(key)
    +          val sampleSize = math.ceil(n * fraction).toLong
    +          val lmbd1 = PoissonBounds.getLowerBound(sampleSize)
    +          val minCount = PoissonBounds.getMinCount(lmbd1)
    --- End diff --
    
    I'm not clear what "it" refers to.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1025#issuecomment-46391364
  
    Merged build finished. All automated tests passed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1025#discussion_r14907358
  
    --- Diff: core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala ---
    @@ -195,6 +193,37 @@ class PairRDDFunctions[K, V](self: RDD[(K, V)])
       }
     
       /**
    +   * Return a subset of this RDD sampled by key (via stratified sampling).
    +   *
    +   * If exact set to true, we guarantee, with high probability, a sample size =
    +   * math.ceil(fraction * S_i), where S_i is the size of the ith stratum (collection of entries
    +   * that share the same key). When sampling without replacement, we need one additional pass over
    +   * the RDD to guarantee sample size with a 99.99% confidence; when sampling with replacement, we
    +   * need two additional passes.
    +   *
    +   * @param withReplacement whether to sample with or without replacement
    +   * @param fractions map of specific keys to sampling rates
    +   * @param seed seed for the random number generator
    +   * @param exact whether sample size needs to be exactly math.ceil(fraction * size) per stratum
    +   * @return RDD containing the sampled subset
    +   */
    +  def sampleByKey(withReplacement: Boolean,
    +      fractions: Map[K, Double],
    +      exact: Boolean = true,
    +      seed: Long = Utils.random.nextLong): RDD[(K, V)]= {
    +
    +    require(fractions.forall {case(k, v) => v >= 0.0}, "Invalid sampling rates.")
    --- End diff --
    
    Sure. But we can do that in a separate PR.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1025#issuecomment-47984396
  
    Merged build started. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by falaki <gi...@git.apache.org>.
Github user falaki commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1025#discussion_r14544027
  
    --- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
    @@ -0,0 +1,296 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.util.random
    +
    +import scala.collection.{Map, mutable}
    +import scala.collection.mutable.ArrayBuffer
    +import scala.reflect.ClassTag
    +
    +import org.apache.commons.math3.random.RandomDataGenerator
    +import org.apache.spark.{Logging, SparkContext, TaskContext}
    +import org.apache.spark.rdd.RDD
    +import org.apache.spark.util.Utils
    +import org.apache.spark.util.random.{PoissonBounds => PB}
    +
    +private[spark] object StratifiedSampler extends Logging {
    +
    +  /**
    +   * A version of {@link #aggregate()} that passes the TaskContext to the function that does
    +   * aggregation for each partition. This function avoids creating an extra depth in the RDD
    +   * lineage, as opposed to using mapPartitionsWithId, which results in slightly improved run time.
    +   */
    +  def aggregateWithContext[U: ClassTag, T: ClassTag](zeroValue: U)
    +      (rdd: RDD[T],
    +       seqOp: ((TaskContext, U), T) => U,
    +       combOp: (U, U) => U): U = {
    +    val sc: SparkContext = rdd.sparkContext
    +    // Clone the zero value since we will also be serializing it as part of tasks
    +    var jobResult = Utils.clone(zeroValue, sc.env.closureSerializer.newInstance())
    +    // pad seqOp and combOp with taskContext to conform to aggregate's signature in TraversableOnce
    +    val paddedSeqOp = (arg1: (TaskContext, U), item: T) => (arg1._1, seqOp(arg1, item))
    +    val paddedcombOp = (arg1: (TaskContext, U), arg2: (TaskContext, U)) =>
    +      (arg1._1, combOp(arg1._2, arg1._2))
    +    val cleanSeqOp = sc.clean(paddedSeqOp)
    +    val cleanCombOp = sc.clean(paddedcombOp)
    +    val aggregatePartition = (tc: TaskContext, it: Iterator[T]) =>
    +      (it.aggregate(tc, zeroValue)(cleanSeqOp, cleanCombOp))._2
    +    val mergeResult = (index: Int, taskResult: U) => jobResult = combOp(jobResult, taskResult)
    +    sc.runJob(rdd, aggregatePartition, mergeResult)
    +    jobResult
    +  }
    +
    +  /**
    +   * Returns the function used by aggregate to collect sampling statistics for each partition.
    +   */
    +  def getSeqOp[K, V](withReplacement: Boolean,
    +      fractionByKey: (K => Double),
    +      counts: Option[Map[K, Long]]): ((TaskContext, Result[K]),(K, V)) => Result[K] = {
    +    val delta = 5e-5
    +    (U: (TaskContext, Result[K]), item: (K, V)) => {
    +      val result = U._2
    +      val tc = U._1
    +      val rng = result.getRand(tc.partitionId)
    +      val fraction = fractionByKey(item._1)
    +      val stratum = result.getEntry(item._1)
    +      if (withReplacement) {
    +        // compute q1 and q2 only if they haven't been computed already
    +        // since they don't change from iteration to iteration.
    +        // TODO change this to the streaming version
    +        if (stratum.q1.isEmpty || stratum.q2.isEmpty) {
    +          val n = counts.get(item._1)
    +          val s = math.ceil(n * fraction).toLong
    +          val lmbd1 = PB.getLowerBound(s)
    +          val minCount = PB.getMinCount(lmbd1)
    +          val lmbd2 = if (lmbd1 == 0) PB.getUpperBound(s) else PB.getUpperBound(s - minCount)
    +          val q1 = lmbd1 / n
    +          val q2 = lmbd2 / n
    +          stratum.q1 = Some(q1)
    +          stratum.q2 = Some(q2)
    +        }
    +        val x1 = if (stratum.q1.get == 0) 0L else rng.nextPoisson(stratum.q1.get)
    +        if (x1 > 0) {
    +          stratum.incrNumAccepted(x1)
    +        }
    +        val x2 = rng.nextPoisson(stratum.q2.get).toInt
    +        if (x2 > 0) {
    +          stratum.addToWaitList(ArrayBuffer.fill(x2)(rng.nextUniform(0.0, 1.0)))
    +        }
    +      } else {
    +        // We use the streaming version of the algorithm for sampling without replacement.
    +        // Hence, q1 and q2 change on every iteration.
    +        val g1 = - math.log(delta) / stratum.numItems
    +        val g2 = (2.0 / 3.0) * g1
    +        val q1 = math.max(0, fraction + g2 - math.sqrt((g2 * g2 + 3 * g2 * fraction)))
    +        val q2 = math.min(1, fraction + g1 + math.sqrt(g1 * g1 + 2 * g1 * fraction))
    +
    +        val x = rng.nextUniform(0.0, 1.0)
    +        if (x < q1) {
    +          stratum.incrNumAccepted()
    +        } else if (x < q2) {
    +          stratum.addToWaitList(x)
    +        }
    +        stratum.q1 = Some(q1)
    +        stratum.q2 = Some(q2)
    +      }
    +      stratum.incrNumItems()
    +      result
    +    }
    +  }
    +
    +  /**
    +   * Returns the function used by aggregate to combine results from different partitions, as
    +   * returned by seqOp.
    +   */
    +  def getCombOp[K](): (Result[K], Result[K]) => Result[K] = {
    +    (r1: Result[K], r2: Result[K]) => {
    +      // take union of both key sets in case one partition doesn't contain all keys
    +      val keyUnion = r1.resultMap.keys.toSet.union(r2.resultMap.keys.toSet)
    +
    +      // Use r2 to keep the combined result since r1 is usual empty
    +      for (key <- keyUnion) {
    +        val entry1 = r1.resultMap.get(key)
    +        val entry2 = r2.resultMap.get(key)
    +        if (entry2.isEmpty && entry1.isDefined) {
    +          r2.resultMap += (key -> entry1.get)
    +        } else if (entry1.isDefined && entry2.isDefined) {
    +          entry2.get.addToWaitList(entry1.get.waitList)
    +          entry2.get.incrNumAccepted(entry1.get.numAccepted)
    +          entry2.get.incrNumItems(entry1.get.numItems)
    +        }
    +      }
    +      r2
    +    }
    +  }
    +
    +  /**
    +   * Given the result returned by the aggregate function, we need to determine the threshold used
    +   * to accept items to generate the exact sample size.
    +   */
    +  def computeThresholdByKey[K](finalResult: Map[K, Stratum], fractionByKey: (K => Double)):
    --- End diff --
    
    Similarly, let's change the type of fractionByKey to Map[K, Double]


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by falaki <gi...@git.apache.org>.
Github user falaki commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1025#discussion_r14688624
  
    --- Diff: core/src/test/scala/org/apache/spark/rdd/PairRDDFunctionsSuite.scala ---
    @@ -83,6 +83,120 @@ class PairRDDFunctionsSuite extends FunSuite with SharedSparkContext {
         assert(valuesFor2.toList.sorted === List(1))
       }
     
    +  test("sampleByKey") {
    +    def stratifier (fractionPositive: Double) = {
    +      (x: Int) => if (x % 10 < (10 * fractionPositive).toInt) "1" else "0"
    +    }
    +
    +    def checkSize(exact: Boolean, withReplacement: Boolean,
    +        expected: Long, actual: Long, p: Double): Boolean = {
    +      if (exact) {
    +        return expected == actual
    +      }
    +      val stdev = if (withReplacement) math.sqrt(expected) else math.sqrt(expected * p * (1 - p))
    +      // Very forgiving margin since we're dealing with very small sample sizes most of the time
    +      math.abs(actual - expected) <= 6 * stdev
    +    }
    +
    +    // Without replacement validation
    +    def takeSampleAndValidateBernoulli(stratifiedData: RDD[(String, Int)],
    +        exact: Boolean,
    +        samplingRate: Double,
    +        seed: Long,
    +        n: Long) = {
    +      val expectedSampleSize = stratifiedData.countByKey().mapValues(count =>
    +        math.ceil(count * samplingRate).toInt)
    +      val fractions = Map("1" -> samplingRate, "0" -> samplingRate)
    +      val sample = stratifiedData.sampleByKey(false, fractions, exact, seed)
    +      val sampleCounts = sample.countByKey()
    +      val takeSample = sample.collect()
    +      assert(sampleCounts.forall({case(k,v) =>
    +        checkSize(exact, false, expectedSampleSize(k), v, samplingRate)}))
    +      assert(takeSample.size === takeSample.toSet.size)
    +      assert(takeSample.forall(x => 1 <= x._2 && x._2 <= n), s"elements not in [1, $n]")
    +    }
    +
    +    // With replacement validation
    +    def takeSampleAndValidatePoisson(stratifiedData: RDD[(String, Int)],
    +        exact: Boolean,
    +        samplingRate: Double,
    +        seed: Long,
    +        n: Long) = {
    +      val expectedSampleSize = stratifiedData.countByKey().mapValues(count =>
    +        math.ceil(count * samplingRate).toInt)
    +      val fractions = Map("1" -> samplingRate, "0" -> samplingRate)
    +      val sample = stratifiedData.sampleByKey(true, fractions, exact, seed)
    +      val sampleCounts = sample.countByKey()
    +      val takeSample = sample.collect()
    +      assert(sampleCounts.forall({case(k,v) =>
    +        checkSize(exact, true, expectedSampleSize(k), v, samplingRate)}))
    +      val groupedByKey = takeSample.groupBy({case(k, v) => k})
    --- End diff --
    
    And here.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1025#issuecomment-45571805
  
    Build finished. All automated tests passed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1025#issuecomment-46389593
  
    Merged build started. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1025#issuecomment-48419506
  
    
    Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16439/


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1025#discussion_r14694394
  
    --- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
    @@ -0,0 +1,311 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.util.random
    +
    +import scala.collection.Map
    +import scala.collection.mutable.{ArrayBuffer, HashMap, Map => MMap}
    +
    +import org.apache.commons.math3.random.RandomDataGenerator
    +import org.apache.spark.Logging
    +import org.apache.spark.rdd.RDD
    +
    +/**
    + * Auxiliary functions and data structures for the sampleByKey method in PairRDDFunctions.
    + *
    + * Essentially, when exact sample size is necessary, we make additional passes over the RDD to
    + * compute the exact threshold value to use for each stratum to guarantee exact sample size with
    + * high probability. This is achieved by maintaining a waitlist of size O(log(s)), where s is the
    + * desired sample size for each stratum.
    + *
    + * Like in simple random sampling, we generate a random value for each item from the
    + * uniform  distribution [0.0, 1.0]. All items with values <= min(values of items in the waitlist)
    + * are accepted into the sample instantly. The threshold for instant accept is designed so that
    + * s - numAccepted = O(log(s)), where s is again the desired sample size. Thus, by maintaining a
    + * waitlist size = O(log(s)), we will be able to create a sample of the exact size s by adding
    + * a portion of the waitlist to the set of items that are instantly accepted. The exact threshold
    + * is computed by sorting the values in the waitlist and picking the value at (s - numAccepted).
    + *
    + * Note that since we use the same seed for the RNG when computing the thresholds and the actual
    + * sample, our computed thresholds are guaranteed to produce the desired sample size.
    + *
    + * For more theoretical background on the sampling techniques used here, please refer to
    + * http://jmlr.org/proceedings/papers/v28/meng13a.html
    + */
    +
    +private[spark] object StratifiedSampler extends Logging {
    +
    +  /**
    +   * Count the number of items instantly accepted and generate the waitlist for each stratum.
    +   *
    +   * This is only invoked when exact sample size is required.
    +   */
    +  def getCounts[K, V](rdd: RDD[(K, V)],
    +      withReplacement: Boolean,
    +      fractions: Map[K, Double],
    +      counts: Option[Map[K, Long]],
    +      seed: Long): MMap[K, Stratum] = {
    +    val combOp = getCombOp[K]
    +    val mappedPartitionRDD = rdd.mapPartitionsWithIndex({ case (partition, iter) =>
    +      val zeroU: MMap[K, Stratum] = new HashMap[K, Stratum]()
    +      val rng = new RandomDataGenerator()
    +      rng.reSeed(seed + partition)
    +      val seqOp = getSeqOp(withReplacement, fractions, rng, counts)
    +      Iterator(iter.aggregate(zeroU)(seqOp, combOp))
    +    }, preservesPartitioning=true)
    +    mappedPartitionRDD.reduce(combOp)
    +  }
    +
    +  /**
    +   * Returns the function used by aggregate to collect sampling statistics for each partition.
    +   */
    +  def getSeqOp[K, V](withReplacement: Boolean,
    +      fractions: Map[K, Double],
    +      rng: RandomDataGenerator,
    +      counts: Option[Map[K, Long]]): (MMap[K, Stratum], (K, V)) => MMap[K, Stratum] = {
    +    val delta = 5e-5
    +    (result: MMap[K, Stratum], item: (K, V)) => {
    +      val key = item._1
    +      val fraction = fractions(key)
    +      if (!result.contains(key)) {
    +        result += (key -> new Stratum())
    +      }
    +      val stratum = result(key)
    +
    +      if (withReplacement) {
    +        // compute acceptBound and waitListBound only if they haven't been computed already
    +        // since they don't change from iteration to iteration.
    +        // TODO change this to the streaming version
    +        if (stratum.areBoundsEmpty) {
    +          val n = counts.get(key)
    +          val sampleSize = math.ceil(n * fraction).toLong
    +          val lmbd1 = PoissonBounds.getLowerBound(sampleSize)
    +          val minCount = PoissonBounds.getMinCount(lmbd1)
    +          val lmbd2 = if (lmbd1 == 0) {
    +            PoissonBounds.getUpperBound(sampleSize)
    +          } else {
    +            PoissonBounds.getUpperBound(sampleSize - minCount)
    +          }
    +          stratum.acceptBound = lmbd1 / n
    +          stratum.waitListBound = lmbd2 / n
    --- End diff --
    
    Sorry, this was my fault.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by falaki <gi...@git.apache.org>.
Github user falaki commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1025#discussion_r14544036
  
    --- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
    @@ -0,0 +1,296 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.util.random
    +
    +import scala.collection.{Map, mutable}
    +import scala.collection.mutable.ArrayBuffer
    +import scala.reflect.ClassTag
    +
    +import org.apache.commons.math3.random.RandomDataGenerator
    +import org.apache.spark.{Logging, SparkContext, TaskContext}
    +import org.apache.spark.rdd.RDD
    +import org.apache.spark.util.Utils
    +import org.apache.spark.util.random.{PoissonBounds => PB}
    +
    +private[spark] object StratifiedSampler extends Logging {
    +
    +  /**
    +   * A version of {@link #aggregate()} that passes the TaskContext to the function that does
    +   * aggregation for each partition. This function avoids creating an extra depth in the RDD
    +   * lineage, as opposed to using mapPartitionsWithId, which results in slightly improved run time.
    +   */
    +  def aggregateWithContext[U: ClassTag, T: ClassTag](zeroValue: U)
    +      (rdd: RDD[T],
    +       seqOp: ((TaskContext, U), T) => U,
    +       combOp: (U, U) => U): U = {
    +    val sc: SparkContext = rdd.sparkContext
    +    // Clone the zero value since we will also be serializing it as part of tasks
    +    var jobResult = Utils.clone(zeroValue, sc.env.closureSerializer.newInstance())
    +    // pad seqOp and combOp with taskContext to conform to aggregate's signature in TraversableOnce
    +    val paddedSeqOp = (arg1: (TaskContext, U), item: T) => (arg1._1, seqOp(arg1, item))
    +    val paddedcombOp = (arg1: (TaskContext, U), arg2: (TaskContext, U)) =>
    +      (arg1._1, combOp(arg1._2, arg1._2))
    +    val cleanSeqOp = sc.clean(paddedSeqOp)
    +    val cleanCombOp = sc.clean(paddedcombOp)
    +    val aggregatePartition = (tc: TaskContext, it: Iterator[T]) =>
    +      (it.aggregate(tc, zeroValue)(cleanSeqOp, cleanCombOp))._2
    +    val mergeResult = (index: Int, taskResult: U) => jobResult = combOp(jobResult, taskResult)
    +    sc.runJob(rdd, aggregatePartition, mergeResult)
    +    jobResult
    +  }
    +
    +  /**
    +   * Returns the function used by aggregate to collect sampling statistics for each partition.
    +   */
    +  def getSeqOp[K, V](withReplacement: Boolean,
    +      fractionByKey: (K => Double),
    +      counts: Option[Map[K, Long]]): ((TaskContext, Result[K]),(K, V)) => Result[K] = {
    +    val delta = 5e-5
    +    (U: (TaskContext, Result[K]), item: (K, V)) => {
    +      val result = U._2
    +      val tc = U._1
    +      val rng = result.getRand(tc.partitionId)
    +      val fraction = fractionByKey(item._1)
    +      val stratum = result.getEntry(item._1)
    +      if (withReplacement) {
    +        // compute q1 and q2 only if they haven't been computed already
    +        // since they don't change from iteration to iteration.
    +        // TODO change this to the streaming version
    +        if (stratum.q1.isEmpty || stratum.q2.isEmpty) {
    +          val n = counts.get(item._1)
    +          val s = math.ceil(n * fraction).toLong
    +          val lmbd1 = PB.getLowerBound(s)
    +          val minCount = PB.getMinCount(lmbd1)
    +          val lmbd2 = if (lmbd1 == 0) PB.getUpperBound(s) else PB.getUpperBound(s - minCount)
    +          val q1 = lmbd1 / n
    +          val q2 = lmbd2 / n
    +          stratum.q1 = Some(q1)
    +          stratum.q2 = Some(q2)
    +        }
    +        val x1 = if (stratum.q1.get == 0) 0L else rng.nextPoisson(stratum.q1.get)
    +        if (x1 > 0) {
    +          stratum.incrNumAccepted(x1)
    +        }
    +        val x2 = rng.nextPoisson(stratum.q2.get).toInt
    +        if (x2 > 0) {
    +          stratum.addToWaitList(ArrayBuffer.fill(x2)(rng.nextUniform(0.0, 1.0)))
    +        }
    +      } else {
    +        // We use the streaming version of the algorithm for sampling without replacement.
    +        // Hence, q1 and q2 change on every iteration.
    +        val g1 = - math.log(delta) / stratum.numItems
    +        val g2 = (2.0 / 3.0) * g1
    +        val q1 = math.max(0, fraction + g2 - math.sqrt((g2 * g2 + 3 * g2 * fraction)))
    +        val q2 = math.min(1, fraction + g1 + math.sqrt(g1 * g1 + 2 * g1 * fraction))
    +
    +        val x = rng.nextUniform(0.0, 1.0)
    +        if (x < q1) {
    +          stratum.incrNumAccepted()
    +        } else if (x < q2) {
    +          stratum.addToWaitList(x)
    +        }
    +        stratum.q1 = Some(q1)
    +        stratum.q2 = Some(q2)
    +      }
    +      stratum.incrNumItems()
    +      result
    +    }
    +  }
    +
    +  /**
    +   * Returns the function used by aggregate to combine results from different partitions, as
    +   * returned by seqOp.
    +   */
    +  def getCombOp[K](): (Result[K], Result[K]) => Result[K] = {
    +    (r1: Result[K], r2: Result[K]) => {
    +      // take union of both key sets in case one partition doesn't contain all keys
    +      val keyUnion = r1.resultMap.keys.toSet.union(r2.resultMap.keys.toSet)
    +
    +      // Use r2 to keep the combined result since r1 is usual empty
    +      for (key <- keyUnion) {
    +        val entry1 = r1.resultMap.get(key)
    +        val entry2 = r2.resultMap.get(key)
    +        if (entry2.isEmpty && entry1.isDefined) {
    +          r2.resultMap += (key -> entry1.get)
    +        } else if (entry1.isDefined && entry2.isDefined) {
    +          entry2.get.addToWaitList(entry1.get.waitList)
    +          entry2.get.incrNumAccepted(entry1.get.numAccepted)
    +          entry2.get.incrNumItems(entry1.get.numItems)
    +        }
    +      }
    +      r2
    +    }
    +  }
    +
    +  /**
    +   * Given the result returned by the aggregate function, we need to determine the threshold used
    +   * to accept items to generate the exact sample size.
    +   */
    +  def computeThresholdByKey[K](finalResult: Map[K, Stratum], fractionByKey: (K => Double)):
    +    (K => Double) = {
    +    val thresholdByKey = new mutable.HashMap[K, Double]()
    +    for ((key, stratum) <- finalResult) {
    +      val fraction = fractionByKey(key)
    +      val s = math.ceil(stratum.numItems * fraction).toLong
    +      if (stratum.numAccepted > s) {
    +        logWarning("Pre-accepted too many")
    +        thresholdByKey += (key -> stratum.q1.get)
    +      } else {
    +        val numWaitListAccepted = (s - stratum.numAccepted).toInt
    +        if (numWaitListAccepted >= stratum.waitList.size) {
    +          logWarning("WaitList too short")
    +          thresholdByKey += (key -> stratum.q2.get)
    +        } else {
    +          thresholdByKey += (key -> stratum.waitList.sorted.apply(numWaitListAccepted))
    +        }
    +      }
    +    }
    +    thresholdByKey
    +  }
    +
    +  /**
    +   * Return the per partition sampling function used for sampling without replacement.
    +   *
    +   * When exact sample size is required, we make an additional pass over the RDD to determine the
    +   * exact sampling rate that guarantees sample size with high confidence.
    +   *
    +   * The sampling function has a unique seed per partition.
    +   */
    +  def getBernoulliSamplingFunction[K, V](rdd: RDD[(K,  V)],
    +      fractionByKey: K => Double,
    --- End diff --
    
    Similarly, let's change the type of fractionByKey to Map[K, Double]


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/1025#issuecomment-50380482
  
    QA tests have started for PR 1025. This patch merges cleanly. <br>View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17296/consoleFull


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by dorx <gi...@git.apache.org>.
Github user dorx commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1025#discussion_r14907202
  
    --- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
    @@ -0,0 +1,311 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.util.random
    +
    +import scala.collection.Map
    +import scala.collection.mutable.{ArrayBuffer, HashMap, Map => MMap}
    +
    +import org.apache.commons.math3.random.RandomDataGenerator
    +import org.apache.spark.Logging
    +import org.apache.spark.rdd.RDD
    +
    +/**
    + * Auxiliary functions and data structures for the sampleByKey method in PairRDDFunctions.
    + *
    + * Essentially, when exact sample size is necessary, we make additional passes over the RDD to
    + * compute the exact threshold value to use for each stratum to guarantee exact sample size with
    + * high probability. This is achieved by maintaining a waitlist of size O(log(s)), where s is the
    + * desired sample size for each stratum.
    + *
    + * Like in simple random sampling, we generate a random value for each item from the
    + * uniform  distribution [0.0, 1.0]. All items with values <= min(values of items in the waitlist)
    + * are accepted into the sample instantly. The threshold for instant accept is designed so that
    + * s - numAccepted = O(log(s)), where s is again the desired sample size. Thus, by maintaining a
    + * waitlist size = O(log(s)), we will be able to create a sample of the exact size s by adding
    + * a portion of the waitlist to the set of items that are instantly accepted. The exact threshold
    + * is computed by sorting the values in the waitlist and picking the value at (s - numAccepted).
    + *
    + * Note that since we use the same seed for the RNG when computing the thresholds and the actual
    + * sample, our computed thresholds are guaranteed to produce the desired sample size.
    + *
    + * For more theoretical background on the sampling techniques used here, please refer to
    + * http://jmlr.org/proceedings/papers/v28/meng13a.html
    + */
    +
    +private[spark] object StratifiedSampler extends Logging {
    +
    +  /**
    +   * Count the number of items instantly accepted and generate the waitlist for each stratum.
    +   *
    +   * This is only invoked when exact sample size is required.
    +   */
    +  def getCounts[K, V](rdd: RDD[(K, V)],
    +      withReplacement: Boolean,
    +      fractions: Map[K, Double],
    +      counts: Option[Map[K, Long]],
    +      seed: Long): MMap[K, Stratum] = {
    +    val combOp = getCombOp[K]
    +    val mappedPartitionRDD = rdd.mapPartitionsWithIndex({ case (partition, iter) =>
    +      val zeroU: MMap[K, Stratum] = new HashMap[K, Stratum]()
    +      val rng = new RandomDataGenerator()
    +      rng.reSeed(seed + partition)
    +      val seqOp = getSeqOp(withReplacement, fractions, rng, counts)
    +      Iterator(iter.aggregate(zeroU)(seqOp, combOp))
    +    }, preservesPartitioning=true)
    +    mappedPartitionRDD.reduce(combOp)
    +  }
    +
    +  /**
    +   * Returns the function used by aggregate to collect sampling statistics for each partition.
    +   */
    +  def getSeqOp[K, V](withReplacement: Boolean,
    +      fractions: Map[K, Double],
    +      rng: RandomDataGenerator,
    +      counts: Option[Map[K, Long]]): (MMap[K, Stratum], (K, V)) => MMap[K, Stratum] = {
    +    val delta = 5e-5
    +    (result: MMap[K, Stratum], item: (K, V)) => {
    +      val key = item._1
    +      val fraction = fractions(key)
    +      if (!result.contains(key)) {
    +        result += (key -> new Stratum())
    +      }
    +      val stratum = result(key)
    +
    +      if (withReplacement) {
    +        // compute acceptBound and waitListBound only if they haven't been computed already
    +        // since they don't change from iteration to iteration.
    +        // TODO change this to the streaming version
    +        if (stratum.areBoundsEmpty) {
    +          val n = counts.get(key)
    +          val sampleSize = math.ceil(n * fraction).toLong
    +          val lmbd1 = PoissonBounds.getLowerBound(sampleSize)
    +          val minCount = PoissonBounds.getMinCount(lmbd1)
    +          val lmbd2 = if (lmbd1 == 0) {
    +            PoissonBounds.getUpperBound(sampleSize)
    +          } else {
    +            PoissonBounds.getUpperBound(sampleSize - minCount)
    +          }
    +          stratum.acceptBound = lmbd1 / n
    +          stratum.waitListBound = lmbd2 / n
    +        }
    +        val acceptBound = stratum.acceptBound
    +        val copiesAccepted = if (acceptBound == 0.0) 0L else rng.nextPoisson(acceptBound)
    +        if (copiesAccepted > 0) {
    +          stratum.incrNumAccepted(copiesAccepted)
    +        }
    +        val copiesWaitlisted = rng.nextPoisson(stratum.waitListBound).toInt
    +        if (copiesWaitlisted > 0) {
    +          stratum.addToWaitList(ArrayBuffer.fill(copiesWaitlisted)(rng.nextUniform(0.0, 1.0)))
    +        }
    +      } else {
    +        // We use the streaming version of the algorithm for sampling without replacement to avoid
    +        // using an extra pass over the RDD for computing the count.
    +        // Hence, acceptBound and waitListBound change on every iteration.
    +        val gamma1 = - math.log(delta) / stratum.numItems
    +        val gamma2 = (2.0 / 3.0) * gamma1
    +        stratum.acceptBound = math.max(0,
    +          fraction + gamma2 - math.sqrt(gamma2 * gamma2 + 3 * gamma2 * fraction))
    +        stratum.waitListBound = math.min(1,
    +          fraction + gamma1 + math.sqrt(gamma1 * gamma1 + 2 * gamma1 * fraction))
    +
    +        val x = rng.nextUniform(0.0, 1.0)
    +        if (x < stratum.acceptBound) {
    +          stratum.incrNumAccepted()
    +        } else if (x < stratum.waitListBound) {
    +          stratum.addToWaitList(x)
    +        }
    +      }
    +      stratum.incrNumItems()
    +      result
    +    }
    +  }
    +
    +  /**
    +   * Returns the function used combine results returned by seqOp from different partitions.
    +   */
    +  def getCombOp[K]: (MMap[K, Stratum], MMap[K, Stratum]) => MMap[K, Stratum] = {
    +    (result1: MMap[K, Stratum], result2: MMap[K, Stratum]) => {
    +      // take union of both key sets in case one partition doesn't contain all keys
    +      for (key <- result1.keySet.union(result2.keySet)) {
    +        // Use result2 to keep the combined result since r1 is usual empty
    +        val entry1 = result1.get(key)
    +        if (result2.contains(key)) {
    +          result2(key).merge(entry1)
    +        } else {
    +          if (entry1.isDefined) {
    +            result2 += (key -> entry1.get)
    +          }
    +        }
    +      }
    +      result2
    +    }
    +  }
    +
    +  /**
    +   * Given the result returned by getCounts, determine the threshold for accepting items to
    +   * generate exact sample size.
    +   *
    +   * To do so, we compute sampleSize = math.ceil(size * samplingRate) for each stratum and compare
    +   * it to the number of items that were accepted instantly and the number of items in the waitlist
    +   * for that stratum. Most of the time, numAccepted <= sampleSize <= (numAccepted + numWaitlisted),
    +   * which means we need to sort the elements in the waitlist by their associated values in order
    +   * to find the value T s.t. |{elements in the stratum whose associated values <= T}| = sampleSize.
    +   * Note that all elements in the waitlist have values >= bound for instant accept, so a T value
    +   * in the waitlist range would allow all elements that were instantly accepted on the first pass
    +   * to be included in the sample.
    +   */
    +  def computeThresholdByKey[K](finalResult: Map[K, Stratum],
    +      fractions: Map[K, Double]): Map[K, Double] = {
    +    val thresholdByKey = new HashMap[K, Double]()
    +    for ((key, stratum) <- finalResult) {
    +      val sampleSize = math.ceil(stratum.numItems * fractions(key)).toLong
    +      if (stratum.numAccepted > sampleSize) {
    +        logWarning("Pre-accepted too many")
    +        thresholdByKey += (key -> stratum.acceptBound)
    +      } else {
    +        val numWaitListAccepted = (sampleSize - stratum.numAccepted).toInt
    +        if (numWaitListAccepted >= stratum.waitList.size) {
    +          logWarning("WaitList too short")
    +          thresholdByKey += (key -> stratum.waitListBound)
    +        } else {
    +          thresholdByKey += (key -> stratum.waitList.sorted.apply(numWaitListAccepted))
    +        }
    +      }
    +    }
    +    thresholdByKey
    +  }
    +
    +  /**
    +   * Return the per partition sampling function used for sampling without replacement.
    +   *
    +   * When exact sample size is required, we make an additional pass over the RDD to determine the
    +   * exact sampling rate that guarantees sample size with high confidence.
    +   *
    +   * The sampling function has a unique seed per partition.
    +   */
    +  def getBernoulliSamplingFunction[K, V](rdd: RDD[(K,  V)],
    +      fractions: Map[K, Double],
    +      exact: Boolean,
    +      seed: Long): (Int, Iterator[(K, V)]) => Iterator[(K, V)] = {
    +    var samplingRateByKey = fractions
    +    if (exact) {
    +      // determine threshold for each stratum and resample
    +      val finalResult = getCounts(rdd, false, fractions, None, seed)
    +      samplingRateByKey = computeThresholdByKey(finalResult, fractions)
    +    }
    +    (idx: Int, iter: Iterator[(K, V)]) => {
    +      val rng = new RandomDataGenerator
    +      rng.reSeed(seed + idx)
    +      iter.filter(t => rng.nextUniform(0.0, 1.0) < samplingRateByKey(t._1))
    +    }
    +  }
    +
    +  /**
    +   * Return the per partition sampling function used for sampling with replacement.
    +   *
    +   * When exact sample size is required, we make two additional passed over the RDD to determine
    +   * the exact sampling rate that guarantees sample size with high confidence. The first pass
    +   * counts the number of items in each stratum (group of items with the same key) in the RDD, and
    +   * the second pass uses the counts to determine exact sampling rates.
    +   *
    +   * The sampling function has a unique seed per partition.
    +   */
    +  def getPoissonSamplingFunction[K, V](rdd: RDD[(K, V)],
    +      fractions: Map[K, Double],
    +      exact: Boolean,
    +      counts: Option[Map[K, Long]],
    +      seed: Long): (Int, Iterator[(K, V)]) => Iterator[(K, V)] = {
    +    // TODO implement the streaming version of sampling w/ replacement that doesn't require counts
    +    if (exact) {
    +      val finalResult = getCounts(rdd, true, fractions, counts, seed)
    +      val thresholdByKey = computeThresholdByKey(finalResult, fractions)
    +      (idx: Int, iter: Iterator[(K, V)]) => {
    +        val rng = new RandomDataGenerator()
    +        rng.reSeed(seed + idx)
    +        iter.flatMap { item =>
    +          val key = item._1
    +          val acceptBound = finalResult(key).acceptBound
    +          val copiesAccepted = if (acceptBound == 0) 0L else rng.nextPoisson(acceptBound)
    +          val copiesWailisted = rng.nextPoisson(finalResult(key).waitListBound).toInt
    +          val copiesInSample = copiesAccepted +
    +            (0 until copiesWailisted).count(i => rng.nextUniform(0.0, 1.0) < thresholdByKey(key))
    +          if (copiesInSample > 0) {
    +            Iterator.fill(copiesInSample.toInt)(item)
    +          } else {
    +            Iterator.empty
    +          }
    +        }
    +      }
    +    } else {
    +      (idx: Int, iter: Iterator[(K, V)]) => {
    +        val rng = new RandomDataGenerator()
    +        rng.reSeed(seed + idx)
    +        iter.flatMap { item =>
    +          val count = rng.nextPoisson(fractions(item._1)).toInt
    +          if (count > 0) {
    +            Iterator.fill(count)(item)
    +          } else {
    +            Iterator.empty
    +          }
    +        }
    +      }
    +    }
    +  }
    +}
    +
    +/**
    + * Object used by seqOp to keep track of the number of items accepted and items waitlisted per
    + * stratum, as well as the bounds for accepting and waitlisting items.
    + *
    + * `[random]` here is necessary since it's in the return type signature of seqOp defined above
    --- End diff --
    
    Wanna to try to compile it for me without `[random]`?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by mateiz <gi...@git.apache.org>.
Github user mateiz commented on the pull request:

    https://github.com/apache/spark/pull/1025#issuecomment-50109108
  
    Well, the other "sample" functions are already approximate anyway. I kind of like this here because it conveys that it's more expensive. The other thing is that if we want the Exact one to be experimental, we can't just make it a parameter.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1025#issuecomment-45964253
  
     Merged build triggered. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1025#issuecomment-46613650
  
    Merged build finished. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by falaki <gi...@git.apache.org>.
Github user falaki commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1025#discussion_r14541686
  
    --- Diff: core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala ---
    @@ -195,6 +193,40 @@ class PairRDDFunctions[K, V](self: RDD[(K, V)])
       }
     
       /**
    +   * Return a subset of this RDD sampled by key (via stratified sampling).
    +   *
    +   * If exact set to true, we guarantee, with high probability, a sample size =
    +   * math.ceil(fraction * S_i), where S_i is the size of the ith stratum (collection of entries
    +   * that share the same key). When sampling without replacement, we need one additional pass over
    +   * the RDD to guarantee sample size with a 99.99% confidence; when sampling with replacement, we
    +   * need two additional passes.
    +   *
    +   * @param withReplacement whether to sample with or without replacement
    +   * @param fractions map of specific keys to sampling rates
    +   * @param seed seed for the random number generator
    +   * @param exact whether sample size needs to be exactly math.ceil(fraction * size) per stratum
    +   * @return RDD containing the sampled subset
    +   */
    +  def sampleByKey(withReplacement: Boolean,
    +      fractions: Map[K, Double],
    +      exact: Boolean = true,
    +      seed: Long = Utils.random.nextLong): RDD[(K, V)]= {
    +
    +    require(fractions.forall({case(k, v) => v >= 0.0}), "Invalid sampling rates.")
    --- End diff --
    
    Could be just:
    ```forall{case(k, v) => v >= 0.0}```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1025#issuecomment-48388110
  
    Merged build finished. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by falaki <gi...@git.apache.org>.
Github user falaki commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1025#discussion_r14622823
  
    --- Diff: core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala ---
    @@ -195,6 +193,37 @@ class PairRDDFunctions[K, V](self: RDD[(K, V)])
       }
     
       /**
    +   * Return a subset of this RDD sampled by key (via stratified sampling).
    +   *
    +   * If exact set to true, we guarantee, with high probability, a sample size =
    +   * math.ceil(fraction * S_i), where S_i is the size of the ith stratum (collection of entries
    +   * that share the same key). When sampling without replacement, we need one additional pass over
    +   * the RDD to guarantee sample size with a 99.99% confidence; when sampling with replacement, we
    +   * need two additional passes.
    +   *
    +   * @param withReplacement whether to sample with or without replacement
    +   * @param fractions map of specific keys to sampling rates
    +   * @param seed seed for the random number generator
    +   * @param exact whether sample size needs to be exactly math.ceil(fraction * size) per stratum
    +   * @return RDD containing the sampled subset
    +   */
    +  def sampleByKey(withReplacement: Boolean,
    +      fractions: Map[K, Double],
    +      exact: Boolean = true,
    +      seed: Long = Utils.random.nextLong): RDD[(K, V)]= {
    +
    +    require(fractions.forall {case(k, v) => v >= 0.0}, "Invalid sampling rates.")
    +
    +    val samplingFunc = if (withReplacement) {
    +      val counts = if (exact) Some(this.countByKey()) else None
    +        StratifiedSampler.getPoissonSamplingFunction(self, fractions, exact, counts, seed)
    +    } else {
    +        StratifiedSampler.getBernoulliSamplingFunction(self, fractions, exact, seed)
    --- End diff --
    
    Remove extra indent.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by dorx <gi...@git.apache.org>.
Github user dorx commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1025#discussion_r14688633
  
    --- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
    @@ -0,0 +1,310 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.util.random
    +
    +import scala.collection.Map
    +import scala.collection.mutable.{ArrayBuffer, HashMap, Map => MMap}
    +
    +import org.apache.commons.math3.random.RandomDataGenerator
    +import org.apache.spark.Logging
    +import org.apache.spark.rdd.RDD
    +
    +/**
    + * Auxiliary functions and data structures for the sampleByKey method in PairRDDFunctions.
    + *
    + * Essentially, when exact sample size is necessary, we make additional passes over the RDD to
    + * compute the exact threshold value to use for each stratum to guarantee exact sample size with
    + * high probability. This is achieved by maintaining a waitlist of size O(log(s)), where s is the
    + * desired sample size for each stratum.
    + *
    + * Like in simple random sampling, we generate a random value for each item from the
    + * uniform  distribution [0.0, 1.0]. All items with values <= min(values of items in the waitlist)
    + * are accepted into the sample instantly. The threshold for instant accept is designed so that
    + * s - numAccepted = O(log(s)), where s is again the desired sample size. Thus, by maintaining a
    + * waitlist size = O(log(s)), we will be able to create a sample of the exact size s by adding
    + * a portion of the waitlist to the set of items that are instantly accepted. The exact threshold
    + * is computed by sorting the values in the waitlist and picking the value at (s - numAccepted).
    + *
    + * Note that since we use the same seed for the RNG when computing the thresholds and the actual
    + * sample, our computed thresholds are guaranteed to produce the desired sample size.
    + *
    + * For more theoretical background on the sampling techniques used here, please refer to
    + * http://jmlr.org/proceedings/papers/v28/meng13a.html
    + */
    +
    +private[spark] object StratifiedSampler extends Logging {
    +
    +  /**
    +   * Count the number of items instantly accepted and generate the waitlist for each stratum.
    +   *
    +   * This is only invoked when exact sample size is required.
    +   */
    +  def getCounts[K, V](rdd: RDD[(K, V)],
    +      withReplacement: Boolean,
    +      fractions: Map[K, Double],
    +      counts: Option[Map[K, Long]],
    +      seed: Long): MMap[K, Stratum] = {
    +    val combOp = getCombOp[K]
    +    val mappedPartitionRDD = rdd.mapPartitionsWithIndex({ case (partition, iter) =>
    +      val zeroU: MMap[K, Stratum] = new HashMap[K, Stratum]()
    +      val rng = new RandomDataGenerator()
    +      rng.reSeed(seed + partition)
    +      val seqOp = getSeqOp(withReplacement, fractions, rng, counts)
    +      Iterator(iter.aggregate(zeroU)(seqOp, combOp))
    +    }, preservesPartitioning=true)
    +    mappedPartitionRDD.reduce(combOp)
    +  }
    +
    +  /**
    +   * Returns the function used by aggregate to collect sampling statistics for each partition.
    +   */
    +  def getSeqOp[K, V](withReplacement: Boolean,
    +      fractions: Map[K, Double],
    +      rng: RandomDataGenerator,
    +      counts: Option[Map[K, Long]]): (MMap[K, Stratum], (K, V)) => MMap[K, Stratum] = {
    +    val delta = 5e-5
    +    (result: MMap[K, Stratum], item: (K, V)) => {
    +      val key = item._1
    +      val fraction = fractions(key)
    +      if (!result.contains(key)) {
    +        result += (key -> new Stratum())
    +      }
    +      val stratum = result(key)
    +
    +      if (withReplacement) {
    +        // compute acceptBound and waitListBound only if they haven't been computed already
    +        // since they don't change from iteration to iteration.
    +        // TODO change this to the streaming version
    +        if (stratum.areBoundsEmpty) {
    +          val n = counts.get(key)
    +          val sampleSize = math.ceil(n * fraction).toLong
    +          val lmbd1 = PoissonBounds.getLowerBound(sampleSize)
    +          val minCount = PoissonBounds.getMinCount(lmbd1)
    +          val lmbd2 = if (lmbd1 == 0) {
    +            PoissonBounds.getUpperBound(sampleSize)
    +          } else {
    +            PoissonBounds.getUpperBound(sampleSize - minCount)
    +          }
    +          stratum.acceptBound = lmbd1 / n
    +          stratum.waitListBound = lmbd2 / n
    +        }
    +        val acceptBound = stratum.acceptBound
    +        val copiesAccepted = if (acceptBound == 0.0) 0L else rng.nextPoisson(acceptBound)
    +        if (copiesAccepted > 0) {
    +          stratum.incrNumAccepted(copiesAccepted)
    +        }
    +        val copiesWaitlisted = rng.nextPoisson(stratum.waitListBound).toInt
    +        if (copiesWaitlisted > 0) {
    +          stratum.addToWaitList(ArrayBuffer.fill(copiesWaitlisted)(rng.nextUniform(0.0, 1.0)))
    +        }
    +      } else {
    +        // We use the streaming version of the algorithm for sampling without replacement to avoid
    +        // using an extra pass over the RDD for computing the count.
    +        // Hence, acceptBound and waitListBound change on every iteration.
    +        val g1 = - math.log(delta) / stratum.numItems // gamma1
    +        val g2 = (2.0 / 3.0) * g1                     // gamma 2
    --- End diff --
    
    I was trying to make the next two lines fit on the same lines, but it doesn't actually look too bad breaking them into two lines each.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1025#issuecomment-48418905
  
     Merged build triggered. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1025#issuecomment-48386807
  
    Merged build started. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by falaki <gi...@git.apache.org>.
Github user falaki commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1025#discussion_r14622781
  
    --- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
    @@ -0,0 +1,335 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.util.random
    +
    +import scala.collection.Map
    +import scala.collection.mutable.{ArrayBuffer, HashMap, Map => MMap}
    +import scala.reflect.ClassTag
    +
    +import org.apache.commons.math3.random.RandomDataGenerator
    +import org.apache.spark.{Logging, SparkContext, TaskContext}
    +import org.apache.spark.rdd.RDD
    +import org.apache.spark.util.Utils
    +
    +/**
    + * Auxiliary functions and data structures for the sampleByKey method in PairRDDFunctions.
    + *
    + * For more theoretical background on the sampling technqiues used here, please refer to
    + * http://jmlr.org/proceedings/papers/v28/meng13a.html
    + */
    +private[spark] object StratifiedSampler extends Logging {
    +
    +  /**
    +   * A version of {@link #aggregate()} that passes the TaskContext to the function that does
    +   * aggregation for each partition. This function avoids creating an extra depth in the RDD
    +   * lineage, as opposed to using mapPartitionsWithIndex, which results in slightly improved
    +   * run time.
    +   */
    +  def aggregateWithContext[U: ClassTag, T: ClassTag](zeroValue: U)
    +      (rdd: RDD[T],
    +       seqOp: ((TaskContext, U), T) => U,
    +       combOp: (U, U) => U): U = {
    +    val sc: SparkContext = rdd.sparkContext
    +    // Clone the zero value since we will also be serializing it as part of tasks
    +    var jobResult = Utils.clone(zeroValue, sc.env.closureSerializer.newInstance())
    +    // pad seqOp and combOp with taskContext to conform to aggregate's signature in TraversableOnce
    +    val paddedSeqOp = (arg1: (TaskContext, U), item: T) => (arg1._1, seqOp(arg1, item))
    +    val paddedCombOp = (arg1: (TaskContext, U), arg2: (TaskContext, U)) =>
    +      (arg1._1, combOp(arg1._2, arg1._2))
    +    val cleanSeqOp = sc.clean(paddedSeqOp)
    +    val cleanCombOp = sc.clean(paddedCombOp)
    +    val aggregatePartition = (tc: TaskContext, it: Iterator[T]) =>
    +      (it.aggregate(tc, zeroValue)(cleanSeqOp, cleanCombOp))._2
    +    val mergeResult = (index: Int, taskResult: U) => jobResult = combOp(jobResult, taskResult)
    +    sc.runJob(rdd, aggregatePartition, mergeResult)
    +    jobResult
    +  }
    +
    +  /**
    +   * Returns the function used by aggregate to collect sampling statistics for each partition.
    +   */
    +  def getSeqOp[K, V](withReplacement: Boolean,
    +      fractions: Map[K, Double],
    +      counts: Option[Map[K, Long]]): ((TaskContext, Result[K]), (K, V)) => Result[K] = {
    +    val delta = 5e-5
    +    (output: (TaskContext, Result[K]), item: (K, V)) => {
    +      val result = output._2
    +      val tc = output._1
    +      val rng = result.getRand(tc.partitionId)
    +      val fraction = fractions(item._1)
    +      val stratum = result.getEntry(item._1)
    +      if (withReplacement) {
    +        // compute acceptBound and waitListBound only if they haven't been computed already
    +        // since they don't change from iteration to iteration.
    +        // TODO change this to the streaming version
    +        if (stratum.areBoundsEmpty) {
    +          val n = counts.get(item._1)
    +          val sampleSize = math.ceil(n * fraction).toLong
    +          val lmbd1 = PoissonBounds.getLowerBound(sampleSize)
    +          val minCount = PoissonBounds.getMinCount(lmbd1)
    +          val lmbd2 = if (lmbd1 == 0) {
    +            PoissonBounds.getUpperBound(sampleSize)
    +          } else {
    +            PoissonBounds.getUpperBound(sampleSize - minCount)
    +          }
    +          stratum.acceptBound = lmbd1 / n
    +          stratum.waitListBound = lmbd2 / n
    +        }
    +        val x1 = if (stratum.acceptBound == 0.0) 0L else rng.nextPoisson(stratum.acceptBound)
    +        if (x1 > 0) {
    +          stratum.incrNumAccepted(x1)
    +        }
    +        val x2 = rng.nextPoisson(stratum.waitListBound).toInt
    +        if (x2 > 0) {
    +          stratum.addToWaitList(ArrayBuffer.fill(x2)(rng.nextUniform(0.0, 1.0)))
    +        }
    +      } else {
    +        // We use the streaming version of the algorithm for sampling without replacement to avoid
    +        // using an extra pass over the RDD for computing the count.
    +        // Hence, acceptBound and waitListBound change on every iteration.
    +        val g1 = - math.log(delta) / stratum.numItems // gamma1
    +        val g2 = (2.0 / 3.0) * g1 // gamma 2
    +        stratum.acceptBound = math.max(0, fraction + g2 - math.sqrt((g2 * g2 + 3 * g2 * fraction)))
    +        stratum.waitListBound = math.min(1, fraction + g1 + math.sqrt(g1 * g1 + 2 * g1 * fraction))
    +
    +        val x = rng.nextUniform(0.0, 1.0)
    +        if (x < stratum.acceptBound) {
    +          stratum.incrNumAccepted()
    +        } else if (x < stratum.waitListBound) {
    +          stratum.addToWaitList(x)
    +        }
    +      }
    +      stratum.incrNumItems()
    +      result
    +    }
    +  }
    +
    +  /**
    +   * Returns the function used by aggregate to combine results from different partitions, as
    +   * returned by seqOp.
    +   */
    +  def getCombOp[K](): (Result[K], Result[K]) => Result[K] = {
    +    (r1: Result[K], r2: Result[K]) => {
    +      // take union of both key sets in case one partition doesn't contain all keys
    +      val keyUnion = r1.resultMap.keySet.union(r2.resultMap.keySet)
    +
    +      // Use r2 to keep the combined result since r1 is usual empty
    +      for (key <- keyUnion) {
    +        val entry1 = r1.resultMap.get(key)
    +        if (r2.resultMap.contains(key)) {
    +          r2.resultMap(key).merge(entry1)
    +        } else {
    +          r2.addEntry(key, entry1)
    +        }
    +      }
    +      r2
    +    }
    +  }
    +
    +  /**
    +   * Given the result returned by the aggregate function, determine the threshold for accepting
    +   * items to generate exact sample size.
    +   *
    +   * To do so, we compute sampleSize = math.ceil(size * samplingRate) for each stratum and compare
    +   * it to the number of items that were accepted instantly and the number of items in the waitlist
    +   * for that stratum. Most of the time, numAccepted <= sampleSize <= (numAccepted + numWaitlisted),
    +   * which means we need to sort the elements in the waitlist by their associated values in order
    +   * to find the value T s.t. |{elements in the stratum whose associated values <= T}| = sampleSize.
    +   * Note that all elements in the waitlist have values >= bound for instant accept, so a T value
    +   * in the waitlist range would allow all elements that were instantly accepted on the first pass
    +   * to be included in the sample.
    +   */
    +  def computeThresholdByKey[K](finalResult: Map[K, Stratum],
    +      fractions: Map[K, Double]):
    --- End diff --
    
    Style nit:  Return type of the function is in a new line. We can join these two lines.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1025#discussion_r15534415
  
    --- Diff: core/src/main/scala/org/apache/spark/util/random/SamplingUtils.scala ---
    @@ -88,14 +91,73 @@ private[spark] object SamplingUtils {
        */
       def computeFractionForSampleSize(sampleSizeLowerBound: Int, total: Long,
           withReplacement: Boolean): Double = {
    -    val fraction = sampleSizeLowerBound.toDouble / total
         if (withReplacement) {
    -      val numStDev = if (sampleSizeLowerBound < 12) 9 else 5
    -      fraction + numStDev * math.sqrt(fraction / total)
    +      PoissonBounds.getUpperBound(sampleSizeLowerBound) / total
         } else {
    -      val delta = 1e-4
    -      val gamma = - math.log(delta) / total
    -      math.min(1, fraction + gamma + math.sqrt(gamma * gamma + 2 * gamma * fraction))
    +      val fraction = sampleSizeLowerBound.toDouble / total
    +      BinomialBounds.getUpperBound(1e-4, total, fraction)
         }
       }
     }
    +
    +/**
    + * Utility functions that help us determine bounds on adjusted sampling rate to guarantee exact
    + * sample sizes with high confidence when sampling with replacement.
    + */
    +private[spark] object PoissonBounds {
    +
    +  /**
    +   * Returns a lambda such that Pr[X > s] is very small, where X ~ Pois(lambda).
    +   */
    +  def getLowerBound(s: Double): Double = {
    +    math.max(s - numStd(s) * math.sqrt(s), 1e-15)
    +  }
    +
    +  /**
    +   * Returns a lambda such that Pr[X < s] is very small, where X ~ Pois(lambda).
    +   *
    +   * @param s sample size
    +   */
    +  def getUpperBound(s: Double): Double = {
    +    math.max(s + numStd(s) * math.sqrt(s), 1e-10)
    +  }
    +
    +  private def numStd(s: Double): Double = {
    +    // TODO: Make it tighter.
    +    if (s < 6.0) {
    +      12.0
    +    } else if (s < 16.0) {
    +      9.0
    +    } else {
    +      6.0
    +    }
    +  }
    +}
    +
    +/**
    + * Utility functions that help us determine bounds on adjusted sampling rate to guarantee exact
    + * sample size with high confidence when sampling without replacement.
    + */
    +private[spark] object BinomialBounds {
    +
    +  val minSamplingRate = 1e-10
    +
    +  /**
    +   * Returns a threshold `p` such that if we conduct n Bernoulli trials with success rate = `p`,
    +   * it is very unlikely to have more than `fraction * n` successes.
    +   */
    +  def getLowerBound(delta: Double, n: Long, fraction: Double): Double = {
    +    val gamma = - math.log(delta) / n * (2.0 / 3.0)
    +    math.max(minSamplingRate,
    --- End diff --
    
    It should be okay to return `0` here. The `minSamplingRate` should be applied to `getUpperBound`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by falaki <gi...@git.apache.org>.
Github user falaki commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1025#discussion_r14544843
  
    --- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
    @@ -0,0 +1,296 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.util.random
    +
    +import scala.collection.{Map, mutable}
    +import scala.collection.mutable.ArrayBuffer
    +import scala.reflect.ClassTag
    +
    +import org.apache.commons.math3.random.RandomDataGenerator
    +import org.apache.spark.{Logging, SparkContext, TaskContext}
    +import org.apache.spark.rdd.RDD
    +import org.apache.spark.util.Utils
    +import org.apache.spark.util.random.{PoissonBounds => PB}
    +
    +private[spark] object StratifiedSampler extends Logging {
    +
    +  /**
    +   * A version of {@link #aggregate()} that passes the TaskContext to the function that does
    +   * aggregation for each partition. This function avoids creating an extra depth in the RDD
    +   * lineage, as opposed to using mapPartitionsWithId, which results in slightly improved run time.
    +   */
    +  def aggregateWithContext[U: ClassTag, T: ClassTag](zeroValue: U)
    +      (rdd: RDD[T],
    +       seqOp: ((TaskContext, U), T) => U,
    +       combOp: (U, U) => U): U = {
    +    val sc: SparkContext = rdd.sparkContext
    +    // Clone the zero value since we will also be serializing it as part of tasks
    +    var jobResult = Utils.clone(zeroValue, sc.env.closureSerializer.newInstance())
    +    // pad seqOp and combOp with taskContext to conform to aggregate's signature in TraversableOnce
    +    val paddedSeqOp = (arg1: (TaskContext, U), item: T) => (arg1._1, seqOp(arg1, item))
    +    val paddedcombOp = (arg1: (TaskContext, U), arg2: (TaskContext, U)) =>
    +      (arg1._1, combOp(arg1._2, arg1._2))
    +    val cleanSeqOp = sc.clean(paddedSeqOp)
    +    val cleanCombOp = sc.clean(paddedcombOp)
    +    val aggregatePartition = (tc: TaskContext, it: Iterator[T]) =>
    +      (it.aggregate(tc, zeroValue)(cleanSeqOp, cleanCombOp))._2
    +    val mergeResult = (index: Int, taskResult: U) => jobResult = combOp(jobResult, taskResult)
    +    sc.runJob(rdd, aggregatePartition, mergeResult)
    +    jobResult
    +  }
    +
    +  /**
    +   * Returns the function used by aggregate to collect sampling statistics for each partition.
    +   */
    +  def getSeqOp[K, V](withReplacement: Boolean,
    +      fractionByKey: (K => Double),
    +      counts: Option[Map[K, Long]]): ((TaskContext, Result[K]),(K, V)) => Result[K] = {
    +    val delta = 5e-5
    +    (U: (TaskContext, Result[K]), item: (K, V)) => {
    +      val result = U._2
    +      val tc = U._1
    +      val rng = result.getRand(tc.partitionId)
    +      val fraction = fractionByKey(item._1)
    +      val stratum = result.getEntry(item._1)
    +      if (withReplacement) {
    +        // compute q1 and q2 only if they haven't been computed already
    +        // since they don't change from iteration to iteration.
    +        // TODO change this to the streaming version
    +        if (stratum.q1.isEmpty || stratum.q2.isEmpty) {
    +          val n = counts.get(item._1)
    +          val s = math.ceil(n * fraction).toLong
    +          val lmbd1 = PB.getLowerBound(s)
    +          val minCount = PB.getMinCount(lmbd1)
    +          val lmbd2 = if (lmbd1 == 0) PB.getUpperBound(s) else PB.getUpperBound(s - minCount)
    +          val q1 = lmbd1 / n
    +          val q2 = lmbd2 / n
    +          stratum.q1 = Some(q1)
    +          stratum.q2 = Some(q2)
    +        }
    +        val x1 = if (stratum.q1.get == 0) 0L else rng.nextPoisson(stratum.q1.get)
    +        if (x1 > 0) {
    +          stratum.incrNumAccepted(x1)
    +        }
    +        val x2 = rng.nextPoisson(stratum.q2.get).toInt
    +        if (x2 > 0) {
    +          stratum.addToWaitList(ArrayBuffer.fill(x2)(rng.nextUniform(0.0, 1.0)))
    +        }
    +      } else {
    +        // We use the streaming version of the algorithm for sampling without replacement.
    +        // Hence, q1 and q2 change on every iteration.
    +        val g1 = - math.log(delta) / stratum.numItems
    +        val g2 = (2.0 / 3.0) * g1
    +        val q1 = math.max(0, fraction + g2 - math.sqrt((g2 * g2 + 3 * g2 * fraction)))
    +        val q2 = math.min(1, fraction + g1 + math.sqrt(g1 * g1 + 2 * g1 * fraction))
    +
    +        val x = rng.nextUniform(0.0, 1.0)
    +        if (x < q1) {
    +          stratum.incrNumAccepted()
    +        } else if (x < q2) {
    +          stratum.addToWaitList(x)
    +        }
    +        stratum.q1 = Some(q1)
    +        stratum.q2 = Some(q2)
    +      }
    +      stratum.incrNumItems()
    +      result
    +    }
    +  }
    +
    +  /**
    +   * Returns the function used by aggregate to combine results from different partitions, as
    +   * returned by seqOp.
    +   */
    +  def getCombOp[K](): (Result[K], Result[K]) => Result[K] = {
    +    (r1: Result[K], r2: Result[K]) => {
    +      // take union of both key sets in case one partition doesn't contain all keys
    +      val keyUnion = r1.resultMap.keys.toSet.union(r2.resultMap.keys.toSet)
    +
    +      // Use r2 to keep the combined result since r1 is usual empty
    +      for (key <- keyUnion) {
    +        val entry1 = r1.resultMap.get(key)
    +        val entry2 = r2.resultMap.get(key)
    +        if (entry2.isEmpty && entry1.isDefined) {
    +          r2.resultMap += (key -> entry1.get)
    +        } else if (entry1.isDefined && entry2.isDefined) {
    +          entry2.get.addToWaitList(entry1.get.waitList)
    +          entry2.get.incrNumAccepted(entry1.get.numAccepted)
    +          entry2.get.incrNumItems(entry1.get.numItems)
    +        }
    +      }
    +      r2
    +    }
    +  }
    +
    +  /**
    +   * Given the result returned by the aggregate function, we need to determine the threshold used
    +   * to accept items to generate the exact sample size.
    +   */
    +  def computeThresholdByKey[K](finalResult: Map[K, Stratum], fractionByKey: (K => Double)):
    +    (K => Double) = {
    +    val thresholdByKey = new mutable.HashMap[K, Double]()
    +    for ((key, stratum) <- finalResult) {
    +      val fraction = fractionByKey(key)
    +      val s = math.ceil(stratum.numItems * fraction).toLong
    +      if (stratum.numAccepted > s) {
    +        logWarning("Pre-accepted too many")
    +        thresholdByKey += (key -> stratum.q1.get)
    +      } else {
    +        val numWaitListAccepted = (s - stratum.numAccepted).toInt
    +        if (numWaitListAccepted >= stratum.waitList.size) {
    +          logWarning("WaitList too short")
    +          thresholdByKey += (key -> stratum.q2.get)
    +        } else {
    +          thresholdByKey += (key -> stratum.waitList.sorted.apply(numWaitListAccepted))
    +        }
    +      }
    +    }
    +    thresholdByKey
    +  }
    +
    +  /**
    +   * Return the per partition sampling function used for sampling without replacement.
    +   *
    +   * When exact sample size is required, we make an additional pass over the RDD to determine the
    +   * exact sampling rate that guarantees sample size with high confidence.
    +   *
    +   * The sampling function has a unique seed per partition.
    +   */
    +  def getBernoulliSamplingFunction[K, V](rdd: RDD[(K,  V)],
    +      fractionByKey: K => Double,
    +      exact: Boolean,
    +      seed: Long): (Int, Iterator[(K, V)]) => Iterator[(K, V)] = {
    +    var samplingRateByKey = fractionByKey
    +    if (exact) {
    +      // determine threshold for each stratum and resample
    +      val seqOp = StratifiedSampler.getSeqOp[K,V](false, fractionByKey, None)
    +      val combOp = StratifiedSampler.getCombOp[K]()
    +      val zeroU = new Result[K](Map[K, Stratum](), seed = seed)
    +      val finalResult = aggregateWithContext(zeroU)(rdd, seqOp, combOp).resultMap
    +      samplingRateByKey = StratifiedSampler.computeThresholdByKey(finalResult, fractionByKey)
    +    }
    +    (idx: Int, iter: Iterator[(K, V)]) => {
    +      val random = new RandomDataGenerator
    +      random.reSeed(seed + idx)
    +      iter.filter(t => random.nextUniform(0.0, 1.0) < samplingRateByKey(t._1))
    +    }
    +  }
    +
    +  /**
    +   * Return the per partition sampling function used for sampling with replacement.
    +   *
    +   * When exact sample size is required, we make two additional passed over the RDD to determine
    +   * the exact sampling rate that guarantees sample size with high confidence. The first pass
    +   * counts the number of items in each stratum (group of items with the same key) in the RDD, and
    +   * the second pass uses the counts to determine exact sampling rates.
    +   *
    +   * The sampling function has a unique seed per partition.
    +   */
    +  def getPoissonSamplingFunction[K, V](rdd:RDD[(K,  V)],
    +      fractionByKey: K => Double,
    +      exact: Boolean,
    +      counts: Option[Map[K, Long]],
    +      seed: Long): (Int, Iterator[(K, V)]) => Iterator[(K, V)] = {
    +    // TODO implement the streaming version of sampling w/ replacement that doesn't require counts
    +    if (exact) {
    +      val seqOp = StratifiedSampler.getSeqOp[K,V](true, fractionByKey, counts)
    +      val combOp = StratifiedSampler.getCombOp[K]()
    +      val zeroU = new Result[K](Map[K, Stratum](), seed = seed)
    +      val finalResult = aggregateWithContext(zeroU)(rdd, seqOp, combOp).resultMap
    +      val thresholdByKey = StratifiedSampler.computeThresholdByKey(finalResult, fractionByKey)
    +      (idx: Int, iter: Iterator[(K, V)]) => {
    +        val random = new RandomDataGenerator()
    +        random.reSeed(seed + idx)
    +        iter.flatMap { t =>
    +          val q1 = finalResult.get(t._1).get.q1.getOrElse(0.0)
    +          val q2 = finalResult.get(t._1).get.q2.getOrElse(0.0)
    +          val x1 = if (q1 == 0) 0L else random.nextPoisson(q1)
    +          val x2 = random.nextPoisson(q2).toInt
    +          val x = x1 + (0 until x2).count(i => random.nextUniform(0.0, 1.0) < thresholdByKey(t._1))
    +          if (x > 0) {
    +            Iterator.fill(x.toInt)(t)
    +          } else {
    +            Iterator.empty
    +          }
    +        }
    +      }
    +    } else {
    +      (idx: Int, iter: Iterator[(K, V)]) => {
    +        val random = new RandomDataGenerator()
    +        random.reSeed(seed + idx)
    +        iter.flatMap { t =>
    +          val count = random.nextPoisson(fractionByKey(t._1)).toInt
    +          if (count > 0) {
    +            Iterator.fill(count)(t)
    +          } else {
    +            Iterator.empty
    +          }
    +        }
    +      }
    +    }
    +  }
    +}
    +
    +/**
    + * Object used by seqOp to keep track of the number of items accepted and items waitlisted per
    + * stratum, as well as the bounds for accepting and waitlisting items.
    + */
    +private[random] class Stratum(var numItems: Long = 0L, var numAccepted: Long = 0L)
    +  extends Serializable {
    +
    +  var waitList: ArrayBuffer[Double] = new ArrayBuffer[Double]
    +  var q1: Option[Double] = None // upper bound for accepting item instantly
    +  var q2: Option[Double] = None // upper bound for adding item to waitlist
    +
    +  def incrNumItems(by: Long = 1L) = numItems += by
    +
    +  def incrNumAccepted(by: Long = 1L) = numAccepted += by
    +
    +  def addToWaitList(elem: Double) = waitList += elem
    +
    +  def addToWaitList(elems: ArrayBuffer[Double]) = waitList ++= elems
    +}
    +
    +/**
    + * Object used by seqOp and combOp to keep track of the sampling statistics for all strata.
    + *
    + * When used by seqOp for each partition, we also keep track of the partition ID in this object
    + * to make sure a single random number generator with a unique seed is used for each partition.
    + */
    +private[random] class Result[K](var resultMap: Map[K, Stratum],
    +                var cachedPartitionId: Option[Int] = None,
    +                val seed: Long)
    +  extends Serializable {
    +
    +  var rand: RandomDataGenerator = new RandomDataGenerator
    --- End diff --
    
    Here you don't need to explicitly declare the type (please apply this comment to other similar places where you explicitly declare type of values or variables).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by dorx <gi...@git.apache.org>.
Github user dorx commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1025#discussion_r14672589
  
    --- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
    @@ -0,0 +1,335 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.util.random
    +
    +import scala.collection.Map
    +import scala.collection.mutable.{ArrayBuffer, HashMap, Map => MMap}
    +import scala.reflect.ClassTag
    +
    +import org.apache.commons.math3.random.RandomDataGenerator
    +import org.apache.spark.{Logging, SparkContext, TaskContext}
    +import org.apache.spark.rdd.RDD
    +import org.apache.spark.util.Utils
    +
    +/**
    + * Auxiliary functions and data structures for the sampleByKey method in PairRDDFunctions.
    + *
    + * For more theoretical background on the sampling technqiues used here, please refer to
    + * http://jmlr.org/proceedings/papers/v28/meng13a.html
    + */
    +private[spark] object StratifiedSampler extends Logging {
    +
    +  /**
    +   * A version of {@link #aggregate()} that passes the TaskContext to the function that does
    +   * aggregation for each partition. This function avoids creating an extra depth in the RDD
    +   * lineage, as opposed to using mapPartitionsWithIndex, which results in slightly improved
    +   * run time.
    +   */
    +  def aggregateWithContext[U: ClassTag, T: ClassTag](zeroValue: U)
    +      (rdd: RDD[T],
    +       seqOp: ((TaskContext, U), T) => U,
    +       combOp: (U, U) => U): U = {
    +    val sc: SparkContext = rdd.sparkContext
    +    // Clone the zero value since we will also be serializing it as part of tasks
    +    var jobResult = Utils.clone(zeroValue, sc.env.closureSerializer.newInstance())
    +    // pad seqOp and combOp with taskContext to conform to aggregate's signature in TraversableOnce
    +    val paddedSeqOp = (arg1: (TaskContext, U), item: T) => (arg1._1, seqOp(arg1, item))
    +    val paddedCombOp = (arg1: (TaskContext, U), arg2: (TaskContext, U)) =>
    +      (arg1._1, combOp(arg1._2, arg1._2))
    +    val cleanSeqOp = sc.clean(paddedSeqOp)
    +    val cleanCombOp = sc.clean(paddedCombOp)
    +    val aggregatePartition = (tc: TaskContext, it: Iterator[T]) =>
    +      (it.aggregate(tc, zeroValue)(cleanSeqOp, cleanCombOp))._2
    +    val mergeResult = (index: Int, taskResult: U) => jobResult = combOp(jobResult, taskResult)
    +    sc.runJob(rdd, aggregatePartition, mergeResult)
    +    jobResult
    +  }
    +
    +  /**
    +   * Returns the function used by aggregate to collect sampling statistics for each partition.
    +   */
    +  def getSeqOp[K, V](withReplacement: Boolean,
    +      fractions: Map[K, Double],
    +      counts: Option[Map[K, Long]]): ((TaskContext, Result[K]), (K, V)) => Result[K] = {
    +    val delta = 5e-5
    +    (output: (TaskContext, Result[K]), item: (K, V)) => {
    +      val result = output._2
    +      val tc = output._1
    +      val rng = result.getRand(tc.partitionId)
    +      val fraction = fractions(item._1)
    +      val stratum = result.getEntry(item._1)
    +      if (withReplacement) {
    +        // compute acceptBound and waitListBound only if they haven't been computed already
    +        // since they don't change from iteration to iteration.
    +        // TODO change this to the streaming version
    +        if (stratum.areBoundsEmpty) {
    +          val n = counts.get(item._1)
    +          val sampleSize = math.ceil(n * fraction).toLong
    +          val lmbd1 = PoissonBounds.getLowerBound(sampleSize)
    +          val minCount = PoissonBounds.getMinCount(lmbd1)
    +          val lmbd2 = if (lmbd1 == 0) {
    +            PoissonBounds.getUpperBound(sampleSize)
    +          } else {
    +            PoissonBounds.getUpperBound(sampleSize - minCount)
    +          }
    +          stratum.acceptBound = lmbd1 / n
    +          stratum.waitListBound = lmbd2 / n
    +        }
    +        val x1 = if (stratum.acceptBound == 0.0) 0L else rng.nextPoisson(stratum.acceptBound)
    +        if (x1 > 0) {
    +          stratum.incrNumAccepted(x1)
    +        }
    +        val x2 = rng.nextPoisson(stratum.waitListBound).toInt
    +        if (x2 > 0) {
    +          stratum.addToWaitList(ArrayBuffer.fill(x2)(rng.nextUniform(0.0, 1.0)))
    +        }
    +      } else {
    +        // We use the streaming version of the algorithm for sampling without replacement to avoid
    +        // using an extra pass over the RDD for computing the count.
    +        // Hence, acceptBound and waitListBound change on every iteration.
    +        val g1 = - math.log(delta) / stratum.numItems // gamma1
    +        val g2 = (2.0 / 3.0) * g1 // gamma 2
    +        stratum.acceptBound = math.max(0, fraction + g2 - math.sqrt((g2 * g2 + 3 * g2 * fraction)))
    +        stratum.waitListBound = math.min(1, fraction + g1 + math.sqrt(g1 * g1 + 2 * g1 * fraction))
    +
    +        val x = rng.nextUniform(0.0, 1.0)
    +        if (x < stratum.acceptBound) {
    +          stratum.incrNumAccepted()
    +        } else if (x < stratum.waitListBound) {
    +          stratum.addToWaitList(x)
    +        }
    +      }
    +      stratum.incrNumItems()
    +      result
    +    }
    +  }
    +
    +  /**
    +   * Returns the function used by aggregate to combine results from different partitions, as
    +   * returned by seqOp.
    +   */
    +  def getCombOp[K](): (Result[K], Result[K]) => Result[K] = {
    +    (r1: Result[K], r2: Result[K]) => {
    +      // take union of both key sets in case one partition doesn't contain all keys
    +      val keyUnion = r1.resultMap.keySet.union(r2.resultMap.keySet)
    +
    +      // Use r2 to keep the combined result since r1 is usual empty
    +      for (key <- keyUnion) {
    +        val entry1 = r1.resultMap.get(key)
    +        if (r2.resultMap.contains(key)) {
    +          r2.resultMap(key).merge(entry1)
    +        } else {
    +          r2.addEntry(key, entry1)
    +        }
    +      }
    +      r2
    +    }
    +  }
    +
    +  /**
    +   * Given the result returned by the aggregate function, determine the threshold for accepting
    +   * items to generate exact sample size.
    +   *
    +   * To do so, we compute sampleSize = math.ceil(size * samplingRate) for each stratum and compare
    +   * it to the number of items that were accepted instantly and the number of items in the waitlist
    +   * for that stratum. Most of the time, numAccepted <= sampleSize <= (numAccepted + numWaitlisted),
    +   * which means we need to sort the elements in the waitlist by their associated values in order
    +   * to find the value T s.t. |{elements in the stratum whose associated values <= T}| = sampleSize.
    +   * Note that all elements in the waitlist have values >= bound for instant accept, so a T value
    +   * in the waitlist range would allow all elements that were instantly accepted on the first pass
    +   * to be included in the sample.
    +   */
    +  def computeThresholdByKey[K](finalResult: Map[K, Stratum],
    +      fractions: Map[K, Double]):
    --- End diff --
    
    Actually what's the rule of thumb here? I was under the impression that if you put your arguments on different lines the return type needs its own line, no?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1025#issuecomment-48384891
  
     Merged build triggered. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by dorx <gi...@git.apache.org>.
Github user dorx commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1025#discussion_r14907155
  
    --- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
    @@ -0,0 +1,311 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.util.random
    +
    +import scala.collection.Map
    +import scala.collection.mutable.{ArrayBuffer, HashMap, Map => MMap}
    +
    +import org.apache.commons.math3.random.RandomDataGenerator
    +import org.apache.spark.Logging
    +import org.apache.spark.rdd.RDD
    +
    +/**
    + * Auxiliary functions and data structures for the sampleByKey method in PairRDDFunctions.
    + *
    + * Essentially, when exact sample size is necessary, we make additional passes over the RDD to
    + * compute the exact threshold value to use for each stratum to guarantee exact sample size with
    + * high probability. This is achieved by maintaining a waitlist of size O(log(s)), where s is the
    + * desired sample size for each stratum.
    + *
    + * Like in simple random sampling, we generate a random value for each item from the
    + * uniform  distribution [0.0, 1.0]. All items with values <= min(values of items in the waitlist)
    + * are accepted into the sample instantly. The threshold for instant accept is designed so that
    + * s - numAccepted = O(log(s)), where s is again the desired sample size. Thus, by maintaining a
    + * waitlist size = O(log(s)), we will be able to create a sample of the exact size s by adding
    + * a portion of the waitlist to the set of items that are instantly accepted. The exact threshold
    + * is computed by sorting the values in the waitlist and picking the value at (s - numAccepted).
    + *
    + * Note that since we use the same seed for the RNG when computing the thresholds and the actual
    + * sample, our computed thresholds are guaranteed to produce the desired sample size.
    + *
    + * For more theoretical background on the sampling techniques used here, please refer to
    + * http://jmlr.org/proceedings/papers/v28/meng13a.html
    + */
    +
    +private[spark] object StratifiedSampler extends Logging {
    +
    +  /**
    +   * Count the number of items instantly accepted and generate the waitlist for each stratum.
    +   *
    +   * This is only invoked when exact sample size is required.
    +   */
    +  def getCounts[K, V](rdd: RDD[(K, V)],
    +      withReplacement: Boolean,
    +      fractions: Map[K, Double],
    +      counts: Option[Map[K, Long]],
    +      seed: Long): MMap[K, Stratum] = {
    +    val combOp = getCombOp[K]
    +    val mappedPartitionRDD = rdd.mapPartitionsWithIndex({ case (partition, iter) =>
    +      val zeroU: MMap[K, Stratum] = new HashMap[K, Stratum]()
    +      val rng = new RandomDataGenerator()
    +      rng.reSeed(seed + partition)
    +      val seqOp = getSeqOp(withReplacement, fractions, rng, counts)
    +      Iterator(iter.aggregate(zeroU)(seqOp, combOp))
    +    }, preservesPartitioning=true)
    +    mappedPartitionRDD.reduce(combOp)
    +  }
    +
    +  /**
    +   * Returns the function used by aggregate to collect sampling statistics for each partition.
    +   */
    +  def getSeqOp[K, V](withReplacement: Boolean,
    +      fractions: Map[K, Double],
    +      rng: RandomDataGenerator,
    +      counts: Option[Map[K, Long]]): (MMap[K, Stratum], (K, V)) => MMap[K, Stratum] = {
    +    val delta = 5e-5
    +    (result: MMap[K, Stratum], item: (K, V)) => {
    +      val key = item._1
    +      val fraction = fractions(key)
    +      if (!result.contains(key)) {
    +        result += (key -> new Stratum())
    +      }
    +      val stratum = result(key)
    +
    +      if (withReplacement) {
    +        // compute acceptBound and waitListBound only if they haven't been computed already
    +        // since they don't change from iteration to iteration.
    +        // TODO change this to the streaming version
    +        if (stratum.areBoundsEmpty) {
    +          val n = counts.get(key)
    +          val sampleSize = math.ceil(n * fraction).toLong
    +          val lmbd1 = PoissonBounds.getLowerBound(sampleSize)
    +          val minCount = PoissonBounds.getMinCount(lmbd1)
    +          val lmbd2 = if (lmbd1 == 0) {
    +            PoissonBounds.getUpperBound(sampleSize)
    +          } else {
    +            PoissonBounds.getUpperBound(sampleSize - minCount)
    +          }
    +          stratum.acceptBound = lmbd1 / n
    +          stratum.waitListBound = lmbd2 / n
    +        }
    +        val acceptBound = stratum.acceptBound
    +        val copiesAccepted = if (acceptBound == 0.0) 0L else rng.nextPoisson(acceptBound)
    +        if (copiesAccepted > 0) {
    +          stratum.incrNumAccepted(copiesAccepted)
    +        }
    +        val copiesWaitlisted = rng.nextPoisson(stratum.waitListBound).toInt
    +        if (copiesWaitlisted > 0) {
    +          stratum.addToWaitList(ArrayBuffer.fill(copiesWaitlisted)(rng.nextUniform(0.0, 1.0)))
    +        }
    +      } else {
    +        // We use the streaming version of the algorithm for sampling without replacement to avoid
    +        // using an extra pass over the RDD for computing the count.
    +        // Hence, acceptBound and waitListBound change on every iteration.
    +        val gamma1 = - math.log(delta) / stratum.numItems
    +        val gamma2 = (2.0 / 3.0) * gamma1
    +        stratum.acceptBound = math.max(0,
    +          fraction + gamma2 - math.sqrt(gamma2 * gamma2 + 3 * gamma2 * fraction))
    +        stratum.waitListBound = math.min(1,
    +          fraction + gamma1 + math.sqrt(gamma1 * gamma1 + 2 * gamma1 * fraction))
    +
    +        val x = rng.nextUniform(0.0, 1.0)
    +        if (x < stratum.acceptBound) {
    +          stratum.incrNumAccepted()
    +        } else if (x < stratum.waitListBound) {
    +          stratum.addToWaitList(x)
    +        }
    +      }
    +      stratum.incrNumItems()
    +      result
    +    }
    +  }
    +
    +  /**
    +   * Returns the function used combine results returned by seqOp from different partitions.
    +   */
    +  def getCombOp[K]: (MMap[K, Stratum], MMap[K, Stratum]) => MMap[K, Stratum] = {
    +    (result1: MMap[K, Stratum], result2: MMap[K, Stratum]) => {
    +      // take union of both key sets in case one partition doesn't contain all keys
    +      for (key <- result1.keySet.union(result2.keySet)) {
    +        // Use result2 to keep the combined result since r1 is usual empty
    --- End diff --
    
    contract per...? I'm not seeing it in the Scala docs: http://www.scala-lang.org/api/2.10.3/index.html#scala.collection.immutable.List



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1025#discussion_r14907639
  
    --- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
    @@ -0,0 +1,311 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.util.random
    +
    +import scala.collection.Map
    +import scala.collection.mutable.{ArrayBuffer, HashMap, Map => MMap}
    +
    +import org.apache.commons.math3.random.RandomDataGenerator
    +import org.apache.spark.Logging
    +import org.apache.spark.rdd.RDD
    +
    +/**
    + * Auxiliary functions and data structures for the sampleByKey method in PairRDDFunctions.
    + *
    + * Essentially, when exact sample size is necessary, we make additional passes over the RDD to
    + * compute the exact threshold value to use for each stratum to guarantee exact sample size with
    + * high probability. This is achieved by maintaining a waitlist of size O(log(s)), where s is the
    + * desired sample size for each stratum.
    + *
    + * Like in simple random sampling, we generate a random value for each item from the
    + * uniform  distribution [0.0, 1.0]. All items with values <= min(values of items in the waitlist)
    + * are accepted into the sample instantly. The threshold for instant accept is designed so that
    + * s - numAccepted = O(log(s)), where s is again the desired sample size. Thus, by maintaining a
    + * waitlist size = O(log(s)), we will be able to create a sample of the exact size s by adding
    + * a portion of the waitlist to the set of items that are instantly accepted. The exact threshold
    + * is computed by sorting the values in the waitlist and picking the value at (s - numAccepted).
    + *
    + * Note that since we use the same seed for the RNG when computing the thresholds and the actual
    + * sample, our computed thresholds are guaranteed to produce the desired sample size.
    + *
    + * For more theoretical background on the sampling techniques used here, please refer to
    + * http://jmlr.org/proceedings/papers/v28/meng13a.html
    + */
    +
    +private[spark] object StratifiedSampler extends Logging {
    +
    +  /**
    +   * Count the number of items instantly accepted and generate the waitlist for each stratum.
    +   *
    +   * This is only invoked when exact sample size is required.
    +   */
    +  def getCounts[K, V](rdd: RDD[(K, V)],
    +      withReplacement: Boolean,
    +      fractions: Map[K, Double],
    +      counts: Option[Map[K, Long]],
    +      seed: Long): MMap[K, Stratum] = {
    +    val combOp = getCombOp[K]
    +    val mappedPartitionRDD = rdd.mapPartitionsWithIndex({ case (partition, iter) =>
    +      val zeroU: MMap[K, Stratum] = new HashMap[K, Stratum]()
    +      val rng = new RandomDataGenerator()
    +      rng.reSeed(seed + partition)
    +      val seqOp = getSeqOp(withReplacement, fractions, rng, counts)
    +      Iterator(iter.aggregate(zeroU)(seqOp, combOp))
    +    }, preservesPartitioning=true)
    +    mappedPartitionRDD.reduce(combOp)
    +  }
    +
    +  /**
    +   * Returns the function used by aggregate to collect sampling statistics for each partition.
    +   */
    +  def getSeqOp[K, V](withReplacement: Boolean,
    +      fractions: Map[K, Double],
    +      rng: RandomDataGenerator,
    +      counts: Option[Map[K, Long]]): (MMap[K, Stratum], (K, V)) => MMap[K, Stratum] = {
    +    val delta = 5e-5
    +    (result: MMap[K, Stratum], item: (K, V)) => {
    +      val key = item._1
    +      val fraction = fractions(key)
    +      if (!result.contains(key)) {
    +        result += (key -> new Stratum())
    +      }
    +      val stratum = result(key)
    +
    +      if (withReplacement) {
    +        // compute acceptBound and waitListBound only if they haven't been computed already
    +        // since they don't change from iteration to iteration.
    +        // TODO change this to the streaming version
    +        if (stratum.areBoundsEmpty) {
    +          val n = counts.get(key)
    +          val sampleSize = math.ceil(n * fraction).toLong
    +          val lmbd1 = PoissonBounds.getLowerBound(sampleSize)
    +          val minCount = PoissonBounds.getMinCount(lmbd1)
    --- End diff --
    
    Having `minCount` here. I think it is only useful in streaming environments but we didn't implement it in that way.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1025#discussion_r14694294
  
    --- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
    @@ -0,0 +1,311 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.util.random
    +
    +import scala.collection.Map
    +import scala.collection.mutable.{ArrayBuffer, HashMap, Map => MMap}
    +
    +import org.apache.commons.math3.random.RandomDataGenerator
    +import org.apache.spark.Logging
    +import org.apache.spark.rdd.RDD
    +
    +/**
    + * Auxiliary functions and data structures for the sampleByKey method in PairRDDFunctions.
    + *
    + * Essentially, when exact sample size is necessary, we make additional passes over the RDD to
    + * compute the exact threshold value to use for each stratum to guarantee exact sample size with
    + * high probability. This is achieved by maintaining a waitlist of size O(log(s)), where s is the
    + * desired sample size for each stratum.
    + *
    + * Like in simple random sampling, we generate a random value for each item from the
    + * uniform  distribution [0.0, 1.0]. All items with values <= min(values of items in the waitlist)
    + * are accepted into the sample instantly. The threshold for instant accept is designed so that
    + * s - numAccepted = O(log(s)), where s is again the desired sample size. Thus, by maintaining a
    + * waitlist size = O(log(s)), we will be able to create a sample of the exact size s by adding
    + * a portion of the waitlist to the set of items that are instantly accepted. The exact threshold
    + * is computed by sorting the values in the waitlist and picking the value at (s - numAccepted).
    + *
    + * Note that since we use the same seed for the RNG when computing the thresholds and the actual
    + * sample, our computed thresholds are guaranteed to produce the desired sample size.
    + *
    + * For more theoretical background on the sampling techniques used here, please refer to
    + * http://jmlr.org/proceedings/papers/v28/meng13a.html
    + */
    +
    +private[spark] object StratifiedSampler extends Logging {
    +
    +  /**
    +   * Count the number of items instantly accepted and generate the waitlist for each stratum.
    +   *
    +   * This is only invoked when exact sample size is required.
    +   */
    +  def getCounts[K, V](rdd: RDD[(K, V)],
    +      withReplacement: Boolean,
    +      fractions: Map[K, Double],
    +      counts: Option[Map[K, Long]],
    +      seed: Long): MMap[K, Stratum] = {
    +    val combOp = getCombOp[K]
    +    val mappedPartitionRDD = rdd.mapPartitionsWithIndex({ case (partition, iter) =>
    +      val zeroU: MMap[K, Stratum] = new HashMap[K, Stratum]()
    +      val rng = new RandomDataGenerator()
    +      rng.reSeed(seed + partition)
    +      val seqOp = getSeqOp(withReplacement, fractions, rng, counts)
    +      Iterator(iter.aggregate(zeroU)(seqOp, combOp))
    +    }, preservesPartitioning=true)
    +    mappedPartitionRDD.reduce(combOp)
    +  }
    +
    +  /**
    +   * Returns the function used by aggregate to collect sampling statistics for each partition.
    +   */
    +  def getSeqOp[K, V](withReplacement: Boolean,
    +      fractions: Map[K, Double],
    +      rng: RandomDataGenerator,
    +      counts: Option[Map[K, Long]]): (MMap[K, Stratum], (K, V)) => MMap[K, Stratum] = {
    +    val delta = 5e-5
    +    (result: MMap[K, Stratum], item: (K, V)) => {
    +      val key = item._1
    +      val fraction = fractions(key)
    +      if (!result.contains(key)) {
    +        result += (key -> new Stratum())
    +      }
    +      val stratum = result(key)
    +
    +      if (withReplacement) {
    +        // compute acceptBound and waitListBound only if they haven't been computed already
    +        // since they don't change from iteration to iteration.
    +        // TODO change this to the streaming version
    +        if (stratum.areBoundsEmpty) {
    +          val n = counts.get(key)
    +          val sampleSize = math.ceil(n * fraction).toLong
    +          val lmbd1 = PoissonBounds.getLowerBound(sampleSize)
    +          val minCount = PoissonBounds.getMinCount(lmbd1)
    +          val lmbd2 = if (lmbd1 == 0) {
    +            PoissonBounds.getUpperBound(sampleSize)
    +          } else {
    +            PoissonBounds.getUpperBound(sampleSize - minCount)
    +          }
    +          stratum.acceptBound = lmbd1 / n
    +          stratum.waitListBound = lmbd2 / n
    +        }
    +        val acceptBound = stratum.acceptBound
    +        val copiesAccepted = if (acceptBound == 0.0) 0L else rng.nextPoisson(acceptBound)
    +        if (copiesAccepted > 0) {
    +          stratum.incrNumAccepted(copiesAccepted)
    +        }
    +        val copiesWaitlisted = rng.nextPoisson(stratum.waitListBound).toInt
    +        if (copiesWaitlisted > 0) {
    +          stratum.addToWaitList(ArrayBuffer.fill(copiesWaitlisted)(rng.nextUniform(0.0, 1.0)))
    +        }
    +      } else {
    +        // We use the streaming version of the algorithm for sampling without replacement to avoid
    +        // using an extra pass over the RDD for computing the count.
    +        // Hence, acceptBound and waitListBound change on every iteration.
    +        val gamma1 = - math.log(delta) / stratum.numItems
    +        val gamma2 = (2.0 / 3.0) * gamma1
    +        stratum.acceptBound = math.max(0,
    +          fraction + gamma2 - math.sqrt(gamma2 * gamma2 + 3 * gamma2 * fraction))
    +        stratum.waitListBound = math.min(1,
    +          fraction + gamma1 + math.sqrt(gamma1 * gamma1 + 2 * gamma1 * fraction))
    +
    +        val x = rng.nextUniform(0.0, 1.0)
    +        if (x < stratum.acceptBound) {
    +          stratum.incrNumAccepted()
    +        } else if (x < stratum.waitListBound) {
    +          stratum.addToWaitList(x)
    +        }
    +      }
    +      stratum.incrNumItems()
    +      result
    +    }
    +  }
    +
    +  /**
    +   * Returns the function used combine results returned by seqOp from different partitions.
    +   */
    +  def getCombOp[K]: (MMap[K, Stratum], MMap[K, Stratum]) => MMap[K, Stratum] = {
    +    (result1: MMap[K, Stratum], result2: MMap[K, Stratum]) => {
    +      // take union of both key sets in case one partition doesn't contain all keys
    +      for (key <- result1.keySet.union(result2.keySet)) {
    +        // Use result2 to keep the combined result since r1 is usual empty
    +        val entry1 = result1.get(key)
    +        if (result2.contains(key)) {
    +          result2(key).merge(entry1)
    +        } else {
    +          if (entry1.isDefined) {
    +            result2 += (key -> entry1.get)
    +          }
    +        }
    +      }
    +      result2
    +    }
    +  }
    +
    +  /**
    +   * Given the result returned by getCounts, determine the threshold for accepting items to
    +   * generate exact sample size.
    +   *
    +   * To do so, we compute sampleSize = math.ceil(size * samplingRate) for each stratum and compare
    +   * it to the number of items that were accepted instantly and the number of items in the waitlist
    +   * for that stratum. Most of the time, numAccepted <= sampleSize <= (numAccepted + numWaitlisted),
    +   * which means we need to sort the elements in the waitlist by their associated values in order
    +   * to find the value T s.t. |{elements in the stratum whose associated values <= T}| = sampleSize.
    +   * Note that all elements in the waitlist have values >= bound for instant accept, so a T value
    +   * in the waitlist range would allow all elements that were instantly accepted on the first pass
    +   * to be included in the sample.
    +   */
    +  def computeThresholdByKey[K](finalResult: Map[K, Stratum],
    +      fractions: Map[K, Double]): Map[K, Double] = {
    +    val thresholdByKey = new HashMap[K, Double]()
    +    for ((key, stratum) <- finalResult) {
    +      val sampleSize = math.ceil(stratum.numItems * fractions(key)).toLong
    +      if (stratum.numAccepted > sampleSize) {
    +        logWarning("Pre-accepted too many")
    +        thresholdByKey += (key -> stratum.acceptBound)
    +      } else {
    +        val numWaitListAccepted = (sampleSize - stratum.numAccepted).toInt
    +        if (numWaitListAccepted >= stratum.waitList.size) {
    +          logWarning("WaitList too short")
    +          thresholdByKey += (key -> stratum.waitListBound)
    +        } else {
    +          thresholdByKey += (key -> stratum.waitList.sorted.apply(numWaitListAccepted))
    +        }
    +      }
    +    }
    +    thresholdByKey
    +  }
    +
    +  /**
    +   * Return the per partition sampling function used for sampling without replacement.
    +   *
    +   * When exact sample size is required, we make an additional pass over the RDD to determine the
    +   * exact sampling rate that guarantees sample size with high confidence.
    +   *
    +   * The sampling function has a unique seed per partition.
    +   */
    +  def getBernoulliSamplingFunction[K, V](rdd: RDD[(K,  V)],
    +      fractions: Map[K, Double],
    +      exact: Boolean,
    +      seed: Long): (Int, Iterator[(K, V)]) => Iterator[(K, V)] = {
    +    var samplingRateByKey = fractions
    +    if (exact) {
    +      // determine threshold for each stratum and resample
    +      val finalResult = getCounts(rdd, false, fractions, None, seed)
    +      samplingRateByKey = computeThresholdByKey(finalResult, fractions)
    +    }
    +    (idx: Int, iter: Iterator[(K, V)]) => {
    +      val rng = new RandomDataGenerator
    +      rng.reSeed(seed + idx)
    +      iter.filter(t => rng.nextUniform(0.0, 1.0) < samplingRateByKey(t._1))
    +    }
    +  }
    +
    +  /**
    +   * Return the per partition sampling function used for sampling with replacement.
    +   *
    +   * When exact sample size is required, we make two additional passed over the RDD to determine
    +   * the exact sampling rate that guarantees sample size with high confidence. The first pass
    +   * counts the number of items in each stratum (group of items with the same key) in the RDD, and
    +   * the second pass uses the counts to determine exact sampling rates.
    +   *
    +   * The sampling function has a unique seed per partition.
    +   */
    +  def getPoissonSamplingFunction[K, V](rdd: RDD[(K, V)],
    +      fractions: Map[K, Double],
    +      exact: Boolean,
    +      counts: Option[Map[K, Long]],
    +      seed: Long): (Int, Iterator[(K, V)]) => Iterator[(K, V)] = {
    +    // TODO implement the streaming version of sampling w/ replacement that doesn't require counts
    +    if (exact) {
    +      val finalResult = getCounts(rdd, true, fractions, counts, seed)
    +      val thresholdByKey = computeThresholdByKey(finalResult, fractions)
    +      (idx: Int, iter: Iterator[(K, V)]) => {
    +        val rng = new RandomDataGenerator()
    +        rng.reSeed(seed + idx)
    +        iter.flatMap { item =>
    +          val key = item._1
    +          val acceptBound = finalResult(key).acceptBound
    +          val copiesAccepted = if (acceptBound == 0) 0L else rng.nextPoisson(acceptBound)
    +          val copiesWailisted = rng.nextPoisson(finalResult(key).waitListBound).toInt
    +          val copiesInSample = copiesAccepted +
    +            (0 until copiesWailisted).count(i => rng.nextUniform(0.0, 1.0) < thresholdByKey(key))
    +          if (copiesInSample > 0) {
    +            Iterator.fill(copiesInSample.toInt)(item)
    +          } else {
    +            Iterator.empty
    +          }
    +        }
    +      }
    +    } else {
    +      (idx: Int, iter: Iterator[(K, V)]) => {
    +        val rng = new RandomDataGenerator()
    +        rng.reSeed(seed + idx)
    +        iter.flatMap { item =>
    +          val count = rng.nextPoisson(fractions(item._1)).toInt
    +          if (count > 0) {
    +            Iterator.fill(count)(item)
    +          } else {
    +            Iterator.empty
    +          }
    +        }
    +      }
    +    }
    +  }
    +}
    +
    +/**
    + * Object used by seqOp to keep track of the number of items accepted and items waitlisted per
    + * stratum, as well as the bounds for accepting and waitlisting items.
    + *
    + * `[random]` here is necessary since it's in the return type signature of seqOp defined above
    --- End diff --
    
    Not necessary.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1025#issuecomment-48422316
  
    Merged build finished. All automated tests passed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1025#issuecomment-46389585
  
     Merged build triggered. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1025#issuecomment-45553453
  
    Build finished. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1025#issuecomment-46381896
  
    Build finished. All automated tests passed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by dorx <gi...@git.apache.org>.
Github user dorx commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1025#discussion_r14578266
  
    --- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
    @@ -0,0 +1,296 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.util.random
    +
    +import scala.collection.{Map, mutable}
    +import scala.collection.mutable.ArrayBuffer
    +import scala.reflect.ClassTag
    +
    +import org.apache.commons.math3.random.RandomDataGenerator
    +import org.apache.spark.{Logging, SparkContext, TaskContext}
    +import org.apache.spark.rdd.RDD
    +import org.apache.spark.util.Utils
    +import org.apache.spark.util.random.{PoissonBounds => PB}
    +
    +private[spark] object StratifiedSampler extends Logging {
    +
    +  /**
    +   * A version of {@link #aggregate()} that passes the TaskContext to the function that does
    +   * aggregation for each partition. This function avoids creating an extra depth in the RDD
    +   * lineage, as opposed to using mapPartitionsWithId, which results in slightly improved run time.
    +   */
    +  def aggregateWithContext[U: ClassTag, T: ClassTag](zeroValue: U)
    +      (rdd: RDD[T],
    +       seqOp: ((TaskContext, U), T) => U,
    +       combOp: (U, U) => U): U = {
    +    val sc: SparkContext = rdd.sparkContext
    +    // Clone the zero value since we will also be serializing it as part of tasks
    +    var jobResult = Utils.clone(zeroValue, sc.env.closureSerializer.newInstance())
    +    // pad seqOp and combOp with taskContext to conform to aggregate's signature in TraversableOnce
    +    val paddedSeqOp = (arg1: (TaskContext, U), item: T) => (arg1._1, seqOp(arg1, item))
    +    val paddedcombOp = (arg1: (TaskContext, U), arg2: (TaskContext, U)) =>
    +      (arg1._1, combOp(arg1._2, arg1._2))
    +    val cleanSeqOp = sc.clean(paddedSeqOp)
    +    val cleanCombOp = sc.clean(paddedcombOp)
    +    val aggregatePartition = (tc: TaskContext, it: Iterator[T]) =>
    +      (it.aggregate(tc, zeroValue)(cleanSeqOp, cleanCombOp))._2
    +    val mergeResult = (index: Int, taskResult: U) => jobResult = combOp(jobResult, taskResult)
    +    sc.runJob(rdd, aggregatePartition, mergeResult)
    +    jobResult
    +  }
    +
    +  /**
    +   * Returns the function used by aggregate to collect sampling statistics for each partition.
    +   */
    +  def getSeqOp[K, V](withReplacement: Boolean,
    +      fractionByKey: (K => Double),
    +      counts: Option[Map[K, Long]]): ((TaskContext, Result[K]),(K, V)) => Result[K] = {
    +    val delta = 5e-5
    +    (U: (TaskContext, Result[K]), item: (K, V)) => {
    +      val result = U._2
    +      val tc = U._1
    +      val rng = result.getRand(tc.partitionId)
    +      val fraction = fractionByKey(item._1)
    +      val stratum = result.getEntry(item._1)
    +      if (withReplacement) {
    +        // compute q1 and q2 only if they haven't been computed already
    +        // since they don't change from iteration to iteration.
    +        // TODO change this to the streaming version
    +        if (stratum.q1.isEmpty || stratum.q2.isEmpty) {
    +          val n = counts.get(item._1)
    +          val s = math.ceil(n * fraction).toLong
    +          val lmbd1 = PB.getLowerBound(s)
    +          val minCount = PB.getMinCount(lmbd1)
    +          val lmbd2 = if (lmbd1 == 0) PB.getUpperBound(s) else PB.getUpperBound(s - minCount)
    +          val q1 = lmbd1 / n
    +          val q2 = lmbd2 / n
    +          stratum.q1 = Some(q1)
    +          stratum.q2 = Some(q2)
    +        }
    +        val x1 = if (stratum.q1.get == 0) 0L else rng.nextPoisson(stratum.q1.get)
    +        if (x1 > 0) {
    +          stratum.incrNumAccepted(x1)
    +        }
    +        val x2 = rng.nextPoisson(stratum.q2.get).toInt
    +        if (x2 > 0) {
    +          stratum.addToWaitList(ArrayBuffer.fill(x2)(rng.nextUniform(0.0, 1.0)))
    +        }
    +      } else {
    +        // We use the streaming version of the algorithm for sampling without replacement.
    +        // Hence, q1 and q2 change on every iteration.
    +        val g1 = - math.log(delta) / stratum.numItems
    +        val g2 = (2.0 / 3.0) * g1
    +        val q1 = math.max(0, fraction + g2 - math.sqrt((g2 * g2 + 3 * g2 * fraction)))
    +        val q2 = math.min(1, fraction + g1 + math.sqrt(g1 * g1 + 2 * g1 * fraction))
    +
    +        val x = rng.nextUniform(0.0, 1.0)
    +        if (x < q1) {
    +          stratum.incrNumAccepted()
    +        } else if (x < q2) {
    +          stratum.addToWaitList(x)
    +        }
    +        stratum.q1 = Some(q1)
    +        stratum.q2 = Some(q2)
    +      }
    +      stratum.incrNumItems()
    +      result
    +    }
    +  }
    +
    +  /**
    +   * Returns the function used by aggregate to combine results from different partitions, as
    +   * returned by seqOp.
    +   */
    +  def getCombOp[K](): (Result[K], Result[K]) => Result[K] = {
    +    (r1: Result[K], r2: Result[K]) => {
    +      // take union of both key sets in case one partition doesn't contain all keys
    +      val keyUnion = r1.resultMap.keys.toSet.union(r2.resultMap.keys.toSet)
    +
    +      // Use r2 to keep the combined result since r1 is usual empty
    +      for (key <- keyUnion) {
    +        val entry1 = r1.resultMap.get(key)
    +        val entry2 = r2.resultMap.get(key)
    +        if (entry2.isEmpty && entry1.isDefined) {
    +          r2.resultMap += (key -> entry1.get)
    +        } else if (entry1.isDefined && entry2.isDefined) {
    +          entry2.get.addToWaitList(entry1.get.waitList)
    +          entry2.get.incrNumAccepted(entry1.get.numAccepted)
    +          entry2.get.incrNumItems(entry1.get.numItems)
    +        }
    +      }
    +      r2
    +    }
    +  }
    +
    +  /**
    +   * Given the result returned by the aggregate function, we need to determine the threshold used
    +   * to accept items to generate the exact sample size.
    +   */
    +  def computeThresholdByKey[K](finalResult: Map[K, Stratum], fractionByKey: (K => Double)):
    +    (K => Double) = {
    +    val thresholdByKey = new mutable.HashMap[K, Double]()
    +    for ((key, stratum) <- finalResult) {
    +      val fraction = fractionByKey(key)
    +      val s = math.ceil(stratum.numItems * fraction).toLong
    +      if (stratum.numAccepted > s) {
    +        logWarning("Pre-accepted too many")
    +        thresholdByKey += (key -> stratum.q1.get)
    +      } else {
    +        val numWaitListAccepted = (s - stratum.numAccepted).toInt
    +        if (numWaitListAccepted >= stratum.waitList.size) {
    +          logWarning("WaitList too short")
    +          thresholdByKey += (key -> stratum.q2.get)
    +        } else {
    +          thresholdByKey += (key -> stratum.waitList.sorted.apply(numWaitListAccepted))
    +        }
    +      }
    +    }
    +    thresholdByKey
    +  }
    +
    +  /**
    +   * Return the per partition sampling function used for sampling without replacement.
    +   *
    +   * When exact sample size is required, we make an additional pass over the RDD to determine the
    +   * exact sampling rate that guarantees sample size with high confidence.
    +   *
    +   * The sampling function has a unique seed per partition.
    +   */
    +  def getBernoulliSamplingFunction[K, V](rdd: RDD[(K,  V)],
    +      fractionByKey: K => Double,
    +      exact: Boolean,
    +      seed: Long): (Int, Iterator[(K, V)]) => Iterator[(K, V)] = {
    +    var samplingRateByKey = fractionByKey
    +    if (exact) {
    +      // determine threshold for each stratum and resample
    +      val seqOp = StratifiedSampler.getSeqOp[K,V](false, fractionByKey, None)
    +      val combOp = StratifiedSampler.getCombOp[K]()
    +      val zeroU = new Result[K](Map[K, Stratum](), seed = seed)
    +      val finalResult = aggregateWithContext(zeroU)(rdd, seqOp, combOp).resultMap
    +      samplingRateByKey = StratifiedSampler.computeThresholdByKey(finalResult, fractionByKey)
    +    }
    +    (idx: Int, iter: Iterator[(K, V)]) => {
    +      val random = new RandomDataGenerator
    +      random.reSeed(seed + idx)
    +      iter.filter(t => random.nextUniform(0.0, 1.0) < samplingRateByKey(t._1))
    +    }
    +  }
    +
    +  /**
    +   * Return the per partition sampling function used for sampling with replacement.
    +   *
    +   * When exact sample size is required, we make two additional passed over the RDD to determine
    +   * the exact sampling rate that guarantees sample size with high confidence. The first pass
    +   * counts the number of items in each stratum (group of items with the same key) in the RDD, and
    +   * the second pass uses the counts to determine exact sampling rates.
    +   *
    +   * The sampling function has a unique seed per partition.
    +   */
    +  def getPoissonSamplingFunction[K, V](rdd:RDD[(K,  V)],
    +      fractionByKey: K => Double,
    +      exact: Boolean,
    +      counts: Option[Map[K, Long]],
    +      seed: Long): (Int, Iterator[(K, V)]) => Iterator[(K, V)] = {
    +    // TODO implement the streaming version of sampling w/ replacement that doesn't require counts
    +    if (exact) {
    +      val seqOp = StratifiedSampler.getSeqOp[K,V](true, fractionByKey, counts)
    +      val combOp = StratifiedSampler.getCombOp[K]()
    +      val zeroU = new Result[K](Map[K, Stratum](), seed = seed)
    +      val finalResult = aggregateWithContext(zeroU)(rdd, seqOp, combOp).resultMap
    +      val thresholdByKey = StratifiedSampler.computeThresholdByKey(finalResult, fractionByKey)
    +      (idx: Int, iter: Iterator[(K, V)]) => {
    +        val random = new RandomDataGenerator()
    +        random.reSeed(seed + idx)
    +        iter.flatMap { t =>
    +          val q1 = finalResult.get(t._1).get.q1.getOrElse(0.0)
    +          val q2 = finalResult.get(t._1).get.q2.getOrElse(0.0)
    +          val x1 = if (q1 == 0) 0L else random.nextPoisson(q1)
    +          val x2 = random.nextPoisson(q2).toInt
    +          val x = x1 + (0 until x2).count(i => random.nextUniform(0.0, 1.0) < thresholdByKey(t._1))
    +          if (x > 0) {
    +            Iterator.fill(x.toInt)(t)
    +          } else {
    +            Iterator.empty
    +          }
    +        }
    +      }
    +    } else {
    +      (idx: Int, iter: Iterator[(K, V)]) => {
    +        val random = new RandomDataGenerator()
    +        random.reSeed(seed + idx)
    +        iter.flatMap { t =>
    +          val count = random.nextPoisson(fractionByKey(t._1)).toInt
    +          if (count > 0) {
    +            Iterator.fill(count)(t)
    +          } else {
    +            Iterator.empty
    +          }
    +        }
    +      }
    +    }
    +  }
    +}
    +
    +/**
    + * Object used by seqOp to keep track of the number of items accepted and items waitlisted per
    + * stratum, as well as the bounds for accepting and waitlisting items.
    + */
    +private[random] class Stratum(var numItems: Long = 0L, var numAccepted: Long = 0L)
    +  extends Serializable {
    +
    +  var waitList: ArrayBuffer[Double] = new ArrayBuffer[Double]
    +  var q1: Option[Double] = None // upper bound for accepting item instantly
    +  var q2: Option[Double] = None // upper bound for adding item to waitlist
    +
    +  def incrNumItems(by: Long = 1L) = numItems += by
    +
    +  def incrNumAccepted(by: Long = 1L) = numAccepted += by
    +
    +  def addToWaitList(elem: Double) = waitList += elem
    +
    +  def addToWaitList(elems: ArrayBuffer[Double]) = waitList ++= elems
    +}
    +
    +/**
    + * Object used by seqOp and combOp to keep track of the sampling statistics for all strata.
    + *
    + * When used by seqOp for each partition, we also keep track of the partition ID in this object
    + * to make sure a single random number generator with a unique seed is used for each partition.
    + */
    +private[random] class Result[K](var resultMap: Map[K, Stratum],
    --- End diff --
    
    Actually if I remove the "[random]" here I get an out-of-scope error: "private class Result escapes its defining scope as part of type (org.apache.spark.util.random.Result[K], org.apache.spark.util.random.Result[K]) => org.apache.spark.util.random.Result[K]"


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by falaki <gi...@git.apache.org>.
Github user falaki commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1025#discussion_r14542021
  
    --- Diff: core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala ---
    @@ -195,6 +193,40 @@ class PairRDDFunctions[K, V](self: RDD[(K, V)])
       }
     
       /**
    +   * Return a subset of this RDD sampled by key (via stratified sampling).
    +   *
    +   * If exact set to true, we guarantee, with high probability, a sample size =
    +   * math.ceil(fraction * S_i), where S_i is the size of the ith stratum (collection of entries
    +   * that share the same key). When sampling without replacement, we need one additional pass over
    +   * the RDD to guarantee sample size with a 99.99% confidence; when sampling with replacement, we
    +   * need two additional passes.
    +   *
    +   * @param withReplacement whether to sample with or without replacement
    +   * @param fractions map of specific keys to sampling rates
    +   * @param seed seed for the random number generator
    +   * @param exact whether sample size needs to be exactly math.ceil(fraction * size) per stratum
    +   * @return RDD containing the sampled subset
    +   */
    +  def sampleByKey(withReplacement: Boolean,
    +      fractions: Map[K, Double],
    +      exact: Boolean = true,
    +      seed: Long = Utils.random.nextLong): RDD[(K, V)]= {
    +
    +    require(fractions.forall({case(k, v) => v >= 0.0}), "Invalid sampling rates.")
    +
    +    if (withReplacement) {
    --- End diff --
    
    This can be written more functional (and readable) like this:
    ```scala
        val samplingFunc = if (withReplacement) {
          val counts = if (exact) Some(this.countByKey()) else None
            StratifiedSampler.getPoissonSamplingFunction(self, fractions, exact, counts, seed)
        } else {
            StratifiedSampler.getBernoulliSamplingFunction(self, fractions, exact, seed)
        }
        self.mapPartitionsWithIndex(samplingFunc, preservesPartitioning = true)
    ```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by falaki <gi...@git.apache.org>.
Github user falaki commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1025#discussion_r14542085
  
    --- Diff: core/src/main/scala/org/apache/spark/rdd/RDD.scala ---
    @@ -350,6 +350,7 @@ abstract class RDD[T: ClassTag](
     
       /**
        * Return a sampled subset of this RDD.
    +   *
    --- End diff --
    
    Let's remove this line.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1025#issuecomment-46381897
  
    All automated tests passed.
    Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15858/


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1025#issuecomment-45558759
  
    Build finished. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1025#issuecomment-46518419
  
    Merged build started. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by falaki <gi...@git.apache.org>.
Github user falaki commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1025#discussion_r14544573
  
    --- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
    @@ -0,0 +1,296 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.util.random
    +
    +import scala.collection.{Map, mutable}
    +import scala.collection.mutable.ArrayBuffer
    +import scala.reflect.ClassTag
    +
    +import org.apache.commons.math3.random.RandomDataGenerator
    +import org.apache.spark.{Logging, SparkContext, TaskContext}
    +import org.apache.spark.rdd.RDD
    +import org.apache.spark.util.Utils
    +import org.apache.spark.util.random.{PoissonBounds => PB}
    +
    +private[spark] object StratifiedSampler extends Logging {
    +
    +  /**
    +   * A version of {@link #aggregate()} that passes the TaskContext to the function that does
    +   * aggregation for each partition. This function avoids creating an extra depth in the RDD
    +   * lineage, as opposed to using mapPartitionsWithId, which results in slightly improved run time.
    +   */
    +  def aggregateWithContext[U: ClassTag, T: ClassTag](zeroValue: U)
    +      (rdd: RDD[T],
    +       seqOp: ((TaskContext, U), T) => U,
    +       combOp: (U, U) => U): U = {
    +    val sc: SparkContext = rdd.sparkContext
    +    // Clone the zero value since we will also be serializing it as part of tasks
    +    var jobResult = Utils.clone(zeroValue, sc.env.closureSerializer.newInstance())
    +    // pad seqOp and combOp with taskContext to conform to aggregate's signature in TraversableOnce
    +    val paddedSeqOp = (arg1: (TaskContext, U), item: T) => (arg1._1, seqOp(arg1, item))
    +    val paddedcombOp = (arg1: (TaskContext, U), arg2: (TaskContext, U)) =>
    +      (arg1._1, combOp(arg1._2, arg1._2))
    +    val cleanSeqOp = sc.clean(paddedSeqOp)
    +    val cleanCombOp = sc.clean(paddedcombOp)
    +    val aggregatePartition = (tc: TaskContext, it: Iterator[T]) =>
    +      (it.aggregate(tc, zeroValue)(cleanSeqOp, cleanCombOp))._2
    +    val mergeResult = (index: Int, taskResult: U) => jobResult = combOp(jobResult, taskResult)
    +    sc.runJob(rdd, aggregatePartition, mergeResult)
    +    jobResult
    +  }
    +
    +  /**
    +   * Returns the function used by aggregate to collect sampling statistics for each partition.
    +   */
    +  def getSeqOp[K, V](withReplacement: Boolean,
    +      fractionByKey: (K => Double),
    +      counts: Option[Map[K, Long]]): ((TaskContext, Result[K]),(K, V)) => Result[K] = {
    +    val delta = 5e-5
    +    (U: (TaskContext, Result[K]), item: (K, V)) => {
    +      val result = U._2
    +      val tc = U._1
    +      val rng = result.getRand(tc.partitionId)
    +      val fraction = fractionByKey(item._1)
    +      val stratum = result.getEntry(item._1)
    +      if (withReplacement) {
    +        // compute q1 and q2 only if they haven't been computed already
    +        // since they don't change from iteration to iteration.
    +        // TODO change this to the streaming version
    +        if (stratum.q1.isEmpty || stratum.q2.isEmpty) {
    +          val n = counts.get(item._1)
    +          val s = math.ceil(n * fraction).toLong
    +          val lmbd1 = PB.getLowerBound(s)
    +          val minCount = PB.getMinCount(lmbd1)
    +          val lmbd2 = if (lmbd1 == 0) PB.getUpperBound(s) else PB.getUpperBound(s - minCount)
    +          val q1 = lmbd1 / n
    +          val q2 = lmbd2 / n
    +          stratum.q1 = Some(q1)
    +          stratum.q2 = Some(q2)
    +        }
    +        val x1 = if (stratum.q1.get == 0) 0L else rng.nextPoisson(stratum.q1.get)
    +        if (x1 > 0) {
    +          stratum.incrNumAccepted(x1)
    +        }
    +        val x2 = rng.nextPoisson(stratum.q2.get).toInt
    +        if (x2 > 0) {
    +          stratum.addToWaitList(ArrayBuffer.fill(x2)(rng.nextUniform(0.0, 1.0)))
    +        }
    +      } else {
    +        // We use the streaming version of the algorithm for sampling without replacement.
    +        // Hence, q1 and q2 change on every iteration.
    +        val g1 = - math.log(delta) / stratum.numItems
    +        val g2 = (2.0 / 3.0) * g1
    +        val q1 = math.max(0, fraction + g2 - math.sqrt((g2 * g2 + 3 * g2 * fraction)))
    +        val q2 = math.min(1, fraction + g1 + math.sqrt(g1 * g1 + 2 * g1 * fraction))
    +
    +        val x = rng.nextUniform(0.0, 1.0)
    +        if (x < q1) {
    +          stratum.incrNumAccepted()
    +        } else if (x < q2) {
    +          stratum.addToWaitList(x)
    +        }
    +        stratum.q1 = Some(q1)
    +        stratum.q2 = Some(q2)
    +      }
    +      stratum.incrNumItems()
    +      result
    +    }
    +  }
    +
    +  /**
    +   * Returns the function used by aggregate to combine results from different partitions, as
    +   * returned by seqOp.
    +   */
    +  def getCombOp[K](): (Result[K], Result[K]) => Result[K] = {
    +    (r1: Result[K], r2: Result[K]) => {
    +      // take union of both key sets in case one partition doesn't contain all keys
    +      val keyUnion = r1.resultMap.keys.toSet.union(r2.resultMap.keys.toSet)
    +
    +      // Use r2 to keep the combined result since r1 is usual empty
    +      for (key <- keyUnion) {
    +        val entry1 = r1.resultMap.get(key)
    +        val entry2 = r2.resultMap.get(key)
    +        if (entry2.isEmpty && entry1.isDefined) {
    +          r2.resultMap += (key -> entry1.get)
    +        } else if (entry1.isDefined && entry2.isDefined) {
    +          entry2.get.addToWaitList(entry1.get.waitList)
    +          entry2.get.incrNumAccepted(entry1.get.numAccepted)
    +          entry2.get.incrNumItems(entry1.get.numItems)
    +        }
    +      }
    +      r2
    +    }
    +  }
    +
    +  /**
    +   * Given the result returned by the aggregate function, we need to determine the threshold used
    +   * to accept items to generate the exact sample size.
    +   */
    +  def computeThresholdByKey[K](finalResult: Map[K, Stratum], fractionByKey: (K => Double)):
    +    (K => Double) = {
    +    val thresholdByKey = new mutable.HashMap[K, Double]()
    +    for ((key, stratum) <- finalResult) {
    +      val fraction = fractionByKey(key)
    +      val s = math.ceil(stratum.numItems * fraction).toLong
    +      if (stratum.numAccepted > s) {
    +        logWarning("Pre-accepted too many")
    +        thresholdByKey += (key -> stratum.q1.get)
    +      } else {
    +        val numWaitListAccepted = (s - stratum.numAccepted).toInt
    +        if (numWaitListAccepted >= stratum.waitList.size) {
    +          logWarning("WaitList too short")
    +          thresholdByKey += (key -> stratum.q2.get)
    +        } else {
    +          thresholdByKey += (key -> stratum.waitList.sorted.apply(numWaitListAccepted))
    +        }
    +      }
    +    }
    +    thresholdByKey
    +  }
    +
    +  /**
    +   * Return the per partition sampling function used for sampling without replacement.
    +   *
    +   * When exact sample size is required, we make an additional pass over the RDD to determine the
    +   * exact sampling rate that guarantees sample size with high confidence.
    +   *
    +   * The sampling function has a unique seed per partition.
    +   */
    +  def getBernoulliSamplingFunction[K, V](rdd: RDD[(K,  V)],
    +      fractionByKey: K => Double,
    +      exact: Boolean,
    +      seed: Long): (Int, Iterator[(K, V)]) => Iterator[(K, V)] = {
    +    var samplingRateByKey = fractionByKey
    +    if (exact) {
    +      // determine threshold for each stratum and resample
    +      val seqOp = StratifiedSampler.getSeqOp[K,V](false, fractionByKey, None)
    +      val combOp = StratifiedSampler.getCombOp[K]()
    +      val zeroU = new Result[K](Map[K, Stratum](), seed = seed)
    +      val finalResult = aggregateWithContext(zeroU)(rdd, seqOp, combOp).resultMap
    +      samplingRateByKey = StratifiedSampler.computeThresholdByKey(finalResult, fractionByKey)
    +    }
    +    (idx: Int, iter: Iterator[(K, V)]) => {
    +      val random = new RandomDataGenerator
    +      random.reSeed(seed + idx)
    +      iter.filter(t => random.nextUniform(0.0, 1.0) < samplingRateByKey(t._1))
    +    }
    +  }
    +
    +  /**
    +   * Return the per partition sampling function used for sampling with replacement.
    +   *
    +   * When exact sample size is required, we make two additional passed over the RDD to determine
    +   * the exact sampling rate that guarantees sample size with high confidence. The first pass
    +   * counts the number of items in each stratum (group of items with the same key) in the RDD, and
    +   * the second pass uses the counts to determine exact sampling rates.
    +   *
    +   * The sampling function has a unique seed per partition.
    +   */
    +  def getPoissonSamplingFunction[K, V](rdd:RDD[(K,  V)],
    +      fractionByKey: K => Double,
    +      exact: Boolean,
    +      counts: Option[Map[K, Long]],
    +      seed: Long): (Int, Iterator[(K, V)]) => Iterator[(K, V)] = {
    +    // TODO implement the streaming version of sampling w/ replacement that doesn't require counts
    +    if (exact) {
    +      val seqOp = StratifiedSampler.getSeqOp[K,V](true, fractionByKey, counts)
    +      val combOp = StratifiedSampler.getCombOp[K]()
    +      val zeroU = new Result[K](Map[K, Stratum](), seed = seed)
    +      val finalResult = aggregateWithContext(zeroU)(rdd, seqOp, combOp).resultMap
    +      val thresholdByKey = StratifiedSampler.computeThresholdByKey(finalResult, fractionByKey)
    +      (idx: Int, iter: Iterator[(K, V)]) => {
    +        val random = new RandomDataGenerator()
    +        random.reSeed(seed + idx)
    +        iter.flatMap { t =>
    +          val q1 = finalResult.get(t._1).get.q1.getOrElse(0.0)
    +          val q2 = finalResult.get(t._1).get.q2.getOrElse(0.0)
    +          val x1 = if (q1 == 0) 0L else random.nextPoisson(q1)
    +          val x2 = random.nextPoisson(q2).toInt
    +          val x = x1 + (0 until x2).count(i => random.nextUniform(0.0, 1.0) < thresholdByKey(t._1))
    +          if (x > 0) {
    +            Iterator.fill(x.toInt)(t)
    +          } else {
    +            Iterator.empty
    +          }
    +        }
    +      }
    +    } else {
    +      (idx: Int, iter: Iterator[(K, V)]) => {
    +        val random = new RandomDataGenerator()
    +        random.reSeed(seed + idx)
    +        iter.flatMap { t =>
    +          val count = random.nextPoisson(fractionByKey(t._1)).toInt
    +          if (count > 0) {
    +            Iterator.fill(count)(t)
    +          } else {
    +            Iterator.empty
    +          }
    +        }
    +      }
    +    }
    +  }
    +}
    +
    +/**
    + * Object used by seqOp to keep track of the number of items accepted and items waitlisted per
    + * stratum, as well as the bounds for accepting and waitlisting items.
    + */
    +private[random] class Stratum(var numItems: Long = 0L, var numAccepted: Long = 0L)
    +  extends Serializable {
    +
    +  var waitList: ArrayBuffer[Double] = new ArrayBuffer[Double]
    +  var q1: Option[Double] = None // upper bound for accepting item instantly
    --- End diff --
    
    We are setting q1 and q2 from outside, let's add more methods to this class for setting and getting values of q1 and q2. Let's use better names than ```updateq1()```. Even better would be if we rename q1 and q2 to more descriptive names (the comments you added help a lot by the way).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by dorx <gi...@git.apache.org>.
Github user dorx commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1025#discussion_r14907781
  
    --- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
    @@ -0,0 +1,311 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.util.random
    +
    +import scala.collection.Map
    +import scala.collection.mutable.{ArrayBuffer, HashMap, Map => MMap}
    +
    +import org.apache.commons.math3.random.RandomDataGenerator
    +import org.apache.spark.Logging
    +import org.apache.spark.rdd.RDD
    +
    +/**
    + * Auxiliary functions and data structures for the sampleByKey method in PairRDDFunctions.
    + *
    + * Essentially, when exact sample size is necessary, we make additional passes over the RDD to
    + * compute the exact threshold value to use for each stratum to guarantee exact sample size with
    + * high probability. This is achieved by maintaining a waitlist of size O(log(s)), where s is the
    + * desired sample size for each stratum.
    + *
    + * Like in simple random sampling, we generate a random value for each item from the
    + * uniform  distribution [0.0, 1.0]. All items with values <= min(values of items in the waitlist)
    + * are accepted into the sample instantly. The threshold for instant accept is designed so that
    + * s - numAccepted = O(log(s)), where s is again the desired sample size. Thus, by maintaining a
    + * waitlist size = O(log(s)), we will be able to create a sample of the exact size s by adding
    + * a portion of the waitlist to the set of items that are instantly accepted. The exact threshold
    + * is computed by sorting the values in the waitlist and picking the value at (s - numAccepted).
    + *
    + * Note that since we use the same seed for the RNG when computing the thresholds and the actual
    + * sample, our computed thresholds are guaranteed to produce the desired sample size.
    + *
    + * For more theoretical background on the sampling techniques used here, please refer to
    + * http://jmlr.org/proceedings/papers/v28/meng13a.html
    + */
    +
    +private[spark] object StratifiedSampler extends Logging {
    +
    +  /**
    +   * Count the number of items instantly accepted and generate the waitlist for each stratum.
    +   *
    +   * This is only invoked when exact sample size is required.
    +   */
    +  def getCounts[K, V](rdd: RDD[(K, V)],
    +      withReplacement: Boolean,
    +      fractions: Map[K, Double],
    +      counts: Option[Map[K, Long]],
    +      seed: Long): MMap[K, Stratum] = {
    +    val combOp = getCombOp[K]
    +    val mappedPartitionRDD = rdd.mapPartitionsWithIndex({ case (partition, iter) =>
    +      val zeroU: MMap[K, Stratum] = new HashMap[K, Stratum]()
    +      val rng = new RandomDataGenerator()
    +      rng.reSeed(seed + partition)
    +      val seqOp = getSeqOp(withReplacement, fractions, rng, counts)
    +      Iterator(iter.aggregate(zeroU)(seqOp, combOp))
    +    }, preservesPartitioning=true)
    +    mappedPartitionRDD.reduce(combOp)
    +  }
    +
    +  /**
    +   * Returns the function used by aggregate to collect sampling statistics for each partition.
    +   */
    +  def getSeqOp[K, V](withReplacement: Boolean,
    +      fractions: Map[K, Double],
    +      rng: RandomDataGenerator,
    +      counts: Option[Map[K, Long]]): (MMap[K, Stratum], (K, V)) => MMap[K, Stratum] = {
    +    val delta = 5e-5
    +    (result: MMap[K, Stratum], item: (K, V)) => {
    +      val key = item._1
    +      val fraction = fractions(key)
    +      if (!result.contains(key)) {
    +        result += (key -> new Stratum())
    +      }
    +      val stratum = result(key)
    +
    +      if (withReplacement) {
    +        // compute acceptBound and waitListBound only if they haven't been computed already
    +        // since they don't change from iteration to iteration.
    +        // TODO change this to the streaming version
    +        if (stratum.areBoundsEmpty) {
    +          val n = counts.get(key)
    +          val sampleSize = math.ceil(n * fraction).toLong
    +          val lmbd1 = PoissonBounds.getLowerBound(sampleSize)
    +          val minCount = PoissonBounds.getMinCount(lmbd1)
    +          val lmbd2 = if (lmbd1 == 0) {
    +            PoissonBounds.getUpperBound(sampleSize)
    +          } else {
    +            PoissonBounds.getUpperBound(sampleSize - minCount)
    +          }
    +          stratum.acceptBound = lmbd1 / n
    +          stratum.waitListBound = lmbd2 / n
    +        }
    +        val acceptBound = stratum.acceptBound
    +        val copiesAccepted = if (acceptBound == 0.0) 0L else rng.nextPoisson(acceptBound)
    +        if (copiesAccepted > 0) {
    +          stratum.incrNumAccepted(copiesAccepted)
    +        }
    +        val copiesWaitlisted = rng.nextPoisson(stratum.waitListBound).toInt
    +        if (copiesWaitlisted > 0) {
    +          stratum.addToWaitList(ArrayBuffer.fill(copiesWaitlisted)(rng.nextUniform(0.0, 1.0)))
    +        }
    +      } else {
    +        // We use the streaming version of the algorithm for sampling without replacement to avoid
    +        // using an extra pass over the RDD for computing the count.
    +        // Hence, acceptBound and waitListBound change on every iteration.
    +        val gamma1 = - math.log(delta) / stratum.numItems
    +        val gamma2 = (2.0 / 3.0) * gamma1
    +        stratum.acceptBound = math.max(0,
    +          fraction + gamma2 - math.sqrt(gamma2 * gamma2 + 3 * gamma2 * fraction))
    +        stratum.waitListBound = math.min(1,
    +          fraction + gamma1 + math.sqrt(gamma1 * gamma1 + 2 * gamma1 * fraction))
    +
    +        val x = rng.nextUniform(0.0, 1.0)
    +        if (x < stratum.acceptBound) {
    +          stratum.incrNumAccepted()
    +        } else if (x < stratum.waitListBound) {
    +          stratum.addToWaitList(x)
    +        }
    +      }
    +      stratum.incrNumItems()
    +      result
    +    }
    +  }
    +
    +  /**
    +   * Returns the function used combine results returned by seqOp from different partitions.
    +   */
    +  def getCombOp[K]: (MMap[K, Stratum], MMap[K, Stratum]) => MMap[K, Stratum] = {
    +    (result1: MMap[K, Stratum], result2: MMap[K, Stratum]) => {
    +      // take union of both key sets in case one partition doesn't contain all keys
    +      for (key <- result1.keySet.union(result2.keySet)) {
    +        // Use result2 to keep the combined result since r1 is usual empty
    --- End diff --
    
    Yeah but we're not using RDD.aggregate here. We're using Scala's TraversableOnce.aggregate


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1025#issuecomment-45951336
  
     Build triggered. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1025#discussion_r14694378
  
    --- Diff: core/src/test/scala/org/apache/spark/rdd/PairRDDFunctionsSuite.scala ---
    @@ -83,6 +83,125 @@ class PairRDDFunctionsSuite extends FunSuite with SharedSparkContext {
         assert(valuesFor2.toList.sorted === List(1))
       }
     
    +  test("sampleByKey") {
    +    def stratifier (fractionPositive: Double) = {
    +      (x: Int) => if (x % 10 < (10 * fractionPositive).toInt) "1" else "0"
    +    }
    +
    +    def checkSize(exact: Boolean,
    +        withReplacement: Boolean,
    +        expected: Long,
    +        actual: Long,
    +        p: Double): Boolean = {
    +      if (exact) {
    +        return expected == actual
    +      }
    +      val stdev = if (withReplacement) math.sqrt(expected) else math.sqrt(expected * p * (1 - p))
    +      // Very forgiving margin since we're dealing with very small sample sizes most of the time
    +      math.abs(actual - expected) <= 6 * stdev
    +    }
    +
    +    // Without replacement validation
    +    def takeSampleAndValidateBernoulli(stratifiedData: RDD[(String, Int)],
    +        exact: Boolean,
    +        samplingRate: Double,
    +        seed: Long,
    +        n: Long) = {
    +      val expectedSampleSize = stratifiedData.countByKey().mapValues(count =>
    --- End diff --
    
    better merge `.mapValues` to the line below.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by dorx <gi...@git.apache.org>.
Github user dorx commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1025#discussion_r14907896
  
    --- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
    @@ -0,0 +1,311 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.util.random
    +
    +import scala.collection.Map
    +import scala.collection.mutable.{ArrayBuffer, HashMap, Map => MMap}
    +
    +import org.apache.commons.math3.random.RandomDataGenerator
    +import org.apache.spark.Logging
    +import org.apache.spark.rdd.RDD
    +
    +/**
    + * Auxiliary functions and data structures for the sampleByKey method in PairRDDFunctions.
    + *
    + * Essentially, when exact sample size is necessary, we make additional passes over the RDD to
    + * compute the exact threshold value to use for each stratum to guarantee exact sample size with
    + * high probability. This is achieved by maintaining a waitlist of size O(log(s)), where s is the
    + * desired sample size for each stratum.
    + *
    + * Like in simple random sampling, we generate a random value for each item from the
    + * uniform  distribution [0.0, 1.0]. All items with values <= min(values of items in the waitlist)
    + * are accepted into the sample instantly. The threshold for instant accept is designed so that
    + * s - numAccepted = O(log(s)), where s is again the desired sample size. Thus, by maintaining a
    + * waitlist size = O(log(s)), we will be able to create a sample of the exact size s by adding
    + * a portion of the waitlist to the set of items that are instantly accepted. The exact threshold
    + * is computed by sorting the values in the waitlist and picking the value at (s - numAccepted).
    + *
    + * Note that since we use the same seed for the RNG when computing the thresholds and the actual
    + * sample, our computed thresholds are guaranteed to produce the desired sample size.
    + *
    + * For more theoretical background on the sampling techniques used here, please refer to
    + * http://jmlr.org/proceedings/papers/v28/meng13a.html
    + */
    +
    +private[spark] object StratifiedSampler extends Logging {
    +
    +  /**
    +   * Count the number of items instantly accepted and generate the waitlist for each stratum.
    +   *
    +   * This is only invoked when exact sample size is required.
    +   */
    +  def getCounts[K, V](rdd: RDD[(K, V)],
    +      withReplacement: Boolean,
    +      fractions: Map[K, Double],
    +      counts: Option[Map[K, Long]],
    +      seed: Long): MMap[K, Stratum] = {
    +    val combOp = getCombOp[K]
    +    val mappedPartitionRDD = rdd.mapPartitionsWithIndex({ case (partition, iter) =>
    +      val zeroU: MMap[K, Stratum] = new HashMap[K, Stratum]()
    +      val rng = new RandomDataGenerator()
    +      rng.reSeed(seed + partition)
    +      val seqOp = getSeqOp(withReplacement, fractions, rng, counts)
    +      Iterator(iter.aggregate(zeroU)(seqOp, combOp))
    +    }, preservesPartitioning=true)
    +    mappedPartitionRDD.reduce(combOp)
    +  }
    +
    +  /**
    +   * Returns the function used by aggregate to collect sampling statistics for each partition.
    +   */
    +  def getSeqOp[K, V](withReplacement: Boolean,
    +      fractions: Map[K, Double],
    +      rng: RandomDataGenerator,
    +      counts: Option[Map[K, Long]]): (MMap[K, Stratum], (K, V)) => MMap[K, Stratum] = {
    +    val delta = 5e-5
    +    (result: MMap[K, Stratum], item: (K, V)) => {
    +      val key = item._1
    +      val fraction = fractions(key)
    +      if (!result.contains(key)) {
    +        result += (key -> new Stratum())
    +      }
    +      val stratum = result(key)
    +
    +      if (withReplacement) {
    +        // compute acceptBound and waitListBound only if they haven't been computed already
    +        // since they don't change from iteration to iteration.
    +        // TODO change this to the streaming version
    +        if (stratum.areBoundsEmpty) {
    +          val n = counts.get(key)
    +          val sampleSize = math.ceil(n * fraction).toLong
    +          val lmbd1 = PoissonBounds.getLowerBound(sampleSize)
    +          val minCount = PoissonBounds.getMinCount(lmbd1)
    +          val lmbd2 = if (lmbd1 == 0) {
    +            PoissonBounds.getUpperBound(sampleSize)
    +          } else {
    +            PoissonBounds.getUpperBound(sampleSize - minCount)
    +          }
    +          stratum.acceptBound = lmbd1 / n
    +          stratum.waitListBound = lmbd2 / n
    +        }
    +        val acceptBound = stratum.acceptBound
    +        val copiesAccepted = if (acceptBound == 0.0) 0L else rng.nextPoisson(acceptBound)
    +        if (copiesAccepted > 0) {
    +          stratum.incrNumAccepted(copiesAccepted)
    +        }
    +        val copiesWaitlisted = rng.nextPoisson(stratum.waitListBound).toInt
    +        if (copiesWaitlisted > 0) {
    +          stratum.addToWaitList(ArrayBuffer.fill(copiesWaitlisted)(rng.nextUniform(0.0, 1.0)))
    +        }
    +      } else {
    +        // We use the streaming version of the algorithm for sampling without replacement to avoid
    +        // using an extra pass over the RDD for computing the count.
    +        // Hence, acceptBound and waitListBound change on every iteration.
    +        val gamma1 = - math.log(delta) / stratum.numItems
    +        val gamma2 = (2.0 / 3.0) * gamma1
    +        stratum.acceptBound = math.max(0,
    +          fraction + gamma2 - math.sqrt(gamma2 * gamma2 + 3 * gamma2 * fraction))
    +        stratum.waitListBound = math.min(1,
    +          fraction + gamma1 + math.sqrt(gamma1 * gamma1 + 2 * gamma1 * fraction))
    +
    +        val x = rng.nextUniform(0.0, 1.0)
    +        if (x < stratum.acceptBound) {
    +          stratum.incrNumAccepted()
    +        } else if (x < stratum.waitListBound) {
    +          stratum.addToWaitList(x)
    +        }
    +      }
    +      stratum.incrNumItems()
    +      result
    +    }
    +  }
    +
    +  /**
    +   * Returns the function used combine results returned by seqOp from different partitions.
    +   */
    +  def getCombOp[K]: (MMap[K, Stratum], MMap[K, Stratum]) => MMap[K, Stratum] = {
    +    (result1: MMap[K, Stratum], result2: MMap[K, Stratum]) => {
    +      // take union of both key sets in case one partition doesn't contain all keys
    +      for (key <- result1.keySet.union(result2.keySet)) {
    +        // Use result2 to keep the combined result since r1 is usual empty
    +        val entry1 = result1.get(key)
    +        if (result2.contains(key)) {
    +          result2(key).merge(entry1)
    +        } else {
    +          if (entry1.isDefined) {
    +            result2 += (key -> entry1.get)
    +          }
    +        }
    +      }
    +      result2
    +    }
    +  }
    +
    +  /**
    +   * Given the result returned by getCounts, determine the threshold for accepting items to
    +   * generate exact sample size.
    +   *
    +   * To do so, we compute sampleSize = math.ceil(size * samplingRate) for each stratum and compare
    +   * it to the number of items that were accepted instantly and the number of items in the waitlist
    +   * for that stratum. Most of the time, numAccepted <= sampleSize <= (numAccepted + numWaitlisted),
    +   * which means we need to sort the elements in the waitlist by their associated values in order
    +   * to find the value T s.t. |{elements in the stratum whose associated values <= T}| = sampleSize.
    +   * Note that all elements in the waitlist have values >= bound for instant accept, so a T value
    +   * in the waitlist range would allow all elements that were instantly accepted on the first pass
    +   * to be included in the sample.
    +   */
    +  def computeThresholdByKey[K](finalResult: Map[K, Stratum],
    +      fractions: Map[K, Double]): Map[K, Double] = {
    +    val thresholdByKey = new HashMap[K, Double]()
    +    for ((key, stratum) <- finalResult) {
    +      val sampleSize = math.ceil(stratum.numItems * fractions(key)).toLong
    +      if (stratum.numAccepted > sampleSize) {
    +        logWarning("Pre-accepted too many")
    +        thresholdByKey += (key -> stratum.acceptBound)
    +      } else {
    +        val numWaitListAccepted = (sampleSize - stratum.numAccepted).toInt
    +        if (numWaitListAccepted >= stratum.waitList.size) {
    +          logWarning("WaitList too short")
    +          thresholdByKey += (key -> stratum.waitListBound)
    +        } else {
    +          thresholdByKey += (key -> stratum.waitList.sorted.apply(numWaitListAccepted))
    +        }
    +      }
    +    }
    +    thresholdByKey
    +  }
    +
    +  /**
    +   * Return the per partition sampling function used for sampling without replacement.
    +   *
    +   * When exact sample size is required, we make an additional pass over the RDD to determine the
    +   * exact sampling rate that guarantees sample size with high confidence.
    +   *
    +   * The sampling function has a unique seed per partition.
    +   */
    +  def getBernoulliSamplingFunction[K, V](rdd: RDD[(K,  V)],
    +      fractions: Map[K, Double],
    +      exact: Boolean,
    +      seed: Long): (Int, Iterator[(K, V)]) => Iterator[(K, V)] = {
    +    var samplingRateByKey = fractions
    +    if (exact) {
    +      // determine threshold for each stratum and resample
    +      val finalResult = getCounts(rdd, false, fractions, None, seed)
    +      samplingRateByKey = computeThresholdByKey(finalResult, fractions)
    +    }
    +    (idx: Int, iter: Iterator[(K, V)]) => {
    +      val rng = new RandomDataGenerator
    +      rng.reSeed(seed + idx)
    +      iter.filter(t => rng.nextUniform(0.0, 1.0) < samplingRateByKey(t._1))
    +    }
    +  }
    +
    +  /**
    +   * Return the per partition sampling function used for sampling with replacement.
    +   *
    +   * When exact sample size is required, we make two additional passed over the RDD to determine
    +   * the exact sampling rate that guarantees sample size with high confidence. The first pass
    +   * counts the number of items in each stratum (group of items with the same key) in the RDD, and
    +   * the second pass uses the counts to determine exact sampling rates.
    +   *
    +   * The sampling function has a unique seed per partition.
    +   */
    +  def getPoissonSamplingFunction[K, V](rdd: RDD[(K, V)],
    +      fractions: Map[K, Double],
    +      exact: Boolean,
    +      counts: Option[Map[K, Long]],
    +      seed: Long): (Int, Iterator[(K, V)]) => Iterator[(K, V)] = {
    +    // TODO implement the streaming version of sampling w/ replacement that doesn't require counts
    +    if (exact) {
    +      val finalResult = getCounts(rdd, true, fractions, counts, seed)
    +      val thresholdByKey = computeThresholdByKey(finalResult, fractions)
    +      (idx: Int, iter: Iterator[(K, V)]) => {
    +        val rng = new RandomDataGenerator()
    +        rng.reSeed(seed + idx)
    +        iter.flatMap { item =>
    +          val key = item._1
    +          val acceptBound = finalResult(key).acceptBound
    +          val copiesAccepted = if (acceptBound == 0) 0L else rng.nextPoisson(acceptBound)
    +          val copiesWailisted = rng.nextPoisson(finalResult(key).waitListBound).toInt
    +          val copiesInSample = copiesAccepted +
    +            (0 until copiesWailisted).count(i => rng.nextUniform(0.0, 1.0) < thresholdByKey(key))
    +          if (copiesInSample > 0) {
    +            Iterator.fill(copiesInSample.toInt)(item)
    +          } else {
    +            Iterator.empty
    +          }
    +        }
    +      }
    +    } else {
    +      (idx: Int, iter: Iterator[(K, V)]) => {
    +        val rng = new RandomDataGenerator()
    +        rng.reSeed(seed + idx)
    +        iter.flatMap { item =>
    +          val count = rng.nextPoisson(fractions(item._1)).toInt
    +          if (count > 0) {
    +            Iterator.fill(count)(item)
    +          } else {
    +            Iterator.empty
    +          }
    +        }
    +      }
    +    }
    +  }
    +}
    +
    +/**
    + * Object used by seqOp to keep track of the number of items accepted and items waitlisted per
    + * stratum, as well as the bounds for accepting and waitlisting items.
    + *
    + * `[random]` here is necessary since it's in the return type signature of seqOp defined above
    + */
    +private[random] class Stratum(var numItems: Long = 0L, var numAccepted: Long = 0L)
    +  extends Serializable {
    +
    +  private val _waitList = new ArrayBuffer[Double]
    +  private var _acceptBound: Option[Double] = None // upper bound for accepting item instantly
    +  private var _waitListBound: Option[Double] = None // upper bound for adding item to waitlist
    +
    +  def incrNumItems(by: Long = 1L) = numItems += by
    +
    +  def incrNumAccepted(by: Long = 1L) = numAccepted += by
    +
    +  def addToWaitList(elem: Double) = _waitList += elem
    +
    +  def addToWaitList(elems: ArrayBuffer[Double]) = _waitList ++= elems
    +
    +  def waitList = _waitList
    +
    +  def acceptBound = _acceptBound.getOrElse(0.0)
    +
    +  def acceptBound_= (value: Double) = _acceptBound = Some(value)
    +
    +  def waitListBound = _waitListBound.getOrElse(0.0)
    +
    +  def waitListBound_= (value: Double) = _waitListBound = Some(value)
    +
    +  def areBoundsEmpty = _acceptBound.isEmpty || _waitListBound.isEmpty
    +
    +  def merge(other: Option[Stratum]): Unit = {
    --- End diff --
    
    Nope.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1025#discussion_r14694277
  
    --- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
    @@ -0,0 +1,311 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.util.random
    +
    +import scala.collection.Map
    +import scala.collection.mutable.{ArrayBuffer, HashMap, Map => MMap}
    +
    +import org.apache.commons.math3.random.RandomDataGenerator
    +import org.apache.spark.Logging
    +import org.apache.spark.rdd.RDD
    +
    +/**
    + * Auxiliary functions and data structures for the sampleByKey method in PairRDDFunctions.
    + *
    + * Essentially, when exact sample size is necessary, we make additional passes over the RDD to
    + * compute the exact threshold value to use for each stratum to guarantee exact sample size with
    + * high probability. This is achieved by maintaining a waitlist of size O(log(s)), where s is the
    + * desired sample size for each stratum.
    + *
    + * Like in simple random sampling, we generate a random value for each item from the
    + * uniform  distribution [0.0, 1.0]. All items with values <= min(values of items in the waitlist)
    + * are accepted into the sample instantly. The threshold for instant accept is designed so that
    + * s - numAccepted = O(log(s)), where s is again the desired sample size. Thus, by maintaining a
    + * waitlist size = O(log(s)), we will be able to create a sample of the exact size s by adding
    + * a portion of the waitlist to the set of items that are instantly accepted. The exact threshold
    + * is computed by sorting the values in the waitlist and picking the value at (s - numAccepted).
    + *
    + * Note that since we use the same seed for the RNG when computing the thresholds and the actual
    + * sample, our computed thresholds are guaranteed to produce the desired sample size.
    + *
    + * For more theoretical background on the sampling techniques used here, please refer to
    + * http://jmlr.org/proceedings/papers/v28/meng13a.html
    + */
    +
    +private[spark] object StratifiedSampler extends Logging {
    +
    +  /**
    +   * Count the number of items instantly accepted and generate the waitlist for each stratum.
    +   *
    +   * This is only invoked when exact sample size is required.
    +   */
    +  def getCounts[K, V](rdd: RDD[(K, V)],
    +      withReplacement: Boolean,
    +      fractions: Map[K, Double],
    +      counts: Option[Map[K, Long]],
    +      seed: Long): MMap[K, Stratum] = {
    +    val combOp = getCombOp[K]
    +    val mappedPartitionRDD = rdd.mapPartitionsWithIndex({ case (partition, iter) =>
    +      val zeroU: MMap[K, Stratum] = new HashMap[K, Stratum]()
    +      val rng = new RandomDataGenerator()
    +      rng.reSeed(seed + partition)
    +      val seqOp = getSeqOp(withReplacement, fractions, rng, counts)
    +      Iterator(iter.aggregate(zeroU)(seqOp, combOp))
    +    }, preservesPartitioning=true)
    +    mappedPartitionRDD.reduce(combOp)
    +  }
    +
    +  /**
    +   * Returns the function used by aggregate to collect sampling statistics for each partition.
    +   */
    +  def getSeqOp[K, V](withReplacement: Boolean,
    +      fractions: Map[K, Double],
    +      rng: RandomDataGenerator,
    +      counts: Option[Map[K, Long]]): (MMap[K, Stratum], (K, V)) => MMap[K, Stratum] = {
    +    val delta = 5e-5
    +    (result: MMap[K, Stratum], item: (K, V)) => {
    +      val key = item._1
    +      val fraction = fractions(key)
    +      if (!result.contains(key)) {
    +        result += (key -> new Stratum())
    +      }
    +      val stratum = result(key)
    +
    +      if (withReplacement) {
    +        // compute acceptBound and waitListBound only if they haven't been computed already
    +        // since they don't change from iteration to iteration.
    +        // TODO change this to the streaming version
    +        if (stratum.areBoundsEmpty) {
    +          val n = counts.get(key)
    +          val sampleSize = math.ceil(n * fraction).toLong
    +          val lmbd1 = PoissonBounds.getLowerBound(sampleSize)
    +          val minCount = PoissonBounds.getMinCount(lmbd1)
    --- End diff --
    
    We already have the exact count for the stratum. Is it necessary?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1025#discussion_r14694259
  
    --- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
    @@ -0,0 +1,311 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.util.random
    +
    +import scala.collection.Map
    +import scala.collection.mutable.{ArrayBuffer, HashMap, Map => MMap}
    +
    +import org.apache.commons.math3.random.RandomDataGenerator
    --- End diff --
    
    Separate 3rd party imports from spark imports.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by falaki <gi...@git.apache.org>.
Github user falaki commented on the pull request:

    https://github.com/apache/spark/pull/1025#issuecomment-48417024
  
    @dorx Thanks for updating this. I left a few style and minor naming comments. Other than those this looks good to me. @mengxr Would you please take a look at this.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1025#issuecomment-45544398
  
     Build triggered. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1025#issuecomment-46376893
  
    Merged build started. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by falaki <gi...@git.apache.org>.
Github user falaki commented on the pull request:

    https://github.com/apache/spark/pull/1025#issuecomment-50104259
  
    This is the first place we introduce 'exact' to our API. We already have 'approx' in function names. I think having both of them is confusing to users. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by falaki <gi...@git.apache.org>.
Github user falaki commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1025#discussion_r14544125
  
    --- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
    @@ -0,0 +1,296 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.util.random
    +
    +import scala.collection.{Map, mutable}
    +import scala.collection.mutable.ArrayBuffer
    +import scala.reflect.ClassTag
    +
    +import org.apache.commons.math3.random.RandomDataGenerator
    +import org.apache.spark.{Logging, SparkContext, TaskContext}
    +import org.apache.spark.rdd.RDD
    +import org.apache.spark.util.Utils
    +import org.apache.spark.util.random.{PoissonBounds => PB}
    +
    +private[spark] object StratifiedSampler extends Logging {
    +
    +  /**
    +   * A version of {@link #aggregate()} that passes the TaskContext to the function that does
    +   * aggregation for each partition. This function avoids creating an extra depth in the RDD
    +   * lineage, as opposed to using mapPartitionsWithId, which results in slightly improved run time.
    +   */
    +  def aggregateWithContext[U: ClassTag, T: ClassTag](zeroValue: U)
    +      (rdd: RDD[T],
    +       seqOp: ((TaskContext, U), T) => U,
    +       combOp: (U, U) => U): U = {
    +    val sc: SparkContext = rdd.sparkContext
    +    // Clone the zero value since we will also be serializing it as part of tasks
    +    var jobResult = Utils.clone(zeroValue, sc.env.closureSerializer.newInstance())
    +    // pad seqOp and combOp with taskContext to conform to aggregate's signature in TraversableOnce
    +    val paddedSeqOp = (arg1: (TaskContext, U), item: T) => (arg1._1, seqOp(arg1, item))
    +    val paddedcombOp = (arg1: (TaskContext, U), arg2: (TaskContext, U)) =>
    +      (arg1._1, combOp(arg1._2, arg1._2))
    +    val cleanSeqOp = sc.clean(paddedSeqOp)
    +    val cleanCombOp = sc.clean(paddedcombOp)
    +    val aggregatePartition = (tc: TaskContext, it: Iterator[T]) =>
    +      (it.aggregate(tc, zeroValue)(cleanSeqOp, cleanCombOp))._2
    +    val mergeResult = (index: Int, taskResult: U) => jobResult = combOp(jobResult, taskResult)
    +    sc.runJob(rdd, aggregatePartition, mergeResult)
    +    jobResult
    +  }
    +
    +  /**
    +   * Returns the function used by aggregate to collect sampling statistics for each partition.
    +   */
    +  def getSeqOp[K, V](withReplacement: Boolean,
    +      fractionByKey: (K => Double),
    +      counts: Option[Map[K, Long]]): ((TaskContext, Result[K]),(K, V)) => Result[K] = {
    +    val delta = 5e-5
    +    (U: (TaskContext, Result[K]), item: (K, V)) => {
    --- End diff --
    
    We use uppercase letters for type parameters. Let's rename 'U' to a more descriptive name.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1025#issuecomment-46609787
  
    Merged build started. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1025#issuecomment-45553585
  
    Build finished. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1025#issuecomment-46609768
  
     Merged build triggered. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by dorx <gi...@git.apache.org>.
Github user dorx commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1025#discussion_r14906919
  
    --- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
    @@ -0,0 +1,311 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.util.random
    +
    +import scala.collection.Map
    +import scala.collection.mutable.{ArrayBuffer, HashMap, Map => MMap}
    +
    +import org.apache.commons.math3.random.RandomDataGenerator
    +import org.apache.spark.Logging
    +import org.apache.spark.rdd.RDD
    +
    +/**
    + * Auxiliary functions and data structures for the sampleByKey method in PairRDDFunctions.
    + *
    + * Essentially, when exact sample size is necessary, we make additional passes over the RDD to
    + * compute the exact threshold value to use for each stratum to guarantee exact sample size with
    + * high probability. This is achieved by maintaining a waitlist of size O(log(s)), where s is the
    + * desired sample size for each stratum.
    + *
    + * Like in simple random sampling, we generate a random value for each item from the
    + * uniform  distribution [0.0, 1.0]. All items with values <= min(values of items in the waitlist)
    + * are accepted into the sample instantly. The threshold for instant accept is designed so that
    + * s - numAccepted = O(log(s)), where s is again the desired sample size. Thus, by maintaining a
    + * waitlist size = O(log(s)), we will be able to create a sample of the exact size s by adding
    + * a portion of the waitlist to the set of items that are instantly accepted. The exact threshold
    + * is computed by sorting the values in the waitlist and picking the value at (s - numAccepted).
    + *
    + * Note that since we use the same seed for the RNG when computing the thresholds and the actual
    + * sample, our computed thresholds are guaranteed to produce the desired sample size.
    + *
    + * For more theoretical background on the sampling techniques used here, please refer to
    + * http://jmlr.org/proceedings/papers/v28/meng13a.html
    + */
    +
    +private[spark] object StratifiedSampler extends Logging {
    +
    +  /**
    +   * Count the number of items instantly accepted and generate the waitlist for each stratum.
    +   *
    +   * This is only invoked when exact sample size is required.
    +   */
    +  def getCounts[K, V](rdd: RDD[(K, V)],
    +      withReplacement: Boolean,
    +      fractions: Map[K, Double],
    +      counts: Option[Map[K, Long]],
    +      seed: Long): MMap[K, Stratum] = {
    +    val combOp = getCombOp[K]
    +    val mappedPartitionRDD = rdd.mapPartitionsWithIndex({ case (partition, iter) =>
    +      val zeroU: MMap[K, Stratum] = new HashMap[K, Stratum]()
    +      val rng = new RandomDataGenerator()
    +      rng.reSeed(seed + partition)
    +      val seqOp = getSeqOp(withReplacement, fractions, rng, counts)
    +      Iterator(iter.aggregate(zeroU)(seqOp, combOp))
    +    }, preservesPartitioning=true)
    +    mappedPartitionRDD.reduce(combOp)
    +  }
    +
    +  /**
    +   * Returns the function used by aggregate to collect sampling statistics for each partition.
    +   */
    +  def getSeqOp[K, V](withReplacement: Boolean,
    +      fractions: Map[K, Double],
    +      rng: RandomDataGenerator,
    +      counts: Option[Map[K, Long]]): (MMap[K, Stratum], (K, V)) => MMap[K, Stratum] = {
    +    val delta = 5e-5
    +    (result: MMap[K, Stratum], item: (K, V)) => {
    +      val key = item._1
    +      val fraction = fractions(key)
    +      if (!result.contains(key)) {
    +        result += (key -> new Stratum())
    +      }
    +      val stratum = result(key)
    +
    +      if (withReplacement) {
    +        // compute acceptBound and waitListBound only if they haven't been computed already
    +        // since they don't change from iteration to iteration.
    +        // TODO change this to the streaming version
    +        if (stratum.areBoundsEmpty) {
    +          val n = counts.get(key)
    +          val sampleSize = math.ceil(n * fraction).toLong
    +          val lmbd1 = PoissonBounds.getLowerBound(sampleSize)
    +          val minCount = PoissonBounds.getMinCount(lmbd1)
    +          val lmbd2 = if (lmbd1 == 0) {
    +            PoissonBounds.getUpperBound(sampleSize)
    +          } else {
    +            PoissonBounds.getUpperBound(sampleSize - minCount)
    +          }
    +          stratum.acceptBound = lmbd1 / n
    +          stratum.waitListBound = lmbd2 / n
    +        }
    +        val acceptBound = stratum.acceptBound
    +        val copiesAccepted = if (acceptBound == 0.0) 0L else rng.nextPoisson(acceptBound)
    +        if (copiesAccepted > 0) {
    +          stratum.incrNumAccepted(copiesAccepted)
    +        }
    +        val copiesWaitlisted = rng.nextPoisson(stratum.waitListBound).toInt
    +        if (copiesWaitlisted > 0) {
    +          stratum.addToWaitList(ArrayBuffer.fill(copiesWaitlisted)(rng.nextUniform(0.0, 1.0)))
    +        }
    +      } else {
    +        // We use the streaming version of the algorithm for sampling without replacement to avoid
    +        // using an extra pass over the RDD for computing the count.
    +        // Hence, acceptBound and waitListBound change on every iteration.
    +        val gamma1 = - math.log(delta) / stratum.numItems
    +        val gamma2 = (2.0 / 3.0) * gamma1
    +        stratum.acceptBound = math.max(0,
    --- End diff --
    
    Sure. Always nice to avoid duplicate logic in different spots.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1025#discussion_r14907579
  
    --- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
    @@ -0,0 +1,311 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.util.random
    +
    +import scala.collection.Map
    +import scala.collection.mutable.{ArrayBuffer, HashMap, Map => MMap}
    +
    +import org.apache.commons.math3.random.RandomDataGenerator
    +import org.apache.spark.Logging
    +import org.apache.spark.rdd.RDD
    +
    +/**
    + * Auxiliary functions and data structures for the sampleByKey method in PairRDDFunctions.
    + *
    + * Essentially, when exact sample size is necessary, we make additional passes over the RDD to
    + * compute the exact threshold value to use for each stratum to guarantee exact sample size with
    + * high probability. This is achieved by maintaining a waitlist of size O(log(s)), where s is the
    + * desired sample size for each stratum.
    + *
    + * Like in simple random sampling, we generate a random value for each item from the
    + * uniform  distribution [0.0, 1.0]. All items with values <= min(values of items in the waitlist)
    + * are accepted into the sample instantly. The threshold for instant accept is designed so that
    + * s - numAccepted = O(log(s)), where s is again the desired sample size. Thus, by maintaining a
    + * waitlist size = O(log(s)), we will be able to create a sample of the exact size s by adding
    + * a portion of the waitlist to the set of items that are instantly accepted. The exact threshold
    + * is computed by sorting the values in the waitlist and picking the value at (s - numAccepted).
    + *
    + * Note that since we use the same seed for the RNG when computing the thresholds and the actual
    + * sample, our computed thresholds are guaranteed to produce the desired sample size.
    + *
    + * For more theoretical background on the sampling techniques used here, please refer to
    + * http://jmlr.org/proceedings/papers/v28/meng13a.html
    + */
    +
    +private[spark] object StratifiedSampler extends Logging {
    --- End diff --
    
    Sounds good.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by falaki <gi...@git.apache.org>.
Github user falaki commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1025#discussion_r14544579
  
    --- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
    @@ -0,0 +1,296 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.util.random
    +
    +import scala.collection.{Map, mutable}
    +import scala.collection.mutable.ArrayBuffer
    +import scala.reflect.ClassTag
    +
    +import org.apache.commons.math3.random.RandomDataGenerator
    +import org.apache.spark.{Logging, SparkContext, TaskContext}
    +import org.apache.spark.rdd.RDD
    +import org.apache.spark.util.Utils
    +import org.apache.spark.util.random.{PoissonBounds => PB}
    +
    +private[spark] object StratifiedSampler extends Logging {
    +
    +  /**
    +   * A version of {@link #aggregate()} that passes the TaskContext to the function that does
    +   * aggregation for each partition. This function avoids creating an extra depth in the RDD
    +   * lineage, as opposed to using mapPartitionsWithId, which results in slightly improved run time.
    +   */
    +  def aggregateWithContext[U: ClassTag, T: ClassTag](zeroValue: U)
    +      (rdd: RDD[T],
    +       seqOp: ((TaskContext, U), T) => U,
    +       combOp: (U, U) => U): U = {
    +    val sc: SparkContext = rdd.sparkContext
    +    // Clone the zero value since we will also be serializing it as part of tasks
    +    var jobResult = Utils.clone(zeroValue, sc.env.closureSerializer.newInstance())
    +    // pad seqOp and combOp with taskContext to conform to aggregate's signature in TraversableOnce
    +    val paddedSeqOp = (arg1: (TaskContext, U), item: T) => (arg1._1, seqOp(arg1, item))
    +    val paddedcombOp = (arg1: (TaskContext, U), arg2: (TaskContext, U)) =>
    +      (arg1._1, combOp(arg1._2, arg1._2))
    +    val cleanSeqOp = sc.clean(paddedSeqOp)
    +    val cleanCombOp = sc.clean(paddedcombOp)
    +    val aggregatePartition = (tc: TaskContext, it: Iterator[T]) =>
    +      (it.aggregate(tc, zeroValue)(cleanSeqOp, cleanCombOp))._2
    +    val mergeResult = (index: Int, taskResult: U) => jobResult = combOp(jobResult, taskResult)
    +    sc.runJob(rdd, aggregatePartition, mergeResult)
    +    jobResult
    +  }
    +
    +  /**
    +   * Returns the function used by aggregate to collect sampling statistics for each partition.
    +   */
    +  def getSeqOp[K, V](withReplacement: Boolean,
    +      fractionByKey: (K => Double),
    +      counts: Option[Map[K, Long]]): ((TaskContext, Result[K]),(K, V)) => Result[K] = {
    +    val delta = 5e-5
    +    (U: (TaskContext, Result[K]), item: (K, V)) => {
    +      val result = U._2
    +      val tc = U._1
    +      val rng = result.getRand(tc.partitionId)
    +      val fraction = fractionByKey(item._1)
    +      val stratum = result.getEntry(item._1)
    +      if (withReplacement) {
    +        // compute q1 and q2 only if they haven't been computed already
    +        // since they don't change from iteration to iteration.
    +        // TODO change this to the streaming version
    +        if (stratum.q1.isEmpty || stratum.q2.isEmpty) {
    +          val n = counts.get(item._1)
    +          val s = math.ceil(n * fraction).toLong
    +          val lmbd1 = PB.getLowerBound(s)
    +          val minCount = PB.getMinCount(lmbd1)
    +          val lmbd2 = if (lmbd1 == 0) PB.getUpperBound(s) else PB.getUpperBound(s - minCount)
    +          val q1 = lmbd1 / n
    +          val q2 = lmbd2 / n
    +          stratum.q1 = Some(q1)
    +          stratum.q2 = Some(q2)
    +        }
    +        val x1 = if (stratum.q1.get == 0) 0L else rng.nextPoisson(stratum.q1.get)
    +        if (x1 > 0) {
    +          stratum.incrNumAccepted(x1)
    +        }
    +        val x2 = rng.nextPoisson(stratum.q2.get).toInt
    +        if (x2 > 0) {
    +          stratum.addToWaitList(ArrayBuffer.fill(x2)(rng.nextUniform(0.0, 1.0)))
    +        }
    +      } else {
    +        // We use the streaming version of the algorithm for sampling without replacement.
    +        // Hence, q1 and q2 change on every iteration.
    +        val g1 = - math.log(delta) / stratum.numItems
    +        val g2 = (2.0 / 3.0) * g1
    +        val q1 = math.max(0, fraction + g2 - math.sqrt((g2 * g2 + 3 * g2 * fraction)))
    +        val q2 = math.min(1, fraction + g1 + math.sqrt(g1 * g1 + 2 * g1 * fraction))
    +
    +        val x = rng.nextUniform(0.0, 1.0)
    +        if (x < q1) {
    +          stratum.incrNumAccepted()
    +        } else if (x < q2) {
    +          stratum.addToWaitList(x)
    +        }
    +        stratum.q1 = Some(q1)
    +        stratum.q2 = Some(q2)
    +      }
    +      stratum.incrNumItems()
    +      result
    +    }
    +  }
    +
    +  /**
    +   * Returns the function used by aggregate to combine results from different partitions, as
    +   * returned by seqOp.
    +   */
    +  def getCombOp[K](): (Result[K], Result[K]) => Result[K] = {
    +    (r1: Result[K], r2: Result[K]) => {
    +      // take union of both key sets in case one partition doesn't contain all keys
    +      val keyUnion = r1.resultMap.keys.toSet.union(r2.resultMap.keys.toSet)
    +
    +      // Use r2 to keep the combined result since r1 is usual empty
    +      for (key <- keyUnion) {
    +        val entry1 = r1.resultMap.get(key)
    +        val entry2 = r2.resultMap.get(key)
    +        if (entry2.isEmpty && entry1.isDefined) {
    +          r2.resultMap += (key -> entry1.get)
    +        } else if (entry1.isDefined && entry2.isDefined) {
    +          entry2.get.addToWaitList(entry1.get.waitList)
    +          entry2.get.incrNumAccepted(entry1.get.numAccepted)
    +          entry2.get.incrNumItems(entry1.get.numItems)
    +        }
    +      }
    +      r2
    +    }
    +  }
    +
    +  /**
    +   * Given the result returned by the aggregate function, we need to determine the threshold used
    +   * to accept items to generate the exact sample size.
    +   */
    +  def computeThresholdByKey[K](finalResult: Map[K, Stratum], fractionByKey: (K => Double)):
    +    (K => Double) = {
    +    val thresholdByKey = new mutable.HashMap[K, Double]()
    +    for ((key, stratum) <- finalResult) {
    +      val fraction = fractionByKey(key)
    +      val s = math.ceil(stratum.numItems * fraction).toLong
    +      if (stratum.numAccepted > s) {
    +        logWarning("Pre-accepted too many")
    +        thresholdByKey += (key -> stratum.q1.get)
    +      } else {
    +        val numWaitListAccepted = (s - stratum.numAccepted).toInt
    +        if (numWaitListAccepted >= stratum.waitList.size) {
    +          logWarning("WaitList too short")
    +          thresholdByKey += (key -> stratum.q2.get)
    +        } else {
    +          thresholdByKey += (key -> stratum.waitList.sorted.apply(numWaitListAccepted))
    +        }
    +      }
    +    }
    +    thresholdByKey
    +  }
    +
    +  /**
    +   * Return the per partition sampling function used for sampling without replacement.
    +   *
    +   * When exact sample size is required, we make an additional pass over the RDD to determine the
    +   * exact sampling rate that guarantees sample size with high confidence.
    +   *
    +   * The sampling function has a unique seed per partition.
    +   */
    +  def getBernoulliSamplingFunction[K, V](rdd: RDD[(K,  V)],
    +      fractionByKey: K => Double,
    +      exact: Boolean,
    +      seed: Long): (Int, Iterator[(K, V)]) => Iterator[(K, V)] = {
    +    var samplingRateByKey = fractionByKey
    +    if (exact) {
    +      // determine threshold for each stratum and resample
    +      val seqOp = StratifiedSampler.getSeqOp[K,V](false, fractionByKey, None)
    +      val combOp = StratifiedSampler.getCombOp[K]()
    +      val zeroU = new Result[K](Map[K, Stratum](), seed = seed)
    +      val finalResult = aggregateWithContext(zeroU)(rdd, seqOp, combOp).resultMap
    +      samplingRateByKey = StratifiedSampler.computeThresholdByKey(finalResult, fractionByKey)
    +    }
    +    (idx: Int, iter: Iterator[(K, V)]) => {
    +      val random = new RandomDataGenerator
    +      random.reSeed(seed + idx)
    +      iter.filter(t => random.nextUniform(0.0, 1.0) < samplingRateByKey(t._1))
    +    }
    +  }
    +
    +  /**
    +   * Return the per partition sampling function used for sampling with replacement.
    +   *
    +   * When exact sample size is required, we make two additional passed over the RDD to determine
    +   * the exact sampling rate that guarantees sample size with high confidence. The first pass
    +   * counts the number of items in each stratum (group of items with the same key) in the RDD, and
    +   * the second pass uses the counts to determine exact sampling rates.
    +   *
    +   * The sampling function has a unique seed per partition.
    +   */
    +  def getPoissonSamplingFunction[K, V](rdd:RDD[(K,  V)],
    +      fractionByKey: K => Double,
    +      exact: Boolean,
    +      counts: Option[Map[K, Long]],
    +      seed: Long): (Int, Iterator[(K, V)]) => Iterator[(K, V)] = {
    +    // TODO implement the streaming version of sampling w/ replacement that doesn't require counts
    +    if (exact) {
    +      val seqOp = StratifiedSampler.getSeqOp[K,V](true, fractionByKey, counts)
    +      val combOp = StratifiedSampler.getCombOp[K]()
    +      val zeroU = new Result[K](Map[K, Stratum](), seed = seed)
    +      val finalResult = aggregateWithContext(zeroU)(rdd, seqOp, combOp).resultMap
    +      val thresholdByKey = StratifiedSampler.computeThresholdByKey(finalResult, fractionByKey)
    +      (idx: Int, iter: Iterator[(K, V)]) => {
    +        val random = new RandomDataGenerator()
    +        random.reSeed(seed + idx)
    +        iter.flatMap { t =>
    +          val q1 = finalResult.get(t._1).get.q1.getOrElse(0.0)
    +          val q2 = finalResult.get(t._1).get.q2.getOrElse(0.0)
    +          val x1 = if (q1 == 0) 0L else random.nextPoisson(q1)
    +          val x2 = random.nextPoisson(q2).toInt
    +          val x = x1 + (0 until x2).count(i => random.nextUniform(0.0, 1.0) < thresholdByKey(t._1))
    +          if (x > 0) {
    +            Iterator.fill(x.toInt)(t)
    +          } else {
    +            Iterator.empty
    +          }
    +        }
    +      }
    +    } else {
    +      (idx: Int, iter: Iterator[(K, V)]) => {
    +        val random = new RandomDataGenerator()
    +        random.reSeed(seed + idx)
    +        iter.flatMap { t =>
    +          val count = random.nextPoisson(fractionByKey(t._1)).toInt
    +          if (count > 0) {
    +            Iterator.fill(count)(t)
    +          } else {
    +            Iterator.empty
    +          }
    +        }
    +      }
    +    }
    +  }
    +}
    +
    +/**
    + * Object used by seqOp to keep track of the number of items accepted and items waitlisted per
    + * stratum, as well as the bounds for accepting and waitlisting items.
    + */
    +private[random] class Stratum(var numItems: Long = 0L, var numAccepted: Long = 0L)
    +  extends Serializable {
    +
    +  var waitList: ArrayBuffer[Double] = new ArrayBuffer[Double]
    --- End diff --
    
    This does not need to be a variable. ArrayBuffer is immutable and that seems enough.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1025#issuecomment-47990759
  
    Merged build finished. All automated tests passed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by falaki <gi...@git.apache.org>.
Github user falaki commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1025#discussion_r14543987
  
    --- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
    @@ -0,0 +1,296 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.util.random
    +
    +import scala.collection.{Map, mutable}
    +import scala.collection.mutable.ArrayBuffer
    +import scala.reflect.ClassTag
    +
    +import org.apache.commons.math3.random.RandomDataGenerator
    +import org.apache.spark.{Logging, SparkContext, TaskContext}
    +import org.apache.spark.rdd.RDD
    +import org.apache.spark.util.Utils
    +import org.apache.spark.util.random.{PoissonBounds => PB}
    +
    +private[spark] object StratifiedSampler extends Logging {
    +
    +  /**
    +   * A version of {@link #aggregate()} that passes the TaskContext to the function that does
    +   * aggregation for each partition. This function avoids creating an extra depth in the RDD
    +   * lineage, as opposed to using mapPartitionsWithId, which results in slightly improved run time.
    +   */
    +  def aggregateWithContext[U: ClassTag, T: ClassTag](zeroValue: U)
    +      (rdd: RDD[T],
    +       seqOp: ((TaskContext, U), T) => U,
    +       combOp: (U, U) => U): U = {
    +    val sc: SparkContext = rdd.sparkContext
    +    // Clone the zero value since we will also be serializing it as part of tasks
    +    var jobResult = Utils.clone(zeroValue, sc.env.closureSerializer.newInstance())
    +    // pad seqOp and combOp with taskContext to conform to aggregate's signature in TraversableOnce
    +    val paddedSeqOp = (arg1: (TaskContext, U), item: T) => (arg1._1, seqOp(arg1, item))
    +    val paddedcombOp = (arg1: (TaskContext, U), arg2: (TaskContext, U)) =>
    +      (arg1._1, combOp(arg1._2, arg1._2))
    +    val cleanSeqOp = sc.clean(paddedSeqOp)
    +    val cleanCombOp = sc.clean(paddedcombOp)
    +    val aggregatePartition = (tc: TaskContext, it: Iterator[T]) =>
    +      (it.aggregate(tc, zeroValue)(cleanSeqOp, cleanCombOp))._2
    +    val mergeResult = (index: Int, taskResult: U) => jobResult = combOp(jobResult, taskResult)
    +    sc.runJob(rdd, aggregatePartition, mergeResult)
    +    jobResult
    +  }
    +
    +  /**
    +   * Returns the function used by aggregate to collect sampling statistics for each partition.
    +   */
    +  def getSeqOp[K, V](withReplacement: Boolean,
    +      fractionByKey: (K => Double),
    --- End diff --
    
    Let's change the type of fractionByKey to Map[K, Double]


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1025#discussion_r14694298
  
    --- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
    @@ -0,0 +1,311 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.util.random
    +
    +import scala.collection.Map
    +import scala.collection.mutable.{ArrayBuffer, HashMap, Map => MMap}
    +
    +import org.apache.commons.math3.random.RandomDataGenerator
    +import org.apache.spark.Logging
    +import org.apache.spark.rdd.RDD
    +
    +/**
    + * Auxiliary functions and data structures for the sampleByKey method in PairRDDFunctions.
    + *
    + * Essentially, when exact sample size is necessary, we make additional passes over the RDD to
    + * compute the exact threshold value to use for each stratum to guarantee exact sample size with
    + * high probability. This is achieved by maintaining a waitlist of size O(log(s)), where s is the
    + * desired sample size for each stratum.
    + *
    + * Like in simple random sampling, we generate a random value for each item from the
    + * uniform  distribution [0.0, 1.0]. All items with values <= min(values of items in the waitlist)
    + * are accepted into the sample instantly. The threshold for instant accept is designed so that
    + * s - numAccepted = O(log(s)), where s is again the desired sample size. Thus, by maintaining a
    + * waitlist size = O(log(s)), we will be able to create a sample of the exact size s by adding
    + * a portion of the waitlist to the set of items that are instantly accepted. The exact threshold
    + * is computed by sorting the values in the waitlist and picking the value at (s - numAccepted).
    + *
    + * Note that since we use the same seed for the RNG when computing the thresholds and the actual
    + * sample, our computed thresholds are guaranteed to produce the desired sample size.
    + *
    + * For more theoretical background on the sampling techniques used here, please refer to
    + * http://jmlr.org/proceedings/papers/v28/meng13a.html
    + */
    +
    +private[spark] object StratifiedSampler extends Logging {
    +
    +  /**
    +   * Count the number of items instantly accepted and generate the waitlist for each stratum.
    +   *
    +   * This is only invoked when exact sample size is required.
    +   */
    +  def getCounts[K, V](rdd: RDD[(K, V)],
    +      withReplacement: Boolean,
    +      fractions: Map[K, Double],
    +      counts: Option[Map[K, Long]],
    +      seed: Long): MMap[K, Stratum] = {
    +    val combOp = getCombOp[K]
    +    val mappedPartitionRDD = rdd.mapPartitionsWithIndex({ case (partition, iter) =>
    +      val zeroU: MMap[K, Stratum] = new HashMap[K, Stratum]()
    +      val rng = new RandomDataGenerator()
    +      rng.reSeed(seed + partition)
    +      val seqOp = getSeqOp(withReplacement, fractions, rng, counts)
    +      Iterator(iter.aggregate(zeroU)(seqOp, combOp))
    +    }, preservesPartitioning=true)
    +    mappedPartitionRDD.reduce(combOp)
    +  }
    +
    +  /**
    +   * Returns the function used by aggregate to collect sampling statistics for each partition.
    +   */
    +  def getSeqOp[K, V](withReplacement: Boolean,
    +      fractions: Map[K, Double],
    +      rng: RandomDataGenerator,
    +      counts: Option[Map[K, Long]]): (MMap[K, Stratum], (K, V)) => MMap[K, Stratum] = {
    +    val delta = 5e-5
    +    (result: MMap[K, Stratum], item: (K, V)) => {
    +      val key = item._1
    +      val fraction = fractions(key)
    +      if (!result.contains(key)) {
    +        result += (key -> new Stratum())
    +      }
    +      val stratum = result(key)
    +
    +      if (withReplacement) {
    +        // compute acceptBound and waitListBound only if they haven't been computed already
    +        // since they don't change from iteration to iteration.
    +        // TODO change this to the streaming version
    +        if (stratum.areBoundsEmpty) {
    +          val n = counts.get(key)
    +          val sampleSize = math.ceil(n * fraction).toLong
    +          val lmbd1 = PoissonBounds.getLowerBound(sampleSize)
    +          val minCount = PoissonBounds.getMinCount(lmbd1)
    +          val lmbd2 = if (lmbd1 == 0) {
    +            PoissonBounds.getUpperBound(sampleSize)
    +          } else {
    +            PoissonBounds.getUpperBound(sampleSize - minCount)
    +          }
    +          stratum.acceptBound = lmbd1 / n
    +          stratum.waitListBound = lmbd2 / n
    +        }
    +        val acceptBound = stratum.acceptBound
    +        val copiesAccepted = if (acceptBound == 0.0) 0L else rng.nextPoisson(acceptBound)
    +        if (copiesAccepted > 0) {
    +          stratum.incrNumAccepted(copiesAccepted)
    +        }
    +        val copiesWaitlisted = rng.nextPoisson(stratum.waitListBound).toInt
    +        if (copiesWaitlisted > 0) {
    +          stratum.addToWaitList(ArrayBuffer.fill(copiesWaitlisted)(rng.nextUniform(0.0, 1.0)))
    +        }
    +      } else {
    +        // We use the streaming version of the algorithm for sampling without replacement to avoid
    +        // using an extra pass over the RDD for computing the count.
    +        // Hence, acceptBound and waitListBound change on every iteration.
    +        val gamma1 = - math.log(delta) / stratum.numItems
    +        val gamma2 = (2.0 / 3.0) * gamma1
    +        stratum.acceptBound = math.max(0,
    +          fraction + gamma2 - math.sqrt(gamma2 * gamma2 + 3 * gamma2 * fraction))
    +        stratum.waitListBound = math.min(1,
    +          fraction + gamma1 + math.sqrt(gamma1 * gamma1 + 2 * gamma1 * fraction))
    +
    +        val x = rng.nextUniform(0.0, 1.0)
    +        if (x < stratum.acceptBound) {
    +          stratum.incrNumAccepted()
    +        } else if (x < stratum.waitListBound) {
    +          stratum.addToWaitList(x)
    +        }
    +      }
    +      stratum.incrNumItems()
    +      result
    +    }
    +  }
    +
    +  /**
    +   * Returns the function used combine results returned by seqOp from different partitions.
    +   */
    +  def getCombOp[K]: (MMap[K, Stratum], MMap[K, Stratum]) => MMap[K, Stratum] = {
    +    (result1: MMap[K, Stratum], result2: MMap[K, Stratum]) => {
    +      // take union of both key sets in case one partition doesn't contain all keys
    +      for (key <- result1.keySet.union(result2.keySet)) {
    +        // Use result2 to keep the combined result since r1 is usual empty
    +        val entry1 = result1.get(key)
    +        if (result2.contains(key)) {
    +          result2(key).merge(entry1)
    +        } else {
    +          if (entry1.isDefined) {
    +            result2 += (key -> entry1.get)
    +          }
    +        }
    +      }
    +      result2
    +    }
    +  }
    +
    +  /**
    +   * Given the result returned by getCounts, determine the threshold for accepting items to
    +   * generate exact sample size.
    +   *
    +   * To do so, we compute sampleSize = math.ceil(size * samplingRate) for each stratum and compare
    +   * it to the number of items that were accepted instantly and the number of items in the waitlist
    +   * for that stratum. Most of the time, numAccepted <= sampleSize <= (numAccepted + numWaitlisted),
    +   * which means we need to sort the elements in the waitlist by their associated values in order
    +   * to find the value T s.t. |{elements in the stratum whose associated values <= T}| = sampleSize.
    +   * Note that all elements in the waitlist have values >= bound for instant accept, so a T value
    +   * in the waitlist range would allow all elements that were instantly accepted on the first pass
    +   * to be included in the sample.
    +   */
    +  def computeThresholdByKey[K](finalResult: Map[K, Stratum],
    +      fractions: Map[K, Double]): Map[K, Double] = {
    +    val thresholdByKey = new HashMap[K, Double]()
    +    for ((key, stratum) <- finalResult) {
    +      val sampleSize = math.ceil(stratum.numItems * fractions(key)).toLong
    +      if (stratum.numAccepted > sampleSize) {
    +        logWarning("Pre-accepted too many")
    +        thresholdByKey += (key -> stratum.acceptBound)
    +      } else {
    +        val numWaitListAccepted = (sampleSize - stratum.numAccepted).toInt
    +        if (numWaitListAccepted >= stratum.waitList.size) {
    +          logWarning("WaitList too short")
    +          thresholdByKey += (key -> stratum.waitListBound)
    +        } else {
    +          thresholdByKey += (key -> stratum.waitList.sorted.apply(numWaitListAccepted))
    +        }
    +      }
    +    }
    +    thresholdByKey
    +  }
    +
    +  /**
    +   * Return the per partition sampling function used for sampling without replacement.
    +   *
    +   * When exact sample size is required, we make an additional pass over the RDD to determine the
    +   * exact sampling rate that guarantees sample size with high confidence.
    +   *
    +   * The sampling function has a unique seed per partition.
    +   */
    +  def getBernoulliSamplingFunction[K, V](rdd: RDD[(K,  V)],
    +      fractions: Map[K, Double],
    +      exact: Boolean,
    +      seed: Long): (Int, Iterator[(K, V)]) => Iterator[(K, V)] = {
    +    var samplingRateByKey = fractions
    +    if (exact) {
    +      // determine threshold for each stratum and resample
    +      val finalResult = getCounts(rdd, false, fractions, None, seed)
    +      samplingRateByKey = computeThresholdByKey(finalResult, fractions)
    +    }
    +    (idx: Int, iter: Iterator[(K, V)]) => {
    +      val rng = new RandomDataGenerator
    +      rng.reSeed(seed + idx)
    +      iter.filter(t => rng.nextUniform(0.0, 1.0) < samplingRateByKey(t._1))
    +    }
    +  }
    +
    +  /**
    +   * Return the per partition sampling function used for sampling with replacement.
    +   *
    +   * When exact sample size is required, we make two additional passed over the RDD to determine
    +   * the exact sampling rate that guarantees sample size with high confidence. The first pass
    +   * counts the number of items in each stratum (group of items with the same key) in the RDD, and
    +   * the second pass uses the counts to determine exact sampling rates.
    +   *
    +   * The sampling function has a unique seed per partition.
    +   */
    +  def getPoissonSamplingFunction[K, V](rdd: RDD[(K, V)],
    +      fractions: Map[K, Double],
    +      exact: Boolean,
    +      counts: Option[Map[K, Long]],
    +      seed: Long): (Int, Iterator[(K, V)]) => Iterator[(K, V)] = {
    +    // TODO implement the streaming version of sampling w/ replacement that doesn't require counts
    +    if (exact) {
    +      val finalResult = getCounts(rdd, true, fractions, counts, seed)
    +      val thresholdByKey = computeThresholdByKey(finalResult, fractions)
    +      (idx: Int, iter: Iterator[(K, V)]) => {
    +        val rng = new RandomDataGenerator()
    +        rng.reSeed(seed + idx)
    +        iter.flatMap { item =>
    +          val key = item._1
    +          val acceptBound = finalResult(key).acceptBound
    +          val copiesAccepted = if (acceptBound == 0) 0L else rng.nextPoisson(acceptBound)
    +          val copiesWailisted = rng.nextPoisson(finalResult(key).waitListBound).toInt
    +          val copiesInSample = copiesAccepted +
    +            (0 until copiesWailisted).count(i => rng.nextUniform(0.0, 1.0) < thresholdByKey(key))
    +          if (copiesInSample > 0) {
    +            Iterator.fill(copiesInSample.toInt)(item)
    +          } else {
    +            Iterator.empty
    +          }
    +        }
    +      }
    +    } else {
    +      (idx: Int, iter: Iterator[(K, V)]) => {
    +        val rng = new RandomDataGenerator()
    +        rng.reSeed(seed + idx)
    +        iter.flatMap { item =>
    +          val count = rng.nextPoisson(fractions(item._1)).toInt
    +          if (count > 0) {
    +            Iterator.fill(count)(item)
    +          } else {
    +            Iterator.empty
    +          }
    +        }
    +      }
    +    }
    +  }
    +}
    +
    +/**
    + * Object used by seqOp to keep track of the number of items accepted and items waitlisted per
    + * stratum, as well as the bounds for accepting and waitlisting items.
    + *
    + * `[random]` here is necessary since it's in the return type signature of seqOp defined above
    + */
    +private[random] class Stratum(var numItems: Long = 0L, var numAccepted: Long = 0L)
    +  extends Serializable {
    +
    +  private val _waitList = new ArrayBuffer[Double]
    +  private var _acceptBound: Option[Double] = None // upper bound for accepting item instantly
    +  private var _waitListBound: Option[Double] = None // upper bound for adding item to waitlist
    +
    +  def incrNumItems(by: Long = 1L) = numItems += by
    +
    +  def incrNumAccepted(by: Long = 1L) = numAccepted += by
    +
    +  def addToWaitList(elem: Double) = _waitList += elem
    +
    +  def addToWaitList(elems: ArrayBuffer[Double]) = _waitList ++= elems
    +
    +  def waitList = _waitList
    +
    +  def acceptBound = _acceptBound.getOrElse(0.0)
    +
    +  def acceptBound_= (value: Double) = _acceptBound = Some(value)
    +
    +  def waitListBound = _waitListBound.getOrElse(0.0)
    +
    +  def waitListBound_= (value: Double) = _waitListBound = Some(value)
    +
    +  def areBoundsEmpty = _acceptBound.isEmpty || _waitListBound.isEmpty
    +
    +  def merge(other: Option[Stratum]): Unit = {
    --- End diff --
    
    Any chance of merging a `None`?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by falaki <gi...@git.apache.org>.
Github user falaki commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1025#discussion_r14688585
  
    --- Diff: core/src/test/scala/org/apache/spark/rdd/PairRDDFunctionsSuite.scala ---
    @@ -83,6 +83,120 @@ class PairRDDFunctionsSuite extends FunSuite with SharedSparkContext {
         assert(valuesFor2.toList.sorted === List(1))
       }
     
    +  test("sampleByKey") {
    +    def stratifier (fractionPositive: Double) = {
    +      (x: Int) => if (x % 10 < (10 * fractionPositive).toInt) "1" else "0"
    +    }
    +
    +    def checkSize(exact: Boolean, withReplacement: Boolean,
    +        expected: Long, actual: Long, p: Double): Boolean = {
    +      if (exact) {
    +        return expected == actual
    +      }
    +      val stdev = if (withReplacement) math.sqrt(expected) else math.sqrt(expected * p * (1 - p))
    +      // Very forgiving margin since we're dealing with very small sample sizes most of the time
    +      math.abs(actual - expected) <= 6 * stdev
    +    }
    +
    +    // Without replacement validation
    +    def takeSampleAndValidateBernoulli(stratifiedData: RDD[(String, Int)],
    +        exact: Boolean,
    +        samplingRate: Double,
    +        seed: Long,
    +        n: Long) = {
    +      val expectedSampleSize = stratifiedData.countByKey().mapValues(count =>
    +        math.ceil(count * samplingRate).toInt)
    +      val fractions = Map("1" -> samplingRate, "0" -> samplingRate)
    +      val sample = stratifiedData.sampleByKey(false, fractions, exact, seed)
    +      val sampleCounts = sample.countByKey()
    +      val takeSample = sample.collect()
    +      assert(sampleCounts.forall({case(k,v) =>
    --- End diff --
    
    Parentheses are superfluous. Square bracket is enough:
    ```sampleCounts.forall { case(k, v) => ```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1025#discussion_r14694278
  
    --- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
    @@ -0,0 +1,311 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.util.random
    +
    +import scala.collection.Map
    +import scala.collection.mutable.{ArrayBuffer, HashMap, Map => MMap}
    +
    +import org.apache.commons.math3.random.RandomDataGenerator
    +import org.apache.spark.Logging
    +import org.apache.spark.rdd.RDD
    +
    +/**
    + * Auxiliary functions and data structures for the sampleByKey method in PairRDDFunctions.
    + *
    + * Essentially, when exact sample size is necessary, we make additional passes over the RDD to
    + * compute the exact threshold value to use for each stratum to guarantee exact sample size with
    + * high probability. This is achieved by maintaining a waitlist of size O(log(s)), where s is the
    + * desired sample size for each stratum.
    + *
    + * Like in simple random sampling, we generate a random value for each item from the
    + * uniform  distribution [0.0, 1.0]. All items with values <= min(values of items in the waitlist)
    + * are accepted into the sample instantly. The threshold for instant accept is designed so that
    + * s - numAccepted = O(log(s)), where s is again the desired sample size. Thus, by maintaining a
    + * waitlist size = O(log(s)), we will be able to create a sample of the exact size s by adding
    + * a portion of the waitlist to the set of items that are instantly accepted. The exact threshold
    + * is computed by sorting the values in the waitlist and picking the value at (s - numAccepted).
    + *
    + * Note that since we use the same seed for the RNG when computing the thresholds and the actual
    + * sample, our computed thresholds are guaranteed to produce the desired sample size.
    + *
    + * For more theoretical background on the sampling techniques used here, please refer to
    + * http://jmlr.org/proceedings/papers/v28/meng13a.html
    + */
    +
    +private[spark] object StratifiedSampler extends Logging {
    +
    +  /**
    +   * Count the number of items instantly accepted and generate the waitlist for each stratum.
    +   *
    +   * This is only invoked when exact sample size is required.
    +   */
    +  def getCounts[K, V](rdd: RDD[(K, V)],
    +      withReplacement: Boolean,
    +      fractions: Map[K, Double],
    +      counts: Option[Map[K, Long]],
    +      seed: Long): MMap[K, Stratum] = {
    +    val combOp = getCombOp[K]
    +    val mappedPartitionRDD = rdd.mapPartitionsWithIndex({ case (partition, iter) =>
    +      val zeroU: MMap[K, Stratum] = new HashMap[K, Stratum]()
    +      val rng = new RandomDataGenerator()
    +      rng.reSeed(seed + partition)
    +      val seqOp = getSeqOp(withReplacement, fractions, rng, counts)
    +      Iterator(iter.aggregate(zeroU)(seqOp, combOp))
    +    }, preservesPartitioning=true)
    +    mappedPartitionRDD.reduce(combOp)
    +  }
    +
    +  /**
    +   * Returns the function used by aggregate to collect sampling statistics for each partition.
    +   */
    +  def getSeqOp[K, V](withReplacement: Boolean,
    +      fractions: Map[K, Double],
    +      rng: RandomDataGenerator,
    +      counts: Option[Map[K, Long]]): (MMap[K, Stratum], (K, V)) => MMap[K, Stratum] = {
    +    val delta = 5e-5
    +    (result: MMap[K, Stratum], item: (K, V)) => {
    +      val key = item._1
    +      val fraction = fractions(key)
    +      if (!result.contains(key)) {
    +        result += (key -> new Stratum())
    +      }
    +      val stratum = result(key)
    +
    +      if (withReplacement) {
    +        // compute acceptBound and waitListBound only if they haven't been computed already
    +        // since they don't change from iteration to iteration.
    +        // TODO change this to the streaming version
    +        if (stratum.areBoundsEmpty) {
    +          val n = counts.get(key)
    +          val sampleSize = math.ceil(n * fraction).toLong
    +          val lmbd1 = PoissonBounds.getLowerBound(sampleSize)
    +          val minCount = PoissonBounds.getMinCount(lmbd1)
    +          val lmbd2 = if (lmbd1 == 0) {
    +            PoissonBounds.getUpperBound(sampleSize)
    +          } else {
    +            PoissonBounds.getUpperBound(sampleSize - minCount)
    +          }
    +          stratum.acceptBound = lmbd1 / n
    +          stratum.waitListBound = lmbd2 / n
    --- End diff --
    
    Should be `(lmbd2 - lmbd1) / n`. Otherwise, we waitlist too many. Please check the waitlist size for some large n.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1025#discussion_r14694386
  
    --- Diff: core/src/test/scala/org/apache/spark/rdd/PairRDDFunctionsSuite.scala ---
    @@ -83,6 +83,125 @@ class PairRDDFunctionsSuite extends FunSuite with SharedSparkContext {
         assert(valuesFor2.toList.sorted === List(1))
       }
     
    +  test("sampleByKey") {
    +    def stratifier (fractionPositive: Double) = {
    +      (x: Int) => if (x % 10 < (10 * fractionPositive).toInt) "1" else "0"
    +    }
    +
    +    def checkSize(exact: Boolean,
    +        withReplacement: Boolean,
    +        expected: Long,
    +        actual: Long,
    +        p: Double): Boolean = {
    +      if (exact) {
    +        return expected == actual
    +      }
    +      val stdev = if (withReplacement) math.sqrt(expected) else math.sqrt(expected * p * (1 - p))
    +      // Very forgiving margin since we're dealing with very small sample sizes most of the time
    +      math.abs(actual - expected) <= 6 * stdev
    +    }
    +
    +    // Without replacement validation
    +    def takeSampleAndValidateBernoulli(stratifiedData: RDD[(String, Int)],
    +        exact: Boolean,
    +        samplingRate: Double,
    +        seed: Long,
    +        n: Long) = {
    +      val expectedSampleSize = stratifiedData.countByKey().mapValues(count =>
    +        math.ceil(count * samplingRate).toInt)
    +      val fractions = Map("1" -> samplingRate, "0" -> samplingRate)
    +      val sample = stratifiedData.sampleByKey(false, fractions, exact, seed)
    +      val sampleCounts = sample.countByKey()
    +      val takeSample = sample.collect()
    +      assert(sampleCounts.forall {case(k,v) =>
    +        checkSize(exact, false, expectedSampleSize(k), v, samplingRate)})
    +      assert(takeSample.size === takeSample.toSet.size)
    +      assert(takeSample.forall(x => 1 <= x._2 && x._2 <= n), s"elements not in [1, $n]")
    +    }
    +
    +    // With replacement validation
    +    def takeSampleAndValidatePoisson(stratifiedData: RDD[(String, Int)],
    +        exact: Boolean,
    +        samplingRate: Double,
    +        seed: Long,
    +        n: Long) = {
    +      val expectedSampleSize = stratifiedData.countByKey().mapValues(count =>
    +        math.ceil(count * samplingRate).toInt)
    +      val fractions = Map("1" -> samplingRate, "0" -> samplingRate)
    +      val sample = stratifiedData.sampleByKey(true, fractions, exact, seed)
    +      val sampleCounts = sample.countByKey()
    +      val takeSample = sample.collect()
    +      assert(sampleCounts.forall {case(k,v) =>
    +        checkSize(exact, true, expectedSampleSize(k), v, samplingRate)})
    +      val groupedByKey = takeSample.groupBy {case(k, v) => k}
    +      for ((key, v) <- groupedByKey) {
    +        if (expectedSampleSize(key) >= 100 && samplingRate >= 0.1) {
    +          // sample large enough for there to be repeats with high likelihood
    +          assert(v.toSet.size < expectedSampleSize(key))
    +        } else {
    +          if (exact) {
    +            assert(v.toSet.size <= expectedSampleSize(key))
    +          } else {
    +            assert(checkSize(false, true, expectedSampleSize(key), v.toSet.size, samplingRate))
    +          }
    +        }
    +      }
    +      assert(takeSample.forall(x => 1 <= x._2 && x._2 <= n), s"elements not in [1, $n]")
    +    }
    +
    +    def checkAllCombos(stratifiedData: RDD[(String, Int)],
    +        samplingRate: Double,
    +        seed: Long,
    +        n: Long) = {
    +      takeSampleAndValidateBernoulli(stratifiedData, true, samplingRate, seed, n)
    +      takeSampleAndValidateBernoulli(stratifiedData, false, samplingRate, seed, n)
    +      takeSampleAndValidatePoisson(stratifiedData, true, samplingRate, seed, n)
    +      takeSampleAndValidatePoisson(stratifiedData, false, samplingRate, seed, n)
    +    }
    +
    +    val defaultSeed = 42L
    +
    +    //vary RDD size
    --- End diff --
    
    add a space after `//`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/1025#issuecomment-50414198
  
    QA results for PR 1025:<br>- This patch PASSES unit tests.<br>- This patch merges cleanly<br>- This patch adds no public classes<br><br>For more information see test ouptut:<br>https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17307/consoleFull


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/1025#issuecomment-48989787
  
    QA results for PR 1025:<br>- This patch PASSES unit tests.<br>- This patch merges cleanly<br>- This patch adds no public classes<br><br>For more information see test ouptut:<br>https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16650/consoleFull


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1025#issuecomment-46644692
  
    Merged build finished. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1025#discussion_r14500845
  
    --- Diff: core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala ---
    @@ -46,6 +48,8 @@ import org.apache.spark.Partitioner.defaultPartitioner
     import org.apache.spark.SparkContext._
     import org.apache.spark.partial.{BoundedDouble, PartialResult}
     import org.apache.spark.serializer.Serializer
    +import org.apache.spark.util.Utils
    +import org.apache.spark.util.random.{Stratum, Result, StratifiedSampler, PoissonBounds => PB}
    --- End diff --
    
    Re-organize the imports (using an intellij plugin: http://plugins.jetbrains.com/plugin/7350?pr=)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by dorx <gi...@git.apache.org>.
Github user dorx commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1025#discussion_r14688550
  
    --- Diff: core/src/main/scala/org/apache/spark/util/random/SamplingUtils.scala ---
    @@ -45,11 +50,75 @@ private[spark] object SamplingUtils {
         val fraction = sampleSizeLowerBound.toDouble / total
         if (withReplacement) {
           val numStDev = if (sampleSizeLowerBound < 12) 9 else 5
    -      fraction + numStDev * math.sqrt(fraction / total)
    +      math.max(1e-10, fraction + numStDev * math.sqrt(fraction / total))
         } else {
           val delta = 1e-4
           val gamma = - math.log(delta) / total
    -      math.min(1, fraction + gamma + math.sqrt(gamma * gamma + 2 * gamma * fraction))
    +      math.min(1,
    +        math.max(1e-10, fraction + gamma + math.sqrt(gamma * gamma + 2 * gamma * fraction)))
    +    }
    +  }
    +}
    +
    +/**
    + * Utility functions that help us determine bounds on adjusted sampling rate to guarantee exact
    + * sample sizes with high confidence when sampling with replacement.
    + *
    + * The algorithm for guaranteeing sample size instantly accepts items whose associated value drawn
    + * from Pois(s) is less than the lower bound and puts items whose value is between the lower and
    + * upper bound in a waitlist. The final sample is consisted of all items accepted on the fly and a
    + * portion of the waitlist needed to make the exact sample size.
    + */
    +private[spark] object PoissonBounds {
    +
    +  val delta = 1e-4 / 3.0
    +  val epsilon = 1e-15
    +
    +  /**
    +   * Compute the threshold for accepting items on the fly. The threshold value is a fairly small
    +   * number, which means if the item has an associated value < threshold, it is highly likely to
    +   * be in the final sample. Hence we accept items with values less than the returned value of this
    +   * function instantly.
    +   *
    +   * @param s sample size
    +   * @return threshold for accepting items on the fly
    +   */
    +  def getLowerBound(s: Double): Double = {
    +    var lb = math.max(0.0, s - math.sqrt(s / delta)) // Chebyshev's inequality
    +    var ub = s
    +    while (lb < ub - 1.0) {
    +      val m = (lb + ub) / 2.0
    +      val poisson = new PoissonDistribution(m, epsilon)
    +      val y = poisson.inverseCumulativeProbability(1 - delta)
    +      if (y > s) ub = m else lb = m
    +    }
    +    lb
    +  }
    +
    +  def getMinCount(lmbd: Double): Double = {
    +    if (lmbd == 0) return 0
    --- End diff --
    
    agreed. this was a copypasta i didn't look too hard at.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1025#discussion_r13578489
  
    --- Diff: core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala ---
    @@ -156,6 +161,182 @@ class PairRDDFunctions[K, V](self: RDD[(K, V)])
       }
     
       /**
    +   * Return a subset of this RDD sampled by key (via stratified sampling).
    +   * We guarantee a sample size = math.ceil(fraction * S_i), where S_i is the size of the ith
    --- End diff --
    
    Also, it would be good to put a comment saying that this method requires two passes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1025#issuecomment-46391365
  
    All automated tests passed.
    Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15864/


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1025#issuecomment-45553302
  
    Build started. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the pull request:

    https://github.com/apache/spark/pull/1025#issuecomment-46643277
  
    Jenkins, retest this please.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1025#issuecomment-46643453
  
    Merged build started. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by dorx <gi...@git.apache.org>.
Github user dorx commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1025#discussion_r14906680
  
    --- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
    @@ -0,0 +1,311 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.util.random
    +
    +import scala.collection.Map
    +import scala.collection.mutable.{ArrayBuffer, HashMap, Map => MMap}
    --- End diff --
    
    There isn't a immutable equivalent of `HashMap`. Also, using `mutable.Map` instead of `MMap` (short for mutable.Map) might push a couple lines over the 100 char limit. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1025#issuecomment-47990764
  
    All automated tests passed.
    Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16315/


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1025#issuecomment-48386184
  
    
    Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16414/


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1025#discussion_r14694289
  
    --- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
    @@ -0,0 +1,311 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.util.random
    +
    +import scala.collection.Map
    +import scala.collection.mutable.{ArrayBuffer, HashMap, Map => MMap}
    +
    +import org.apache.commons.math3.random.RandomDataGenerator
    +import org.apache.spark.Logging
    +import org.apache.spark.rdd.RDD
    +
    +/**
    + * Auxiliary functions and data structures for the sampleByKey method in PairRDDFunctions.
    + *
    + * Essentially, when exact sample size is necessary, we make additional passes over the RDD to
    + * compute the exact threshold value to use for each stratum to guarantee exact sample size with
    + * high probability. This is achieved by maintaining a waitlist of size O(log(s)), where s is the
    + * desired sample size for each stratum.
    + *
    + * Like in simple random sampling, we generate a random value for each item from the
    + * uniform  distribution [0.0, 1.0]. All items with values <= min(values of items in the waitlist)
    + * are accepted into the sample instantly. The threshold for instant accept is designed so that
    + * s - numAccepted = O(log(s)), where s is again the desired sample size. Thus, by maintaining a
    + * waitlist size = O(log(s)), we will be able to create a sample of the exact size s by adding
    + * a portion of the waitlist to the set of items that are instantly accepted. The exact threshold
    + * is computed by sorting the values in the waitlist and picking the value at (s - numAccepted).
    + *
    + * Note that since we use the same seed for the RNG when computing the thresholds and the actual
    + * sample, our computed thresholds are guaranteed to produce the desired sample size.
    + *
    + * For more theoretical background on the sampling techniques used here, please refer to
    + * http://jmlr.org/proceedings/papers/v28/meng13a.html
    + */
    +
    +private[spark] object StratifiedSampler extends Logging {
    +
    +  /**
    +   * Count the number of items instantly accepted and generate the waitlist for each stratum.
    +   *
    +   * This is only invoked when exact sample size is required.
    +   */
    +  def getCounts[K, V](rdd: RDD[(K, V)],
    +      withReplacement: Boolean,
    +      fractions: Map[K, Double],
    +      counts: Option[Map[K, Long]],
    +      seed: Long): MMap[K, Stratum] = {
    +    val combOp = getCombOp[K]
    +    val mappedPartitionRDD = rdd.mapPartitionsWithIndex({ case (partition, iter) =>
    +      val zeroU: MMap[K, Stratum] = new HashMap[K, Stratum]()
    +      val rng = new RandomDataGenerator()
    +      rng.reSeed(seed + partition)
    +      val seqOp = getSeqOp(withReplacement, fractions, rng, counts)
    +      Iterator(iter.aggregate(zeroU)(seqOp, combOp))
    +    }, preservesPartitioning=true)
    +    mappedPartitionRDD.reduce(combOp)
    +  }
    +
    +  /**
    +   * Returns the function used by aggregate to collect sampling statistics for each partition.
    +   */
    +  def getSeqOp[K, V](withReplacement: Boolean,
    +      fractions: Map[K, Double],
    +      rng: RandomDataGenerator,
    +      counts: Option[Map[K, Long]]): (MMap[K, Stratum], (K, V)) => MMap[K, Stratum] = {
    +    val delta = 5e-5
    +    (result: MMap[K, Stratum], item: (K, V)) => {
    +      val key = item._1
    +      val fraction = fractions(key)
    +      if (!result.contains(key)) {
    +        result += (key -> new Stratum())
    +      }
    +      val stratum = result(key)
    +
    +      if (withReplacement) {
    +        // compute acceptBound and waitListBound only if they haven't been computed already
    +        // since they don't change from iteration to iteration.
    +        // TODO change this to the streaming version
    +        if (stratum.areBoundsEmpty) {
    +          val n = counts.get(key)
    +          val sampleSize = math.ceil(n * fraction).toLong
    +          val lmbd1 = PoissonBounds.getLowerBound(sampleSize)
    +          val minCount = PoissonBounds.getMinCount(lmbd1)
    +          val lmbd2 = if (lmbd1 == 0) {
    +            PoissonBounds.getUpperBound(sampleSize)
    +          } else {
    +            PoissonBounds.getUpperBound(sampleSize - minCount)
    +          }
    +          stratum.acceptBound = lmbd1 / n
    +          stratum.waitListBound = lmbd2 / n
    +        }
    +        val acceptBound = stratum.acceptBound
    +        val copiesAccepted = if (acceptBound == 0.0) 0L else rng.nextPoisson(acceptBound)
    +        if (copiesAccepted > 0) {
    +          stratum.incrNumAccepted(copiesAccepted)
    +        }
    +        val copiesWaitlisted = rng.nextPoisson(stratum.waitListBound).toInt
    +        if (copiesWaitlisted > 0) {
    +          stratum.addToWaitList(ArrayBuffer.fill(copiesWaitlisted)(rng.nextUniform(0.0, 1.0)))
    +        }
    +      } else {
    +        // We use the streaming version of the algorithm for sampling without replacement to avoid
    +        // using an extra pass over the RDD for computing the count.
    +        // Hence, acceptBound and waitListBound change on every iteration.
    +        val gamma1 = - math.log(delta) / stratum.numItems
    +        val gamma2 = (2.0 / 3.0) * gamma1
    +        stratum.acceptBound = math.max(0,
    +          fraction + gamma2 - math.sqrt(gamma2 * gamma2 + 3 * gamma2 * fraction))
    +        stratum.waitListBound = math.min(1,
    +          fraction + gamma1 + math.sqrt(gamma1 * gamma1 + 2 * gamma1 * fraction))
    +
    +        val x = rng.nextUniform(0.0, 1.0)
    +        if (x < stratum.acceptBound) {
    +          stratum.incrNumAccepted()
    +        } else if (x < stratum.waitListBound) {
    +          stratum.addToWaitList(x)
    +        }
    +      }
    +      stratum.incrNumItems()
    +      result
    +    }
    +  }
    +
    +  /**
    +   * Returns the function used combine results returned by seqOp from different partitions.
    +   */
    +  def getCombOp[K]: (MMap[K, Stratum], MMap[K, Stratum]) => MMap[K, Stratum] = {
    +    (result1: MMap[K, Stratum], result2: MMap[K, Stratum]) => {
    +      // take union of both key sets in case one partition doesn't contain all keys
    +      for (key <- result1.keySet.union(result2.keySet)) {
    +        // Use result2 to keep the combined result since r1 is usual empty
    +        val entry1 = result1.get(key)
    +        if (result2.contains(key)) {
    +          result2(key).merge(entry1)
    +        } else {
    +          if (entry1.isDefined) {
    +            result2 += (key -> entry1.get)
    +          }
    +        }
    +      }
    +      result2
    +    }
    +  }
    +
    +  /**
    +   * Given the result returned by getCounts, determine the threshold for accepting items to
    +   * generate exact sample size.
    +   *
    +   * To do so, we compute sampleSize = math.ceil(size * samplingRate) for each stratum and compare
    +   * it to the number of items that were accepted instantly and the number of items in the waitlist
    +   * for that stratum. Most of the time, numAccepted <= sampleSize <= (numAccepted + numWaitlisted),
    +   * which means we need to sort the elements in the waitlist by their associated values in order
    +   * to find the value T s.t. |{elements in the stratum whose associated values <= T}| = sampleSize.
    +   * Note that all elements in the waitlist have values >= bound for instant accept, so a T value
    +   * in the waitlist range would allow all elements that were instantly accepted on the first pass
    +   * to be included in the sample.
    +   */
    +  def computeThresholdByKey[K](finalResult: Map[K, Stratum],
    +      fractions: Map[K, Double]): Map[K, Double] = {
    +    val thresholdByKey = new HashMap[K, Double]()
    +    for ((key, stratum) <- finalResult) {
    +      val sampleSize = math.ceil(stratum.numItems * fractions(key)).toLong
    +      if (stratum.numAccepted > sampleSize) {
    +        logWarning("Pre-accepted too many")
    +        thresholdByKey += (key -> stratum.acceptBound)
    +      } else {
    +        val numWaitListAccepted = (sampleSize - stratum.numAccepted).toInt
    +        if (numWaitListAccepted >= stratum.waitList.size) {
    +          logWarning("WaitList too short")
    +          thresholdByKey += (key -> stratum.waitListBound)
    +        } else {
    +          thresholdByKey += (key -> stratum.waitList.sorted.apply(numWaitListAccepted))
    +        }
    +      }
    +    }
    +    thresholdByKey
    +  }
    +
    +  /**
    +   * Return the per partition sampling function used for sampling without replacement.
    +   *
    +   * When exact sample size is required, we make an additional pass over the RDD to determine the
    +   * exact sampling rate that guarantees sample size with high confidence.
    +   *
    +   * The sampling function has a unique seed per partition.
    +   */
    +  def getBernoulliSamplingFunction[K, V](rdd: RDD[(K,  V)],
    +      fractions: Map[K, Double],
    +      exact: Boolean,
    +      seed: Long): (Int, Iterator[(K, V)]) => Iterator[(K, V)] = {
    +    var samplingRateByKey = fractions
    +    if (exact) {
    +      // determine threshold for each stratum and resample
    +      val finalResult = getCounts(rdd, false, fractions, None, seed)
    +      samplingRateByKey = computeThresholdByKey(finalResult, fractions)
    +    }
    +    (idx: Int, iter: Iterator[(K, V)]) => {
    +      val rng = new RandomDataGenerator
    +      rng.reSeed(seed + idx)
    +      iter.filter(t => rng.nextUniform(0.0, 1.0) < samplingRateByKey(t._1))
    +    }
    +  }
    +
    +  /**
    +   * Return the per partition sampling function used for sampling with replacement.
    +   *
    +   * When exact sample size is required, we make two additional passed over the RDD to determine
    +   * the exact sampling rate that guarantees sample size with high confidence. The first pass
    +   * counts the number of items in each stratum (group of items with the same key) in the RDD, and
    +   * the second pass uses the counts to determine exact sampling rates.
    +   *
    +   * The sampling function has a unique seed per partition.
    +   */
    +  def getPoissonSamplingFunction[K, V](rdd: RDD[(K, V)],
    +      fractions: Map[K, Double],
    +      exact: Boolean,
    +      counts: Option[Map[K, Long]],
    --- End diff --
    
    Remove counts and compute it inside the function. Since this is only needed for Poisson and when `exact == true`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1025#discussion_r14694233
  
    --- Diff: core/src/main/scala/org/apache/spark/api/java/JavaPairRDD.scala ---
    @@ -130,6 +130,38 @@ class JavaPairRDD[K, V](val rdd: RDD[(K, V)])
         new JavaPairRDD[K, V](rdd.sample(withReplacement, fraction, seed))
     
       /**
    +   * Return a subset of this RDD sampled by key (via stratified sampling).
    --- End diff --
    
    Needs more doc here. And for the overloaded methods below, please write down the default values, especially when `exact` is omitted.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1025#discussion_r14694262
  
    --- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
    @@ -0,0 +1,311 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.util.random
    +
    +import scala.collection.Map
    +import scala.collection.mutable.{ArrayBuffer, HashMap, Map => MMap}
    +
    +import org.apache.commons.math3.random.RandomDataGenerator
    +import org.apache.spark.Logging
    +import org.apache.spark.rdd.RDD
    +
    +/**
    + * Auxiliary functions and data structures for the sampleByKey method in PairRDDFunctions.
    + *
    + * Essentially, when exact sample size is necessary, we make additional passes over the RDD to
    + * compute the exact threshold value to use for each stratum to guarantee exact sample size with
    + * high probability. This is achieved by maintaining a waitlist of size O(log(s)), where s is the
    --- End diff --
    
    Should be `O(sqrt(s))` for sampling without replacement.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1025#issuecomment-45547151
  
     Build triggered. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1025#issuecomment-48419784
  
    Merged build started. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1025#issuecomment-46520000
  
    
    Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15892/


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by falaki <gi...@git.apache.org>.
Github user falaki commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1025#discussion_r14543734
  
    --- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
    @@ -0,0 +1,296 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.util.random
    +
    +import scala.collection.{Map, mutable}
    +import scala.collection.mutable.ArrayBuffer
    +import scala.reflect.ClassTag
    +
    +import org.apache.commons.math3.random.RandomDataGenerator
    +import org.apache.spark.{Logging, SparkContext, TaskContext}
    +import org.apache.spark.rdd.RDD
    +import org.apache.spark.util.Utils
    +import org.apache.spark.util.random.{PoissonBounds => PB}
    +
    +private[spark] object StratifiedSampler extends Logging {
    +
    +  /**
    +   * A version of {@link #aggregate()} that passes the TaskContext to the function that does
    +   * aggregation for each partition. This function avoids creating an extra depth in the RDD
    +   * lineage, as opposed to using mapPartitionsWithId, which results in slightly improved run time.
    +   */
    +  def aggregateWithContext[U: ClassTag, T: ClassTag](zeroValue: U)
    +      (rdd: RDD[T],
    +       seqOp: ((TaskContext, U), T) => U,
    +       combOp: (U, U) => U): U = {
    +    val sc: SparkContext = rdd.sparkContext
    +    // Clone the zero value since we will also be serializing it as part of tasks
    +    var jobResult = Utils.clone(zeroValue, sc.env.closureSerializer.newInstance())
    +    // pad seqOp and combOp with taskContext to conform to aggregate's signature in TraversableOnce
    +    val paddedSeqOp = (arg1: (TaskContext, U), item: T) => (arg1._1, seqOp(arg1, item))
    +    val paddedcombOp = (arg1: (TaskContext, U), arg2: (TaskContext, U)) =>
    +      (arg1._1, combOp(arg1._2, arg1._2))
    +    val cleanSeqOp = sc.clean(paddedSeqOp)
    +    val cleanCombOp = sc.clean(paddedcombOp)
    +    val aggregatePartition = (tc: TaskContext, it: Iterator[T]) =>
    +      (it.aggregate(tc, zeroValue)(cleanSeqOp, cleanCombOp))._2
    +    val mergeResult = (index: Int, taskResult: U) => jobResult = combOp(jobResult, taskResult)
    +    sc.runJob(rdd, aggregatePartition, mergeResult)
    +    jobResult
    +  }
    +
    +  /**
    +   * Returns the function used by aggregate to collect sampling statistics for each partition.
    +   */
    +  def getSeqOp[K, V](withReplacement: Boolean,
    +      fractionByKey: (K => Double),
    +      counts: Option[Map[K, Long]]): ((TaskContext, Result[K]),(K, V)) => Result[K] = {
    +    val delta = 5e-5
    +    (U: (TaskContext, Result[K]), item: (K, V)) => {
    +      val result = U._2
    +      val tc = U._1
    +      val rng = result.getRand(tc.partitionId)
    +      val fraction = fractionByKey(item._1)
    +      val stratum = result.getEntry(item._1)
    +      if (withReplacement) {
    +        // compute q1 and q2 only if they haven't been computed already
    +        // since they don't change from iteration to iteration.
    +        // TODO change this to the streaming version
    +        if (stratum.q1.isEmpty || stratum.q2.isEmpty) {
    +          val n = counts.get(item._1)
    +          val s = math.ceil(n * fraction).toLong
    +          val lmbd1 = PB.getLowerBound(s)
    +          val minCount = PB.getMinCount(lmbd1)
    +          val lmbd2 = if (lmbd1 == 0) PB.getUpperBound(s) else PB.getUpperBound(s - minCount)
    +          val q1 = lmbd1 / n
    +          val q2 = lmbd2 / n
    +          stratum.q1 = Some(q1)
    +          stratum.q2 = Some(q2)
    +        }
    +        val x1 = if (stratum.q1.get == 0) 0L else rng.nextPoisson(stratum.q1.get)
    +        if (x1 > 0) {
    +          stratum.incrNumAccepted(x1)
    +        }
    +        val x2 = rng.nextPoisson(stratum.q2.get).toInt
    +        if (x2 > 0) {
    +          stratum.addToWaitList(ArrayBuffer.fill(x2)(rng.nextUniform(0.0, 1.0)))
    +        }
    +      } else {
    +        // We use the streaming version of the algorithm for sampling without replacement.
    +        // Hence, q1 and q2 change on every iteration.
    +        val g1 = - math.log(delta) / stratum.numItems
    +        val g2 = (2.0 / 3.0) * g1
    +        val q1 = math.max(0, fraction + g2 - math.sqrt((g2 * g2 + 3 * g2 * fraction)))
    +        val q2 = math.min(1, fraction + g1 + math.sqrt(g1 * g1 + 2 * g1 * fraction))
    +
    +        val x = rng.nextUniform(0.0, 1.0)
    +        if (x < q1) {
    +          stratum.incrNumAccepted()
    +        } else if (x < q2) {
    +          stratum.addToWaitList(x)
    +        }
    +        stratum.q1 = Some(q1)
    +        stratum.q2 = Some(q2)
    +      }
    +      stratum.incrNumItems()
    +      result
    +    }
    +  }
    +
    +  /**
    +   * Returns the function used by aggregate to combine results from different partitions, as
    +   * returned by seqOp.
    +   */
    +  def getCombOp[K](): (Result[K], Result[K]) => Result[K] = {
    +    (r1: Result[K], r2: Result[K]) => {
    +      // take union of both key sets in case one partition doesn't contain all keys
    +      val keyUnion = r1.resultMap.keys.toSet.union(r2.resultMap.keys.toSet)
    +
    +      // Use r2 to keep the combined result since r1 is usual empty
    +      for (key <- keyUnion) {
    +        val entry1 = r1.resultMap.get(key)
    +        val entry2 = r2.resultMap.get(key)
    +        if (entry2.isEmpty && entry1.isDefined) {
    +          r2.resultMap += (key -> entry1.get)
    +        } else if (entry1.isDefined && entry2.isDefined) {
    +          entry2.get.addToWaitList(entry1.get.waitList)
    +          entry2.get.incrNumAccepted(entry1.get.numAccepted)
    +          entry2.get.incrNumItems(entry1.get.numItems)
    +        }
    +      }
    +      r2
    +    }
    +  }
    +
    +  /**
    +   * Given the result returned by the aggregate function, we need to determine the threshold used
    +   * to accept items to generate the exact sample size.
    +   */
    +  def computeThresholdByKey[K](finalResult: Map[K, Stratum], fractionByKey: (K => Double)):
    +    (K => Double) = {
    +    val thresholdByKey = new mutable.HashMap[K, Double]()
    +    for ((key, stratum) <- finalResult) {
    +      val fraction = fractionByKey(key)
    +      val s = math.ceil(stratum.numItems * fraction).toLong
    +      if (stratum.numAccepted > s) {
    +        logWarning("Pre-accepted too many")
    +        thresholdByKey += (key -> stratum.q1.get)
    +      } else {
    +        val numWaitListAccepted = (s - stratum.numAccepted).toInt
    +        if (numWaitListAccepted >= stratum.waitList.size) {
    +          logWarning("WaitList too short")
    +          thresholdByKey += (key -> stratum.q2.get)
    +        } else {
    +          thresholdByKey += (key -> stratum.waitList.sorted.apply(numWaitListAccepted))
    +        }
    +      }
    +    }
    +    thresholdByKey
    +  }
    +
    +  /**
    +   * Return the per partition sampling function used for sampling without replacement.
    +   *
    +   * When exact sample size is required, we make an additional pass over the RDD to determine the
    +   * exact sampling rate that guarantees sample size with high confidence.
    +   *
    +   * The sampling function has a unique seed per partition.
    +   */
    +  def getBernoulliSamplingFunction[K, V](rdd: RDD[(K,  V)],
    +      fractionByKey: K => Double,
    +      exact: Boolean,
    +      seed: Long): (Int, Iterator[(K, V)]) => Iterator[(K, V)] = {
    +    var samplingRateByKey = fractionByKey
    +    if (exact) {
    +      // determine threshold for each stratum and resample
    +      val seqOp = StratifiedSampler.getSeqOp[K,V](false, fractionByKey, None)
    +      val combOp = StratifiedSampler.getCombOp[K]()
    +      val zeroU = new Result[K](Map[K, Stratum](), seed = seed)
    +      val finalResult = aggregateWithContext(zeroU)(rdd, seqOp, combOp).resultMap
    +      samplingRateByKey = StratifiedSampler.computeThresholdByKey(finalResult, fractionByKey)
    +    }
    +    (idx: Int, iter: Iterator[(K, V)]) => {
    +      val random = new RandomDataGenerator
    +      random.reSeed(seed + idx)
    +      iter.filter(t => random.nextUniform(0.0, 1.0) < samplingRateByKey(t._1))
    +    }
    +  }
    +
    +  /**
    +   * Return the per partition sampling function used for sampling with replacement.
    +   *
    +   * When exact sample size is required, we make two additional passed over the RDD to determine
    +   * the exact sampling rate that guarantees sample size with high confidence. The first pass
    +   * counts the number of items in each stratum (group of items with the same key) in the RDD, and
    +   * the second pass uses the counts to determine exact sampling rates.
    +   *
    +   * The sampling function has a unique seed per partition.
    +   */
    +  def getPoissonSamplingFunction[K, V](rdd:RDD[(K,  V)],
    --- End diff --
    
    Nit: add a space after colon 
    ```rdd: Rdd[(K, V)]```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by falaki <gi...@git.apache.org>.
Github user falaki commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1025#discussion_r14688702
  
    --- Diff: core/src/test/scala/org/apache/spark/rdd/PairRDDFunctionsSuite.scala ---
    @@ -83,6 +83,120 @@ class PairRDDFunctionsSuite extends FunSuite with SharedSparkContext {
         assert(valuesFor2.toList.sorted === List(1))
       }
     
    +  test("sampleByKey") {
    +    def stratifier (fractionPositive: Double) = {
    +      (x: Int) => if (x % 10 < (10 * fractionPositive).toInt) "1" else "0"
    +    }
    +
    +    def checkSize(exact: Boolean, withReplacement: Boolean,
    +        expected: Long, actual: Long, p: Double): Boolean = {
    +      if (exact) {
    +        return expected == actual
    +      }
    +      val stdev = if (withReplacement) math.sqrt(expected) else math.sqrt(expected * p * (1 - p))
    +      // Very forgiving margin since we're dealing with very small sample sizes most of the time
    +      math.abs(actual - expected) <= 6 * stdev
    +    }
    +
    +    // Without replacement validation
    +    def takeSampleAndValidateBernoulli(stratifiedData: RDD[(String, Int)],
    +        exact: Boolean,
    +        samplingRate: Double,
    +        seed: Long,
    +        n: Long) = {
    +      val expectedSampleSize = stratifiedData.countByKey().mapValues(count =>
    +        math.ceil(count * samplingRate).toInt)
    +      val fractions = Map("1" -> samplingRate, "0" -> samplingRate)
    +      val sample = stratifiedData.sampleByKey(false, fractions, exact, seed)
    +      val sampleCounts = sample.countByKey()
    +      val takeSample = sample.collect()
    +      assert(sampleCounts.forall({case(k,v) =>
    +        checkSize(exact, false, expectedSampleSize(k), v, samplingRate)}))
    +      assert(takeSample.size === takeSample.toSet.size)
    +      assert(takeSample.forall(x => 1 <= x._2 && x._2 <= n), s"elements not in [1, $n]")
    +    }
    +
    +    // With replacement validation
    +    def takeSampleAndValidatePoisson(stratifiedData: RDD[(String, Int)],
    +        exact: Boolean,
    +        samplingRate: Double,
    +        seed: Long,
    +        n: Long) = {
    +      val expectedSampleSize = stratifiedData.countByKey().mapValues(count =>
    +        math.ceil(count * samplingRate).toInt)
    +      val fractions = Map("1" -> samplingRate, "0" -> samplingRate)
    +      val sample = stratifiedData.sampleByKey(true, fractions, exact, seed)
    +      val sampleCounts = sample.countByKey()
    +      val takeSample = sample.collect()
    +      assert(sampleCounts.forall({case(k,v) =>
    +        checkSize(exact, true, expectedSampleSize(k), v, samplingRate)}))
    +      val groupedByKey = takeSample.groupBy({case(k, v) => k})
    +      for ((key, v) <- groupedByKey) {
    +        if (expectedSampleSize(key) >= 100 && samplingRate >= 0.1) {
    +          // sample large enough for there to be repeats with high likelihood
    +          assert(v.toSet.size < expectedSampleSize(key))
    +        } else {
    +          if (exact) {
    +            assert(v.toSet.size <= expectedSampleSize(key))
    +          } else {
    +            assert(checkSize(false, true, expectedSampleSize(key), v.toSet.size, samplingRate))
    +          }
    +        }
    +      }
    +      assert(takeSample.forall(x => 1 <= x._2 && x._2 <= n), s"elements not in [1, $n]")
    +    }
    +
    +    def checkAllCombos(stratifiedData: RDD[(String, Int)], samplingRate: Double,
    --- End diff --
    
    Style nit: When you break lines between parameters of a function, do it for ever one of them.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/1025#issuecomment-50527473
  
    QA results for PR 1025:<br>- This patch PASSES unit tests.<br>- This patch merges cleanly<br>- This patch adds no public classes<br><br>For more information see test ouptut:<br>https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17368/consoleFull


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1025#issuecomment-45951353
  
    Build started. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by dorx <gi...@git.apache.org>.
Github user dorx commented on the pull request:

    https://github.com/apache/spark/pull/1025#issuecomment-46389694
  
    @falaki @mengxr Barring the API related issues, everything else should be in the "final" state. commons-math3 added as a Spark dependency (okay'ed by Matei). mvn clean install compiled.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1025#issuecomment-46377057
  
    Merged build finished. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by dorx <gi...@git.apache.org>.
Github user dorx commented on the pull request:

    https://github.com/apache/spark/pull/1025#issuecomment-45666934
  
    @falaki


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1025#issuecomment-48384908
  
    Merged build started. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1025#discussion_r14694383
  
    --- Diff: core/src/test/scala/org/apache/spark/rdd/PairRDDFunctionsSuite.scala ---
    @@ -83,6 +83,125 @@ class PairRDDFunctionsSuite extends FunSuite with SharedSparkContext {
         assert(valuesFor2.toList.sorted === List(1))
       }
     
    +  test("sampleByKey") {
    +    def stratifier (fractionPositive: Double) = {
    +      (x: Int) => if (x % 10 < (10 * fractionPositive).toInt) "1" else "0"
    +    }
    +
    +    def checkSize(exact: Boolean,
    +        withReplacement: Boolean,
    +        expected: Long,
    +        actual: Long,
    +        p: Double): Boolean = {
    +      if (exact) {
    +        return expected == actual
    +      }
    +      val stdev = if (withReplacement) math.sqrt(expected) else math.sqrt(expected * p * (1 - p))
    +      // Very forgiving margin since we're dealing with very small sample sizes most of the time
    +      math.abs(actual - expected) <= 6 * stdev
    +    }
    +
    +    // Without replacement validation
    +    def takeSampleAndValidateBernoulli(stratifiedData: RDD[(String, Int)],
    +        exact: Boolean,
    +        samplingRate: Double,
    +        seed: Long,
    +        n: Long) = {
    +      val expectedSampleSize = stratifiedData.countByKey().mapValues(count =>
    +        math.ceil(count * samplingRate).toInt)
    +      val fractions = Map("1" -> samplingRate, "0" -> samplingRate)
    +      val sample = stratifiedData.sampleByKey(false, fractions, exact, seed)
    +      val sampleCounts = sample.countByKey()
    +      val takeSample = sample.collect()
    +      assert(sampleCounts.forall {case(k,v) =>
    +        checkSize(exact, false, expectedSampleSize(k), v, samplingRate)})
    +      assert(takeSample.size === takeSample.toSet.size)
    +      assert(takeSample.forall(x => 1 <= x._2 && x._2 <= n), s"elements not in [1, $n]")
    +    }
    +
    +    // With replacement validation
    +    def takeSampleAndValidatePoisson(stratifiedData: RDD[(String, Int)],
    +        exact: Boolean,
    +        samplingRate: Double,
    +        seed: Long,
    +        n: Long) = {
    +      val expectedSampleSize = stratifiedData.countByKey().mapValues(count =>
    +        math.ceil(count * samplingRate).toInt)
    +      val fractions = Map("1" -> samplingRate, "0" -> samplingRate)
    +      val sample = stratifiedData.sampleByKey(true, fractions, exact, seed)
    +      val sampleCounts = sample.countByKey()
    +      val takeSample = sample.collect()
    +      assert(sampleCounts.forall {case(k,v) =>
    +        checkSize(exact, true, expectedSampleSize(k), v, samplingRate)})
    +      val groupedByKey = takeSample.groupBy {case(k, v) => k}
    --- End diff --
    
    `groupBy(_._1)`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by dorx <gi...@git.apache.org>.
Github user dorx commented on the pull request:

    https://github.com/apache/spark/pull/1025#issuecomment-48386518
  
    Jenkins, retest this please.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...

Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1025#discussion_r14694254
  
    --- Diff: core/src/main/scala/org/apache/spark/util/random/SamplingUtils.scala ---
    @@ -45,11 +50,78 @@ private[spark] object SamplingUtils {
         val fraction = sampleSizeLowerBound.toDouble / total
         if (withReplacement) {
           val numStDev = if (sampleSizeLowerBound < 12) 9 else 5
    -      fraction + numStDev * math.sqrt(fraction / total)
    +      math.max(1e-10, fraction + numStDev * math.sqrt(fraction / total))
         } else {
           val delta = 1e-4
           val gamma = - math.log(delta) / total
    -      math.min(1, fraction + gamma + math.sqrt(gamma * gamma + 2 * gamma * fraction))
    +      math.min(1,
    +        math.max(1e-10, fraction + gamma + math.sqrt(gamma * gamma + 2 * gamma * fraction)))
    +    }
    +  }
    +}
    +
    +/**
    + * Utility functions that help us determine bounds on adjusted sampling rate to guarantee exact
    + * sample sizes with high confidence when sampling with replacement.
    + *
    + * The algorithm for guaranteeing sample size instantly accepts items whose associated value drawn
    + * from Pois(s) is less than the lower bound and puts items whose value is between the lower and
    + * upper bound in a waitlist. The final sample is consisted of all items accepted on the fly and a
    + * portion of the waitlist needed to make the exact sample size.
    + */
    +private[spark] object PoissonBounds {
    +
    +  val delta = 1e-4 / 3.0
    +  val epsilon = 1e-15
    +
    +  /**
    +   * Compute the threshold for accepting items on the fly. The threshold value is a fairly small
    +   * number, which means if the item has an associated value < threshold, it is highly likely to
    +   * be in the final sample. Hence we accept items with values less than the returned value of this
    +   * function instantly.
    +   *
    +   * @param s sample size
    +   * @return threshold for accepting items on the fly
    +   */
    +  def getLowerBound(s: Double): Double = {
    +    var lb = math.max(0.0, s - math.sqrt(s / delta)) // Chebyshev's inequality
    +    var ub = s
    +    while (lb < ub - 1.0) {
    +      val m = (lb + ub) / 2.0
    +      val poisson = new PoissonDistribution(m, epsilon)
    +      val y = poisson.inverseCumulativeProbability(1 - delta)
    +      if (y > s) ub = m else lb = m
    +    }
    +    lb
    +  }
    +
    +  def getMinCount(lmbd: Double): Double = {
    --- End diff --
    
    doc?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---