You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by dorx <gi...@git.apache.org> on 2014/06/09 23:06:49 UTC
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
GitHub user dorx opened a pull request:
https://github.com/apache/spark/pull/1025
[SPARK-2082] stratified sampling in PairRDDFunctions that guarantees exact sample size
Implemented stratified sampling that guarantees exact sample size using ScaRSR with two passes over the RDD for sampling without replacement and three passes for sampling with replacement.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/dorx/spark stratified
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/1025.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1025
----
commit 14419775202e6eef1f0e1f0c74c7be9030aca73d
Author: Doris Xin <do...@gmail.com>
Date: 2014-05-29T22:22:14Z
SPARK-1939 Refactor takeSample method in RDD to use ScaSRS
commit ffea61a67d228edb476d29ca13a84bb3f9a22887
Author: Doris Xin <do...@gmail.com>
Date: 2014-05-30T00:55:54Z
SPARK-1939: Refactor takeSample method in RDD
Reviewer comments addressed:
- commons-math3 is now a test-only dependency. bumped up to v3.3
- comments added to explain what computeFraction is doing
- fixed the unit for computeFraction to use BinomialDitro for without
replacement sampling
- stylistic fixes
commit 7cab53a3926f4351432e5e3600b0796b9a4146e4
Author: Doris Xin <do...@gmail.com>
Date: 2014-06-02T19:00:38Z
fixed import bug in rdd.py
commit e3fd6a628317d559a08a7a20421e9c0618180902
Author: Doris Xin <do...@gmail.com>
Date: 2014-06-02T19:06:18Z
Merge branch 'master' into takeSample
commit 9ee94ee3c28e8d808063fef4e5d39f06ab738e0b
Author: Doris Xin <do...@gmail.com>
Date: 2014-06-09T20:15:23Z
[SPARK-2082] stratified sampling in PairRDDFunctions that guarantees exact sample size
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the pull request:
https://github.com/apache/spark/pull/1025#issuecomment-50290028
@dorx I removed commons-math3 from dependencies, separated `sampleByKey` and `sampleByKeyExact`, and corrected the math in waitlisting in sampling with replacement.
Other than those changes, the refactoring was completely based on your implementation. I did some renaming but I'm not taking any credit for the implementation. Some of the refactoring was unnecessary, which was partially because I did this around 5am and my head was not clear. I'm really sorry for the unnecessary code changes.
Since this is apparently your contribution, do you mind re-opening the PR? Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by dorx <gi...@git.apache.org>.
Github user dorx closed the pull request at:
https://github.com/apache/spark/pull/1025
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1025#issuecomment-46518417
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1025#issuecomment-47984385
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by dorx <gi...@git.apache.org>.
Github user dorx commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r14577393
--- Diff: core/src/main/scala/org/apache/spark/util/random/SamplingUtils.scala ---
@@ -45,11 +50,74 @@ private[spark] object SamplingUtils {
val fraction = sampleSizeLowerBound.toDouble / total
if (withReplacement) {
val numStDev = if (sampleSizeLowerBound < 12) 9 else 5
- fraction + numStDev * math.sqrt(fraction / total)
+ math.max(1e-10, fraction + numStDev * math.sqrt(fraction / total))
} else {
val delta = 1e-4
val gamma = - math.log(delta) / total
- math.min(1, fraction + gamma + math.sqrt(gamma * gamma + 2 * gamma * fraction))
+ math.min(1,
+ math.max(1e-10, fraction + gamma + math.sqrt(gamma * gamma + 2 * gamma * fraction)))
+ }
+ }
+}
+
+/**
+ * Utility functions that help us determine bounds on adjusted sampling rate to guarantee exact
+ * sample sizes with high confidence when sampling with replacement.
+ *
+ * The algorithm for guaranteeing sample size instantly accepts items whose associated value drawn
+ * from Pois(s) is less than the lower bound and puts items whose value is between the lower and
+ * upper bound in a waitlist. The final sample is consisted of all items accepted on the fly and a
+ * portion of the waitlist needed to make the exact sample size.
+ */
+private[spark] object PoissonBounds {
+
+ val delta = 1e-4 / 3.0
+
+ /**
+ * Compute the threshold for accepting items on the fly. The threshold value is a fairly small
+ * number, which means if the item has an associated value < threshold, it is highly likely to
+ * be in the final sample. Hence we accept items with values less than the returned value of this
+ * function instantly.
+ *
+ * @param s sample size
+ * @return threshold for accepting items on the fly
+ */
+ def getLowerBound(s: Double): Double = {
--- End diff --
Actually on a second thought, the implementation becomes unnecessarily muddled if we combine the two functions. I'd prefer to leave them as is for clarity.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by falaki <gi...@git.apache.org>.
Github user falaki commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r14544802
--- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util.random
+
+import scala.collection.{Map, mutable}
+import scala.collection.mutable.ArrayBuffer
+import scala.reflect.ClassTag
+
+import org.apache.commons.math3.random.RandomDataGenerator
+import org.apache.spark.{Logging, SparkContext, TaskContext}
+import org.apache.spark.rdd.RDD
+import org.apache.spark.util.Utils
+import org.apache.spark.util.random.{PoissonBounds => PB}
+
+private[spark] object StratifiedSampler extends Logging {
+
+ /**
+ * A version of {@link #aggregate()} that passes the TaskContext to the function that does
+ * aggregation for each partition. This function avoids creating an extra depth in the RDD
+ * lineage, as opposed to using mapPartitionsWithId, which results in slightly improved run time.
+ */
+ def aggregateWithContext[U: ClassTag, T: ClassTag](zeroValue: U)
+ (rdd: RDD[T],
+ seqOp: ((TaskContext, U), T) => U,
+ combOp: (U, U) => U): U = {
+ val sc: SparkContext = rdd.sparkContext
+ // Clone the zero value since we will also be serializing it as part of tasks
+ var jobResult = Utils.clone(zeroValue, sc.env.closureSerializer.newInstance())
+ // pad seqOp and combOp with taskContext to conform to aggregate's signature in TraversableOnce
+ val paddedSeqOp = (arg1: (TaskContext, U), item: T) => (arg1._1, seqOp(arg1, item))
+ val paddedcombOp = (arg1: (TaskContext, U), arg2: (TaskContext, U)) =>
+ (arg1._1, combOp(arg1._2, arg1._2))
+ val cleanSeqOp = sc.clean(paddedSeqOp)
+ val cleanCombOp = sc.clean(paddedcombOp)
+ val aggregatePartition = (tc: TaskContext, it: Iterator[T]) =>
+ (it.aggregate(tc, zeroValue)(cleanSeqOp, cleanCombOp))._2
+ val mergeResult = (index: Int, taskResult: U) => jobResult = combOp(jobResult, taskResult)
+ sc.runJob(rdd, aggregatePartition, mergeResult)
+ jobResult
+ }
+
+ /**
+ * Returns the function used by aggregate to collect sampling statistics for each partition.
+ */
+ def getSeqOp[K, V](withReplacement: Boolean,
+ fractionByKey: (K => Double),
+ counts: Option[Map[K, Long]]): ((TaskContext, Result[K]),(K, V)) => Result[K] = {
+ val delta = 5e-5
+ (U: (TaskContext, Result[K]), item: (K, V)) => {
+ val result = U._2
+ val tc = U._1
+ val rng = result.getRand(tc.partitionId)
+ val fraction = fractionByKey(item._1)
+ val stratum = result.getEntry(item._1)
+ if (withReplacement) {
+ // compute q1 and q2 only if they haven't been computed already
+ // since they don't change from iteration to iteration.
+ // TODO change this to the streaming version
+ if (stratum.q1.isEmpty || stratum.q2.isEmpty) {
+ val n = counts.get(item._1)
+ val s = math.ceil(n * fraction).toLong
+ val lmbd1 = PB.getLowerBound(s)
+ val minCount = PB.getMinCount(lmbd1)
+ val lmbd2 = if (lmbd1 == 0) PB.getUpperBound(s) else PB.getUpperBound(s - minCount)
+ val q1 = lmbd1 / n
+ val q2 = lmbd2 / n
+ stratum.q1 = Some(q1)
+ stratum.q2 = Some(q2)
+ }
+ val x1 = if (stratum.q1.get == 0) 0L else rng.nextPoisson(stratum.q1.get)
+ if (x1 > 0) {
+ stratum.incrNumAccepted(x1)
+ }
+ val x2 = rng.nextPoisson(stratum.q2.get).toInt
+ if (x2 > 0) {
+ stratum.addToWaitList(ArrayBuffer.fill(x2)(rng.nextUniform(0.0, 1.0)))
+ }
+ } else {
+ // We use the streaming version of the algorithm for sampling without replacement.
+ // Hence, q1 and q2 change on every iteration.
+ val g1 = - math.log(delta) / stratum.numItems
+ val g2 = (2.0 / 3.0) * g1
+ val q1 = math.max(0, fraction + g2 - math.sqrt((g2 * g2 + 3 * g2 * fraction)))
+ val q2 = math.min(1, fraction + g1 + math.sqrt(g1 * g1 + 2 * g1 * fraction))
+
+ val x = rng.nextUniform(0.0, 1.0)
+ if (x < q1) {
+ stratum.incrNumAccepted()
+ } else if (x < q2) {
+ stratum.addToWaitList(x)
+ }
+ stratum.q1 = Some(q1)
+ stratum.q2 = Some(q2)
+ }
+ stratum.incrNumItems()
+ result
+ }
+ }
+
+ /**
+ * Returns the function used by aggregate to combine results from different partitions, as
+ * returned by seqOp.
+ */
+ def getCombOp[K](): (Result[K], Result[K]) => Result[K] = {
+ (r1: Result[K], r2: Result[K]) => {
+ // take union of both key sets in case one partition doesn't contain all keys
+ val keyUnion = r1.resultMap.keys.toSet.union(r2.resultMap.keys.toSet)
+
+ // Use r2 to keep the combined result since r1 is usual empty
+ for (key <- keyUnion) {
+ val entry1 = r1.resultMap.get(key)
+ val entry2 = r2.resultMap.get(key)
+ if (entry2.isEmpty && entry1.isDefined) {
+ r2.resultMap += (key -> entry1.get)
+ } else if (entry1.isDefined && entry2.isDefined) {
+ entry2.get.addToWaitList(entry1.get.waitList)
+ entry2.get.incrNumAccepted(entry1.get.numAccepted)
+ entry2.get.incrNumItems(entry1.get.numItems)
+ }
+ }
+ r2
+ }
+ }
+
+ /**
+ * Given the result returned by the aggregate function, we need to determine the threshold used
+ * to accept items to generate the exact sample size.
+ */
+ def computeThresholdByKey[K](finalResult: Map[K, Stratum], fractionByKey: (K => Double)):
+ (K => Double) = {
+ val thresholdByKey = new mutable.HashMap[K, Double]()
+ for ((key, stratum) <- finalResult) {
+ val fraction = fractionByKey(key)
+ val s = math.ceil(stratum.numItems * fraction).toLong
+ if (stratum.numAccepted > s) {
+ logWarning("Pre-accepted too many")
+ thresholdByKey += (key -> stratum.q1.get)
+ } else {
+ val numWaitListAccepted = (s - stratum.numAccepted).toInt
+ if (numWaitListAccepted >= stratum.waitList.size) {
+ logWarning("WaitList too short")
+ thresholdByKey += (key -> stratum.q2.get)
+ } else {
+ thresholdByKey += (key -> stratum.waitList.sorted.apply(numWaitListAccepted))
+ }
+ }
+ }
+ thresholdByKey
+ }
+
+ /**
+ * Return the per partition sampling function used for sampling without replacement.
+ *
+ * When exact sample size is required, we make an additional pass over the RDD to determine the
+ * exact sampling rate that guarantees sample size with high confidence.
+ *
+ * The sampling function has a unique seed per partition.
+ */
+ def getBernoulliSamplingFunction[K, V](rdd: RDD[(K, V)],
+ fractionByKey: K => Double,
+ exact: Boolean,
+ seed: Long): (Int, Iterator[(K, V)]) => Iterator[(K, V)] = {
+ var samplingRateByKey = fractionByKey
+ if (exact) {
+ // determine threshold for each stratum and resample
+ val seqOp = StratifiedSampler.getSeqOp[K,V](false, fractionByKey, None)
+ val combOp = StratifiedSampler.getCombOp[K]()
+ val zeroU = new Result[K](Map[K, Stratum](), seed = seed)
+ val finalResult = aggregateWithContext(zeroU)(rdd, seqOp, combOp).resultMap
+ samplingRateByKey = StratifiedSampler.computeThresholdByKey(finalResult, fractionByKey)
+ }
+ (idx: Int, iter: Iterator[(K, V)]) => {
+ val random = new RandomDataGenerator
+ random.reSeed(seed + idx)
+ iter.filter(t => random.nextUniform(0.0, 1.0) < samplingRateByKey(t._1))
+ }
+ }
+
+ /**
+ * Return the per partition sampling function used for sampling with replacement.
+ *
+ * When exact sample size is required, we make two additional passed over the RDD to determine
+ * the exact sampling rate that guarantees sample size with high confidence. The first pass
+ * counts the number of items in each stratum (group of items with the same key) in the RDD, and
+ * the second pass uses the counts to determine exact sampling rates.
+ *
+ * The sampling function has a unique seed per partition.
+ */
+ def getPoissonSamplingFunction[K, V](rdd:RDD[(K, V)],
+ fractionByKey: K => Double,
+ exact: Boolean,
+ counts: Option[Map[K, Long]],
+ seed: Long): (Int, Iterator[(K, V)]) => Iterator[(K, V)] = {
+ // TODO implement the streaming version of sampling w/ replacement that doesn't require counts
+ if (exact) {
+ val seqOp = StratifiedSampler.getSeqOp[K,V](true, fractionByKey, counts)
+ val combOp = StratifiedSampler.getCombOp[K]()
+ val zeroU = new Result[K](Map[K, Stratum](), seed = seed)
+ val finalResult = aggregateWithContext(zeroU)(rdd, seqOp, combOp).resultMap
+ val thresholdByKey = StratifiedSampler.computeThresholdByKey(finalResult, fractionByKey)
+ (idx: Int, iter: Iterator[(K, V)]) => {
+ val random = new RandomDataGenerator()
+ random.reSeed(seed + idx)
+ iter.flatMap { t =>
+ val q1 = finalResult.get(t._1).get.q1.getOrElse(0.0)
--- End diff --
Similarly, better names instead of q1, q2, g1, and g2.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by dorx <gi...@git.apache.org>.
Github user dorx commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r14691577
--- Diff: pom.xml ---
@@ -257,6 +257,11 @@
<version>1.5</version>
</dependency>
<dependency>
+ <groupId>org.apache.commons</groupId>
+ <artifactId>commons-math3</artifactId>
+ <version>3.3</version>
--- End diff --
3.1.1 is good for me
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by dorx <gi...@git.apache.org>.
Github user dorx commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r14906754
--- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
@@ -0,0 +1,311 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util.random
+
+import scala.collection.Map
+import scala.collection.mutable.{ArrayBuffer, HashMap, Map => MMap}
+
+import org.apache.commons.math3.random.RandomDataGenerator
+import org.apache.spark.Logging
+import org.apache.spark.rdd.RDD
+
+/**
+ * Auxiliary functions and data structures for the sampleByKey method in PairRDDFunctions.
+ *
+ * Essentially, when exact sample size is necessary, we make additional passes over the RDD to
+ * compute the exact threshold value to use for each stratum to guarantee exact sample size with
+ * high probability. This is achieved by maintaining a waitlist of size O(log(s)), where s is the
+ * desired sample size for each stratum.
+ *
+ * Like in simple random sampling, we generate a random value for each item from the
+ * uniform distribution [0.0, 1.0]. All items with values <= min(values of items in the waitlist)
+ * are accepted into the sample instantly. The threshold for instant accept is designed so that
+ * s - numAccepted = O(log(s)), where s is again the desired sample size. Thus, by maintaining a
+ * waitlist size = O(log(s)), we will be able to create a sample of the exact size s by adding
+ * a portion of the waitlist to the set of items that are instantly accepted. The exact threshold
+ * is computed by sorting the values in the waitlist and picking the value at (s - numAccepted).
+ *
+ * Note that since we use the same seed for the RNG when computing the thresholds and the actual
+ * sample, our computed thresholds are guaranteed to produce the desired sample size.
+ *
+ * For more theoretical background on the sampling techniques used here, please refer to
+ * http://jmlr.org/proceedings/papers/v28/meng13a.html
+ */
+
+private[spark] object StratifiedSampler extends Logging {
--- End diff --
I'd prefer renaming the class to `StratifiedSamplingUtils` instead of adding too many things into `SamplingUtils`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1025#issuecomment-45957666
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15725/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by dorx <gi...@git.apache.org>.
Github user dorx commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r13889473
--- Diff: core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala ---
@@ -156,6 +161,182 @@ class PairRDDFunctions[K, V](self: RDD[(K, V)])
}
/**
+ * Return a subset of this RDD sampled by key (via stratified sampling).
+ * We guarantee a sample size = math.ceil(fraction * S_i), where S_i is the size of the ith
+ * stratum.
+ *
+ * @param withReplacement whether to sample with or without replacement
+ * @param fraction sampling rate
+ * @param seed seed for the random number generator
+ * @return RDD containing the sampled subset
+ */
+ def sampleByKey(withReplacement: Boolean,
+ fraction: Double,
+ seed: Long = Utils.random.nextLong): RDD[(K, V)]= {
+
+ class Stratum(var numItems: Long = 0L, var numAccepted: Long = 0L) extends Serializable {
+ var waitList: ArrayBuffer[Double] = new ArrayBuffer[Double]
+ var q1: Option[Double] = None
+ var q2: Option[Double] = None
+
+ def incrNumItems(by: Long = 1L) = numItems += by
+
+ def incrNumAccepted(by: Long = 1L) = numAccepted += by
+
+ def addToWaitList(elem: Double) = waitList += elem
+
+ def addToWaitList(elems: ArrayBuffer[Double]) = waitList ++= elems
+
+ override def toString() = {
+ "numItems: " + numItems + " numAccepted: " + numAccepted + " q1: " + q1 + " q2: " + q2 +
+ " waitListSize:" + waitList.size
+ }
+ }
+
+ class Result(var resultMap: Map[K, Stratum], var cachedPartitionId: Option[Int] = None)
+ extends Serializable {
+ var rand: RandomDataGenerator = new RandomDataGenerator
+
+ def getEntry(key: K, numItems: Long = 0L): Stratum = {
+ if (resultMap.get(key).isEmpty) {
--- End diff --
actually
if (!resultMap.contains(key)) {
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1025#issuecomment-46643449
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by falaki <gi...@git.apache.org>.
Github user falaki commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r14688145
--- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
@@ -0,0 +1,310 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util.random
+
+import scala.collection.Map
+import scala.collection.mutable.{ArrayBuffer, HashMap, Map => MMap}
+
+import org.apache.commons.math3.random.RandomDataGenerator
+import org.apache.spark.Logging
+import org.apache.spark.rdd.RDD
+
+/**
+ * Auxiliary functions and data structures for the sampleByKey method in PairRDDFunctions.
+ *
+ * Essentially, when exact sample size is necessary, we make additional passes over the RDD to
+ * compute the exact threshold value to use for each stratum to guarantee exact sample size with
+ * high probability. This is achieved by maintaining a waitlist of size O(log(s)), where s is the
+ * desired sample size for each stratum.
+ *
+ * Like in simple random sampling, we generate a random value for each item from the
+ * uniform distribution [0.0, 1.0]. All items with values <= min(values of items in the waitlist)
+ * are accepted into the sample instantly. The threshold for instant accept is designed so that
+ * s - numAccepted = O(log(s)), where s is again the desired sample size. Thus, by maintaining a
+ * waitlist size = O(log(s)), we will be able to create a sample of the exact size s by adding
+ * a portion of the waitlist to the set of items that are instantly accepted. The exact threshold
+ * is computed by sorting the values in the waitlist and picking the value at (s - numAccepted).
+ *
+ * Note that since we use the same seed for the RNG when computing the thresholds and the actual
+ * sample, our computed thresholds are guaranteed to produce the desired sample size.
+ *
+ * For more theoretical background on the sampling techniques used here, please refer to
+ * http://jmlr.org/proceedings/papers/v28/meng13a.html
+ */
+
+private[spark] object StratifiedSampler extends Logging {
+
+ /**
+ * Count the number of items instantly accepted and generate the waitlist for each stratum.
+ *
+ * This is only invoked when exact sample size is required.
+ */
+ def getCounts[K, V](rdd: RDD[(K, V)],
+ withReplacement: Boolean,
+ fractions: Map[K, Double],
+ counts: Option[Map[K, Long]],
+ seed: Long): MMap[K, Stratum] = {
+ val combOp = getCombOp[K]
+ val mappedPartitionRDD = rdd.mapPartitionsWithIndex({ case (partition, iter) =>
--- End diff --
Parenthesis is superfluous (the bracket is enough).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1025#issuecomment-50393886
QA results for PR 1025:<br>- This patch FAILED unit tests.<br><br>For more information see test ouptut:<br>https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17293/consoleFull
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1025#issuecomment-45553455
Build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1025#issuecomment-45968209
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15741/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1025#issuecomment-46608074
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by mateiz <gi...@git.apache.org>.
Github user mateiz commented on the pull request:
https://github.com/apache/spark/pull/1025#issuecomment-50101767
Sorry, how was the API changed, was it making `sampleByKeyExact` a separate method and making it experimental? That actually seems okay to me, the algorithm there is quite a bit more involved.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r14694248
--- Diff: core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala ---
@@ -195,6 +193,37 @@ class PairRDDFunctions[K, V](self: RDD[(K, V)])
}
/**
+ * Return a subset of this RDD sampled by key (via stratified sampling).
+ *
+ * If exact set to true, we guarantee, with high probability, a sample size =
+ * math.ceil(fraction * S_i), where S_i is the size of the ith stratum (collection of entries
+ * that share the same key). When sampling without replacement, we need one additional pass over
+ * the RDD to guarantee sample size with a 99.99% confidence; when sampling with replacement, we
+ * need two additional passes.
+ *
+ * @param withReplacement whether to sample with or without replacement
+ * @param fractions map of specific keys to sampling rates
+ * @param seed seed for the random number generator
+ * @param exact whether sample size needs to be exactly math.ceil(fraction * size) per stratum
+ * @return RDD containing the sampled subset
+ */
+ def sampleByKey(withReplacement: Boolean,
+ fractions: Map[K, Double],
+ exact: Boolean = true,
--- End diff --
I would suggest using `false` as the default value, to match the behavior of `RDD.sample`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by dorx <gi...@git.apache.org>.
Github user dorx commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r14577966
--- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util.random
+
+import scala.collection.{Map, mutable}
+import scala.collection.mutable.ArrayBuffer
+import scala.reflect.ClassTag
+
+import org.apache.commons.math3.random.RandomDataGenerator
+import org.apache.spark.{Logging, SparkContext, TaskContext}
+import org.apache.spark.rdd.RDD
+import org.apache.spark.util.Utils
+import org.apache.spark.util.random.{PoissonBounds => PB}
+
+private[spark] object StratifiedSampler extends Logging {
+
+ /**
+ * A version of {@link #aggregate()} that passes the TaskContext to the function that does
+ * aggregation for each partition. This function avoids creating an extra depth in the RDD
+ * lineage, as opposed to using mapPartitionsWithId, which results in slightly improved run time.
+ */
+ def aggregateWithContext[U: ClassTag, T: ClassTag](zeroValue: U)
+ (rdd: RDD[T],
+ seqOp: ((TaskContext, U), T) => U,
+ combOp: (U, U) => U): U = {
+ val sc: SparkContext = rdd.sparkContext
+ // Clone the zero value since we will also be serializing it as part of tasks
+ var jobResult = Utils.clone(zeroValue, sc.env.closureSerializer.newInstance())
+ // pad seqOp and combOp with taskContext to conform to aggregate's signature in TraversableOnce
+ val paddedSeqOp = (arg1: (TaskContext, U), item: T) => (arg1._1, seqOp(arg1, item))
+ val paddedcombOp = (arg1: (TaskContext, U), arg2: (TaskContext, U)) =>
+ (arg1._1, combOp(arg1._2, arg1._2))
+ val cleanSeqOp = sc.clean(paddedSeqOp)
+ val cleanCombOp = sc.clean(paddedcombOp)
+ val aggregatePartition = (tc: TaskContext, it: Iterator[T]) =>
+ (it.aggregate(tc, zeroValue)(cleanSeqOp, cleanCombOp))._2
+ val mergeResult = (index: Int, taskResult: U) => jobResult = combOp(jobResult, taskResult)
+ sc.runJob(rdd, aggregatePartition, mergeResult)
+ jobResult
+ }
+
+ /**
+ * Returns the function used by aggregate to collect sampling statistics for each partition.
+ */
+ def getSeqOp[K, V](withReplacement: Boolean,
+ fractionByKey: (K => Double),
+ counts: Option[Map[K, Long]]): ((TaskContext, Result[K]),(K, V)) => Result[K] = {
+ val delta = 5e-5
+ (U: (TaskContext, Result[K]), item: (K, V)) => {
+ val result = U._2
+ val tc = U._1
+ val rng = result.getRand(tc.partitionId)
+ val fraction = fractionByKey(item._1)
+ val stratum = result.getEntry(item._1)
+ if (withReplacement) {
+ // compute q1 and q2 only if they haven't been computed already
+ // since they don't change from iteration to iteration.
+ // TODO change this to the streaming version
+ if (stratum.q1.isEmpty || stratum.q2.isEmpty) {
+ val n = counts.get(item._1)
+ val s = math.ceil(n * fraction).toLong
+ val lmbd1 = PB.getLowerBound(s)
+ val minCount = PB.getMinCount(lmbd1)
+ val lmbd2 = if (lmbd1 == 0) PB.getUpperBound(s) else PB.getUpperBound(s - minCount)
+ val q1 = lmbd1 / n
+ val q2 = lmbd2 / n
+ stratum.q1 = Some(q1)
+ stratum.q2 = Some(q2)
+ }
+ val x1 = if (stratum.q1.get == 0) 0L else rng.nextPoisson(stratum.q1.get)
+ if (x1 > 0) {
+ stratum.incrNumAccepted(x1)
+ }
+ val x2 = rng.nextPoisson(stratum.q2.get).toInt
+ if (x2 > 0) {
+ stratum.addToWaitList(ArrayBuffer.fill(x2)(rng.nextUniform(0.0, 1.0)))
+ }
+ } else {
+ // We use the streaming version of the algorithm for sampling without replacement.
+ // Hence, q1 and q2 change on every iteration.
+ val g1 = - math.log(delta) / stratum.numItems
+ val g2 = (2.0 / 3.0) * g1
+ val q1 = math.max(0, fraction + g2 - math.sqrt((g2 * g2 + 3 * g2 * fraction)))
+ val q2 = math.min(1, fraction + g1 + math.sqrt(g1 * g1 + 2 * g1 * fraction))
+
+ val x = rng.nextUniform(0.0, 1.0)
+ if (x < q1) {
+ stratum.incrNumAccepted()
+ } else if (x < q2) {
+ stratum.addToWaitList(x)
+ }
+ stratum.q1 = Some(q1)
+ stratum.q2 = Some(q2)
+ }
+ stratum.incrNumItems()
+ result
+ }
+ }
+
+ /**
+ * Returns the function used by aggregate to combine results from different partitions, as
+ * returned by seqOp.
+ */
+ def getCombOp[K](): (Result[K], Result[K]) => Result[K] = {
+ (r1: Result[K], r2: Result[K]) => {
+ // take union of both key sets in case one partition doesn't contain all keys
+ val keyUnion = r1.resultMap.keys.toSet.union(r2.resultMap.keys.toSet)
+
+ // Use r2 to keep the combined result since r1 is usual empty
+ for (key <- keyUnion) {
+ val entry1 = r1.resultMap.get(key)
--- End diff --
r1.resultMap(key) is equivalent to r1.resultMap.get(key).get, which removed the ability to check for null entries. So the gets here aren't superfluous. But I'll move some of the logic into helper functions in Result instead.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1025#issuecomment-48386790
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by falaki <gi...@git.apache.org>.
Github user falaki commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r14544806
--- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util.random
+
+import scala.collection.{Map, mutable}
+import scala.collection.mutable.ArrayBuffer
+import scala.reflect.ClassTag
+
+import org.apache.commons.math3.random.RandomDataGenerator
+import org.apache.spark.{Logging, SparkContext, TaskContext}
+import org.apache.spark.rdd.RDD
+import org.apache.spark.util.Utils
+import org.apache.spark.util.random.{PoissonBounds => PB}
+
+private[spark] object StratifiedSampler extends Logging {
+
+ /**
+ * A version of {@link #aggregate()} that passes the TaskContext to the function that does
+ * aggregation for each partition. This function avoids creating an extra depth in the RDD
+ * lineage, as opposed to using mapPartitionsWithId, which results in slightly improved run time.
+ */
+ def aggregateWithContext[U: ClassTag, T: ClassTag](zeroValue: U)
+ (rdd: RDD[T],
+ seqOp: ((TaskContext, U), T) => U,
+ combOp: (U, U) => U): U = {
+ val sc: SparkContext = rdd.sparkContext
+ // Clone the zero value since we will also be serializing it as part of tasks
+ var jobResult = Utils.clone(zeroValue, sc.env.closureSerializer.newInstance())
+ // pad seqOp and combOp with taskContext to conform to aggregate's signature in TraversableOnce
+ val paddedSeqOp = (arg1: (TaskContext, U), item: T) => (arg1._1, seqOp(arg1, item))
+ val paddedcombOp = (arg1: (TaskContext, U), arg2: (TaskContext, U)) =>
+ (arg1._1, combOp(arg1._2, arg1._2))
+ val cleanSeqOp = sc.clean(paddedSeqOp)
+ val cleanCombOp = sc.clean(paddedcombOp)
+ val aggregatePartition = (tc: TaskContext, it: Iterator[T]) =>
+ (it.aggregate(tc, zeroValue)(cleanSeqOp, cleanCombOp))._2
+ val mergeResult = (index: Int, taskResult: U) => jobResult = combOp(jobResult, taskResult)
+ sc.runJob(rdd, aggregatePartition, mergeResult)
+ jobResult
+ }
+
+ /**
+ * Returns the function used by aggregate to collect sampling statistics for each partition.
+ */
+ def getSeqOp[K, V](withReplacement: Boolean,
+ fractionByKey: (K => Double),
+ counts: Option[Map[K, Long]]): ((TaskContext, Result[K]),(K, V)) => Result[K] = {
+ val delta = 5e-5
+ (U: (TaskContext, Result[K]), item: (K, V)) => {
+ val result = U._2
+ val tc = U._1
+ val rng = result.getRand(tc.partitionId)
+ val fraction = fractionByKey(item._1)
+ val stratum = result.getEntry(item._1)
+ if (withReplacement) {
+ // compute q1 and q2 only if they haven't been computed already
+ // since they don't change from iteration to iteration.
+ // TODO change this to the streaming version
+ if (stratum.q1.isEmpty || stratum.q2.isEmpty) {
+ val n = counts.get(item._1)
+ val s = math.ceil(n * fraction).toLong
+ val lmbd1 = PB.getLowerBound(s)
+ val minCount = PB.getMinCount(lmbd1)
+ val lmbd2 = if (lmbd1 == 0) PB.getUpperBound(s) else PB.getUpperBound(s - minCount)
+ val q1 = lmbd1 / n
+ val q2 = lmbd2 / n
+ stratum.q1 = Some(q1)
+ stratum.q2 = Some(q2)
+ }
+ val x1 = if (stratum.q1.get == 0) 0L else rng.nextPoisson(stratum.q1.get)
+ if (x1 > 0) {
+ stratum.incrNumAccepted(x1)
+ }
+ val x2 = rng.nextPoisson(stratum.q2.get).toInt
+ if (x2 > 0) {
+ stratum.addToWaitList(ArrayBuffer.fill(x2)(rng.nextUniform(0.0, 1.0)))
+ }
+ } else {
+ // We use the streaming version of the algorithm for sampling without replacement.
+ // Hence, q1 and q2 change on every iteration.
+ val g1 = - math.log(delta) / stratum.numItems
+ val g2 = (2.0 / 3.0) * g1
+ val q1 = math.max(0, fraction + g2 - math.sqrt((g2 * g2 + 3 * g2 * fraction)))
+ val q2 = math.min(1, fraction + g1 + math.sqrt(g1 * g1 + 2 * g1 * fraction))
+
+ val x = rng.nextUniform(0.0, 1.0)
+ if (x < q1) {
+ stratum.incrNumAccepted()
+ } else if (x < q2) {
+ stratum.addToWaitList(x)
+ }
+ stratum.q1 = Some(q1)
+ stratum.q2 = Some(q2)
+ }
+ stratum.incrNumItems()
+ result
+ }
+ }
+
+ /**
+ * Returns the function used by aggregate to combine results from different partitions, as
+ * returned by seqOp.
+ */
+ def getCombOp[K](): (Result[K], Result[K]) => Result[K] = {
+ (r1: Result[K], r2: Result[K]) => {
+ // take union of both key sets in case one partition doesn't contain all keys
+ val keyUnion = r1.resultMap.keys.toSet.union(r2.resultMap.keys.toSet)
+
+ // Use r2 to keep the combined result since r1 is usual empty
+ for (key <- keyUnion) {
+ val entry1 = r1.resultMap.get(key)
+ val entry2 = r2.resultMap.get(key)
+ if (entry2.isEmpty && entry1.isDefined) {
+ r2.resultMap += (key -> entry1.get)
+ } else if (entry1.isDefined && entry2.isDefined) {
+ entry2.get.addToWaitList(entry1.get.waitList)
+ entry2.get.incrNumAccepted(entry1.get.numAccepted)
+ entry2.get.incrNumItems(entry1.get.numItems)
+ }
+ }
+ r2
+ }
+ }
+
+ /**
+ * Given the result returned by the aggregate function, we need to determine the threshold used
+ * to accept items to generate the exact sample size.
+ */
+ def computeThresholdByKey[K](finalResult: Map[K, Stratum], fractionByKey: (K => Double)):
+ (K => Double) = {
+ val thresholdByKey = new mutable.HashMap[K, Double]()
+ for ((key, stratum) <- finalResult) {
+ val fraction = fractionByKey(key)
+ val s = math.ceil(stratum.numItems * fraction).toLong
+ if (stratum.numAccepted > s) {
+ logWarning("Pre-accepted too many")
+ thresholdByKey += (key -> stratum.q1.get)
+ } else {
+ val numWaitListAccepted = (s - stratum.numAccepted).toInt
+ if (numWaitListAccepted >= stratum.waitList.size) {
+ logWarning("WaitList too short")
+ thresholdByKey += (key -> stratum.q2.get)
+ } else {
+ thresholdByKey += (key -> stratum.waitList.sorted.apply(numWaitListAccepted))
+ }
+ }
+ }
+ thresholdByKey
+ }
+
+ /**
+ * Return the per partition sampling function used for sampling without replacement.
+ *
+ * When exact sample size is required, we make an additional pass over the RDD to determine the
+ * exact sampling rate that guarantees sample size with high confidence.
+ *
+ * The sampling function has a unique seed per partition.
+ */
+ def getBernoulliSamplingFunction[K, V](rdd: RDD[(K, V)],
+ fractionByKey: K => Double,
+ exact: Boolean,
+ seed: Long): (Int, Iterator[(K, V)]) => Iterator[(K, V)] = {
+ var samplingRateByKey = fractionByKey
+ if (exact) {
+ // determine threshold for each stratum and resample
+ val seqOp = StratifiedSampler.getSeqOp[K,V](false, fractionByKey, None)
+ val combOp = StratifiedSampler.getCombOp[K]()
+ val zeroU = new Result[K](Map[K, Stratum](), seed = seed)
+ val finalResult = aggregateWithContext(zeroU)(rdd, seqOp, combOp).resultMap
+ samplingRateByKey = StratifiedSampler.computeThresholdByKey(finalResult, fractionByKey)
+ }
+ (idx: Int, iter: Iterator[(K, V)]) => {
+ val random = new RandomDataGenerator
+ random.reSeed(seed + idx)
+ iter.filter(t => random.nextUniform(0.0, 1.0) < samplingRateByKey(t._1))
+ }
+ }
+
+ /**
+ * Return the per partition sampling function used for sampling with replacement.
+ *
+ * When exact sample size is required, we make two additional passed over the RDD to determine
+ * the exact sampling rate that guarantees sample size with high confidence. The first pass
+ * counts the number of items in each stratum (group of items with the same key) in the RDD, and
+ * the second pass uses the counts to determine exact sampling rates.
+ *
+ * The sampling function has a unique seed per partition.
+ */
+ def getPoissonSamplingFunction[K, V](rdd:RDD[(K, V)],
+ fractionByKey: K => Double,
+ exact: Boolean,
+ counts: Option[Map[K, Long]],
+ seed: Long): (Int, Iterator[(K, V)]) => Iterator[(K, V)] = {
+ // TODO implement the streaming version of sampling w/ replacement that doesn't require counts
+ if (exact) {
+ val seqOp = StratifiedSampler.getSeqOp[K,V](true, fractionByKey, counts)
+ val combOp = StratifiedSampler.getCombOp[K]()
+ val zeroU = new Result[K](Map[K, Stratum](), seed = seed)
+ val finalResult = aggregateWithContext(zeroU)(rdd, seqOp, combOp).resultMap
+ val thresholdByKey = StratifiedSampler.computeThresholdByKey(finalResult, fractionByKey)
+ (idx: Int, iter: Iterator[(K, V)]) => {
+ val random = new RandomDataGenerator()
+ random.reSeed(seed + idx)
+ iter.flatMap { t =>
+ val q1 = finalResult.get(t._1).get.q1.getOrElse(0.0)
+ val q2 = finalResult.get(t._1).get.q2.getOrElse(0.0)
--- End diff --
No need to call get on a Map.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r14691480
--- Diff: pom.xml ---
@@ -257,6 +257,11 @@
<version>1.5</version>
</dependency>
<dependency>
+ <groupId>org.apache.commons</groupId>
+ <artifactId>commons-math3</artifactId>
+ <version>3.3</version>
--- End diff --
Just found that hadoop-common depends on commons-math3-3.1.1 since v2.3.0 (HADOOP-9594). 3.3 is not compatible with 3.1.1. So there is (or will be) a problem no matter which version we include. Do we have alternative solutions?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by dorx <gi...@git.apache.org>.
Github user dorx commented on the pull request:
https://github.com/apache/spark/pull/1025#issuecomment-48419578
Jenkins, retest this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1025#issuecomment-46519999
Merged build finished.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by dorx <gi...@git.apache.org>.
Github user dorx commented on the pull request:
https://github.com/apache/spark/pull/1025#issuecomment-50073867
Also, seems like there wasn't a single line of code preserved from before the updates. We should probably close this PR and let Xiangrui submit his version in a separate PR at this point.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by falaki <gi...@git.apache.org>.
Github user falaki commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r14544795
--- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util.random
+
+import scala.collection.{Map, mutable}
+import scala.collection.mutable.ArrayBuffer
+import scala.reflect.ClassTag
+
+import org.apache.commons.math3.random.RandomDataGenerator
+import org.apache.spark.{Logging, SparkContext, TaskContext}
+import org.apache.spark.rdd.RDD
+import org.apache.spark.util.Utils
+import org.apache.spark.util.random.{PoissonBounds => PB}
+
+private[spark] object StratifiedSampler extends Logging {
+
+ /**
+ * A version of {@link #aggregate()} that passes the TaskContext to the function that does
+ * aggregation for each partition. This function avoids creating an extra depth in the RDD
+ * lineage, as opposed to using mapPartitionsWithId, which results in slightly improved run time.
+ */
+ def aggregateWithContext[U: ClassTag, T: ClassTag](zeroValue: U)
+ (rdd: RDD[T],
+ seqOp: ((TaskContext, U), T) => U,
+ combOp: (U, U) => U): U = {
+ val sc: SparkContext = rdd.sparkContext
+ // Clone the zero value since we will also be serializing it as part of tasks
+ var jobResult = Utils.clone(zeroValue, sc.env.closureSerializer.newInstance())
+ // pad seqOp and combOp with taskContext to conform to aggregate's signature in TraversableOnce
+ val paddedSeqOp = (arg1: (TaskContext, U), item: T) => (arg1._1, seqOp(arg1, item))
+ val paddedcombOp = (arg1: (TaskContext, U), arg2: (TaskContext, U)) =>
+ (arg1._1, combOp(arg1._2, arg1._2))
+ val cleanSeqOp = sc.clean(paddedSeqOp)
+ val cleanCombOp = sc.clean(paddedcombOp)
+ val aggregatePartition = (tc: TaskContext, it: Iterator[T]) =>
+ (it.aggregate(tc, zeroValue)(cleanSeqOp, cleanCombOp))._2
+ val mergeResult = (index: Int, taskResult: U) => jobResult = combOp(jobResult, taskResult)
+ sc.runJob(rdd, aggregatePartition, mergeResult)
+ jobResult
+ }
+
+ /**
+ * Returns the function used by aggregate to collect sampling statistics for each partition.
+ */
+ def getSeqOp[K, V](withReplacement: Boolean,
+ fractionByKey: (K => Double),
+ counts: Option[Map[K, Long]]): ((TaskContext, Result[K]),(K, V)) => Result[K] = {
+ val delta = 5e-5
+ (U: (TaskContext, Result[K]), item: (K, V)) => {
+ val result = U._2
+ val tc = U._1
+ val rng = result.getRand(tc.partitionId)
+ val fraction = fractionByKey(item._1)
+ val stratum = result.getEntry(item._1)
+ if (withReplacement) {
+ // compute q1 and q2 only if they haven't been computed already
+ // since they don't change from iteration to iteration.
+ // TODO change this to the streaming version
+ if (stratum.q1.isEmpty || stratum.q2.isEmpty) {
+ val n = counts.get(item._1)
+ val s = math.ceil(n * fraction).toLong
+ val lmbd1 = PB.getLowerBound(s)
+ val minCount = PB.getMinCount(lmbd1)
+ val lmbd2 = if (lmbd1 == 0) PB.getUpperBound(s) else PB.getUpperBound(s - minCount)
+ val q1 = lmbd1 / n
+ val q2 = lmbd2 / n
+ stratum.q1 = Some(q1)
+ stratum.q2 = Some(q2)
+ }
+ val x1 = if (stratum.q1.get == 0) 0L else rng.nextPoisson(stratum.q1.get)
+ if (x1 > 0) {
+ stratum.incrNumAccepted(x1)
+ }
+ val x2 = rng.nextPoisson(stratum.q2.get).toInt
+ if (x2 > 0) {
+ stratum.addToWaitList(ArrayBuffer.fill(x2)(rng.nextUniform(0.0, 1.0)))
+ }
+ } else {
+ // We use the streaming version of the algorithm for sampling without replacement.
+ // Hence, q1 and q2 change on every iteration.
+ val g1 = - math.log(delta) / stratum.numItems
+ val g2 = (2.0 / 3.0) * g1
+ val q1 = math.max(0, fraction + g2 - math.sqrt((g2 * g2 + 3 * g2 * fraction)))
+ val q2 = math.min(1, fraction + g1 + math.sqrt(g1 * g1 + 2 * g1 * fraction))
+
+ val x = rng.nextUniform(0.0, 1.0)
+ if (x < q1) {
+ stratum.incrNumAccepted()
+ } else if (x < q2) {
+ stratum.addToWaitList(x)
+ }
+ stratum.q1 = Some(q1)
+ stratum.q2 = Some(q2)
+ }
+ stratum.incrNumItems()
+ result
+ }
+ }
+
+ /**
+ * Returns the function used by aggregate to combine results from different partitions, as
+ * returned by seqOp.
+ */
+ def getCombOp[K](): (Result[K], Result[K]) => Result[K] = {
+ (r1: Result[K], r2: Result[K]) => {
+ // take union of both key sets in case one partition doesn't contain all keys
+ val keyUnion = r1.resultMap.keys.toSet.union(r2.resultMap.keys.toSet)
+
+ // Use r2 to keep the combined result since r1 is usual empty
+ for (key <- keyUnion) {
+ val entry1 = r1.resultMap.get(key)
+ val entry2 = r2.resultMap.get(key)
+ if (entry2.isEmpty && entry1.isDefined) {
+ r2.resultMap += (key -> entry1.get)
+ } else if (entry1.isDefined && entry2.isDefined) {
+ entry2.get.addToWaitList(entry1.get.waitList)
+ entry2.get.incrNumAccepted(entry1.get.numAccepted)
+ entry2.get.incrNumItems(entry1.get.numItems)
+ }
+ }
+ r2
+ }
+ }
+
+ /**
+ * Given the result returned by the aggregate function, we need to determine the threshold used
+ * to accept items to generate the exact sample size.
+ */
+ def computeThresholdByKey[K](finalResult: Map[K, Stratum], fractionByKey: (K => Double)):
+ (K => Double) = {
+ val thresholdByKey = new mutable.HashMap[K, Double]()
+ for ((key, stratum) <- finalResult) {
+ val fraction = fractionByKey(key)
+ val s = math.ceil(stratum.numItems * fraction).toLong
+ if (stratum.numAccepted > s) {
+ logWarning("Pre-accepted too many")
+ thresholdByKey += (key -> stratum.q1.get)
+ } else {
+ val numWaitListAccepted = (s - stratum.numAccepted).toInt
+ if (numWaitListAccepted >= stratum.waitList.size) {
+ logWarning("WaitList too short")
+ thresholdByKey += (key -> stratum.q2.get)
+ } else {
+ thresholdByKey += (key -> stratum.waitList.sorted.apply(numWaitListAccepted))
+ }
+ }
+ }
+ thresholdByKey
+ }
+
+ /**
+ * Return the per partition sampling function used for sampling without replacement.
+ *
+ * When exact sample size is required, we make an additional pass over the RDD to determine the
+ * exact sampling rate that guarantees sample size with high confidence.
+ *
+ * The sampling function has a unique seed per partition.
+ */
+ def getBernoulliSamplingFunction[K, V](rdd: RDD[(K, V)],
+ fractionByKey: K => Double,
+ exact: Boolean,
+ seed: Long): (Int, Iterator[(K, V)]) => Iterator[(K, V)] = {
+ var samplingRateByKey = fractionByKey
+ if (exact) {
+ // determine threshold for each stratum and resample
+ val seqOp = StratifiedSampler.getSeqOp[K,V](false, fractionByKey, None)
+ val combOp = StratifiedSampler.getCombOp[K]()
+ val zeroU = new Result[K](Map[K, Stratum](), seed = seed)
+ val finalResult = aggregateWithContext(zeroU)(rdd, seqOp, combOp).resultMap
+ samplingRateByKey = StratifiedSampler.computeThresholdByKey(finalResult, fractionByKey)
+ }
+ (idx: Int, iter: Iterator[(K, V)]) => {
+ val random = new RandomDataGenerator
+ random.reSeed(seed + idx)
+ iter.filter(t => random.nextUniform(0.0, 1.0) < samplingRateByKey(t._1))
+ }
+ }
+
+ /**
+ * Return the per partition sampling function used for sampling with replacement.
+ *
+ * When exact sample size is required, we make two additional passed over the RDD to determine
+ * the exact sampling rate that guarantees sample size with high confidence. The first pass
+ * counts the number of items in each stratum (group of items with the same key) in the RDD, and
+ * the second pass uses the counts to determine exact sampling rates.
+ *
+ * The sampling function has a unique seed per partition.
+ */
+ def getPoissonSamplingFunction[K, V](rdd:RDD[(K, V)],
+ fractionByKey: K => Double,
+ exact: Boolean,
+ counts: Option[Map[K, Long]],
+ seed: Long): (Int, Iterator[(K, V)]) => Iterator[(K, V)] = {
+ // TODO implement the streaming version of sampling w/ replacement that doesn't require counts
+ if (exact) {
+ val seqOp = StratifiedSampler.getSeqOp[K,V](true, fractionByKey, counts)
--- End diff --
Similarly remove ```StratifiedSampler``` object name from these lines
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by falaki <gi...@git.apache.org>.
Github user falaki commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r14544074
--- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util.random
+
+import scala.collection.{Map, mutable}
+import scala.collection.mutable.ArrayBuffer
+import scala.reflect.ClassTag
+
+import org.apache.commons.math3.random.RandomDataGenerator
+import org.apache.spark.{Logging, SparkContext, TaskContext}
+import org.apache.spark.rdd.RDD
+import org.apache.spark.util.Utils
+import org.apache.spark.util.random.{PoissonBounds => PB}
+
+private[spark] object StratifiedSampler extends Logging {
+
+ /**
+ * A version of {@link #aggregate()} that passes the TaskContext to the function that does
+ * aggregation for each partition. This function avoids creating an extra depth in the RDD
+ * lineage, as opposed to using mapPartitionsWithId, which results in slightly improved run time.
+ */
+ def aggregateWithContext[U: ClassTag, T: ClassTag](zeroValue: U)
+ (rdd: RDD[T],
+ seqOp: ((TaskContext, U), T) => U,
+ combOp: (U, U) => U): U = {
+ val sc: SparkContext = rdd.sparkContext
+ // Clone the zero value since we will also be serializing it as part of tasks
+ var jobResult = Utils.clone(zeroValue, sc.env.closureSerializer.newInstance())
+ // pad seqOp and combOp with taskContext to conform to aggregate's signature in TraversableOnce
+ val paddedSeqOp = (arg1: (TaskContext, U), item: T) => (arg1._1, seqOp(arg1, item))
+ val paddedcombOp = (arg1: (TaskContext, U), arg2: (TaskContext, U)) =>
+ (arg1._1, combOp(arg1._2, arg1._2))
+ val cleanSeqOp = sc.clean(paddedSeqOp)
+ val cleanCombOp = sc.clean(paddedcombOp)
+ val aggregatePartition = (tc: TaskContext, it: Iterator[T]) =>
+ (it.aggregate(tc, zeroValue)(cleanSeqOp, cleanCombOp))._2
+ val mergeResult = (index: Int, taskResult: U) => jobResult = combOp(jobResult, taskResult)
+ sc.runJob(rdd, aggregatePartition, mergeResult)
+ jobResult
+ }
+
+ /**
+ * Returns the function used by aggregate to collect sampling statistics for each partition.
+ */
+ def getSeqOp[K, V](withReplacement: Boolean,
+ fractionByKey: (K => Double),
+ counts: Option[Map[K, Long]]): ((TaskContext, Result[K]),(K, V)) => Result[K] = {
--- End diff --
Space after comma
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r14694292
--- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
@@ -0,0 +1,311 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util.random
+
+import scala.collection.Map
+import scala.collection.mutable.{ArrayBuffer, HashMap, Map => MMap}
+
+import org.apache.commons.math3.random.RandomDataGenerator
+import org.apache.spark.Logging
+import org.apache.spark.rdd.RDD
+
+/**
+ * Auxiliary functions and data structures for the sampleByKey method in PairRDDFunctions.
+ *
+ * Essentially, when exact sample size is necessary, we make additional passes over the RDD to
+ * compute the exact threshold value to use for each stratum to guarantee exact sample size with
+ * high probability. This is achieved by maintaining a waitlist of size O(log(s)), where s is the
+ * desired sample size for each stratum.
+ *
+ * Like in simple random sampling, we generate a random value for each item from the
+ * uniform distribution [0.0, 1.0]. All items with values <= min(values of items in the waitlist)
+ * are accepted into the sample instantly. The threshold for instant accept is designed so that
+ * s - numAccepted = O(log(s)), where s is again the desired sample size. Thus, by maintaining a
+ * waitlist size = O(log(s)), we will be able to create a sample of the exact size s by adding
+ * a portion of the waitlist to the set of items that are instantly accepted. The exact threshold
+ * is computed by sorting the values in the waitlist and picking the value at (s - numAccepted).
+ *
+ * Note that since we use the same seed for the RNG when computing the thresholds and the actual
+ * sample, our computed thresholds are guaranteed to produce the desired sample size.
+ *
+ * For more theoretical background on the sampling techniques used here, please refer to
+ * http://jmlr.org/proceedings/papers/v28/meng13a.html
+ */
+
+private[spark] object StratifiedSampler extends Logging {
+
+ /**
+ * Count the number of items instantly accepted and generate the waitlist for each stratum.
+ *
+ * This is only invoked when exact sample size is required.
+ */
+ def getCounts[K, V](rdd: RDD[(K, V)],
+ withReplacement: Boolean,
+ fractions: Map[K, Double],
+ counts: Option[Map[K, Long]],
+ seed: Long): MMap[K, Stratum] = {
+ val combOp = getCombOp[K]
+ val mappedPartitionRDD = rdd.mapPartitionsWithIndex({ case (partition, iter) =>
+ val zeroU: MMap[K, Stratum] = new HashMap[K, Stratum]()
+ val rng = new RandomDataGenerator()
+ rng.reSeed(seed + partition)
+ val seqOp = getSeqOp(withReplacement, fractions, rng, counts)
+ Iterator(iter.aggregate(zeroU)(seqOp, combOp))
+ }, preservesPartitioning=true)
+ mappedPartitionRDD.reduce(combOp)
+ }
+
+ /**
+ * Returns the function used by aggregate to collect sampling statistics for each partition.
+ */
+ def getSeqOp[K, V](withReplacement: Boolean,
+ fractions: Map[K, Double],
+ rng: RandomDataGenerator,
+ counts: Option[Map[K, Long]]): (MMap[K, Stratum], (K, V)) => MMap[K, Stratum] = {
+ val delta = 5e-5
+ (result: MMap[K, Stratum], item: (K, V)) => {
+ val key = item._1
+ val fraction = fractions(key)
+ if (!result.contains(key)) {
+ result += (key -> new Stratum())
+ }
+ val stratum = result(key)
+
+ if (withReplacement) {
+ // compute acceptBound and waitListBound only if they haven't been computed already
+ // since they don't change from iteration to iteration.
+ // TODO change this to the streaming version
+ if (stratum.areBoundsEmpty) {
+ val n = counts.get(key)
+ val sampleSize = math.ceil(n * fraction).toLong
+ val lmbd1 = PoissonBounds.getLowerBound(sampleSize)
+ val minCount = PoissonBounds.getMinCount(lmbd1)
+ val lmbd2 = if (lmbd1 == 0) {
+ PoissonBounds.getUpperBound(sampleSize)
+ } else {
+ PoissonBounds.getUpperBound(sampleSize - minCount)
+ }
+ stratum.acceptBound = lmbd1 / n
+ stratum.waitListBound = lmbd2 / n
+ }
+ val acceptBound = stratum.acceptBound
+ val copiesAccepted = if (acceptBound == 0.0) 0L else rng.nextPoisson(acceptBound)
+ if (copiesAccepted > 0) {
+ stratum.incrNumAccepted(copiesAccepted)
+ }
+ val copiesWaitlisted = rng.nextPoisson(stratum.waitListBound).toInt
+ if (copiesWaitlisted > 0) {
+ stratum.addToWaitList(ArrayBuffer.fill(copiesWaitlisted)(rng.nextUniform(0.0, 1.0)))
+ }
+ } else {
+ // We use the streaming version of the algorithm for sampling without replacement to avoid
+ // using an extra pass over the RDD for computing the count.
+ // Hence, acceptBound and waitListBound change on every iteration.
+ val gamma1 = - math.log(delta) / stratum.numItems
+ val gamma2 = (2.0 / 3.0) * gamma1
+ stratum.acceptBound = math.max(0,
+ fraction + gamma2 - math.sqrt(gamma2 * gamma2 + 3 * gamma2 * fraction))
+ stratum.waitListBound = math.min(1,
+ fraction + gamma1 + math.sqrt(gamma1 * gamma1 + 2 * gamma1 * fraction))
+
+ val x = rng.nextUniform(0.0, 1.0)
+ if (x < stratum.acceptBound) {
+ stratum.incrNumAccepted()
+ } else if (x < stratum.waitListBound) {
+ stratum.addToWaitList(x)
+ }
+ }
+ stratum.incrNumItems()
+ result
+ }
+ }
+
+ /**
+ * Returns the function used combine results returned by seqOp from different partitions.
+ */
+ def getCombOp[K]: (MMap[K, Stratum], MMap[K, Stratum]) => MMap[K, Stratum] = {
+ (result1: MMap[K, Stratum], result2: MMap[K, Stratum]) => {
+ // take union of both key sets in case one partition doesn't contain all keys
+ for (key <- result1.keySet.union(result2.keySet)) {
+ // Use result2 to keep the combined result since r1 is usual empty
+ val entry1 = result1.get(key)
+ if (result2.contains(key)) {
+ result2(key).merge(entry1)
+ } else {
+ if (entry1.isDefined) {
+ result2 += (key -> entry1.get)
+ }
+ }
+ }
+ result2
+ }
+ }
+
+ /**
+ * Given the result returned by getCounts, determine the threshold for accepting items to
+ * generate exact sample size.
+ *
+ * To do so, we compute sampleSize = math.ceil(size * samplingRate) for each stratum and compare
+ * it to the number of items that were accepted instantly and the number of items in the waitlist
+ * for that stratum. Most of the time, numAccepted <= sampleSize <= (numAccepted + numWaitlisted),
+ * which means we need to sort the elements in the waitlist by their associated values in order
+ * to find the value T s.t. |{elements in the stratum whose associated values <= T}| = sampleSize.
+ * Note that all elements in the waitlist have values >= bound for instant accept, so a T value
+ * in the waitlist range would allow all elements that were instantly accepted on the first pass
+ * to be included in the sample.
+ */
+ def computeThresholdByKey[K](finalResult: Map[K, Stratum],
+ fractions: Map[K, Double]): Map[K, Double] = {
+ val thresholdByKey = new HashMap[K, Double]()
+ for ((key, stratum) <- finalResult) {
+ val sampleSize = math.ceil(stratum.numItems * fractions(key)).toLong
+ if (stratum.numAccepted > sampleSize) {
+ logWarning("Pre-accepted too many")
+ thresholdByKey += (key -> stratum.acceptBound)
+ } else {
+ val numWaitListAccepted = (sampleSize - stratum.numAccepted).toInt
+ if (numWaitListAccepted >= stratum.waitList.size) {
+ logWarning("WaitList too short")
+ thresholdByKey += (key -> stratum.waitListBound)
+ } else {
+ thresholdByKey += (key -> stratum.waitList.sorted.apply(numWaitListAccepted))
+ }
+ }
+ }
+ thresholdByKey
+ }
+
+ /**
+ * Return the per partition sampling function used for sampling without replacement.
+ *
+ * When exact sample size is required, we make an additional pass over the RDD to determine the
+ * exact sampling rate that guarantees sample size with high confidence.
+ *
+ * The sampling function has a unique seed per partition.
+ */
+ def getBernoulliSamplingFunction[K, V](rdd: RDD[(K, V)],
+ fractions: Map[K, Double],
+ exact: Boolean,
+ seed: Long): (Int, Iterator[(K, V)]) => Iterator[(K, V)] = {
+ var samplingRateByKey = fractions
+ if (exact) {
+ // determine threshold for each stratum and resample
+ val finalResult = getCounts(rdd, false, fractions, None, seed)
+ samplingRateByKey = computeThresholdByKey(finalResult, fractions)
+ }
+ (idx: Int, iter: Iterator[(K, V)]) => {
+ val rng = new RandomDataGenerator
+ rng.reSeed(seed + idx)
+ iter.filter(t => rng.nextUniform(0.0, 1.0) < samplingRateByKey(t._1))
+ }
+ }
+
+ /**
+ * Return the per partition sampling function used for sampling with replacement.
+ *
+ * When exact sample size is required, we make two additional passed over the RDD to determine
+ * the exact sampling rate that guarantees sample size with high confidence. The first pass
+ * counts the number of items in each stratum (group of items with the same key) in the RDD, and
+ * the second pass uses the counts to determine exact sampling rates.
+ *
+ * The sampling function has a unique seed per partition.
+ */
+ def getPoissonSamplingFunction[K, V](rdd: RDD[(K, V)],
+ fractions: Map[K, Double],
+ exact: Boolean,
+ counts: Option[Map[K, Long]],
+ seed: Long): (Int, Iterator[(K, V)]) => Iterator[(K, V)] = {
+ // TODO implement the streaming version of sampling w/ replacement that doesn't require counts
+ if (exact) {
+ val finalResult = getCounts(rdd, true, fractions, counts, seed)
+ val thresholdByKey = computeThresholdByKey(finalResult, fractions)
+ (idx: Int, iter: Iterator[(K, V)]) => {
+ val rng = new RandomDataGenerator()
+ rng.reSeed(seed + idx)
+ iter.flatMap { item =>
+ val key = item._1
+ val acceptBound = finalResult(key).acceptBound
+ val copiesAccepted = if (acceptBound == 0) 0L else rng.nextPoisson(acceptBound)
+ val copiesWailisted = rng.nextPoisson(finalResult(key).waitListBound).toInt
+ val copiesInSample = copiesAccepted +
+ (0 until copiesWailisted).count(i => rng.nextUniform(0.0, 1.0) < thresholdByKey(key))
--- End diff --
key doesn't change here. So we don't need to recompute it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by dorx <gi...@git.apache.org>.
Github user dorx commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r14906412
--- Diff: core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala ---
@@ -195,6 +193,37 @@ class PairRDDFunctions[K, V](self: RDD[(K, V)])
}
/**
+ * Return a subset of this RDD sampled by key (via stratified sampling).
+ *
+ * If exact set to true, we guarantee, with high probability, a sample size =
+ * math.ceil(fraction * S_i), where S_i is the size of the ith stratum (collection of entries
+ * that share the same key). When sampling without replacement, we need one additional pass over
+ * the RDD to guarantee sample size with a 99.99% confidence; when sampling with replacement, we
+ * need two additional passes.
+ *
+ * @param withReplacement whether to sample with or without replacement
+ * @param fractions map of specific keys to sampling rates
+ * @param seed seed for the random number generator
+ * @param exact whether sample size needs to be exactly math.ceil(fraction * size) per stratum
+ * @return RDD containing the sampled subset
+ */
+ def sampleByKey(withReplacement: Boolean,
+ fractions: Map[K, Double],
+ exact: Boolean = true,
+ seed: Long = Utils.random.nextLong): RDD[(K, V)]= {
+
+ require(fractions.forall {case(k, v) => v >= 0.0}, "Invalid sampling rates.")
--- End diff --
This is done to be consistent with sample. Would you like `Invalid` in sample to be changed as well?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r14907668
--- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
@@ -0,0 +1,311 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util.random
+
+import scala.collection.Map
+import scala.collection.mutable.{ArrayBuffer, HashMap, Map => MMap}
+
+import org.apache.commons.math3.random.RandomDataGenerator
+import org.apache.spark.Logging
+import org.apache.spark.rdd.RDD
+
+/**
+ * Auxiliary functions and data structures for the sampleByKey method in PairRDDFunctions.
+ *
+ * Essentially, when exact sample size is necessary, we make additional passes over the RDD to
+ * compute the exact threshold value to use for each stratum to guarantee exact sample size with
+ * high probability. This is achieved by maintaining a waitlist of size O(log(s)), where s is the
+ * desired sample size for each stratum.
+ *
+ * Like in simple random sampling, we generate a random value for each item from the
+ * uniform distribution [0.0, 1.0]. All items with values <= min(values of items in the waitlist)
+ * are accepted into the sample instantly. The threshold for instant accept is designed so that
+ * s - numAccepted = O(log(s)), where s is again the desired sample size. Thus, by maintaining a
+ * waitlist size = O(log(s)), we will be able to create a sample of the exact size s by adding
+ * a portion of the waitlist to the set of items that are instantly accepted. The exact threshold
+ * is computed by sorting the values in the waitlist and picking the value at (s - numAccepted).
+ *
+ * Note that since we use the same seed for the RNG when computing the thresholds and the actual
+ * sample, our computed thresholds are guaranteed to produce the desired sample size.
+ *
+ * For more theoretical background on the sampling techniques used here, please refer to
+ * http://jmlr.org/proceedings/papers/v28/meng13a.html
+ */
+
+private[spark] object StratifiedSampler extends Logging {
+
+ /**
+ * Count the number of items instantly accepted and generate the waitlist for each stratum.
+ *
+ * This is only invoked when exact sample size is required.
+ */
+ def getCounts[K, V](rdd: RDD[(K, V)],
+ withReplacement: Boolean,
+ fractions: Map[K, Double],
+ counts: Option[Map[K, Long]],
+ seed: Long): MMap[K, Stratum] = {
+ val combOp = getCombOp[K]
+ val mappedPartitionRDD = rdd.mapPartitionsWithIndex({ case (partition, iter) =>
+ val zeroU: MMap[K, Stratum] = new HashMap[K, Stratum]()
+ val rng = new RandomDataGenerator()
+ rng.reSeed(seed + partition)
+ val seqOp = getSeqOp(withReplacement, fractions, rng, counts)
+ Iterator(iter.aggregate(zeroU)(seqOp, combOp))
+ }, preservesPartitioning=true)
+ mappedPartitionRDD.reduce(combOp)
+ }
+
+ /**
+ * Returns the function used by aggregate to collect sampling statistics for each partition.
+ */
+ def getSeqOp[K, V](withReplacement: Boolean,
+ fractions: Map[K, Double],
+ rng: RandomDataGenerator,
+ counts: Option[Map[K, Long]]): (MMap[K, Stratum], (K, V)) => MMap[K, Stratum] = {
+ val delta = 5e-5
+ (result: MMap[K, Stratum], item: (K, V)) => {
+ val key = item._1
+ val fraction = fractions(key)
+ if (!result.contains(key)) {
+ result += (key -> new Stratum())
+ }
+ val stratum = result(key)
+
+ if (withReplacement) {
+ // compute acceptBound and waitListBound only if they haven't been computed already
+ // since they don't change from iteration to iteration.
+ // TODO change this to the streaming version
+ if (stratum.areBoundsEmpty) {
+ val n = counts.get(key)
+ val sampleSize = math.ceil(n * fraction).toLong
+ val lmbd1 = PoissonBounds.getLowerBound(sampleSize)
+ val minCount = PoissonBounds.getMinCount(lmbd1)
+ val lmbd2 = if (lmbd1 == 0) {
+ PoissonBounds.getUpperBound(sampleSize)
+ } else {
+ PoissonBounds.getUpperBound(sampleSize - minCount)
+ }
+ stratum.acceptBound = lmbd1 / n
+ stratum.waitListBound = lmbd2 / n
+ }
+ val acceptBound = stratum.acceptBound
+ val copiesAccepted = if (acceptBound == 0.0) 0L else rng.nextPoisson(acceptBound)
+ if (copiesAccepted > 0) {
+ stratum.incrNumAccepted(copiesAccepted)
+ }
+ val copiesWaitlisted = rng.nextPoisson(stratum.waitListBound).toInt
+ if (copiesWaitlisted > 0) {
+ stratum.addToWaitList(ArrayBuffer.fill(copiesWaitlisted)(rng.nextUniform(0.0, 1.0)))
+ }
+ } else {
+ // We use the streaming version of the algorithm for sampling without replacement to avoid
+ // using an extra pass over the RDD for computing the count.
+ // Hence, acceptBound and waitListBound change on every iteration.
+ val gamma1 = - math.log(delta) / stratum.numItems
+ val gamma2 = (2.0 / 3.0) * gamma1
+ stratum.acceptBound = math.max(0,
+ fraction + gamma2 - math.sqrt(gamma2 * gamma2 + 3 * gamma2 * fraction))
+ stratum.waitListBound = math.min(1,
+ fraction + gamma1 + math.sqrt(gamma1 * gamma1 + 2 * gamma1 * fraction))
+
+ val x = rng.nextUniform(0.0, 1.0)
+ if (x < stratum.acceptBound) {
+ stratum.incrNumAccepted()
+ } else if (x < stratum.waitListBound) {
+ stratum.addToWaitList(x)
+ }
+ }
+ stratum.incrNumItems()
+ result
+ }
+ }
+
+ /**
+ * Returns the function used combine results returned by seqOp from different partitions.
+ */
+ def getCombOp[K]: (MMap[K, Stratum], MMap[K, Stratum]) => MMap[K, Stratum] = {
+ (result1: MMap[K, Stratum], result2: MMap[K, Stratum]) => {
+ // take union of both key sets in case one partition doesn't contain all keys
+ for (key <- result1.keySet.union(result2.keySet)) {
+ // Use result2 to keep the combined result since r1 is usual empty
--- End diff --
Check Spark's RDD.aggregate doc.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1025#issuecomment-50381620
QA results for PR 1025:<br>- This patch FAILED unit tests.<br>- This patch merges cleanly<br>- This patch adds no public classes<br><br>For more information see test ouptut:<br>https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17296/consoleFull
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1025#issuecomment-45562050
Build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r14694252
--- Diff: core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala ---
@@ -195,6 +193,37 @@ class PairRDDFunctions[K, V](self: RDD[(K, V)])
}
/**
+ * Return a subset of this RDD sampled by key (via stratified sampling).
+ *
+ * If exact set to true, we guarantee, with high probability, a sample size =
+ * math.ceil(fraction * S_i), where S_i is the size of the ith stratum (collection of entries
+ * that share the same key). When sampling without replacement, we need one additional pass over
+ * the RDD to guarantee sample size with a 99.99% confidence; when sampling with replacement, we
+ * need two additional passes.
+ *
+ * @param withReplacement whether to sample with or without replacement
+ * @param fractions map of specific keys to sampling rates
+ * @param seed seed for the random number generator
+ * @param exact whether sample size needs to be exactly math.ceil(fraction * size) per stratum
+ * @return RDD containing the sampled subset
+ */
+ def sampleByKey(withReplacement: Boolean,
+ fractions: Map[K, Double],
+ exact: Boolean = true,
+ seed: Long = Utils.random.nextLong): RDD[(K, V)]= {
+
+ require(fractions.forall {case(k, v) => v >= 0.0}, "Invalid sampling rates.")
+
+ val samplingFunc = if (withReplacement) {
+ val counts = if (exact) Some(this.countByKey()) else None
+ StratifiedSampler.getPoissonSamplingFunction(self, fractions, exact, counts, seed)
+ } else {
+ StratifiedSampler.getBernoulliSamplingFunction(self, fractions, exact, seed)
+ }
+ self.mapPartitionsWithIndex(samplingFunc, preservesPartitioning=true)
--- End diff --
spaces around `=`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1025#issuecomment-48388111
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16416/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r13578453
--- Diff: core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala ---
@@ -156,6 +161,182 @@ class PairRDDFunctions[K, V](self: RDD[(K, V)])
}
/**
+ * Return a subset of this RDD sampled by key (via stratified sampling).
+ * We guarantee a sample size = math.ceil(fraction * S_i), where S_i is the size of the ith
--- End diff --
+ "with high probability"
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1025#issuecomment-45568512
Build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by dorx <gi...@git.apache.org>.
Github user dorx commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r13617209
--- Diff: core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala ---
@@ -27,8 +27,12 @@ import scala.collection.Map
import scala.collection.mutable
import scala.collection.mutable.ArrayBuffer
import scala.reflect.ClassTag
+import scala.util.control.Breaks._
--- End diff --
any reason why this is disallowed? It's actually used inside a for loop for "continue"
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by dorx <gi...@git.apache.org>.
Github user dorx commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r14906349
--- Diff: core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala ---
@@ -195,6 +193,37 @@ class PairRDDFunctions[K, V](self: RDD[(K, V)])
}
/**
+ * Return a subset of this RDD sampled by key (via stratified sampling).
+ *
+ * If exact set to true, we guarantee, with high probability, a sample size =
--- End diff --
So I was told that a rule of thumb is summarize what the function does in one line first and then dive into the details. I can add the explanation in a second paragraph (with a line skip) for elaboration on sampleByKey. For future reference, do you prefer a big chunk of text up front?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r14694281
--- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
@@ -0,0 +1,311 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util.random
+
+import scala.collection.Map
+import scala.collection.mutable.{ArrayBuffer, HashMap, Map => MMap}
+
+import org.apache.commons.math3.random.RandomDataGenerator
+import org.apache.spark.Logging
+import org.apache.spark.rdd.RDD
+
+/**
+ * Auxiliary functions and data structures for the sampleByKey method in PairRDDFunctions.
+ *
+ * Essentially, when exact sample size is necessary, we make additional passes over the RDD to
+ * compute the exact threshold value to use for each stratum to guarantee exact sample size with
+ * high probability. This is achieved by maintaining a waitlist of size O(log(s)), where s is the
+ * desired sample size for each stratum.
+ *
+ * Like in simple random sampling, we generate a random value for each item from the
+ * uniform distribution [0.0, 1.0]. All items with values <= min(values of items in the waitlist)
+ * are accepted into the sample instantly. The threshold for instant accept is designed so that
+ * s - numAccepted = O(log(s)), where s is again the desired sample size. Thus, by maintaining a
+ * waitlist size = O(log(s)), we will be able to create a sample of the exact size s by adding
+ * a portion of the waitlist to the set of items that are instantly accepted. The exact threshold
+ * is computed by sorting the values in the waitlist and picking the value at (s - numAccepted).
+ *
+ * Note that since we use the same seed for the RNG when computing the thresholds and the actual
+ * sample, our computed thresholds are guaranteed to produce the desired sample size.
+ *
+ * For more theoretical background on the sampling techniques used here, please refer to
+ * http://jmlr.org/proceedings/papers/v28/meng13a.html
+ */
+
+private[spark] object StratifiedSampler extends Logging {
+
+ /**
+ * Count the number of items instantly accepted and generate the waitlist for each stratum.
+ *
+ * This is only invoked when exact sample size is required.
+ */
+ def getCounts[K, V](rdd: RDD[(K, V)],
+ withReplacement: Boolean,
+ fractions: Map[K, Double],
+ counts: Option[Map[K, Long]],
+ seed: Long): MMap[K, Stratum] = {
+ val combOp = getCombOp[K]
+ val mappedPartitionRDD = rdd.mapPartitionsWithIndex({ case (partition, iter) =>
+ val zeroU: MMap[K, Stratum] = new HashMap[K, Stratum]()
+ val rng = new RandomDataGenerator()
+ rng.reSeed(seed + partition)
+ val seqOp = getSeqOp(withReplacement, fractions, rng, counts)
+ Iterator(iter.aggregate(zeroU)(seqOp, combOp))
+ }, preservesPartitioning=true)
+ mappedPartitionRDD.reduce(combOp)
+ }
+
+ /**
+ * Returns the function used by aggregate to collect sampling statistics for each partition.
+ */
+ def getSeqOp[K, V](withReplacement: Boolean,
+ fractions: Map[K, Double],
+ rng: RandomDataGenerator,
+ counts: Option[Map[K, Long]]): (MMap[K, Stratum], (K, V)) => MMap[K, Stratum] = {
+ val delta = 5e-5
+ (result: MMap[K, Stratum], item: (K, V)) => {
+ val key = item._1
+ val fraction = fractions(key)
+ if (!result.contains(key)) {
+ result += (key -> new Stratum())
+ }
+ val stratum = result(key)
+
+ if (withReplacement) {
+ // compute acceptBound and waitListBound only if they haven't been computed already
+ // since they don't change from iteration to iteration.
+ // TODO change this to the streaming version
+ if (stratum.areBoundsEmpty) {
+ val n = counts.get(key)
+ val sampleSize = math.ceil(n * fraction).toLong
+ val lmbd1 = PoissonBounds.getLowerBound(sampleSize)
+ val minCount = PoissonBounds.getMinCount(lmbd1)
+ val lmbd2 = if (lmbd1 == 0) {
+ PoissonBounds.getUpperBound(sampleSize)
+ } else {
+ PoissonBounds.getUpperBound(sampleSize - minCount)
+ }
+ stratum.acceptBound = lmbd1 / n
+ stratum.waitListBound = lmbd2 / n
+ }
+ val acceptBound = stratum.acceptBound
+ val copiesAccepted = if (acceptBound == 0.0) 0L else rng.nextPoisson(acceptBound)
+ if (copiesAccepted > 0) {
+ stratum.incrNumAccepted(copiesAccepted)
+ }
+ val copiesWaitlisted = rng.nextPoisson(stratum.waitListBound).toInt
+ if (copiesWaitlisted > 0) {
+ stratum.addToWaitList(ArrayBuffer.fill(copiesWaitlisted)(rng.nextUniform(0.0, 1.0)))
+ }
+ } else {
+ // We use the streaming version of the algorithm for sampling without replacement to avoid
+ // using an extra pass over the RDD for computing the count.
+ // Hence, acceptBound and waitListBound change on every iteration.
+ val gamma1 = - math.log(delta) / stratum.numItems
+ val gamma2 = (2.0 / 3.0) * gamma1
+ stratum.acceptBound = math.max(0,
--- End diff --
Those bounds are also used in `takeSample`. Should we have BernoulliBounds in SamplingUtils, similar to PoissonBounds?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r14694250
--- Diff: core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala ---
@@ -195,6 +193,37 @@ class PairRDDFunctions[K, V](self: RDD[(K, V)])
}
/**
+ * Return a subset of this RDD sampled by key (via stratified sampling).
+ *
+ * If exact set to true, we guarantee, with high probability, a sample size =
+ * math.ceil(fraction * S_i), where S_i is the size of the ith stratum (collection of entries
+ * that share the same key). When sampling without replacement, we need one additional pass over
+ * the RDD to guarantee sample size with a 99.99% confidence; when sampling with replacement, we
+ * need two additional passes.
+ *
+ * @param withReplacement whether to sample with or without replacement
+ * @param fractions map of specific keys to sampling rates
+ * @param seed seed for the random number generator
+ * @param exact whether sample size needs to be exactly math.ceil(fraction * size) per stratum
+ * @return RDD containing the sampled subset
+ */
+ def sampleByKey(withReplacement: Boolean,
+ fractions: Map[K, Double],
+ exact: Boolean = true,
+ seed: Long = Utils.random.nextLong): RDD[(K, V)]= {
+
+ require(fractions.forall {case(k, v) => v >= 0.0}, "Invalid sampling rates.")
--- End diff --
`fractions.values.forall(v => v >= 0.0)`
Change `Invalid` to `Negative` to be more specific.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by dorx <gi...@git.apache.org>.
Github user dorx commented on the pull request:
https://github.com/apache/spark/pull/1025#issuecomment-46376803
working on migrating from commons-math3 to cern.jet.random
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r13578415
--- Diff: core/src/main/scala/org/apache/spark/util/random/SamplingUtils.scala ---
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util.random
+
+import org.apache.commons.math3.distribution.{PoissonDistribution, NormalDistribution}
+
+private[spark] object PoissonBounds {
+
+ val delta = 1e-4 / 3.0
+ val phi = new NormalDistribution().cumulativeProbability(1.0 - delta)
--- End diff --
`phi` is not used.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1025#issuecomment-50070185
QA results for PR 1025:<br>- This patch FAILED unit tests.<br>- This patch merges cleanly<br>- This patch adds the following public classes (experimental):<br>case class Task[K](stratum: K, count: Long, sampleSize: Long)<br>trait Reviewer[K] extends Serializable with Logging {<br>class BernoulliReviewer[K](@transient tasks: Iterable[Task[K]]) extends Reviewer[K] {<br>case class Policy(preselect: Double, waitlist: Double)<br>class PoissonReviewer[K](@transient tasks: Iterable[Task[K]]) extends Reviewer[K] {<br>case class Policy(preselect: Double, waitlist: Double, var threshold: Double = Double.NaN)<br>class SelectionResultMap[K] extends Serializable {<br>class SelectionResult extends Serializable {<br><br>For more information see test ouptut:<br>https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17130/consoleFull
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1025#issuecomment-47983319
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1025#issuecomment-48130352
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by falaki <gi...@git.apache.org>.
Github user falaki commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r13630894
--- Diff: core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala ---
@@ -156,6 +161,182 @@ class PairRDDFunctions[K, V](self: RDD[(K, V)])
}
/**
+ * Return a subset of this RDD sampled by key (via stratified sampling).
+ * We guarantee a sample size = math.ceil(fraction * S_i), where S_i is the size of the ith
+ * stratum.
+ *
+ * @param withReplacement whether to sample with or without replacement
+ * @param fraction sampling rate
+ * @param seed seed for the random number generator
+ * @return RDD containing the sampled subset
+ */
+ def sampleByKey(withReplacement: Boolean,
+ fraction: Double,
+ seed: Long = Utils.random.nextLong): RDD[(K, V)]= {
+
+ class Stratum(var numItems: Long = 0L, var numAccepted: Long = 0L) extends Serializable {
+ var waitList: ArrayBuffer[Double] = new ArrayBuffer[Double]
+ var q1: Option[Double] = None
+ var q2: Option[Double] = None
+
+ def incrNumItems(by: Long = 1L) = numItems += by
+
+ def incrNumAccepted(by: Long = 1L) = numAccepted += by
+
+ def addToWaitList(elem: Double) = waitList += elem
+
+ def addToWaitList(elems: ArrayBuffer[Double]) = waitList ++= elems
+
+ override def toString() = {
+ "numItems: " + numItems + " numAccepted: " + numAccepted + " q1: " + q1 + " q2: " + q2 +
+ " waitListSize:" + waitList.size
+ }
+ }
+
+ class Result(var resultMap: Map[K, Stratum], var cachedPartitionId: Option[Int] = None)
+ extends Serializable {
+ var rand: RandomDataGenerator = new RandomDataGenerator
+
+ def getEntry(key: K, numItems: Long = 0L): Stratum = {
+ if (resultMap.get(key).isEmpty) {
+ resultMap += (key -> new Stratum(numItems))
+ }
+ resultMap.get(key).get
+ }
+
+ def getRand(partitionId: Int): RandomDataGenerator = {
+ if (cachedPartitionId.isEmpty || cachedPartitionId.get != partitionId) {
+ cachedPartitionId = Some(partitionId)
+ rand.reSeed(seed + partitionId)
+ }
+ rand
+ }
+ }
+
+ // TODO implement the streaming version of sampling w/ replacement that doesn't require counts
+ // in order to save one pass over the RDD
+ val counts = if (withReplacement) Some(this.countByKey()) else None
+
+ val seqOp = (U: (TaskContext, Result), item: (K, V)) => {
+ val delta = 5e-5
+ val result = U._2
+ val tc = U._1
+ val rng = result.getRand(tc.partitionId)
+ val stratum = result.getEntry(item._1)
+ if (withReplacement) {
+ // compute q1 and q2 only if they haven't been computed already
+ // since they don't change from iteration to iteration.
+ // TODO change this to the streaming version
+ if (stratum.q1.isEmpty || stratum.q2.isEmpty) {
+ val n = counts.get(item._1)
+ val s = math.ceil(n * fraction).toLong
+ val lmbd1 = PB.getLambda1(s)
+ val minCount = PB.getMinCount(lmbd1)
+ val lmbd2 = if (lmbd1 == 0) PB.getLambda2(s) else PB.getLambda2(s - minCount)
+ val q1 = lmbd1 / n
+ val q2 = lmbd2 / n
+ stratum.q1 = Some(q1)
+ stratum.q2 = Some(q2)
+ }
+ val x1 = if (stratum.q1.get == 0) 0L else rng.nextPoisson(stratum.q1.get)
+ if (x1 > 0) {
+ stratum.incrNumAccepted(x1)
+ }
+ val x2 = rng.nextPoisson(stratum.q2.get).toInt
+ if (x2 > 0) {
+ stratum.addToWaitList(ArrayBuffer.fill(x2)(rng.nextUniform(0.0, 1.0)))
+ }
+ } else {
+ val g1 = - math.log(delta) / stratum.numItems
+ val g2 = (2.0 / 3.0) * g1
+ val q1 = math.max(0, fraction + g2 - math.sqrt((g2 * g2 + 3 * g2 * fraction)))
+ val q2 = math.min(1, fraction + g1 + math.sqrt(g1 * g1 + 2 * g1 * fraction))
+
+ val x = rng.nextUniform(0.0, 1.0)
+ if (x < q1) {
+ stratum.incrNumAccepted()
+ } else if ( x < q2) {
+ stratum.addToWaitList(x)
+ }
+ stratum.q1 = Some(q1)
+ stratum.q2 = Some(q2)
+ }
+ stratum.incrNumItems()
+ result
+ }
+
+ val combOp = (r1: Result, r2: Result) => {
+ // take union of both key sets in case one partion doesn't contain all keys
+ val keyUnion = r1.resultMap.keys.toSet.union(r2.resultMap.keys.toSet)
+
+ // Use r2 to keep the combined result since r1 is usual empty
+ for (key <- keyUnion) {
+ val entry1 = r1.resultMap.get(key)
+ val entry2 = r2.resultMap.get(key)
+ if (entry2.isEmpty && entry1.isDefined) {
+ r2.resultMap += (key -> entry1.get)
+ } else if (entry1.isDefined && entry2.isDefined) {
+ entry2.get.addToWaitList(entry1.get.waitList)
+ entry2.get.incrNumAccepted(entry1.get.numAccepted)
+ entry2.get.incrNumItems(entry1.get.numItems)
+ }
+ }
+ r2
+ }
+
+ val zeroU = new Result(Map[K, Stratum]())
+
+ // determine threshold for each stratum and resample
+ val finalResult = self.aggregateWithContext(zeroU)(seqOp, combOp).resultMap
+ val thresholdByKey = new mutable.HashMap[K, Double]()
+ for ((key, stratum) <- finalResult) {
+ val s = math.ceil(stratum.numItems * fraction).toLong
+ breakable {
+ if (stratum.numAccepted > s) {
+ logWarning("Pre-accepted too many")
+ thresholdByKey += (key -> stratum.q1.get)
+ break
+ }
+ val numWaitListAccepted = (s - stratum.numAccepted).toInt
+ if (numWaitListAccepted >= stratum.waitList.size) {
+ logWarning("WaitList too short")
+ thresholdByKey += (key -> stratum.q2.get)
+ } else {
+ thresholdByKey += (key -> stratum.waitList.sorted.apply(numWaitListAccepted))
+ }
+ }
+ }
+
+ if (withReplacement) {
+ // Poisson sampler
+ self.mapPartitionsWithIndex((idx: Int, iter: Iterator[(K, V)]) => {
+ val random = new RandomDataGenerator()
+ random.reSeed(seed + idx)
+ iter.flatMap { t =>
+ val q1 = finalResult.get(t._1).get.q1.get
+ val q2 = finalResult.get(t._1).get.q2.get
+ val x1 = if (q1 == 0) 0L else random.nextPoisson(q1)
+ val x2 = random.nextPoisson(q2).toInt
+ val x = x1 + (0 until x2).filter(i => random.nextUniform(0.0, 1.0) <
+ thresholdByKey.get(t._1).get).size
--- End diff --
You can make it simpler by using count:
```scala
(0 until x2).count(i => random.nextUniform(0.0, 1.0) < thresholdByKey.get(t._1).get)
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1025#issuecomment-50379743
QA tests have started for PR 1025. This patch DID NOT merge cleanly! <br>View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17293/consoleFull
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by falaki <gi...@git.apache.org>.
Github user falaki commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r14544689
--- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util.random
+
+import scala.collection.{Map, mutable}
+import scala.collection.mutable.ArrayBuffer
+import scala.reflect.ClassTag
+
+import org.apache.commons.math3.random.RandomDataGenerator
+import org.apache.spark.{Logging, SparkContext, TaskContext}
+import org.apache.spark.rdd.RDD
+import org.apache.spark.util.Utils
+import org.apache.spark.util.random.{PoissonBounds => PB}
+
+private[spark] object StratifiedSampler extends Logging {
+
+ /**
+ * A version of {@link #aggregate()} that passes the TaskContext to the function that does
+ * aggregation for each partition. This function avoids creating an extra depth in the RDD
+ * lineage, as opposed to using mapPartitionsWithId, which results in slightly improved run time.
+ */
+ def aggregateWithContext[U: ClassTag, T: ClassTag](zeroValue: U)
+ (rdd: RDD[T],
+ seqOp: ((TaskContext, U), T) => U,
+ combOp: (U, U) => U): U = {
+ val sc: SparkContext = rdd.sparkContext
+ // Clone the zero value since we will also be serializing it as part of tasks
+ var jobResult = Utils.clone(zeroValue, sc.env.closureSerializer.newInstance())
+ // pad seqOp and combOp with taskContext to conform to aggregate's signature in TraversableOnce
+ val paddedSeqOp = (arg1: (TaskContext, U), item: T) => (arg1._1, seqOp(arg1, item))
+ val paddedcombOp = (arg1: (TaskContext, U), arg2: (TaskContext, U)) =>
+ (arg1._1, combOp(arg1._2, arg1._2))
+ val cleanSeqOp = sc.clean(paddedSeqOp)
+ val cleanCombOp = sc.clean(paddedcombOp)
+ val aggregatePartition = (tc: TaskContext, it: Iterator[T]) =>
+ (it.aggregate(tc, zeroValue)(cleanSeqOp, cleanCombOp))._2
+ val mergeResult = (index: Int, taskResult: U) => jobResult = combOp(jobResult, taskResult)
+ sc.runJob(rdd, aggregatePartition, mergeResult)
+ jobResult
+ }
+
+ /**
+ * Returns the function used by aggregate to collect sampling statistics for each partition.
+ */
+ def getSeqOp[K, V](withReplacement: Boolean,
+ fractionByKey: (K => Double),
+ counts: Option[Map[K, Long]]): ((TaskContext, Result[K]),(K, V)) => Result[K] = {
+ val delta = 5e-5
+ (U: (TaskContext, Result[K]), item: (K, V)) => {
+ val result = U._2
+ val tc = U._1
+ val rng = result.getRand(tc.partitionId)
+ val fraction = fractionByKey(item._1)
+ val stratum = result.getEntry(item._1)
+ if (withReplacement) {
+ // compute q1 and q2 only if they haven't been computed already
+ // since they don't change from iteration to iteration.
+ // TODO change this to the streaming version
+ if (stratum.q1.isEmpty || stratum.q2.isEmpty) {
+ val n = counts.get(item._1)
+ val s = math.ceil(n * fraction).toLong
+ val lmbd1 = PB.getLowerBound(s)
+ val minCount = PB.getMinCount(lmbd1)
+ val lmbd2 = if (lmbd1 == 0) PB.getUpperBound(s) else PB.getUpperBound(s - minCount)
+ val q1 = lmbd1 / n
+ val q2 = lmbd2 / n
+ stratum.q1 = Some(q1)
+ stratum.q2 = Some(q2)
+ }
+ val x1 = if (stratum.q1.get == 0) 0L else rng.nextPoisson(stratum.q1.get)
+ if (x1 > 0) {
+ stratum.incrNumAccepted(x1)
+ }
+ val x2 = rng.nextPoisson(stratum.q2.get).toInt
+ if (x2 > 0) {
+ stratum.addToWaitList(ArrayBuffer.fill(x2)(rng.nextUniform(0.0, 1.0)))
+ }
+ } else {
+ // We use the streaming version of the algorithm for sampling without replacement.
+ // Hence, q1 and q2 change on every iteration.
+ val g1 = - math.log(delta) / stratum.numItems
+ val g2 = (2.0 / 3.0) * g1
+ val q1 = math.max(0, fraction + g2 - math.sqrt((g2 * g2 + 3 * g2 * fraction)))
+ val q2 = math.min(1, fraction + g1 + math.sqrt(g1 * g1 + 2 * g1 * fraction))
+
+ val x = rng.nextUniform(0.0, 1.0)
+ if (x < q1) {
+ stratum.incrNumAccepted()
+ } else if (x < q2) {
+ stratum.addToWaitList(x)
+ }
+ stratum.q1 = Some(q1)
+ stratum.q2 = Some(q2)
+ }
+ stratum.incrNumItems()
+ result
+ }
+ }
+
+ /**
+ * Returns the function used by aggregate to combine results from different partitions, as
+ * returned by seqOp.
+ */
+ def getCombOp[K](): (Result[K], Result[K]) => Result[K] = {
+ (r1: Result[K], r2: Result[K]) => {
+ // take union of both key sets in case one partition doesn't contain all keys
+ val keyUnion = r1.resultMap.keys.toSet.union(r2.resultMap.keys.toSet)
+
+ // Use r2 to keep the combined result since r1 is usual empty
+ for (key <- keyUnion) {
+ val entry1 = r1.resultMap.get(key)
--- End diff --
get() is superfluous, this can just be ```r1.resultMap(key)```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the pull request:
https://github.com/apache/spark/pull/1025#issuecomment-50528615
LGTM. Merged into master. Thanks!!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by falaki <gi...@git.apache.org>.
Github user falaki commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r14544520
--- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util.random
+
+import scala.collection.{Map, mutable}
+import scala.collection.mutable.ArrayBuffer
+import scala.reflect.ClassTag
+
+import org.apache.commons.math3.random.RandomDataGenerator
+import org.apache.spark.{Logging, SparkContext, TaskContext}
+import org.apache.spark.rdd.RDD
+import org.apache.spark.util.Utils
+import org.apache.spark.util.random.{PoissonBounds => PB}
--- End diff --
Let's not rename this. The original name is more descriptive.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r14694285
--- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
@@ -0,0 +1,311 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util.random
+
+import scala.collection.Map
+import scala.collection.mutable.{ArrayBuffer, HashMap, Map => MMap}
+
+import org.apache.commons.math3.random.RandomDataGenerator
+import org.apache.spark.Logging
+import org.apache.spark.rdd.RDD
+
+/**
+ * Auxiliary functions and data structures for the sampleByKey method in PairRDDFunctions.
+ *
+ * Essentially, when exact sample size is necessary, we make additional passes over the RDD to
+ * compute the exact threshold value to use for each stratum to guarantee exact sample size with
+ * high probability. This is achieved by maintaining a waitlist of size O(log(s)), where s is the
+ * desired sample size for each stratum.
+ *
+ * Like in simple random sampling, we generate a random value for each item from the
+ * uniform distribution [0.0, 1.0]. All items with values <= min(values of items in the waitlist)
+ * are accepted into the sample instantly. The threshold for instant accept is designed so that
+ * s - numAccepted = O(log(s)), where s is again the desired sample size. Thus, by maintaining a
+ * waitlist size = O(log(s)), we will be able to create a sample of the exact size s by adding
+ * a portion of the waitlist to the set of items that are instantly accepted. The exact threshold
+ * is computed by sorting the values in the waitlist and picking the value at (s - numAccepted).
+ *
+ * Note that since we use the same seed for the RNG when computing the thresholds and the actual
+ * sample, our computed thresholds are guaranteed to produce the desired sample size.
+ *
+ * For more theoretical background on the sampling techniques used here, please refer to
+ * http://jmlr.org/proceedings/papers/v28/meng13a.html
+ */
+
+private[spark] object StratifiedSampler extends Logging {
+
+ /**
+ * Count the number of items instantly accepted and generate the waitlist for each stratum.
+ *
+ * This is only invoked when exact sample size is required.
+ */
+ def getCounts[K, V](rdd: RDD[(K, V)],
+ withReplacement: Boolean,
+ fractions: Map[K, Double],
+ counts: Option[Map[K, Long]],
+ seed: Long): MMap[K, Stratum] = {
+ val combOp = getCombOp[K]
+ val mappedPartitionRDD = rdd.mapPartitionsWithIndex({ case (partition, iter) =>
+ val zeroU: MMap[K, Stratum] = new HashMap[K, Stratum]()
+ val rng = new RandomDataGenerator()
+ rng.reSeed(seed + partition)
+ val seqOp = getSeqOp(withReplacement, fractions, rng, counts)
+ Iterator(iter.aggregate(zeroU)(seqOp, combOp))
+ }, preservesPartitioning=true)
+ mappedPartitionRDD.reduce(combOp)
+ }
+
+ /**
+ * Returns the function used by aggregate to collect sampling statistics for each partition.
+ */
+ def getSeqOp[K, V](withReplacement: Boolean,
+ fractions: Map[K, Double],
+ rng: RandomDataGenerator,
+ counts: Option[Map[K, Long]]): (MMap[K, Stratum], (K, V)) => MMap[K, Stratum] = {
+ val delta = 5e-5
+ (result: MMap[K, Stratum], item: (K, V)) => {
+ val key = item._1
+ val fraction = fractions(key)
+ if (!result.contains(key)) {
+ result += (key -> new Stratum())
+ }
+ val stratum = result(key)
+
+ if (withReplacement) {
+ // compute acceptBound and waitListBound only if they haven't been computed already
+ // since they don't change from iteration to iteration.
+ // TODO change this to the streaming version
+ if (stratum.areBoundsEmpty) {
+ val n = counts.get(key)
+ val sampleSize = math.ceil(n * fraction).toLong
+ val lmbd1 = PoissonBounds.getLowerBound(sampleSize)
+ val minCount = PoissonBounds.getMinCount(lmbd1)
+ val lmbd2 = if (lmbd1 == 0) {
+ PoissonBounds.getUpperBound(sampleSize)
+ } else {
+ PoissonBounds.getUpperBound(sampleSize - minCount)
+ }
+ stratum.acceptBound = lmbd1 / n
+ stratum.waitListBound = lmbd2 / n
+ }
+ val acceptBound = stratum.acceptBound
+ val copiesAccepted = if (acceptBound == 0.0) 0L else rng.nextPoisson(acceptBound)
+ if (copiesAccepted > 0) {
+ stratum.incrNumAccepted(copiesAccepted)
+ }
+ val copiesWaitlisted = rng.nextPoisson(stratum.waitListBound).toInt
+ if (copiesWaitlisted > 0) {
+ stratum.addToWaitList(ArrayBuffer.fill(copiesWaitlisted)(rng.nextUniform(0.0, 1.0)))
+ }
+ } else {
+ // We use the streaming version of the algorithm for sampling without replacement to avoid
+ // using an extra pass over the RDD for computing the count.
+ // Hence, acceptBound and waitListBound change on every iteration.
+ val gamma1 = - math.log(delta) / stratum.numItems
+ val gamma2 = (2.0 / 3.0) * gamma1
+ stratum.acceptBound = math.max(0,
+ fraction + gamma2 - math.sqrt(gamma2 * gamma2 + 3 * gamma2 * fraction))
+ stratum.waitListBound = math.min(1,
+ fraction + gamma1 + math.sqrt(gamma1 * gamma1 + 2 * gamma1 * fraction))
+
+ val x = rng.nextUniform(0.0, 1.0)
+ if (x < stratum.acceptBound) {
+ stratum.incrNumAccepted()
+ } else if (x < stratum.waitListBound) {
+ stratum.addToWaitList(x)
+ }
+ }
+ stratum.incrNumItems()
+ result
+ }
+ }
+
+ /**
+ * Returns the function used combine results returned by seqOp from different partitions.
+ */
+ def getCombOp[K]: (MMap[K, Stratum], MMap[K, Stratum]) => MMap[K, Stratum] = {
+ (result1: MMap[K, Stratum], result2: MMap[K, Stratum]) => {
+ // take union of both key sets in case one partition doesn't contain all keys
+ for (key <- result1.keySet.union(result2.keySet)) {
+ // Use result2 to keep the combined result since r1 is usual empty
--- End diff --
The contract of aggregate only allows modification to the first argument.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1025#issuecomment-46644693
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15938/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1025#issuecomment-48416874
All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16431/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1025#issuecomment-46377060
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15859/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r14694267
--- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
@@ -0,0 +1,311 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util.random
+
+import scala.collection.Map
+import scala.collection.mutable.{ArrayBuffer, HashMap, Map => MMap}
+
+import org.apache.commons.math3.random.RandomDataGenerator
+import org.apache.spark.Logging
+import org.apache.spark.rdd.RDD
+
+/**
+ * Auxiliary functions and data structures for the sampleByKey method in PairRDDFunctions.
+ *
+ * Essentially, when exact sample size is necessary, we make additional passes over the RDD to
+ * compute the exact threshold value to use for each stratum to guarantee exact sample size with
+ * high probability. This is achieved by maintaining a waitlist of size O(log(s)), where s is the
+ * desired sample size for each stratum.
+ *
+ * Like in simple random sampling, we generate a random value for each item from the
+ * uniform distribution [0.0, 1.0]. All items with values <= min(values of items in the waitlist)
+ * are accepted into the sample instantly. The threshold for instant accept is designed so that
+ * s - numAccepted = O(log(s)), where s is again the desired sample size. Thus, by maintaining a
+ * waitlist size = O(log(s)), we will be able to create a sample of the exact size s by adding
+ * a portion of the waitlist to the set of items that are instantly accepted. The exact threshold
+ * is computed by sorting the values in the waitlist and picking the value at (s - numAccepted).
+ *
+ * Note that since we use the same seed for the RNG when computing the thresholds and the actual
+ * sample, our computed thresholds are guaranteed to produce the desired sample size.
+ *
+ * For more theoretical background on the sampling techniques used here, please refer to
+ * http://jmlr.org/proceedings/papers/v28/meng13a.html
+ */
+
+private[spark] object StratifiedSampler extends Logging {
--- End diff --
As mentioned in the doc, this is not a sampler but something like `StratifiedSamplingUtils`. I think it is okay to merge the functions into `SamplingUtils`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by dorx <gi...@git.apache.org>.
GitHub user dorx reopened a pull request:
https://github.com/apache/spark/pull/1025
[SPARK-2082] stratified sampling in PairRDDFunctions that guarantees exact sample size
Implemented stratified sampling that guarantees exact sample size using ScaRSR with two passes over the RDD for sampling without replacement and three passes for sampling with replacement.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/dorx/spark stratified
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/1025.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1025
----
commit 14419775202e6eef1f0e1f0c74c7be9030aca73d
Author: Doris Xin <do...@gmail.com>
Date: 2014-05-29T22:22:14Z
SPARK-1939 Refactor takeSample method in RDD to use ScaSRS
commit ffea61a67d228edb476d29ca13a84bb3f9a22887
Author: Doris Xin <do...@gmail.com>
Date: 2014-05-30T00:55:54Z
SPARK-1939: Refactor takeSample method in RDD
Reviewer comments addressed:
- commons-math3 is now a test-only dependency. bumped up to v3.3
- comments added to explain what computeFraction is doing
- fixed the unit for computeFraction to use BinomialDitro for without
replacement sampling
- stylistic fixes
commit 7cab53a3926f4351432e5e3600b0796b9a4146e4
Author: Doris Xin <do...@gmail.com>
Date: 2014-06-02T19:00:38Z
fixed import bug in rdd.py
commit e3fd6a628317d559a08a7a20421e9c0618180902
Author: Doris Xin <do...@gmail.com>
Date: 2014-06-02T19:06:18Z
Merge branch 'master' into takeSample
commit 9ee94ee3c28e8d808063fef4e5d39f06ab738e0b
Author: Doris Xin <do...@gmail.com>
Date: 2014-06-09T20:15:23Z
[SPARK-2082] stratified sampling in PairRDDFunctions that guarantees exact sample size
commit 1d413ce877a67379a0a74afefba071c018b0ca70
Author: Doris Xin <do...@gmail.com>
Date: 2014-06-09T21:14:26Z
fixed checkstyle issues
commit 7e1a48182ebec54cd3a6a290b1dc27b928f57dba
Author: Doris Xin <do...@gmail.com>
Date: 2014-06-09T21:29:30Z
changed the permission on SamplingUtil
commit 46f6c8c86f3e1fdaf49b63796f1cd5bd6db79ec7
Author: Doris Xin <do...@gmail.com>
Date: 2014-06-10T00:39:50Z
fixed the NPE caused by closures being cleaned before being passed into the aggregate function
commit 50581fc8b08bd5f18cdf2288772c22f2549af0a5
Author: Doris Xin <do...@gmail.com>
Date: 2014-06-12T21:36:37Z
added a TODO for logging in python
commit 73276111a2c4a9354bfbf6414afceabf70fb21e5
Author: Doris Xin <do...@gmail.com>
Date: 2014-06-13T00:28:14Z
merge master
commit 9e74ab505e5441eedfed5dbfbeac37566d3de1f0
Author: Doris Xin <do...@gmail.com>
Date: 2014-06-17T22:24:22Z
Separated out most of the logic in sampleByKey
into StratifiedSampler in util.random
commit 90d94c0d4f4d909fb99b8e72f9a09ca5329e070c
Author: Doris Xin <do...@gmail.com>
Date: 2014-06-17T22:57:11Z
merge master
commit 0214a7659c62e4ff0f68f6e09cd7846547cd3bcb
Author: Doris Xin <do...@gmail.com>
Date: 2014-06-18T02:22:32Z
cleanUp
Addressed reviewer comments and added better documentation of code.
Added commons-math3 as a dependency of spark (okay’ed by Matei). “mvm
clean install” compiled. Recovered files that were reverted by accident
in the merge.
TODOs: figure out API for sampleByKeyExact and update Java, Python, and
the markdown file accordingly.
commit 944a10cff3c218bafcb8b43e7e3f309cc644633e
Author: Doris Xin <do...@gmail.com>
Date: 2014-06-19T02:48:22Z
[SPARK-2145] Add lower bound on sampling rate
to guarantee sampling performance
commit 1fe1cff58d99f336406f67f89f53fe6ab3bfde5e
Author: Doris Xin <do...@gmail.com>
Date: 2014-06-19T19:46:59Z
Changed fractionByKey to a map to enable arg check
commit bd9dc6e08444ec83c5d6adc4c452b49d1ac2b154
Author: Doris Xin <do...@gmail.com>
Date: 2014-06-19T20:00:47Z
unit bug and style violation fixed
commit 4ad516b14559f06a32d65ef0ce2fa2d7526610bc
Author: Doris Xin <do...@gmail.com>
Date: 2014-06-20T18:38:09Z
remove unused imports from PairRDDFunctions
commit 254e03c96e1f2aaa5baa9c3d384adeb117e0b7ab
Author: Doris Xin <do...@gmail.com>
Date: 2014-07-03T20:49:46Z
minor fixes and Java API.
punting on python for now. moved aggregateWithContext out of RDD
commit 6b5b10b71669590f862ec2b7d004ba976d59676c
Author: Doris Xin <do...@gmail.com>
Date: 2014-07-03T20:58:29Z
Merge branch 'master' into stratified
commit ee9d260e5eb6ee2d6912e57491daeded1704248c
Author: Doris Xin <do...@gmail.com>
Date: 2014-07-06T23:20:39Z
addressed reviewer comments
commit bbfb8c91a68deae076db7d9fdc518b6cee8a99f1
Author: Doris Xin <do...@gmail.com>
Date: 2014-07-06T23:21:20Z
Merge branch 'master' into stratified
commit 9884a9f03b18f3170d90d168c246d78fc463028e
Author: Doris Xin <do...@gmail.com>
Date: 2014-07-08T18:58:39Z
style fix
commit 680b677bc5276e1499c59c7e24abfae7d85e5c7d
Author: Doris Xin <do...@gmail.com>
Date: 2014-07-08T23:49:25Z
use mapPartitionWithIndex instead
also better readability and lots more comments.
commit a2bf756454b2ae6b71ddc67457c0f69f165b937d
Author: Doris Xin <do...@gmail.com>
Date: 2014-07-08T23:50:17Z
Merge branch 'master' into stratified
commit a10e68dd9c23b0fc1f25effd9ccd0ac3e7299206
Author: Doris Xin <do...@gmail.com>
Date: 2014-07-09T01:09:14Z
style fix
commit f4c21f324075bc5e7b8cba07a5c47d23fade542f
Author: Doris Xin <do...@gmail.com>
Date: 2014-07-15T01:34:10Z
Reviewer comments
Added BernoulliBounds
commit eecee5fb31aa678db872eccbd7385461ea3a6a96
Author: Doris Xin <do...@gmail.com>
Date: 2014-07-15T02:56:58Z
Merge branch 'master' into stratified
Conflicts:
project/SparkBuild.scala
commit 1f9e2654921194baba29d58b862f21bd53c29470
Author: Xiangrui Meng <me...@databricks.com>
Date: 2014-07-24T04:15:05Z
merge master
commit bd6652d8e69c18dc8ee60ae00272fb99958bfae2
Author: Xiangrui Meng <me...@databricks.com>
Date: 2014-07-24T04:15:54Z
change commons-math3 scope to test
commit 3f6f0b784a0456f98619e40dd97a04352efd2192
Author: Xiangrui Meng <me...@databricks.com>
Date: 2014-07-24T12:41:58Z
refractor stratified sampling impl
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1025#issuecomment-46608301
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15919/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1025#issuecomment-48131246
All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16359/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by dorx <gi...@git.apache.org>.
Github user dorx commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r14542699
--- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util.random
+
+import scala.collection.{Map, mutable}
+import scala.collection.mutable.ArrayBuffer
+import scala.reflect.ClassTag
+
+import org.apache.commons.math3.random.RandomDataGenerator
+import org.apache.spark.{Logging, SparkContext, TaskContext}
+import org.apache.spark.rdd.RDD
+import org.apache.spark.util.Utils
+import org.apache.spark.util.random.{PoissonBounds => PB}
+
+private[spark] object StratifiedSampler extends Logging {
+
+ /**
+ * A version of {@link #aggregate()} that passes the TaskContext to the function that does
+ * aggregation for each partition. This function avoids creating an extra depth in the RDD
+ * lineage, as opposed to using mapPartitionsWithId, which results in slightly improved run time.
+ */
+ def aggregateWithContext[U: ClassTag, T: ClassTag](zeroValue: U)
+ (rdd: RDD[T],
+ seqOp: ((TaskContext, U), T) => U,
+ combOp: (U, U) => U): U = {
+ val sc: SparkContext = rdd.sparkContext
+ // Clone the zero value since we will also be serializing it as part of tasks
+ var jobResult = Utils.clone(zeroValue, sc.env.closureSerializer.newInstance())
+ // pad seqOp and combOp with taskContext to conform to aggregate's signature in TraversableOnce
+ val paddedSeqOp = (arg1: (TaskContext, U), item: T) => (arg1._1, seqOp(arg1, item))
+ val paddedcombOp = (arg1: (TaskContext, U), arg2: (TaskContext, U)) =>
+ (arg1._1, combOp(arg1._2, arg1._2))
+ val cleanSeqOp = sc.clean(paddedSeqOp)
+ val cleanCombOp = sc.clean(paddedcombOp)
+ val aggregatePartition = (tc: TaskContext, it: Iterator[T]) =>
+ (it.aggregate(tc, zeroValue)(cleanSeqOp, cleanCombOp))._2
+ val mergeResult = (index: Int, taskResult: U) => jobResult = combOp(jobResult, taskResult)
+ sc.runJob(rdd, aggregatePartition, mergeResult)
+ jobResult
+ }
+
+ /**
+ * Returns the function used by aggregate to collect sampling statistics for each partition.
+ */
+ def getSeqOp[K, V](withReplacement: Boolean,
+ fractionByKey: (K => Double),
+ counts: Option[Map[K, Long]]): ((TaskContext, Result[K]),(K, V)) => Result[K] = {
+ val delta = 5e-5
+ (U: (TaskContext, Result[K]), item: (K, V)) => {
+ val result = U._2
+ val tc = U._1
+ val rng = result.getRand(tc.partitionId)
+ val fraction = fractionByKey(item._1)
+ val stratum = result.getEntry(item._1)
+ if (withReplacement) {
+ // compute q1 and q2 only if they haven't been computed already
+ // since they don't change from iteration to iteration.
+ // TODO change this to the streaming version
+ if (stratum.q1.isEmpty || stratum.q2.isEmpty) {
+ val n = counts.get(item._1)
+ val s = math.ceil(n * fraction).toLong
+ val lmbd1 = PB.getLowerBound(s)
+ val minCount = PB.getMinCount(lmbd1)
+ val lmbd2 = if (lmbd1 == 0) PB.getUpperBound(s) else PB.getUpperBound(s - minCount)
+ val q1 = lmbd1 / n
+ val q2 = lmbd2 / n
+ stratum.q1 = Some(q1)
+ stratum.q2 = Some(q2)
+ }
+ val x1 = if (stratum.q1.get == 0) 0L else rng.nextPoisson(stratum.q1.get)
+ if (x1 > 0) {
+ stratum.incrNumAccepted(x1)
+ }
+ val x2 = rng.nextPoisson(stratum.q2.get).toInt
+ if (x2 > 0) {
+ stratum.addToWaitList(ArrayBuffer.fill(x2)(rng.nextUniform(0.0, 1.0)))
+ }
+ } else {
+ // We use the streaming version of the algorithm for sampling without replacement.
+ // Hence, q1 and q2 change on every iteration.
+ val g1 = - math.log(delta) / stratum.numItems
+ val g2 = (2.0 / 3.0) * g1
+ val q1 = math.max(0, fraction + g2 - math.sqrt((g2 * g2 + 3 * g2 * fraction)))
+ val q2 = math.min(1, fraction + g1 + math.sqrt(g1 * g1 + 2 * g1 * fraction))
+
+ val x = rng.nextUniform(0.0, 1.0)
+ if (x < q1) {
+ stratum.incrNumAccepted()
+ } else if (x < q2) {
+ stratum.addToWaitList(x)
+ }
+ stratum.q1 = Some(q1)
+ stratum.q2 = Some(q2)
+ }
+ stratum.incrNumItems()
+ result
+ }
+ }
+
+ /**
+ * Returns the function used by aggregate to combine results from different partitions, as
+ * returned by seqOp.
+ */
+ def getCombOp[K](): (Result[K], Result[K]) => Result[K] = {
+ (r1: Result[K], r2: Result[K]) => {
+ // take union of both key sets in case one partition doesn't contain all keys
+ val keyUnion = r1.resultMap.keys.toSet.union(r2.resultMap.keys.toSet)
+
+ // Use r2 to keep the combined result since r1 is usual empty
+ for (key <- keyUnion) {
+ val entry1 = r1.resultMap.get(key)
+ val entry2 = r2.resultMap.get(key)
+ if (entry2.isEmpty && entry1.isDefined) {
+ r2.resultMap += (key -> entry1.get)
+ } else if (entry1.isDefined && entry2.isDefined) {
+ entry2.get.addToWaitList(entry1.get.waitList)
+ entry2.get.incrNumAccepted(entry1.get.numAccepted)
+ entry2.get.incrNumItems(entry1.get.numItems)
+ }
+ }
+ r2
+ }
+ }
+
+ /**
+ * Given the result returned by the aggregate function, we need to determine the threshold used
+ * to accept items to generate the exact sample size.
+ */
+ def computeThresholdByKey[K](finalResult: Map[K, Stratum], fractionByKey: (K => Double)):
+ (K => Double) = {
+ val thresholdByKey = new mutable.HashMap[K, Double]()
+ for ((key, stratum) <- finalResult) {
+ val fraction = fractionByKey(key)
+ val s = math.ceil(stratum.numItems * fraction).toLong
+ if (stratum.numAccepted > s) {
+ logWarning("Pre-accepted too many")
+ thresholdByKey += (key -> stratum.q1.get)
+ } else {
+ val numWaitListAccepted = (s - stratum.numAccepted).toInt
+ if (numWaitListAccepted >= stratum.waitList.size) {
+ logWarning("WaitList too short")
+ thresholdByKey += (key -> stratum.q2.get)
+ } else {
+ thresholdByKey += (key -> stratum.waitList.sorted.apply(numWaitListAccepted))
+ }
+ }
+ }
+ thresholdByKey
+ }
+
+ /**
+ * Return the per partition sampling function used for sampling without replacement.
+ *
+ * When exact sample size is required, we make an additional pass over the RDD to determine the
+ * exact sampling rate that guarantees sample size with high confidence.
+ *
+ * The sampling function has a unique seed per partition.
+ */
+ def getBernoulliSamplingFunction[K, V](rdd: RDD[(K, V)],
+ fractionByKey: K => Double,
+ exact: Boolean,
+ seed: Long): (Int, Iterator[(K, V)]) => Iterator[(K, V)] = {
+ var samplingRateByKey = fractionByKey
+ if (exact) {
+ // determine threshold for each stratum and resample
+ val seqOp = StratifiedSampler.getSeqOp[K,V](false, fractionByKey, None)
+ val combOp = StratifiedSampler.getCombOp[K]()
+ val zeroU = new Result[K](Map[K, Stratum](), seed = seed)
+ val finalResult = aggregateWithContext(zeroU)(rdd, seqOp, combOp).resultMap
+ samplingRateByKey = StratifiedSampler.computeThresholdByKey(finalResult, fractionByKey)
+ }
+ (idx: Int, iter: Iterator[(K, V)]) => {
+ val random = new RandomDataGenerator
+ random.reSeed(seed + idx)
+ iter.filter(t => random.nextUniform(0.0, 1.0) < samplingRateByKey(t._1))
+ }
+ }
+
+ /**
+ * Return the per partition sampling function used for sampling with replacement.
+ *
+ * When exact sample size is required, we make two additional passed over the RDD to determine
+ * the exact sampling rate that guarantees sample size with high confidence. The first pass
+ * counts the number of items in each stratum (group of items with the same key) in the RDD, and
+ * the second pass uses the counts to determine exact sampling rates.
+ *
+ * The sampling function has a unique seed per partition.
+ */
+ def getPoissonSamplingFunction[K, V](rdd:RDD[(K, V)],
+ fractionByKey: K => Double,
+ exact: Boolean,
+ counts: Option[Map[K, Long]],
+ seed: Long): (Int, Iterator[(K, V)]) => Iterator[(K, V)] = {
+ // TODO implement the streaming version of sampling w/ replacement that doesn't require counts
+ if (exact) {
+ val seqOp = StratifiedSampler.getSeqOp[K,V](true, fractionByKey, counts)
+ val combOp = StratifiedSampler.getCombOp[K]()
+ val zeroU = new Result[K](Map[K, Stratum](), seed = seed)
+ val finalResult = aggregateWithContext(zeroU)(rdd, seqOp, combOp).resultMap
+ val thresholdByKey = StratifiedSampler.computeThresholdByKey(finalResult, fractionByKey)
+ (idx: Int, iter: Iterator[(K, V)]) => {
+ val random = new RandomDataGenerator()
+ random.reSeed(seed + idx)
+ iter.flatMap { t =>
+ val q1 = finalResult.get(t._1).get.q1.getOrElse(0.0)
+ val q2 = finalResult.get(t._1).get.q2.getOrElse(0.0)
+ val x1 = if (q1 == 0) 0L else random.nextPoisson(q1)
+ val x2 = random.nextPoisson(q2).toInt
+ val x = x1 + (0 until x2).count(i => random.nextUniform(0.0, 1.0) < thresholdByKey(t._1))
+ if (x > 0) {
+ Iterator.fill(x.toInt)(t)
+ } else {
+ Iterator.empty
+ }
+ }
+ }
+ } else {
+ (idx: Int, iter: Iterator[(K, V)]) => {
+ val random = new RandomDataGenerator()
+ random.reSeed(seed + idx)
+ iter.flatMap { t =>
+ val count = random.nextPoisson(fractionByKey(t._1)).toInt
+ if (count > 0) {
+ Iterator.fill(count)(t)
+ } else {
+ Iterator.empty
+ }
+ }
+ }
+ }
+ }
+}
+
+/**
+ * Object used by seqOp to keep track of the number of items accepted and items waitlisted per
+ * stratum, as well as the bounds for accepting and waitlisting items.
+ */
+private[random] class Stratum(var numItems: Long = 0L, var numAccepted: Long = 0L)
--- End diff --
"[random]" can be removed. it's only used in this class
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1025#issuecomment-47983304
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by falaki <gi...@git.apache.org>.
Github user falaki commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r14544595
--- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util.random
+
+import scala.collection.{Map, mutable}
+import scala.collection.mutable.ArrayBuffer
+import scala.reflect.ClassTag
+
+import org.apache.commons.math3.random.RandomDataGenerator
+import org.apache.spark.{Logging, SparkContext, TaskContext}
+import org.apache.spark.rdd.RDD
+import org.apache.spark.util.Utils
+import org.apache.spark.util.random.{PoissonBounds => PB}
+
+private[spark] object StratifiedSampler extends Logging {
+
+ /**
+ * A version of {@link #aggregate()} that passes the TaskContext to the function that does
+ * aggregation for each partition. This function avoids creating an extra depth in the RDD
+ * lineage, as opposed to using mapPartitionsWithId, which results in slightly improved run time.
+ */
+ def aggregateWithContext[U: ClassTag, T: ClassTag](zeroValue: U)
+ (rdd: RDD[T],
+ seqOp: ((TaskContext, U), T) => U,
+ combOp: (U, U) => U): U = {
+ val sc: SparkContext = rdd.sparkContext
+ // Clone the zero value since we will also be serializing it as part of tasks
+ var jobResult = Utils.clone(zeroValue, sc.env.closureSerializer.newInstance())
+ // pad seqOp and combOp with taskContext to conform to aggregate's signature in TraversableOnce
+ val paddedSeqOp = (arg1: (TaskContext, U), item: T) => (arg1._1, seqOp(arg1, item))
+ val paddedcombOp = (arg1: (TaskContext, U), arg2: (TaskContext, U)) =>
+ (arg1._1, combOp(arg1._2, arg1._2))
+ val cleanSeqOp = sc.clean(paddedSeqOp)
+ val cleanCombOp = sc.clean(paddedcombOp)
+ val aggregatePartition = (tc: TaskContext, it: Iterator[T]) =>
+ (it.aggregate(tc, zeroValue)(cleanSeqOp, cleanCombOp))._2
+ val mergeResult = (index: Int, taskResult: U) => jobResult = combOp(jobResult, taskResult)
+ sc.runJob(rdd, aggregatePartition, mergeResult)
+ jobResult
+ }
+
+ /**
+ * Returns the function used by aggregate to collect sampling statistics for each partition.
+ */
+ def getSeqOp[K, V](withReplacement: Boolean,
+ fractionByKey: (K => Double),
+ counts: Option[Map[K, Long]]): ((TaskContext, Result[K]),(K, V)) => Result[K] = {
+ val delta = 5e-5
+ (U: (TaskContext, Result[K]), item: (K, V)) => {
+ val result = U._2
+ val tc = U._1
+ val rng = result.getRand(tc.partitionId)
+ val fraction = fractionByKey(item._1)
+ val stratum = result.getEntry(item._1)
+ if (withReplacement) {
+ // compute q1 and q2 only if they haven't been computed already
+ // since they don't change from iteration to iteration.
+ // TODO change this to the streaming version
+ if (stratum.q1.isEmpty || stratum.q2.isEmpty) {
+ val n = counts.get(item._1)
+ val s = math.ceil(n * fraction).toLong
+ val lmbd1 = PB.getLowerBound(s)
+ val minCount = PB.getMinCount(lmbd1)
+ val lmbd2 = if (lmbd1 == 0) PB.getUpperBound(s) else PB.getUpperBound(s - minCount)
+ val q1 = lmbd1 / n
+ val q2 = lmbd2 / n
+ stratum.q1 = Some(q1)
+ stratum.q2 = Some(q2)
+ }
+ val x1 = if (stratum.q1.get == 0) 0L else rng.nextPoisson(stratum.q1.get)
+ if (x1 > 0) {
+ stratum.incrNumAccepted(x1)
+ }
+ val x2 = rng.nextPoisson(stratum.q2.get).toInt
+ if (x2 > 0) {
+ stratum.addToWaitList(ArrayBuffer.fill(x2)(rng.nextUniform(0.0, 1.0)))
+ }
+ } else {
+ // We use the streaming version of the algorithm for sampling without replacement.
+ // Hence, q1 and q2 change on every iteration.
--- End diff --
We are implementing a non-trivial algorithm here. Lets use more descriptive names instead of g1, q2, q1, and q2.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r14500775
--- Diff: core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala ---
@@ -27,8 +27,12 @@ import scala.collection.Map
import scala.collection.mutable
import scala.collection.mutable.ArrayBuffer
import scala.reflect.ClassTag
+import scala.util.control.Breaks._
--- End diff --
This is not a native feature of Scala, neither a for loop. Try to use while with a flag that terminates the loop.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by falaki <gi...@git.apache.org>.
Github user falaki commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r14544621
--- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util.random
+
+import scala.collection.{Map, mutable}
+import scala.collection.mutable.ArrayBuffer
+import scala.reflect.ClassTag
+
+import org.apache.commons.math3.random.RandomDataGenerator
+import org.apache.spark.{Logging, SparkContext, TaskContext}
+import org.apache.spark.rdd.RDD
+import org.apache.spark.util.Utils
+import org.apache.spark.util.random.{PoissonBounds => PB}
+
+private[spark] object StratifiedSampler extends Logging {
+
+ /**
+ * A version of {@link #aggregate()} that passes the TaskContext to the function that does
+ * aggregation for each partition. This function avoids creating an extra depth in the RDD
+ * lineage, as opposed to using mapPartitionsWithId, which results in slightly improved run time.
+ */
+ def aggregateWithContext[U: ClassTag, T: ClassTag](zeroValue: U)
+ (rdd: RDD[T],
+ seqOp: ((TaskContext, U), T) => U,
+ combOp: (U, U) => U): U = {
+ val sc: SparkContext = rdd.sparkContext
+ // Clone the zero value since we will also be serializing it as part of tasks
+ var jobResult = Utils.clone(zeroValue, sc.env.closureSerializer.newInstance())
+ // pad seqOp and combOp with taskContext to conform to aggregate's signature in TraversableOnce
+ val paddedSeqOp = (arg1: (TaskContext, U), item: T) => (arg1._1, seqOp(arg1, item))
+ val paddedcombOp = (arg1: (TaskContext, U), arg2: (TaskContext, U)) =>
+ (arg1._1, combOp(arg1._2, arg1._2))
+ val cleanSeqOp = sc.clean(paddedSeqOp)
+ val cleanCombOp = sc.clean(paddedcombOp)
+ val aggregatePartition = (tc: TaskContext, it: Iterator[T]) =>
+ (it.aggregate(tc, zeroValue)(cleanSeqOp, cleanCombOp))._2
+ val mergeResult = (index: Int, taskResult: U) => jobResult = combOp(jobResult, taskResult)
+ sc.runJob(rdd, aggregatePartition, mergeResult)
+ jobResult
+ }
+
+ /**
+ * Returns the function used by aggregate to collect sampling statistics for each partition.
+ */
+ def getSeqOp[K, V](withReplacement: Boolean,
+ fractionByKey: (K => Double),
+ counts: Option[Map[K, Long]]): ((TaskContext, Result[K]),(K, V)) => Result[K] = {
+ val delta = 5e-5
+ (U: (TaskContext, Result[K]), item: (K, V)) => {
+ val result = U._2
+ val tc = U._1
+ val rng = result.getRand(tc.partitionId)
+ val fraction = fractionByKey(item._1)
+ val stratum = result.getEntry(item._1)
+ if (withReplacement) {
+ // compute q1 and q2 only if they haven't been computed already
+ // since they don't change from iteration to iteration.
+ // TODO change this to the streaming version
+ if (stratum.q1.isEmpty || stratum.q2.isEmpty) {
+ val n = counts.get(item._1)
+ val s = math.ceil(n * fraction).toLong
+ val lmbd1 = PB.getLowerBound(s)
+ val minCount = PB.getMinCount(lmbd1)
+ val lmbd2 = if (lmbd1 == 0) PB.getUpperBound(s) else PB.getUpperBound(s - minCount)
+ val q1 = lmbd1 / n
+ val q2 = lmbd2 / n
+ stratum.q1 = Some(q1)
+ stratum.q2 = Some(q2)
+ }
+ val x1 = if (stratum.q1.get == 0) 0L else rng.nextPoisson(stratum.q1.get)
+ if (x1 > 0) {
+ stratum.incrNumAccepted(x1)
+ }
+ val x2 = rng.nextPoisson(stratum.q2.get).toInt
+ if (x2 > 0) {
+ stratum.addToWaitList(ArrayBuffer.fill(x2)(rng.nextUniform(0.0, 1.0)))
+ }
+ } else {
+ // We use the streaming version of the algorithm for sampling without replacement.
+ // Hence, q1 and q2 change on every iteration.
+ val g1 = - math.log(delta) / stratum.numItems
+ val g2 = (2.0 / 3.0) * g1
+ val q1 = math.max(0, fraction + g2 - math.sqrt((g2 * g2 + 3 * g2 * fraction)))
+ val q2 = math.min(1, fraction + g1 + math.sqrt(g1 * g1 + 2 * g1 * fraction))
+
+ val x = rng.nextUniform(0.0, 1.0)
+ if (x < q1) {
+ stratum.incrNumAccepted()
+ } else if (x < q2) {
+ stratum.addToWaitList(x)
+ }
+ stratum.q1 = Some(q1)
+ stratum.q2 = Some(q2)
+ }
+ stratum.incrNumItems()
+ result
+ }
+ }
+
+ /**
+ * Returns the function used by aggregate to combine results from different partitions, as
+ * returned by seqOp.
+ */
+ def getCombOp[K](): (Result[K], Result[K]) => Result[K] = {
+ (r1: Result[K], r2: Result[K]) => {
+ // take union of both key sets in case one partition doesn't contain all keys
+ val keyUnion = r1.resultMap.keys.toSet.union(r2.resultMap.keys.toSet)
+
+ // Use r2 to keep the combined result since r1 is usual empty
+ for (key <- keyUnion) {
+ val entry1 = r1.resultMap.get(key)
+ val entry2 = r2.resultMap.get(key)
+ if (entry2.isEmpty && entry1.isDefined) {
+ r2.resultMap += (key -> entry1.get)
+ } else if (entry1.isDefined && entry2.isDefined) {
+ entry2.get.addToWaitList(entry1.get.waitList)
+ entry2.get.incrNumAccepted(entry1.get.numAccepted)
+ entry2.get.incrNumItems(entry1.get.numItems)
+ }
+ }
+ r2
+ }
+ }
+
+ /**
+ * Given the result returned by the aggregate function, we need to determine the threshold used
+ * to accept items to generate the exact sample size.
+ */
+ def computeThresholdByKey[K](finalResult: Map[K, Stratum], fractionByKey: (K => Double)):
+ (K => Double) = {
+ val thresholdByKey = new mutable.HashMap[K, Double]()
+ for ((key, stratum) <- finalResult) {
+ val fraction = fractionByKey(key)
+ val s = math.ceil(stratum.numItems * fraction).toLong
+ if (stratum.numAccepted > s) {
+ logWarning("Pre-accepted too many")
+ thresholdByKey += (key -> stratum.q1.get)
+ } else {
+ val numWaitListAccepted = (s - stratum.numAccepted).toInt
+ if (numWaitListAccepted >= stratum.waitList.size) {
+ logWarning("WaitList too short")
+ thresholdByKey += (key -> stratum.q2.get)
+ } else {
+ thresholdByKey += (key -> stratum.waitList.sorted.apply(numWaitListAccepted))
+ }
+ }
+ }
+ thresholdByKey
+ }
+
+ /**
+ * Return the per partition sampling function used for sampling without replacement.
+ *
+ * When exact sample size is required, we make an additional pass over the RDD to determine the
+ * exact sampling rate that guarantees sample size with high confidence.
+ *
+ * The sampling function has a unique seed per partition.
+ */
+ def getBernoulliSamplingFunction[K, V](rdd: RDD[(K, V)],
+ fractionByKey: K => Double,
+ exact: Boolean,
+ seed: Long): (Int, Iterator[(K, V)]) => Iterator[(K, V)] = {
+ var samplingRateByKey = fractionByKey
+ if (exact) {
+ // determine threshold for each stratum and resample
+ val seqOp = StratifiedSampler.getSeqOp[K,V](false, fractionByKey, None)
+ val combOp = StratifiedSampler.getCombOp[K]()
+ val zeroU = new Result[K](Map[K, Stratum](), seed = seed)
+ val finalResult = aggregateWithContext(zeroU)(rdd, seqOp, combOp).resultMap
+ samplingRateByKey = StratifiedSampler.computeThresholdByKey(finalResult, fractionByKey)
+ }
+ (idx: Int, iter: Iterator[(K, V)]) => {
+ val random = new RandomDataGenerator
+ random.reSeed(seed + idx)
+ iter.filter(t => random.nextUniform(0.0, 1.0) < samplingRateByKey(t._1))
+ }
+ }
+
+ /**
+ * Return the per partition sampling function used for sampling with replacement.
+ *
+ * When exact sample size is required, we make two additional passed over the RDD to determine
+ * the exact sampling rate that guarantees sample size with high confidence. The first pass
+ * counts the number of items in each stratum (group of items with the same key) in the RDD, and
+ * the second pass uses the counts to determine exact sampling rates.
+ *
+ * The sampling function has a unique seed per partition.
+ */
+ def getPoissonSamplingFunction[K, V](rdd:RDD[(K, V)],
+ fractionByKey: K => Double,
+ exact: Boolean,
+ counts: Option[Map[K, Long]],
+ seed: Long): (Int, Iterator[(K, V)]) => Iterator[(K, V)] = {
+ // TODO implement the streaming version of sampling w/ replacement that doesn't require counts
+ if (exact) {
+ val seqOp = StratifiedSampler.getSeqOp[K,V](true, fractionByKey, counts)
+ val combOp = StratifiedSampler.getCombOp[K]()
+ val zeroU = new Result[K](Map[K, Stratum](), seed = seed)
+ val finalResult = aggregateWithContext(zeroU)(rdd, seqOp, combOp).resultMap
+ val thresholdByKey = StratifiedSampler.computeThresholdByKey(finalResult, fractionByKey)
+ (idx: Int, iter: Iterator[(K, V)]) => {
+ val random = new RandomDataGenerator()
+ random.reSeed(seed + idx)
+ iter.flatMap { t =>
+ val q1 = finalResult.get(t._1).get.q1.getOrElse(0.0)
+ val q2 = finalResult.get(t._1).get.q2.getOrElse(0.0)
+ val x1 = if (q1 == 0) 0L else random.nextPoisson(q1)
+ val x2 = random.nextPoisson(q2).toInt
+ val x = x1 + (0 until x2).count(i => random.nextUniform(0.0, 1.0) < thresholdByKey(t._1))
+ if (x > 0) {
+ Iterator.fill(x.toInt)(t)
+ } else {
+ Iterator.empty
+ }
+ }
+ }
+ } else {
+ (idx: Int, iter: Iterator[(K, V)]) => {
+ val random = new RandomDataGenerator()
+ random.reSeed(seed + idx)
+ iter.flatMap { t =>
+ val count = random.nextPoisson(fractionByKey(t._1)).toInt
+ if (count > 0) {
+ Iterator.fill(count)(t)
+ } else {
+ Iterator.empty
+ }
+ }
+ }
+ }
+ }
+}
+
+/**
+ * Object used by seqOp to keep track of the number of items accepted and items waitlisted per
+ * stratum, as well as the bounds for accepting and waitlisting items.
+ */
+private[random] class Stratum(var numItems: Long = 0L, var numAccepted: Long = 0L)
+ extends Serializable {
+
+ var waitList: ArrayBuffer[Double] = new ArrayBuffer[Double]
+ var q1: Option[Double] = None // upper bound for accepting item instantly
+ var q2: Option[Double] = None // upper bound for adding item to waitlist
+
+ def incrNumItems(by: Long = 1L) = numItems += by
+
+ def incrNumAccepted(by: Long = 1L) = numAccepted += by
+
+ def addToWaitList(elem: Double) = waitList += elem
+
+ def addToWaitList(elems: ArrayBuffer[Double]) = waitList ++= elems
+}
+
+/**
+ * Object used by seqOp and combOp to keep track of the sampling statistics for all strata.
+ *
+ * When used by seqOp for each partition, we also keep track of the partition ID in this object
+ * to make sure a single random number generator with a unique seed is used for each partition.
+ */
+private[random] class Result[K](var resultMap: Map[K, Stratum],
--- End diff --
resultMap can be a value, of type mutable Map.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1025#issuecomment-45553454
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15579/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by falaki <gi...@git.apache.org>.
Github user falaki commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r13630674
--- Diff: core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala ---
@@ -156,6 +161,182 @@ class PairRDDFunctions[K, V](self: RDD[(K, V)])
}
/**
+ * Return a subset of this RDD sampled by key (via stratified sampling).
+ * We guarantee a sample size = math.ceil(fraction * S_i), where S_i is the size of the ith
+ * stratum.
+ *
+ * @param withReplacement whether to sample with or without replacement
+ * @param fraction sampling rate
+ * @param seed seed for the random number generator
+ * @return RDD containing the sampled subset
+ */
+ def sampleByKey(withReplacement: Boolean,
+ fraction: Double,
+ seed: Long = Utils.random.nextLong): RDD[(K, V)]= {
+
+ class Stratum(var numItems: Long = 0L, var numAccepted: Long = 0L) extends Serializable {
+ var waitList: ArrayBuffer[Double] = new ArrayBuffer[Double]
+ var q1: Option[Double] = None
+ var q2: Option[Double] = None
--- End diff --
Either pick more clear names (for q1 and q2) or add a line comment for what they represent.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r13578427
--- Diff: core/src/main/scala/org/apache/spark/util/random/SamplingUtils.scala ---
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util.random
+
+import org.apache.commons.math3.distribution.{PoissonDistribution, NormalDistribution}
+
+private[spark] object PoissonBounds {
+
+ val delta = 1e-4 / 3.0
+ val phi = new NormalDistribution().cumulativeProbability(1.0 - delta)
+
+ def getLambda1(s: Double): Double = {
+ var lb = math.max(0.0, s - math.sqrt(s / delta)) // Chebyshev's inequality
+ var ub = s
+ while (lb < ub - 1.0) {
+ val m = (lb + ub) / 2.0
+ val poisson = new PoissonDistribution(m, 1e-15)
+ val y = poisson.inverseCumulativeProbability(1 - delta)
+ if (y > s) ub = m else lb = m
+ }
+ lb
+ }
+
+ def getMinCount(lmbd: Double): Double = {
+ if(lmbd == 0) return 0
+ val poisson = new PoissonDistribution(lmbd, 1e-15)
+ poisson.inverseCumulativeProbability(delta)
+ }
+
+ def getLambda2(s: Double): Double = {
--- End diff --
`getUpperBound`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by dorx <gi...@git.apache.org>.
Github user dorx commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r14907670
--- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
@@ -0,0 +1,311 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util.random
+
+import scala.collection.Map
+import scala.collection.mutable.{ArrayBuffer, HashMap, Map => MMap}
+
+import org.apache.commons.math3.random.RandomDataGenerator
+import org.apache.spark.Logging
+import org.apache.spark.rdd.RDD
+
+/**
+ * Auxiliary functions and data structures for the sampleByKey method in PairRDDFunctions.
+ *
+ * Essentially, when exact sample size is necessary, we make additional passes over the RDD to
+ * compute the exact threshold value to use for each stratum to guarantee exact sample size with
+ * high probability. This is achieved by maintaining a waitlist of size O(log(s)), where s is the
+ * desired sample size for each stratum.
+ *
+ * Like in simple random sampling, we generate a random value for each item from the
+ * uniform distribution [0.0, 1.0]. All items with values <= min(values of items in the waitlist)
+ * are accepted into the sample instantly. The threshold for instant accept is designed so that
+ * s - numAccepted = O(log(s)), where s is again the desired sample size. Thus, by maintaining a
+ * waitlist size = O(log(s)), we will be able to create a sample of the exact size s by adding
+ * a portion of the waitlist to the set of items that are instantly accepted. The exact threshold
+ * is computed by sorting the values in the waitlist and picking the value at (s - numAccepted).
+ *
+ * Note that since we use the same seed for the RNG when computing the thresholds and the actual
+ * sample, our computed thresholds are guaranteed to produce the desired sample size.
+ *
+ * For more theoretical background on the sampling techniques used here, please refer to
+ * http://jmlr.org/proceedings/papers/v28/meng13a.html
+ */
+
+private[spark] object StratifiedSampler extends Logging {
+
+ /**
+ * Count the number of items instantly accepted and generate the waitlist for each stratum.
+ *
+ * This is only invoked when exact sample size is required.
+ */
+ def getCounts[K, V](rdd: RDD[(K, V)],
+ withReplacement: Boolean,
+ fractions: Map[K, Double],
+ counts: Option[Map[K, Long]],
+ seed: Long): MMap[K, Stratum] = {
+ val combOp = getCombOp[K]
+ val mappedPartitionRDD = rdd.mapPartitionsWithIndex({ case (partition, iter) =>
+ val zeroU: MMap[K, Stratum] = new HashMap[K, Stratum]()
+ val rng = new RandomDataGenerator()
+ rng.reSeed(seed + partition)
+ val seqOp = getSeqOp(withReplacement, fractions, rng, counts)
+ Iterator(iter.aggregate(zeroU)(seqOp, combOp))
+ }, preservesPartitioning=true)
+ mappedPartitionRDD.reduce(combOp)
+ }
+
+ /**
+ * Returns the function used by aggregate to collect sampling statistics for each partition.
+ */
+ def getSeqOp[K, V](withReplacement: Boolean,
+ fractions: Map[K, Double],
+ rng: RandomDataGenerator,
+ counts: Option[Map[K, Long]]): (MMap[K, Stratum], (K, V)) => MMap[K, Stratum] = {
+ val delta = 5e-5
+ (result: MMap[K, Stratum], item: (K, V)) => {
+ val key = item._1
+ val fraction = fractions(key)
+ if (!result.contains(key)) {
+ result += (key -> new Stratum())
+ }
+ val stratum = result(key)
+
+ if (withReplacement) {
+ // compute acceptBound and waitListBound only if they haven't been computed already
+ // since they don't change from iteration to iteration.
+ // TODO change this to the streaming version
+ if (stratum.areBoundsEmpty) {
+ val n = counts.get(key)
+ val sampleSize = math.ceil(n * fraction).toLong
+ val lmbd1 = PoissonBounds.getLowerBound(sampleSize)
+ val minCount = PoissonBounds.getMinCount(lmbd1)
+ val lmbd2 = if (lmbd1 == 0) {
+ PoissonBounds.getUpperBound(sampleSize)
+ } else {
+ PoissonBounds.getUpperBound(sampleSize - minCount)
+ }
+ stratum.acceptBound = lmbd1 / n
+ stratum.waitListBound = lmbd2 / n
+ }
+ val acceptBound = stratum.acceptBound
+ val copiesAccepted = if (acceptBound == 0.0) 0L else rng.nextPoisson(acceptBound)
+ if (copiesAccepted > 0) {
+ stratum.incrNumAccepted(copiesAccepted)
+ }
+ val copiesWaitlisted = rng.nextPoisson(stratum.waitListBound).toInt
+ if (copiesWaitlisted > 0) {
+ stratum.addToWaitList(ArrayBuffer.fill(copiesWaitlisted)(rng.nextUniform(0.0, 1.0)))
+ }
+ } else {
+ // We use the streaming version of the algorithm for sampling without replacement to avoid
+ // using an extra pass over the RDD for computing the count.
+ // Hence, acceptBound and waitListBound change on every iteration.
+ val gamma1 = - math.log(delta) / stratum.numItems
+ val gamma2 = (2.0 / 3.0) * gamma1
+ stratum.acceptBound = math.max(0,
+ fraction + gamma2 - math.sqrt(gamma2 * gamma2 + 3 * gamma2 * fraction))
+ stratum.waitListBound = math.min(1,
+ fraction + gamma1 + math.sqrt(gamma1 * gamma1 + 2 * gamma1 * fraction))
+
+ val x = rng.nextUniform(0.0, 1.0)
+ if (x < stratum.acceptBound) {
+ stratum.incrNumAccepted()
+ } else if (x < stratum.waitListBound) {
+ stratum.addToWaitList(x)
+ }
+ }
+ stratum.incrNumItems()
+ result
+ }
+ }
+
+ /**
+ * Returns the function used combine results returned by seqOp from different partitions.
+ */
+ def getCombOp[K]: (MMap[K, Stratum], MMap[K, Stratum]) => MMap[K, Stratum] = {
+ (result1: MMap[K, Stratum], result2: MMap[K, Stratum]) => {
+ // take union of both key sets in case one partition doesn't contain all keys
+ for (key <- result1.keySet.union(result2.keySet)) {
+ // Use result2 to keep the combined result since r1 is usual empty
+ val entry1 = result1.get(key)
+ if (result2.contains(key)) {
+ result2(key).merge(entry1)
+ } else {
+ if (entry1.isDefined) {
+ result2 += (key -> entry1.get)
+ }
+ }
+ }
+ result2
+ }
+ }
+
+ /**
+ * Given the result returned by getCounts, determine the threshold for accepting items to
+ * generate exact sample size.
+ *
+ * To do so, we compute sampleSize = math.ceil(size * samplingRate) for each stratum and compare
+ * it to the number of items that were accepted instantly and the number of items in the waitlist
+ * for that stratum. Most of the time, numAccepted <= sampleSize <= (numAccepted + numWaitlisted),
+ * which means we need to sort the elements in the waitlist by their associated values in order
+ * to find the value T s.t. |{elements in the stratum whose associated values <= T}| = sampleSize.
+ * Note that all elements in the waitlist have values >= bound for instant accept, so a T value
+ * in the waitlist range would allow all elements that were instantly accepted on the first pass
+ * to be included in the sample.
+ */
+ def computeThresholdByKey[K](finalResult: Map[K, Stratum],
+ fractions: Map[K, Double]): Map[K, Double] = {
+ val thresholdByKey = new HashMap[K, Double]()
+ for ((key, stratum) <- finalResult) {
+ val sampleSize = math.ceil(stratum.numItems * fractions(key)).toLong
+ if (stratum.numAccepted > sampleSize) {
+ logWarning("Pre-accepted too many")
+ thresholdByKey += (key -> stratum.acceptBound)
+ } else {
+ val numWaitListAccepted = (sampleSize - stratum.numAccepted).toInt
+ if (numWaitListAccepted >= stratum.waitList.size) {
+ logWarning("WaitList too short")
+ thresholdByKey += (key -> stratum.waitListBound)
+ } else {
+ thresholdByKey += (key -> stratum.waitList.sorted.apply(numWaitListAccepted))
+ }
+ }
+ }
+ thresholdByKey
+ }
+
+ /**
+ * Return the per partition sampling function used for sampling without replacement.
+ *
+ * When exact sample size is required, we make an additional pass over the RDD to determine the
+ * exact sampling rate that guarantees sample size with high confidence.
+ *
+ * The sampling function has a unique seed per partition.
+ */
+ def getBernoulliSamplingFunction[K, V](rdd: RDD[(K, V)],
+ fractions: Map[K, Double],
+ exact: Boolean,
+ seed: Long): (Int, Iterator[(K, V)]) => Iterator[(K, V)] = {
+ var samplingRateByKey = fractions
+ if (exact) {
+ // determine threshold for each stratum and resample
+ val finalResult = getCounts(rdd, false, fractions, None, seed)
+ samplingRateByKey = computeThresholdByKey(finalResult, fractions)
+ }
+ (idx: Int, iter: Iterator[(K, V)]) => {
+ val rng = new RandomDataGenerator
+ rng.reSeed(seed + idx)
+ iter.filter(t => rng.nextUniform(0.0, 1.0) < samplingRateByKey(t._1))
+ }
+ }
+
+ /**
+ * Return the per partition sampling function used for sampling with replacement.
+ *
+ * When exact sample size is required, we make two additional passed over the RDD to determine
+ * the exact sampling rate that guarantees sample size with high confidence. The first pass
+ * counts the number of items in each stratum (group of items with the same key) in the RDD, and
+ * the second pass uses the counts to determine exact sampling rates.
+ *
+ * The sampling function has a unique seed per partition.
+ */
+ def getPoissonSamplingFunction[K, V](rdd: RDD[(K, V)],
+ fractions: Map[K, Double],
+ exact: Boolean,
+ counts: Option[Map[K, Long]],
+ seed: Long): (Int, Iterator[(K, V)]) => Iterator[(K, V)] = {
+ // TODO implement the streaming version of sampling w/ replacement that doesn't require counts
+ if (exact) {
+ val finalResult = getCounts(rdd, true, fractions, counts, seed)
+ val thresholdByKey = computeThresholdByKey(finalResult, fractions)
+ (idx: Int, iter: Iterator[(K, V)]) => {
+ val rng = new RandomDataGenerator()
+ rng.reSeed(seed + idx)
+ iter.flatMap { item =>
+ val key = item._1
+ val acceptBound = finalResult(key).acceptBound
+ val copiesAccepted = if (acceptBound == 0) 0L else rng.nextPoisson(acceptBound)
+ val copiesWailisted = rng.nextPoisson(finalResult(key).waitListBound).toInt
+ val copiesInSample = copiesAccepted +
+ (0 until copiesWailisted).count(i => rng.nextUniform(0.0, 1.0) < thresholdByKey(key))
--- End diff --
Where is the key recomputed?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by falaki <gi...@git.apache.org>.
Github user falaki commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r14688338
--- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
@@ -0,0 +1,310 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util.random
+
+import scala.collection.Map
+import scala.collection.mutable.{ArrayBuffer, HashMap, Map => MMap}
+
+import org.apache.commons.math3.random.RandomDataGenerator
+import org.apache.spark.Logging
+import org.apache.spark.rdd.RDD
+
+/**
+ * Auxiliary functions and data structures for the sampleByKey method in PairRDDFunctions.
+ *
+ * Essentially, when exact sample size is necessary, we make additional passes over the RDD to
+ * compute the exact threshold value to use for each stratum to guarantee exact sample size with
+ * high probability. This is achieved by maintaining a waitlist of size O(log(s)), where s is the
+ * desired sample size for each stratum.
+ *
+ * Like in simple random sampling, we generate a random value for each item from the
+ * uniform distribution [0.0, 1.0]. All items with values <= min(values of items in the waitlist)
+ * are accepted into the sample instantly. The threshold for instant accept is designed so that
+ * s - numAccepted = O(log(s)), where s is again the desired sample size. Thus, by maintaining a
+ * waitlist size = O(log(s)), we will be able to create a sample of the exact size s by adding
+ * a portion of the waitlist to the set of items that are instantly accepted. The exact threshold
+ * is computed by sorting the values in the waitlist and picking the value at (s - numAccepted).
+ *
+ * Note that since we use the same seed for the RNG when computing the thresholds and the actual
+ * sample, our computed thresholds are guaranteed to produce the desired sample size.
+ *
+ * For more theoretical background on the sampling techniques used here, please refer to
+ * http://jmlr.org/proceedings/papers/v28/meng13a.html
+ */
+
+private[spark] object StratifiedSampler extends Logging {
+
+ /**
+ * Count the number of items instantly accepted and generate the waitlist for each stratum.
+ *
+ * This is only invoked when exact sample size is required.
+ */
+ def getCounts[K, V](rdd: RDD[(K, V)],
+ withReplacement: Boolean,
+ fractions: Map[K, Double],
+ counts: Option[Map[K, Long]],
+ seed: Long): MMap[K, Stratum] = {
+ val combOp = getCombOp[K]
+ val mappedPartitionRDD = rdd.mapPartitionsWithIndex({ case (partition, iter) =>
+ val zeroU: MMap[K, Stratum] = new HashMap[K, Stratum]()
+ val rng = new RandomDataGenerator()
+ rng.reSeed(seed + partition)
+ val seqOp = getSeqOp(withReplacement, fractions, rng, counts)
+ Iterator(iter.aggregate(zeroU)(seqOp, combOp))
+ }, preservesPartitioning=true)
+ mappedPartitionRDD.reduce(combOp)
+ }
+
+ /**
+ * Returns the function used by aggregate to collect sampling statistics for each partition.
+ */
+ def getSeqOp[K, V](withReplacement: Boolean,
+ fractions: Map[K, Double],
+ rng: RandomDataGenerator,
+ counts: Option[Map[K, Long]]): (MMap[K, Stratum], (K, V)) => MMap[K, Stratum] = {
+ val delta = 5e-5
+ (result: MMap[K, Stratum], item: (K, V)) => {
+ val key = item._1
+ val fraction = fractions(key)
+ if (!result.contains(key)) {
+ result += (key -> new Stratum())
+ }
+ val stratum = result(key)
+
+ if (withReplacement) {
+ // compute acceptBound and waitListBound only if they haven't been computed already
+ // since they don't change from iteration to iteration.
+ // TODO change this to the streaming version
+ if (stratum.areBoundsEmpty) {
+ val n = counts.get(key)
+ val sampleSize = math.ceil(n * fraction).toLong
+ val lmbd1 = PoissonBounds.getLowerBound(sampleSize)
+ val minCount = PoissonBounds.getMinCount(lmbd1)
+ val lmbd2 = if (lmbd1 == 0) {
+ PoissonBounds.getUpperBound(sampleSize)
+ } else {
+ PoissonBounds.getUpperBound(sampleSize - minCount)
+ }
+ stratum.acceptBound = lmbd1 / n
+ stratum.waitListBound = lmbd2 / n
+ }
+ val acceptBound = stratum.acceptBound
+ val copiesAccepted = if (acceptBound == 0.0) 0L else rng.nextPoisson(acceptBound)
+ if (copiesAccepted > 0) {
+ stratum.incrNumAccepted(copiesAccepted)
+ }
+ val copiesWaitlisted = rng.nextPoisson(stratum.waitListBound).toInt
+ if (copiesWaitlisted > 0) {
+ stratum.addToWaitList(ArrayBuffer.fill(copiesWaitlisted)(rng.nextUniform(0.0, 1.0)))
+ }
+ } else {
+ // We use the streaming version of the algorithm for sampling without replacement to avoid
+ // using an extra pass over the RDD for computing the count.
+ // Hence, acceptBound and waitListBound change on every iteration.
+ val g1 = - math.log(delta) / stratum.numItems // gamma1
+ val g2 = (2.0 / 3.0) * g1 // gamma 2
--- End diff --
Minor: I still think it is nicer to just rename ```g1``` to ```gamma1```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by falaki <gi...@git.apache.org>.
Github user falaki commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r14544684
--- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util.random
+
+import scala.collection.{Map, mutable}
+import scala.collection.mutable.ArrayBuffer
+import scala.reflect.ClassTag
+
+import org.apache.commons.math3.random.RandomDataGenerator
+import org.apache.spark.{Logging, SparkContext, TaskContext}
+import org.apache.spark.rdd.RDD
+import org.apache.spark.util.Utils
+import org.apache.spark.util.random.{PoissonBounds => PB}
+
+private[spark] object StratifiedSampler extends Logging {
+
+ /**
+ * A version of {@link #aggregate()} that passes the TaskContext to the function that does
+ * aggregation for each partition. This function avoids creating an extra depth in the RDD
+ * lineage, as opposed to using mapPartitionsWithId, which results in slightly improved run time.
+ */
+ def aggregateWithContext[U: ClassTag, T: ClassTag](zeroValue: U)
+ (rdd: RDD[T],
+ seqOp: ((TaskContext, U), T) => U,
+ combOp: (U, U) => U): U = {
+ val sc: SparkContext = rdd.sparkContext
+ // Clone the zero value since we will also be serializing it as part of tasks
+ var jobResult = Utils.clone(zeroValue, sc.env.closureSerializer.newInstance())
+ // pad seqOp and combOp with taskContext to conform to aggregate's signature in TraversableOnce
+ val paddedSeqOp = (arg1: (TaskContext, U), item: T) => (arg1._1, seqOp(arg1, item))
+ val paddedcombOp = (arg1: (TaskContext, U), arg2: (TaskContext, U)) =>
+ (arg1._1, combOp(arg1._2, arg1._2))
+ val cleanSeqOp = sc.clean(paddedSeqOp)
+ val cleanCombOp = sc.clean(paddedcombOp)
+ val aggregatePartition = (tc: TaskContext, it: Iterator[T]) =>
+ (it.aggregate(tc, zeroValue)(cleanSeqOp, cleanCombOp))._2
+ val mergeResult = (index: Int, taskResult: U) => jobResult = combOp(jobResult, taskResult)
+ sc.runJob(rdd, aggregatePartition, mergeResult)
+ jobResult
+ }
+
+ /**
+ * Returns the function used by aggregate to collect sampling statistics for each partition.
+ */
+ def getSeqOp[K, V](withReplacement: Boolean,
+ fractionByKey: (K => Double),
+ counts: Option[Map[K, Long]]): ((TaskContext, Result[K]),(K, V)) => Result[K] = {
+ val delta = 5e-5
+ (U: (TaskContext, Result[K]), item: (K, V)) => {
+ val result = U._2
+ val tc = U._1
+ val rng = result.getRand(tc.partitionId)
+ val fraction = fractionByKey(item._1)
+ val stratum = result.getEntry(item._1)
+ if (withReplacement) {
+ // compute q1 and q2 only if they haven't been computed already
+ // since they don't change from iteration to iteration.
+ // TODO change this to the streaming version
+ if (stratum.q1.isEmpty || stratum.q2.isEmpty) {
+ val n = counts.get(item._1)
+ val s = math.ceil(n * fraction).toLong
+ val lmbd1 = PB.getLowerBound(s)
+ val minCount = PB.getMinCount(lmbd1)
+ val lmbd2 = if (lmbd1 == 0) PB.getUpperBound(s) else PB.getUpperBound(s - minCount)
+ val q1 = lmbd1 / n
+ val q2 = lmbd2 / n
+ stratum.q1 = Some(q1)
+ stratum.q2 = Some(q2)
+ }
+ val x1 = if (stratum.q1.get == 0) 0L else rng.nextPoisson(stratum.q1.get)
+ if (x1 > 0) {
+ stratum.incrNumAccepted(x1)
+ }
+ val x2 = rng.nextPoisson(stratum.q2.get).toInt
+ if (x2 > 0) {
+ stratum.addToWaitList(ArrayBuffer.fill(x2)(rng.nextUniform(0.0, 1.0)))
+ }
+ } else {
+ // We use the streaming version of the algorithm for sampling without replacement.
+ // Hence, q1 and q2 change on every iteration.
+ val g1 = - math.log(delta) / stratum.numItems
+ val g2 = (2.0 / 3.0) * g1
+ val q1 = math.max(0, fraction + g2 - math.sqrt((g2 * g2 + 3 * g2 * fraction)))
+ val q2 = math.min(1, fraction + g1 + math.sqrt(g1 * g1 + 2 * g1 * fraction))
+
+ val x = rng.nextUniform(0.0, 1.0)
+ if (x < q1) {
+ stratum.incrNumAccepted()
+ } else if (x < q2) {
+ stratum.addToWaitList(x)
+ }
+ stratum.q1 = Some(q1)
+ stratum.q2 = Some(q2)
+ }
+ stratum.incrNumItems()
+ result
+ }
+ }
+
+ /**
+ * Returns the function used by aggregate to combine results from different partitions, as
+ * returned by seqOp.
+ */
+ def getCombOp[K](): (Result[K], Result[K]) => Result[K] = {
+ (r1: Result[K], r2: Result[K]) => {
+ // take union of both key sets in case one partition doesn't contain all keys
+ val keyUnion = r1.resultMap.keys.toSet.union(r2.resultMap.keys.toSet)
+
+ // Use r2 to keep the combined result since r1 is usual empty
+ for (key <- keyUnion) {
+ val entry1 = r1.resultMap.get(key)
+ val entry2 = r2.resultMap.get(key)
+ if (entry2.isEmpty && entry1.isDefined) {
+ r2.resultMap += (key -> entry1.get)
--- End diff --
Since we already have a helper class for Result, we can make these lines simpler by adding some helper methods to Result. For example this could become something like:
```scala
r2.addMapping(key, entry1)
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by falaki <gi...@git.apache.org>.
Github user falaki commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r14544754
--- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util.random
+
+import scala.collection.{Map, mutable}
+import scala.collection.mutable.ArrayBuffer
+import scala.reflect.ClassTag
+
+import org.apache.commons.math3.random.RandomDataGenerator
+import org.apache.spark.{Logging, SparkContext, TaskContext}
+import org.apache.spark.rdd.RDD
+import org.apache.spark.util.Utils
+import org.apache.spark.util.random.{PoissonBounds => PB}
+
+private[spark] object StratifiedSampler extends Logging {
+
+ /**
+ * A version of {@link #aggregate()} that passes the TaskContext to the function that does
+ * aggregation for each partition. This function avoids creating an extra depth in the RDD
+ * lineage, as opposed to using mapPartitionsWithId, which results in slightly improved run time.
+ */
+ def aggregateWithContext[U: ClassTag, T: ClassTag](zeroValue: U)
+ (rdd: RDD[T],
+ seqOp: ((TaskContext, U), T) => U,
+ combOp: (U, U) => U): U = {
+ val sc: SparkContext = rdd.sparkContext
+ // Clone the zero value since we will also be serializing it as part of tasks
+ var jobResult = Utils.clone(zeroValue, sc.env.closureSerializer.newInstance())
+ // pad seqOp and combOp with taskContext to conform to aggregate's signature in TraversableOnce
+ val paddedSeqOp = (arg1: (TaskContext, U), item: T) => (arg1._1, seqOp(arg1, item))
+ val paddedcombOp = (arg1: (TaskContext, U), arg2: (TaskContext, U)) =>
+ (arg1._1, combOp(arg1._2, arg1._2))
+ val cleanSeqOp = sc.clean(paddedSeqOp)
+ val cleanCombOp = sc.clean(paddedcombOp)
+ val aggregatePartition = (tc: TaskContext, it: Iterator[T]) =>
+ (it.aggregate(tc, zeroValue)(cleanSeqOp, cleanCombOp))._2
+ val mergeResult = (index: Int, taskResult: U) => jobResult = combOp(jobResult, taskResult)
+ sc.runJob(rdd, aggregatePartition, mergeResult)
+ jobResult
+ }
+
+ /**
+ * Returns the function used by aggregate to collect sampling statistics for each partition.
+ */
+ def getSeqOp[K, V](withReplacement: Boolean,
+ fractionByKey: (K => Double),
+ counts: Option[Map[K, Long]]): ((TaskContext, Result[K]),(K, V)) => Result[K] = {
+ val delta = 5e-5
+ (U: (TaskContext, Result[K]), item: (K, V)) => {
+ val result = U._2
+ val tc = U._1
+ val rng = result.getRand(tc.partitionId)
+ val fraction = fractionByKey(item._1)
+ val stratum = result.getEntry(item._1)
+ if (withReplacement) {
+ // compute q1 and q2 only if they haven't been computed already
+ // since they don't change from iteration to iteration.
+ // TODO change this to the streaming version
+ if (stratum.q1.isEmpty || stratum.q2.isEmpty) {
+ val n = counts.get(item._1)
+ val s = math.ceil(n * fraction).toLong
+ val lmbd1 = PB.getLowerBound(s)
+ val minCount = PB.getMinCount(lmbd1)
+ val lmbd2 = if (lmbd1 == 0) PB.getUpperBound(s) else PB.getUpperBound(s - minCount)
+ val q1 = lmbd1 / n
+ val q2 = lmbd2 / n
+ stratum.q1 = Some(q1)
+ stratum.q2 = Some(q2)
+ }
+ val x1 = if (stratum.q1.get == 0) 0L else rng.nextPoisson(stratum.q1.get)
+ if (x1 > 0) {
+ stratum.incrNumAccepted(x1)
+ }
+ val x2 = rng.nextPoisson(stratum.q2.get).toInt
+ if (x2 > 0) {
+ stratum.addToWaitList(ArrayBuffer.fill(x2)(rng.nextUniform(0.0, 1.0)))
+ }
+ } else {
+ // We use the streaming version of the algorithm for sampling without replacement.
+ // Hence, q1 and q2 change on every iteration.
+ val g1 = - math.log(delta) / stratum.numItems
+ val g2 = (2.0 / 3.0) * g1
+ val q1 = math.max(0, fraction + g2 - math.sqrt((g2 * g2 + 3 * g2 * fraction)))
+ val q2 = math.min(1, fraction + g1 + math.sqrt(g1 * g1 + 2 * g1 * fraction))
+
+ val x = rng.nextUniform(0.0, 1.0)
+ if (x < q1) {
+ stratum.incrNumAccepted()
+ } else if (x < q2) {
+ stratum.addToWaitList(x)
+ }
+ stratum.q1 = Some(q1)
+ stratum.q2 = Some(q2)
+ }
+ stratum.incrNumItems()
+ result
+ }
+ }
+
+ /**
+ * Returns the function used by aggregate to combine results from different partitions, as
+ * returned by seqOp.
+ */
+ def getCombOp[K](): (Result[K], Result[K]) => Result[K] = {
+ (r1: Result[K], r2: Result[K]) => {
+ // take union of both key sets in case one partition doesn't contain all keys
+ val keyUnion = r1.resultMap.keys.toSet.union(r2.resultMap.keys.toSet)
+
+ // Use r2 to keep the combined result since r1 is usual empty
+ for (key <- keyUnion) {
+ val entry1 = r1.resultMap.get(key)
+ val entry2 = r2.resultMap.get(key)
+ if (entry2.isEmpty && entry1.isDefined) {
+ r2.resultMap += (key -> entry1.get)
+ } else if (entry1.isDefined && entry2.isDefined) {
+ entry2.get.addToWaitList(entry1.get.waitList)
+ entry2.get.incrNumAccepted(entry1.get.numAccepted)
+ entry2.get.incrNumItems(entry1.get.numItems)
+ }
+ }
+ r2
+ }
+ }
+
+ /**
+ * Given the result returned by the aggregate function, we need to determine the threshold used
+ * to accept items to generate the exact sample size.
+ */
+ def computeThresholdByKey[K](finalResult: Map[K, Stratum], fractionByKey: (K => Double)):
+ (K => Double) = {
+ val thresholdByKey = new mutable.HashMap[K, Double]()
+ for ((key, stratum) <- finalResult) {
--- End diff --
Some comments that explain the logic here would be great.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r14694381
--- Diff: core/src/test/scala/org/apache/spark/rdd/PairRDDFunctionsSuite.scala ---
@@ -83,6 +83,125 @@ class PairRDDFunctionsSuite extends FunSuite with SharedSparkContext {
assert(valuesFor2.toList.sorted === List(1))
}
+ test("sampleByKey") {
+ def stratifier (fractionPositive: Double) = {
+ (x: Int) => if (x % 10 < (10 * fractionPositive).toInt) "1" else "0"
+ }
+
+ def checkSize(exact: Boolean,
+ withReplacement: Boolean,
+ expected: Long,
+ actual: Long,
+ p: Double): Boolean = {
+ if (exact) {
+ return expected == actual
+ }
+ val stdev = if (withReplacement) math.sqrt(expected) else math.sqrt(expected * p * (1 - p))
+ // Very forgiving margin since we're dealing with very small sample sizes most of the time
+ math.abs(actual - expected) <= 6 * stdev
+ }
+
+ // Without replacement validation
+ def takeSampleAndValidateBernoulli(stratifiedData: RDD[(String, Int)],
+ exact: Boolean,
+ samplingRate: Double,
+ seed: Long,
+ n: Long) = {
+ val expectedSampleSize = stratifiedData.countByKey().mapValues(count =>
+ math.ceil(count * samplingRate).toInt)
+ val fractions = Map("1" -> samplingRate, "0" -> samplingRate)
+ val sample = stratifiedData.sampleByKey(false, fractions, exact, seed)
+ val sampleCounts = sample.countByKey()
+ val takeSample = sample.collect()
+ assert(sampleCounts.forall {case(k,v) =>
+ checkSize(exact, false, expectedSampleSize(k), v, samplingRate)})
+ assert(takeSample.size === takeSample.toSet.size)
+ assert(takeSample.forall(x => 1 <= x._2 && x._2 <= n), s"elements not in [1, $n]")
+ }
+
+ // With replacement validation
+ def takeSampleAndValidatePoisson(stratifiedData: RDD[(String, Int)],
+ exact: Boolean,
+ samplingRate: Double,
+ seed: Long,
+ n: Long) = {
+ val expectedSampleSize = stratifiedData.countByKey().mapValues(count =>
+ math.ceil(count * samplingRate).toInt)
+ val fractions = Map("1" -> samplingRate, "0" -> samplingRate)
+ val sample = stratifiedData.sampleByKey(true, fractions, exact, seed)
+ val sampleCounts = sample.countByKey()
+ val takeSample = sample.collect()
+ assert(sampleCounts.forall {case(k,v) =>
--- End diff --
same style issus
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by dorx <gi...@git.apache.org>.
Github user dorx commented on the pull request:
https://github.com/apache/spark/pull/1025#issuecomment-48419179
Holding out on updating the docs until the python version is supported.
For the python version, any objections to using _jrdd to invoke the java version of sampleByKey?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r14694287
--- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
@@ -0,0 +1,311 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util.random
+
+import scala.collection.Map
+import scala.collection.mutable.{ArrayBuffer, HashMap, Map => MMap}
+
+import org.apache.commons.math3.random.RandomDataGenerator
+import org.apache.spark.Logging
+import org.apache.spark.rdd.RDD
+
+/**
+ * Auxiliary functions and data structures for the sampleByKey method in PairRDDFunctions.
+ *
+ * Essentially, when exact sample size is necessary, we make additional passes over the RDD to
+ * compute the exact threshold value to use for each stratum to guarantee exact sample size with
+ * high probability. This is achieved by maintaining a waitlist of size O(log(s)), where s is the
+ * desired sample size for each stratum.
+ *
+ * Like in simple random sampling, we generate a random value for each item from the
+ * uniform distribution [0.0, 1.0]. All items with values <= min(values of items in the waitlist)
+ * are accepted into the sample instantly. The threshold for instant accept is designed so that
+ * s - numAccepted = O(log(s)), where s is again the desired sample size. Thus, by maintaining a
+ * waitlist size = O(log(s)), we will be able to create a sample of the exact size s by adding
+ * a portion of the waitlist to the set of items that are instantly accepted. The exact threshold
+ * is computed by sorting the values in the waitlist and picking the value at (s - numAccepted).
+ *
+ * Note that since we use the same seed for the RNG when computing the thresholds and the actual
+ * sample, our computed thresholds are guaranteed to produce the desired sample size.
+ *
+ * For more theoretical background on the sampling techniques used here, please refer to
+ * http://jmlr.org/proceedings/papers/v28/meng13a.html
+ */
+
+private[spark] object StratifiedSampler extends Logging {
+
+ /**
+ * Count the number of items instantly accepted and generate the waitlist for each stratum.
+ *
+ * This is only invoked when exact sample size is required.
+ */
+ def getCounts[K, V](rdd: RDD[(K, V)],
+ withReplacement: Boolean,
+ fractions: Map[K, Double],
+ counts: Option[Map[K, Long]],
+ seed: Long): MMap[K, Stratum] = {
+ val combOp = getCombOp[K]
+ val mappedPartitionRDD = rdd.mapPartitionsWithIndex({ case (partition, iter) =>
+ val zeroU: MMap[K, Stratum] = new HashMap[K, Stratum]()
+ val rng = new RandomDataGenerator()
+ rng.reSeed(seed + partition)
+ val seqOp = getSeqOp(withReplacement, fractions, rng, counts)
+ Iterator(iter.aggregate(zeroU)(seqOp, combOp))
+ }, preservesPartitioning=true)
+ mappedPartitionRDD.reduce(combOp)
+ }
+
+ /**
+ * Returns the function used by aggregate to collect sampling statistics for each partition.
+ */
+ def getSeqOp[K, V](withReplacement: Boolean,
+ fractions: Map[K, Double],
+ rng: RandomDataGenerator,
+ counts: Option[Map[K, Long]]): (MMap[K, Stratum], (K, V)) => MMap[K, Stratum] = {
+ val delta = 5e-5
+ (result: MMap[K, Stratum], item: (K, V)) => {
+ val key = item._1
+ val fraction = fractions(key)
+ if (!result.contains(key)) {
+ result += (key -> new Stratum())
+ }
+ val stratum = result(key)
+
+ if (withReplacement) {
+ // compute acceptBound and waitListBound only if they haven't been computed already
+ // since they don't change from iteration to iteration.
+ // TODO change this to the streaming version
+ if (stratum.areBoundsEmpty) {
+ val n = counts.get(key)
+ val sampleSize = math.ceil(n * fraction).toLong
+ val lmbd1 = PoissonBounds.getLowerBound(sampleSize)
+ val minCount = PoissonBounds.getMinCount(lmbd1)
+ val lmbd2 = if (lmbd1 == 0) {
+ PoissonBounds.getUpperBound(sampleSize)
+ } else {
+ PoissonBounds.getUpperBound(sampleSize - minCount)
+ }
+ stratum.acceptBound = lmbd1 / n
+ stratum.waitListBound = lmbd2 / n
+ }
+ val acceptBound = stratum.acceptBound
+ val copiesAccepted = if (acceptBound == 0.0) 0L else rng.nextPoisson(acceptBound)
+ if (copiesAccepted > 0) {
+ stratum.incrNumAccepted(copiesAccepted)
+ }
+ val copiesWaitlisted = rng.nextPoisson(stratum.waitListBound).toInt
+ if (copiesWaitlisted > 0) {
+ stratum.addToWaitList(ArrayBuffer.fill(copiesWaitlisted)(rng.nextUniform(0.0, 1.0)))
+ }
+ } else {
+ // We use the streaming version of the algorithm for sampling without replacement to avoid
+ // using an extra pass over the RDD for computing the count.
+ // Hence, acceptBound and waitListBound change on every iteration.
+ val gamma1 = - math.log(delta) / stratum.numItems
+ val gamma2 = (2.0 / 3.0) * gamma1
+ stratum.acceptBound = math.max(0,
+ fraction + gamma2 - math.sqrt(gamma2 * gamma2 + 3 * gamma2 * fraction))
+ stratum.waitListBound = math.min(1,
+ fraction + gamma1 + math.sqrt(gamma1 * gamma1 + 2 * gamma1 * fraction))
+
+ val x = rng.nextUniform(0.0, 1.0)
+ if (x < stratum.acceptBound) {
+ stratum.incrNumAccepted()
+ } else if (x < stratum.waitListBound) {
+ stratum.addToWaitList(x)
+ }
+ }
+ stratum.incrNumItems()
+ result
+ }
+ }
+
+ /**
+ * Returns the function used combine results returned by seqOp from different partitions.
+ */
+ def getCombOp[K]: (MMap[K, Stratum], MMap[K, Stratum]) => MMap[K, Stratum] = {
+ (result1: MMap[K, Stratum], result2: MMap[K, Stratum]) => {
+ // take union of both key sets in case one partition doesn't contain all keys
+ for (key <- result1.keySet.union(result2.keySet)) {
+ // Use result2 to keep the combined result since r1 is usual empty
+ val entry1 = result1.get(key)
+ if (result2.contains(key)) {
+ result2(key).merge(entry1)
+ } else {
+ if (entry1.isDefined) {
+ result2 += (key -> entry1.get)
+ }
+ }
+ }
+ result2
+ }
+ }
+
+ /**
+ * Given the result returned by getCounts, determine the threshold for accepting items to
+ * generate exact sample size.
+ *
+ * To do so, we compute sampleSize = math.ceil(size * samplingRate) for each stratum and compare
+ * it to the number of items that were accepted instantly and the number of items in the waitlist
+ * for that stratum. Most of the time, numAccepted <= sampleSize <= (numAccepted + numWaitlisted),
+ * which means we need to sort the elements in the waitlist by their associated values in order
+ * to find the value T s.t. |{elements in the stratum whose associated values <= T}| = sampleSize.
+ * Note that all elements in the waitlist have values >= bound for instant accept, so a T value
+ * in the waitlist range would allow all elements that were instantly accepted on the first pass
+ * to be included in the sample.
+ */
+ def computeThresholdByKey[K](finalResult: Map[K, Stratum],
+ fractions: Map[K, Double]): Map[K, Double] = {
+ val thresholdByKey = new HashMap[K, Double]()
+ for ((key, stratum) <- finalResult) {
+ val sampleSize = math.ceil(stratum.numItems * fractions(key)).toLong
+ if (stratum.numAccepted > sampleSize) {
+ logWarning("Pre-accepted too many")
+ thresholdByKey += (key -> stratum.acceptBound)
+ } else {
+ val numWaitListAccepted = (sampleSize - stratum.numAccepted).toInt
+ if (numWaitListAccepted >= stratum.waitList.size) {
+ logWarning("WaitList too short")
+ thresholdByKey += (key -> stratum.waitListBound)
+ } else {
+ thresholdByKey += (key -> stratum.waitList.sorted.apply(numWaitListAccepted))
+ }
+ }
+ }
+ thresholdByKey
+ }
+
+ /**
+ * Return the per partition sampling function used for sampling without replacement.
+ *
+ * When exact sample size is required, we make an additional pass over the RDD to determine the
+ * exact sampling rate that guarantees sample size with high confidence.
+ *
+ * The sampling function has a unique seed per partition.
+ */
+ def getBernoulliSamplingFunction[K, V](rdd: RDD[(K, V)],
+ fractions: Map[K, Double],
+ exact: Boolean,
+ seed: Long): (Int, Iterator[(K, V)]) => Iterator[(K, V)] = {
+ var samplingRateByKey = fractions
+ if (exact) {
+ // determine threshold for each stratum and resample
+ val finalResult = getCounts(rdd, false, fractions, None, seed)
+ samplingRateByKey = computeThresholdByKey(finalResult, fractions)
+ }
+ (idx: Int, iter: Iterator[(K, V)]) => {
+ val rng = new RandomDataGenerator
+ rng.reSeed(seed + idx)
+ iter.filter(t => rng.nextUniform(0.0, 1.0) < samplingRateByKey(t._1))
--- End diff --
Please add a comment here saying the way we call rng must be the same as in `getCounts`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by falaki <gi...@git.apache.org>.
Github user falaki commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r14622797
--- Diff: core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala ---
@@ -195,6 +193,37 @@ class PairRDDFunctions[K, V](self: RDD[(K, V)])
}
/**
+ * Return a subset of this RDD sampled by key (via stratified sampling).
+ *
+ * If exact set to true, we guarantee, with high probability, a sample size =
+ * math.ceil(fraction * S_i), where S_i is the size of the ith stratum (collection of entries
+ * that share the same key). When sampling without replacement, we need one additional pass over
+ * the RDD to guarantee sample size with a 99.99% confidence; when sampling with replacement, we
+ * need two additional passes.
+ *
+ * @param withReplacement whether to sample with or without replacement
+ * @param fractions map of specific keys to sampling rates
+ * @param seed seed for the random number generator
+ * @param exact whether sample size needs to be exactly math.ceil(fraction * size) per stratum
+ * @return RDD containing the sampled subset
+ */
+ def sampleByKey(withReplacement: Boolean,
+ fractions: Map[K, Double],
+ exact: Boolean = true,
+ seed: Long = Utils.random.nextLong): RDD[(K, V)]= {
+
+ require(fractions.forall {case(k, v) => v >= 0.0}, "Invalid sampling rates.")
+
+ val samplingFunc = if (withReplacement) {
+ val counts = if (exact) Some(this.countByKey()) else None
+ StratifiedSampler.getPoissonSamplingFunction(self, fractions, exact, counts, seed)
--- End diff --
Fix indent.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by dorx <gi...@git.apache.org>.
Github user dorx commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r14906065
--- Diff: pom.xml ---
@@ -257,6 +257,11 @@
<version>1.5</version>
</dependency>
<dependency>
+ <groupId>org.apache.commons</groupId>
+ <artifactId>commons-math3</artifactId>
+ <version>3.3</version>
--- End diff --
The final decision was to remove dependency on commons-math3 and use approximation for Poisson inverseCDF instead. @srowen
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1025#issuecomment-48422320
All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16441/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1025#issuecomment-48386183
Merged build finished.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1025#issuecomment-45545540
Build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1025#issuecomment-48130356
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1025#issuecomment-48414132
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by dorx <gi...@git.apache.org>.
Github user dorx commented on the pull request:
https://github.com/apache/spark/pull/1025#issuecomment-46402513
@srowen Hey Sean, turns out Colt(or breeze for that matter) doesn't have inverseCDF for the distributions, which I need for the implementation. The original thought was to avoid increasing the number of dependencies for Spark, as we need to support multiple build frameworks (and it's a lot of compatibility testing work for the person merging the PR). As it turns out, there isn't really a viable substitute in the current dependencies, which was why I went and got the approval for adding commons-math3 as a dependency.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r14694234
--- Diff: core/src/main/scala/org/apache/spark/api/java/JavaPairRDD.scala ---
@@ -130,6 +130,38 @@ class JavaPairRDD[K, V](val rdd: RDD[(K, V)])
new JavaPairRDD[K, V](rdd.sample(withReplacement, fraction, seed))
/**
+ * Return a subset of this RDD sampled by key (via stratified sampling).
+ */
+ def sampleByKey(withReplacement: Boolean,
+ fractions: JMap[K, Double],
+ exact: Boolean,
+ seed: Long): JavaPairRDD[K, V] =
+ new JavaPairRDD[K, V](rdd.sampleByKey(withReplacement, fractions, exact, seed))
+
+
--- End diff --
remove empty line
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r14907687
--- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
@@ -0,0 +1,311 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util.random
+
+import scala.collection.Map
+import scala.collection.mutable.{ArrayBuffer, HashMap, Map => MMap}
+
+import org.apache.commons.math3.random.RandomDataGenerator
+import org.apache.spark.Logging
+import org.apache.spark.rdd.RDD
+
+/**
+ * Auxiliary functions and data structures for the sampleByKey method in PairRDDFunctions.
+ *
+ * Essentially, when exact sample size is necessary, we make additional passes over the RDD to
+ * compute the exact threshold value to use for each stratum to guarantee exact sample size with
+ * high probability. This is achieved by maintaining a waitlist of size O(log(s)), where s is the
+ * desired sample size for each stratum.
+ *
+ * Like in simple random sampling, we generate a random value for each item from the
+ * uniform distribution [0.0, 1.0]. All items with values <= min(values of items in the waitlist)
+ * are accepted into the sample instantly. The threshold for instant accept is designed so that
+ * s - numAccepted = O(log(s)), where s is again the desired sample size. Thus, by maintaining a
+ * waitlist size = O(log(s)), we will be able to create a sample of the exact size s by adding
+ * a portion of the waitlist to the set of items that are instantly accepted. The exact threshold
+ * is computed by sorting the values in the waitlist and picking the value at (s - numAccepted).
+ *
+ * Note that since we use the same seed for the RNG when computing the thresholds and the actual
+ * sample, our computed thresholds are guaranteed to produce the desired sample size.
+ *
+ * For more theoretical background on the sampling techniques used here, please refer to
+ * http://jmlr.org/proceedings/papers/v28/meng13a.html
+ */
+
+private[spark] object StratifiedSampler extends Logging {
+
+ /**
+ * Count the number of items instantly accepted and generate the waitlist for each stratum.
+ *
+ * This is only invoked when exact sample size is required.
+ */
+ def getCounts[K, V](rdd: RDD[(K, V)],
+ withReplacement: Boolean,
+ fractions: Map[K, Double],
+ counts: Option[Map[K, Long]],
+ seed: Long): MMap[K, Stratum] = {
+ val combOp = getCombOp[K]
+ val mappedPartitionRDD = rdd.mapPartitionsWithIndex({ case (partition, iter) =>
+ val zeroU: MMap[K, Stratum] = new HashMap[K, Stratum]()
+ val rng = new RandomDataGenerator()
+ rng.reSeed(seed + partition)
+ val seqOp = getSeqOp(withReplacement, fractions, rng, counts)
+ Iterator(iter.aggregate(zeroU)(seqOp, combOp))
+ }, preservesPartitioning=true)
+ mappedPartitionRDD.reduce(combOp)
+ }
+
+ /**
+ * Returns the function used by aggregate to collect sampling statistics for each partition.
+ */
+ def getSeqOp[K, V](withReplacement: Boolean,
+ fractions: Map[K, Double],
+ rng: RandomDataGenerator,
+ counts: Option[Map[K, Long]]): (MMap[K, Stratum], (K, V)) => MMap[K, Stratum] = {
+ val delta = 5e-5
+ (result: MMap[K, Stratum], item: (K, V)) => {
+ val key = item._1
+ val fraction = fractions(key)
+ if (!result.contains(key)) {
+ result += (key -> new Stratum())
+ }
+ val stratum = result(key)
+
+ if (withReplacement) {
+ // compute acceptBound and waitListBound only if they haven't been computed already
+ // since they don't change from iteration to iteration.
+ // TODO change this to the streaming version
+ if (stratum.areBoundsEmpty) {
+ val n = counts.get(key)
+ val sampleSize = math.ceil(n * fraction).toLong
+ val lmbd1 = PoissonBounds.getLowerBound(sampleSize)
+ val minCount = PoissonBounds.getMinCount(lmbd1)
+ val lmbd2 = if (lmbd1 == 0) {
+ PoissonBounds.getUpperBound(sampleSize)
+ } else {
+ PoissonBounds.getUpperBound(sampleSize - minCount)
+ }
+ stratum.acceptBound = lmbd1 / n
+ stratum.waitListBound = lmbd2 / n
+ }
+ val acceptBound = stratum.acceptBound
+ val copiesAccepted = if (acceptBound == 0.0) 0L else rng.nextPoisson(acceptBound)
+ if (copiesAccepted > 0) {
+ stratum.incrNumAccepted(copiesAccepted)
+ }
+ val copiesWaitlisted = rng.nextPoisson(stratum.waitListBound).toInt
+ if (copiesWaitlisted > 0) {
+ stratum.addToWaitList(ArrayBuffer.fill(copiesWaitlisted)(rng.nextUniform(0.0, 1.0)))
+ }
+ } else {
+ // We use the streaming version of the algorithm for sampling without replacement to avoid
+ // using an extra pass over the RDD for computing the count.
+ // Hence, acceptBound and waitListBound change on every iteration.
+ val gamma1 = - math.log(delta) / stratum.numItems
+ val gamma2 = (2.0 / 3.0) * gamma1
+ stratum.acceptBound = math.max(0,
+ fraction + gamma2 - math.sqrt(gamma2 * gamma2 + 3 * gamma2 * fraction))
+ stratum.waitListBound = math.min(1,
+ fraction + gamma1 + math.sqrt(gamma1 * gamma1 + 2 * gamma1 * fraction))
+
+ val x = rng.nextUniform(0.0, 1.0)
+ if (x < stratum.acceptBound) {
+ stratum.incrNumAccepted()
+ } else if (x < stratum.waitListBound) {
+ stratum.addToWaitList(x)
+ }
+ }
+ stratum.incrNumItems()
+ result
+ }
+ }
+
+ /**
+ * Returns the function used combine results returned by seqOp from different partitions.
+ */
+ def getCombOp[K]: (MMap[K, Stratum], MMap[K, Stratum]) => MMap[K, Stratum] = {
+ (result1: MMap[K, Stratum], result2: MMap[K, Stratum]) => {
+ // take union of both key sets in case one partition doesn't contain all keys
+ for (key <- result1.keySet.union(result2.keySet)) {
+ // Use result2 to keep the combined result since r1 is usual empty
+ val entry1 = result1.get(key)
+ if (result2.contains(key)) {
+ result2(key).merge(entry1)
+ } else {
+ if (entry1.isDefined) {
+ result2 += (key -> entry1.get)
+ }
+ }
+ }
+ result2
+ }
+ }
+
+ /**
+ * Given the result returned by getCounts, determine the threshold for accepting items to
+ * generate exact sample size.
+ *
+ * To do so, we compute sampleSize = math.ceil(size * samplingRate) for each stratum and compare
+ * it to the number of items that were accepted instantly and the number of items in the waitlist
+ * for that stratum. Most of the time, numAccepted <= sampleSize <= (numAccepted + numWaitlisted),
+ * which means we need to sort the elements in the waitlist by their associated values in order
+ * to find the value T s.t. |{elements in the stratum whose associated values <= T}| = sampleSize.
+ * Note that all elements in the waitlist have values >= bound for instant accept, so a T value
+ * in the waitlist range would allow all elements that were instantly accepted on the first pass
+ * to be included in the sample.
+ */
+ def computeThresholdByKey[K](finalResult: Map[K, Stratum],
+ fractions: Map[K, Double]): Map[K, Double] = {
+ val thresholdByKey = new HashMap[K, Double]()
+ for ((key, stratum) <- finalResult) {
+ val sampleSize = math.ceil(stratum.numItems * fractions(key)).toLong
+ if (stratum.numAccepted > sampleSize) {
+ logWarning("Pre-accepted too many")
+ thresholdByKey += (key -> stratum.acceptBound)
+ } else {
+ val numWaitListAccepted = (sampleSize - stratum.numAccepted).toInt
+ if (numWaitListAccepted >= stratum.waitList.size) {
+ logWarning("WaitList too short")
+ thresholdByKey += (key -> stratum.waitListBound)
+ } else {
+ thresholdByKey += (key -> stratum.waitList.sorted.apply(numWaitListAccepted))
+ }
+ }
+ }
+ thresholdByKey
+ }
+
+ /**
+ * Return the per partition sampling function used for sampling without replacement.
+ *
+ * When exact sample size is required, we make an additional pass over the RDD to determine the
+ * exact sampling rate that guarantees sample size with high confidence.
+ *
+ * The sampling function has a unique seed per partition.
+ */
+ def getBernoulliSamplingFunction[K, V](rdd: RDD[(K, V)],
+ fractions: Map[K, Double],
+ exact: Boolean,
+ seed: Long): (Int, Iterator[(K, V)]) => Iterator[(K, V)] = {
+ var samplingRateByKey = fractions
+ if (exact) {
+ // determine threshold for each stratum and resample
+ val finalResult = getCounts(rdd, false, fractions, None, seed)
+ samplingRateByKey = computeThresholdByKey(finalResult, fractions)
+ }
+ (idx: Int, iter: Iterator[(K, V)]) => {
+ val rng = new RandomDataGenerator
+ rng.reSeed(seed + idx)
+ iter.filter(t => rng.nextUniform(0.0, 1.0) < samplingRateByKey(t._1))
+ }
+ }
+
+ /**
+ * Return the per partition sampling function used for sampling with replacement.
+ *
+ * When exact sample size is required, we make two additional passed over the RDD to determine
+ * the exact sampling rate that guarantees sample size with high confidence. The first pass
+ * counts the number of items in each stratum (group of items with the same key) in the RDD, and
+ * the second pass uses the counts to determine exact sampling rates.
+ *
+ * The sampling function has a unique seed per partition.
+ */
+ def getPoissonSamplingFunction[K, V](rdd: RDD[(K, V)],
+ fractions: Map[K, Double],
+ exact: Boolean,
+ counts: Option[Map[K, Long]],
+ seed: Long): (Int, Iterator[(K, V)]) => Iterator[(K, V)] = {
+ // TODO implement the streaming version of sampling w/ replacement that doesn't require counts
+ if (exact) {
+ val finalResult = getCounts(rdd, true, fractions, counts, seed)
+ val thresholdByKey = computeThresholdByKey(finalResult, fractions)
+ (idx: Int, iter: Iterator[(K, V)]) => {
+ val rng = new RandomDataGenerator()
+ rng.reSeed(seed + idx)
+ iter.flatMap { item =>
+ val key = item._1
+ val acceptBound = finalResult(key).acceptBound
+ val copiesAccepted = if (acceptBound == 0) 0L else rng.nextPoisson(acceptBound)
+ val copiesWailisted = rng.nextPoisson(finalResult(key).waitListBound).toInt
+ val copiesInSample = copiesAccepted +
+ (0 until copiesWailisted).count(i => rng.nextUniform(0.0, 1.0) < thresholdByKey(key))
+ if (copiesInSample > 0) {
+ Iterator.fill(copiesInSample.toInt)(item)
+ } else {
+ Iterator.empty
+ }
+ }
+ }
+ } else {
+ (idx: Int, iter: Iterator[(K, V)]) => {
+ val rng = new RandomDataGenerator()
+ rng.reSeed(seed + idx)
+ iter.flatMap { item =>
+ val count = rng.nextPoisson(fractions(item._1)).toInt
+ if (count > 0) {
+ Iterator.fill(count)(item)
+ } else {
+ Iterator.empty
+ }
+ }
+ }
+ }
+ }
+}
+
+/**
+ * Object used by seqOp to keep track of the number of items accepted and items waitlisted per
+ * stratum, as well as the bounds for accepting and waitlisting items.
+ *
+ * `[random]` here is necessary since it's in the return type signature of seqOp defined above
--- End diff --
I meant the comment is not necessary.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r14694251
--- Diff: core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala ---
@@ -195,6 +193,37 @@ class PairRDDFunctions[K, V](self: RDD[(K, V)])
}
/**
+ * Return a subset of this RDD sampled by key (via stratified sampling).
+ *
+ * If exact set to true, we guarantee, with high probability, a sample size =
+ * math.ceil(fraction * S_i), where S_i is the size of the ith stratum (collection of entries
+ * that share the same key). When sampling without replacement, we need one additional pass over
+ * the RDD to guarantee sample size with a 99.99% confidence; when sampling with replacement, we
+ * need two additional passes.
+ *
+ * @param withReplacement whether to sample with or without replacement
+ * @param fractions map of specific keys to sampling rates
+ * @param seed seed for the random number generator
+ * @param exact whether sample size needs to be exactly math.ceil(fraction * size) per stratum
+ * @return RDD containing the sampled subset
+ */
+ def sampleByKey(withReplacement: Boolean,
+ fractions: Map[K, Double],
+ exact: Boolean = true,
+ seed: Long = Utils.random.nextLong): RDD[(K, V)]= {
+
+ require(fractions.forall {case(k, v) => v >= 0.0}, "Invalid sampling rates.")
+
+ val samplingFunc = if (withReplacement) {
+ val counts = if (exact) Some(this.countByKey()) else None
--- End diff --
`counts` is not used outside `getPoissionSamplingFunction`. Shall we move it into `getPoissonSamplingFunction`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by falaki <gi...@git.apache.org>.
Github user falaki commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r13659499
--- Diff: core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala ---
@@ -156,6 +161,182 @@ class PairRDDFunctions[K, V](self: RDD[(K, V)])
}
/**
+ * Return a subset of this RDD sampled by key (via stratified sampling).
+ * We guarantee a sample size = math.ceil(fraction * S_i), where S_i is the size of the ith
+ * stratum.
+ *
+ * @param withReplacement whether to sample with or without replacement
+ * @param fraction sampling rate
+ * @param seed seed for the random number generator
+ * @return RDD containing the sampled subset
+ */
+ def sampleByKey(withReplacement: Boolean,
+ fraction: Double,
+ seed: Long = Utils.random.nextLong): RDD[(K, V)]= {
+
+ class Stratum(var numItems: Long = 0L, var numAccepted: Long = 0L) extends Serializable {
+ var waitList: ArrayBuffer[Double] = new ArrayBuffer[Double]
+ var q1: Option[Double] = None
+ var q2: Option[Double] = None
+
+ def incrNumItems(by: Long = 1L) = numItems += by
+
+ def incrNumAccepted(by: Long = 1L) = numAccepted += by
+
+ def addToWaitList(elem: Double) = waitList += elem
+
+ def addToWaitList(elems: ArrayBuffer[Double]) = waitList ++= elems
+
+ override def toString() = {
+ "numItems: " + numItems + " numAccepted: " + numAccepted + " q1: " + q1 + " q2: " + q2 +
+ " waitListSize:" + waitList.size
+ }
+ }
+
+ class Result(var resultMap: Map[K, Stratum], var cachedPartitionId: Option[Int] = None)
+ extends Serializable {
+ var rand: RandomDataGenerator = new RandomDataGenerator
+
+ def getEntry(key: K, numItems: Long = 0L): Stratum = {
+ if (resultMap.get(key).isEmpty) {
+ resultMap += (key -> new Stratum(numItems))
+ }
+ resultMap.get(key).get
--- End diff --
```resultMap(key)```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r14907544
--- Diff: core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala ---
@@ -195,6 +193,37 @@ class PairRDDFunctions[K, V](self: RDD[(K, V)])
}
/**
+ * Return a subset of this RDD sampled by key (via stratified sampling).
+ *
+ * If exact set to true, we guarantee, with high probability, a sample size =
--- End diff --
The first sentence should provide a summary and the following paragraph should expand this first sentence if some terms there are not well-defined, e.g., `sample by key` in this case.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1025#issuecomment-48416873
Merged build finished. All automated tests passed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r14694380
--- Diff: core/src/test/scala/org/apache/spark/rdd/PairRDDFunctionsSuite.scala ---
@@ -83,6 +83,125 @@ class PairRDDFunctionsSuite extends FunSuite with SharedSparkContext {
assert(valuesFor2.toList.sorted === List(1))
}
+ test("sampleByKey") {
+ def stratifier (fractionPositive: Double) = {
+ (x: Int) => if (x % 10 < (10 * fractionPositive).toInt) "1" else "0"
+ }
+
+ def checkSize(exact: Boolean,
+ withReplacement: Boolean,
+ expected: Long,
+ actual: Long,
+ p: Double): Boolean = {
+ if (exact) {
+ return expected == actual
+ }
+ val stdev = if (withReplacement) math.sqrt(expected) else math.sqrt(expected * p * (1 - p))
+ // Very forgiving margin since we're dealing with very small sample sizes most of the time
+ math.abs(actual - expected) <= 6 * stdev
+ }
+
+ // Without replacement validation
+ def takeSampleAndValidateBernoulli(stratifiedData: RDD[(String, Int)],
+ exact: Boolean,
+ samplingRate: Double,
+ seed: Long,
+ n: Long) = {
+ val expectedSampleSize = stratifiedData.countByKey().mapValues(count =>
+ math.ceil(count * samplingRate).toInt)
+ val fractions = Map("1" -> samplingRate, "0" -> samplingRate)
+ val sample = stratifiedData.sampleByKey(false, fractions, exact, seed)
+ val sampleCounts = sample.countByKey()
+ val takeSample = sample.collect()
+ assert(sampleCounts.forall {case(k,v) =>
--- End diff --
Put spaces after `{` and `,`.
Better put assert inside foreach so we can output more info if something goes wrong.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r14694253
--- Diff: core/src/main/scala/org/apache/spark/util/random/SamplingUtils.scala ---
@@ -45,11 +50,78 @@ private[spark] object SamplingUtils {
val fraction = sampleSizeLowerBound.toDouble / total
if (withReplacement) {
val numStDev = if (sampleSizeLowerBound < 12) 9 else 5
- fraction + numStDev * math.sqrt(fraction / total)
+ math.max(1e-10, fraction + numStDev * math.sqrt(fraction / total))
} else {
val delta = 1e-4
val gamma = - math.log(delta) / total
- math.min(1, fraction + gamma + math.sqrt(gamma * gamma + 2 * gamma * fraction))
+ math.min(1,
+ math.max(1e-10, fraction + gamma + math.sqrt(gamma * gamma + 2 * gamma * fraction)))
+ }
+ }
+}
+
+/**
+ * Utility functions that help us determine bounds on adjusted sampling rate to guarantee exact
+ * sample sizes with high confidence when sampling with replacement.
+ *
+ * The algorithm for guaranteeing sample size instantly accepts items whose associated value drawn
--- End diff --
I think the documentation is wrong. The bounds are not used as thresholds in sampling with replacement. The lower bound q1 guarantees that if sampling according to Pois(q1/n) the sample size is smaller than s, while the upper bound q2 guarantees that if sampling according to Pois(q2/n) the sample size is greater than s.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1025#issuecomment-48419772
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by falaki <gi...@git.apache.org>.
Github user falaki commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r14688613
--- Diff: core/src/test/scala/org/apache/spark/rdd/PairRDDFunctionsSuite.scala ---
@@ -83,6 +83,120 @@ class PairRDDFunctionsSuite extends FunSuite with SharedSparkContext {
assert(valuesFor2.toList.sorted === List(1))
}
+ test("sampleByKey") {
+ def stratifier (fractionPositive: Double) = {
+ (x: Int) => if (x % 10 < (10 * fractionPositive).toInt) "1" else "0"
+ }
+
+ def checkSize(exact: Boolean, withReplacement: Boolean,
+ expected: Long, actual: Long, p: Double): Boolean = {
+ if (exact) {
+ return expected == actual
+ }
+ val stdev = if (withReplacement) math.sqrt(expected) else math.sqrt(expected * p * (1 - p))
+ // Very forgiving margin since we're dealing with very small sample sizes most of the time
+ math.abs(actual - expected) <= 6 * stdev
+ }
+
+ // Without replacement validation
+ def takeSampleAndValidateBernoulli(stratifiedData: RDD[(String, Int)],
+ exact: Boolean,
+ samplingRate: Double,
+ seed: Long,
+ n: Long) = {
+ val expectedSampleSize = stratifiedData.countByKey().mapValues(count =>
+ math.ceil(count * samplingRate).toInt)
+ val fractions = Map("1" -> samplingRate, "0" -> samplingRate)
+ val sample = stratifiedData.sampleByKey(false, fractions, exact, seed)
+ val sampleCounts = sample.countByKey()
+ val takeSample = sample.collect()
+ assert(sampleCounts.forall({case(k,v) =>
+ checkSize(exact, false, expectedSampleSize(k), v, samplingRate)}))
+ assert(takeSample.size === takeSample.toSet.size)
+ assert(takeSample.forall(x => 1 <= x._2 && x._2 <= n), s"elements not in [1, $n]")
+ }
+
+ // With replacement validation
+ def takeSampleAndValidatePoisson(stratifiedData: RDD[(String, Int)],
+ exact: Boolean,
+ samplingRate: Double,
+ seed: Long,
+ n: Long) = {
+ val expectedSampleSize = stratifiedData.countByKey().mapValues(count =>
+ math.ceil(count * samplingRate).toInt)
+ val fractions = Map("1" -> samplingRate, "0" -> samplingRate)
+ val sample = stratifiedData.sampleByKey(true, fractions, exact, seed)
+ val sampleCounts = sample.countByKey()
+ val takeSample = sample.collect()
+ assert(sampleCounts.forall({case(k,v) =>
--- End diff --
here as well.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by falaki <gi...@git.apache.org>.
Github user falaki commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r13630611
--- Diff: core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala ---
@@ -156,6 +161,182 @@ class PairRDDFunctions[K, V](self: RDD[(K, V)])
}
/**
+ * Return a subset of this RDD sampled by key (via stratified sampling).
+ * We guarantee a sample size = math.ceil(fraction * S_i), where S_i is the size of the ith
+ * stratum.
+ *
+ * @param withReplacement whether to sample with or without replacement
+ * @param fraction sampling rate
+ * @param seed seed for the random number generator
+ * @return RDD containing the sampled subset
+ */
+ def sampleByKey(withReplacement: Boolean,
+ fraction: Double,
+ seed: Long = Utils.random.nextLong): RDD[(K, V)]= {
+
+ class Stratum(var numItems: Long = 0L, var numAccepted: Long = 0L) extends Serializable {
+ var waitList: ArrayBuffer[Double] = new ArrayBuffer[Double]
+ var q1: Option[Double] = None
+ var q2: Option[Double] = None
+
+ def incrNumItems(by: Long = 1L) = numItems += by
+
+ def incrNumAccepted(by: Long = 1L) = numAccepted += by
+
+ def addToWaitList(elem: Double) = waitList += elem
+
+ def addToWaitList(elems: ArrayBuffer[Double]) = waitList ++= elems
+
+ override def toString() = {
+ "numItems: " + numItems + " numAccepted: " + numAccepted + " q1: " + q1 + " q2: " + q2 +
+ " waitListSize:" + waitList.size
+ }
+ }
+
+ class Result(var resultMap: Map[K, Stratum], var cachedPartitionId: Option[Int] = None)
+ extends Serializable {
+ var rand: RandomDataGenerator = new RandomDataGenerator
+
+ def getEntry(key: K, numItems: Long = 0L): Stratum = {
+ if (resultMap.get(key).isEmpty) {
--- End diff --
This would be more readable:
```scala
resultMap.contains(key)
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1025#issuecomment-46376522
Build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1025#issuecomment-46376884
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1025#issuecomment-45553588
Build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r13566493
--- Diff: core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala ---
@@ -46,7 +50,8 @@ import org.apache.spark.Partitioner.defaultPartitioner
import org.apache.spark.SparkContext._
import org.apache.spark.partial.{BoundedDouble, PartialResult}
import org.apache.spark.serializer.Serializer
-import org.apache.spark.util.SerializableHyperLogLog
+import org.apache.spark.util.{Utils, SerializableHyperLogLog}
--- End diff --
SerializableHyperLogLog no longer exists
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1025#issuecomment-45571806
All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15589/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1025#issuecomment-45968207
Merged build finished.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/1025#issuecomment-46402006
@dorx you said above you are trying to use the RNG from Jet? why, if Commons math3 is usable now? I prefer the latter FWIW. I probably misunderstand your comment.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1025#issuecomment-50388808
QA results for PR 1025:<br>- This patch FAILED unit tests.<br>- This patch merges cleanly<br>- This patch adds no public classes<br><br>For more information see test ouptut:<br>https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17300/consoleFull
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1025#issuecomment-46608300
Merged build finished.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1025#issuecomment-46376511
Build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by falaki <gi...@git.apache.org>.
Github user falaki commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r14544785
--- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util.random
+
+import scala.collection.{Map, mutable}
+import scala.collection.mutable.ArrayBuffer
+import scala.reflect.ClassTag
+
+import org.apache.commons.math3.random.RandomDataGenerator
+import org.apache.spark.{Logging, SparkContext, TaskContext}
+import org.apache.spark.rdd.RDD
+import org.apache.spark.util.Utils
+import org.apache.spark.util.random.{PoissonBounds => PB}
+
+private[spark] object StratifiedSampler extends Logging {
+
+ /**
+ * A version of {@link #aggregate()} that passes the TaskContext to the function that does
+ * aggregation for each partition. This function avoids creating an extra depth in the RDD
+ * lineage, as opposed to using mapPartitionsWithId, which results in slightly improved run time.
+ */
+ def aggregateWithContext[U: ClassTag, T: ClassTag](zeroValue: U)
+ (rdd: RDD[T],
+ seqOp: ((TaskContext, U), T) => U,
+ combOp: (U, U) => U): U = {
+ val sc: SparkContext = rdd.sparkContext
+ // Clone the zero value since we will also be serializing it as part of tasks
+ var jobResult = Utils.clone(zeroValue, sc.env.closureSerializer.newInstance())
+ // pad seqOp and combOp with taskContext to conform to aggregate's signature in TraversableOnce
+ val paddedSeqOp = (arg1: (TaskContext, U), item: T) => (arg1._1, seqOp(arg1, item))
+ val paddedcombOp = (arg1: (TaskContext, U), arg2: (TaskContext, U)) =>
+ (arg1._1, combOp(arg1._2, arg1._2))
+ val cleanSeqOp = sc.clean(paddedSeqOp)
+ val cleanCombOp = sc.clean(paddedcombOp)
+ val aggregatePartition = (tc: TaskContext, it: Iterator[T]) =>
+ (it.aggregate(tc, zeroValue)(cleanSeqOp, cleanCombOp))._2
+ val mergeResult = (index: Int, taskResult: U) => jobResult = combOp(jobResult, taskResult)
+ sc.runJob(rdd, aggregatePartition, mergeResult)
+ jobResult
+ }
+
+ /**
+ * Returns the function used by aggregate to collect sampling statistics for each partition.
+ */
+ def getSeqOp[K, V](withReplacement: Boolean,
+ fractionByKey: (K => Double),
+ counts: Option[Map[K, Long]]): ((TaskContext, Result[K]),(K, V)) => Result[K] = {
+ val delta = 5e-5
+ (U: (TaskContext, Result[K]), item: (K, V)) => {
+ val result = U._2
+ val tc = U._1
+ val rng = result.getRand(tc.partitionId)
+ val fraction = fractionByKey(item._1)
+ val stratum = result.getEntry(item._1)
+ if (withReplacement) {
+ // compute q1 and q2 only if they haven't been computed already
+ // since they don't change from iteration to iteration.
+ // TODO change this to the streaming version
+ if (stratum.q1.isEmpty || stratum.q2.isEmpty) {
+ val n = counts.get(item._1)
+ val s = math.ceil(n * fraction).toLong
+ val lmbd1 = PB.getLowerBound(s)
+ val minCount = PB.getMinCount(lmbd1)
+ val lmbd2 = if (lmbd1 == 0) PB.getUpperBound(s) else PB.getUpperBound(s - minCount)
+ val q1 = lmbd1 / n
+ val q2 = lmbd2 / n
+ stratum.q1 = Some(q1)
+ stratum.q2 = Some(q2)
+ }
+ val x1 = if (stratum.q1.get == 0) 0L else rng.nextPoisson(stratum.q1.get)
+ if (x1 > 0) {
+ stratum.incrNumAccepted(x1)
+ }
+ val x2 = rng.nextPoisson(stratum.q2.get).toInt
+ if (x2 > 0) {
+ stratum.addToWaitList(ArrayBuffer.fill(x2)(rng.nextUniform(0.0, 1.0)))
+ }
+ } else {
+ // We use the streaming version of the algorithm for sampling without replacement.
+ // Hence, q1 and q2 change on every iteration.
+ val g1 = - math.log(delta) / stratum.numItems
+ val g2 = (2.0 / 3.0) * g1
+ val q1 = math.max(0, fraction + g2 - math.sqrt((g2 * g2 + 3 * g2 * fraction)))
+ val q2 = math.min(1, fraction + g1 + math.sqrt(g1 * g1 + 2 * g1 * fraction))
+
+ val x = rng.nextUniform(0.0, 1.0)
+ if (x < q1) {
+ stratum.incrNumAccepted()
+ } else if (x < q2) {
+ stratum.addToWaitList(x)
+ }
+ stratum.q1 = Some(q1)
+ stratum.q2 = Some(q2)
+ }
+ stratum.incrNumItems()
+ result
+ }
+ }
+
+ /**
+ * Returns the function used by aggregate to combine results from different partitions, as
+ * returned by seqOp.
+ */
+ def getCombOp[K](): (Result[K], Result[K]) => Result[K] = {
+ (r1: Result[K], r2: Result[K]) => {
+ // take union of both key sets in case one partition doesn't contain all keys
+ val keyUnion = r1.resultMap.keys.toSet.union(r2.resultMap.keys.toSet)
+
+ // Use r2 to keep the combined result since r1 is usual empty
+ for (key <- keyUnion) {
+ val entry1 = r1.resultMap.get(key)
+ val entry2 = r2.resultMap.get(key)
+ if (entry2.isEmpty && entry1.isDefined) {
+ r2.resultMap += (key -> entry1.get)
+ } else if (entry1.isDefined && entry2.isDefined) {
+ entry2.get.addToWaitList(entry1.get.waitList)
+ entry2.get.incrNumAccepted(entry1.get.numAccepted)
+ entry2.get.incrNumItems(entry1.get.numItems)
+ }
+ }
+ r2
+ }
+ }
+
+ /**
+ * Given the result returned by the aggregate function, we need to determine the threshold used
+ * to accept items to generate the exact sample size.
+ */
+ def computeThresholdByKey[K](finalResult: Map[K, Stratum], fractionByKey: (K => Double)):
+ (K => Double) = {
+ val thresholdByKey = new mutable.HashMap[K, Double]()
+ for ((key, stratum) <- finalResult) {
+ val fraction = fractionByKey(key)
+ val s = math.ceil(stratum.numItems * fraction).toLong
+ if (stratum.numAccepted > s) {
+ logWarning("Pre-accepted too many")
+ thresholdByKey += (key -> stratum.q1.get)
+ } else {
+ val numWaitListAccepted = (s - stratum.numAccepted).toInt
+ if (numWaitListAccepted >= stratum.waitList.size) {
+ logWarning("WaitList too short")
+ thresholdByKey += (key -> stratum.q2.get)
+ } else {
+ thresholdByKey += (key -> stratum.waitList.sorted.apply(numWaitListAccepted))
+ }
+ }
+ }
+ thresholdByKey
+ }
+
+ /**
+ * Return the per partition sampling function used for sampling without replacement.
+ *
+ * When exact sample size is required, we make an additional pass over the RDD to determine the
+ * exact sampling rate that guarantees sample size with high confidence.
+ *
+ * The sampling function has a unique seed per partition.
+ */
+ def getBernoulliSamplingFunction[K, V](rdd: RDD[(K, V)],
+ fractionByKey: K => Double,
+ exact: Boolean,
+ seed: Long): (Int, Iterator[(K, V)]) => Iterator[(K, V)] = {
+ var samplingRateByKey = fractionByKey
+ if (exact) {
+ // determine threshold for each stratum and resample
+ val seqOp = StratifiedSampler.getSeqOp[K,V](false, fractionByKey, None)
--- End diff --
In these lines we can remove ```StratifiedSampler```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by falaki <gi...@git.apache.org>.
Github user falaki commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r13630716
--- Diff: core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala ---
@@ -156,6 +161,182 @@ class PairRDDFunctions[K, V](self: RDD[(K, V)])
}
/**
+ * Return a subset of this RDD sampled by key (via stratified sampling).
+ * We guarantee a sample size = math.ceil(fraction * S_i), where S_i is the size of the ith
+ * stratum.
+ *
+ * @param withReplacement whether to sample with or without replacement
+ * @param fraction sampling rate
+ * @param seed seed for the random number generator
+ * @return RDD containing the sampled subset
+ */
+ def sampleByKey(withReplacement: Boolean,
+ fraction: Double,
+ seed: Long = Utils.random.nextLong): RDD[(K, V)]= {
+
+ class Stratum(var numItems: Long = 0L, var numAccepted: Long = 0L) extends Serializable {
+ var waitList: ArrayBuffer[Double] = new ArrayBuffer[Double]
+ var q1: Option[Double] = None
+ var q2: Option[Double] = None
+
+ def incrNumItems(by: Long = 1L) = numItems += by
+
+ def incrNumAccepted(by: Long = 1L) = numAccepted += by
+
+ def addToWaitList(elem: Double) = waitList += elem
+
+ def addToWaitList(elems: ArrayBuffer[Double]) = waitList ++= elems
+
+ override def toString() = {
+ "numItems: " + numItems + " numAccepted: " + numAccepted + " q1: " + q1 + " q2: " + q2 +
+ " waitListSize:" + waitList.size
+ }
+ }
+
+ class Result(var resultMap: Map[K, Stratum], var cachedPartitionId: Option[Int] = None)
+ extends Serializable {
+ var rand: RandomDataGenerator = new RandomDataGenerator
+
+ def getEntry(key: K, numItems: Long = 0L): Stratum = {
+ if (resultMap.get(key).isEmpty) {
+ resultMap += (key -> new Stratum(numItems))
+ }
+ resultMap.get(key).get
+ }
+
+ def getRand(partitionId: Int): RandomDataGenerator = {
+ if (cachedPartitionId.isEmpty || cachedPartitionId.get != partitionId) {
+ cachedPartitionId = Some(partitionId)
+ rand.reSeed(seed + partitionId)
+ }
+ rand
+ }
+ }
+
+ // TODO implement the streaming version of sampling w/ replacement that doesn't require counts
+ // in order to save one pass over the RDD
+ val counts = if (withReplacement) Some(this.countByKey()) else None
+
+ val seqOp = (U: (TaskContext, Result), item: (K, V)) => {
+ val delta = 5e-5
+ val result = U._2
+ val tc = U._1
+ val rng = result.getRand(tc.partitionId)
+ val stratum = result.getEntry(item._1)
+ if (withReplacement) {
+ // compute q1 and q2 only if they haven't been computed already
+ // since they don't change from iteration to iteration.
+ // TODO change this to the streaming version
+ if (stratum.q1.isEmpty || stratum.q2.isEmpty) {
+ val n = counts.get(item._1)
+ val s = math.ceil(n * fraction).toLong
+ val lmbd1 = PB.getLambda1(s)
+ val minCount = PB.getMinCount(lmbd1)
+ val lmbd2 = if (lmbd1 == 0) PB.getLambda2(s) else PB.getLambda2(s - minCount)
+ val q1 = lmbd1 / n
+ val q2 = lmbd2 / n
+ stratum.q1 = Some(q1)
+ stratum.q2 = Some(q2)
+ }
+ val x1 = if (stratum.q1.get == 0) 0L else rng.nextPoisson(stratum.q1.get)
+ if (x1 > 0) {
+ stratum.incrNumAccepted(x1)
+ }
+ val x2 = rng.nextPoisson(stratum.q2.get).toInt
+ if (x2 > 0) {
+ stratum.addToWaitList(ArrayBuffer.fill(x2)(rng.nextUniform(0.0, 1.0)))
+ }
+ } else {
+ val g1 = - math.log(delta) / stratum.numItems
+ val g2 = (2.0 / 3.0) * g1
+ val q1 = math.max(0, fraction + g2 - math.sqrt((g2 * g2 + 3 * g2 * fraction)))
+ val q2 = math.min(1, fraction + g1 + math.sqrt(g1 * g1 + 2 * g1 * fraction))
+
+ val x = rng.nextUniform(0.0, 1.0)
+ if (x < q1) {
+ stratum.incrNumAccepted()
+ } else if ( x < q2) {
+ stratum.addToWaitList(x)
+ }
+ stratum.q1 = Some(q1)
+ stratum.q2 = Some(q2)
+ }
+ stratum.incrNumItems()
+ result
+ }
+
+ val combOp = (r1: Result, r2: Result) => {
+ // take union of both key sets in case one partion doesn't contain all keys
--- End diff --
partition
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r13578424
--- Diff: core/src/main/scala/org/apache/spark/util/random/SamplingUtils.scala ---
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util.random
+
+import org.apache.commons.math3.distribution.{PoissonDistribution, NormalDistribution}
+
+private[spark] object PoissonBounds {
+
+ val delta = 1e-4 / 3.0
+ val phi = new NormalDistribution().cumulativeProbability(1.0 - delta)
+
+ def getLambda1(s: Double): Double = {
--- End diff --
`getLowerBound`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1025#issuecomment-50065418
QA tests have started for PR 1025. This patch merges cleanly. <br>View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17130/consoleFull
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r13566500
--- Diff: core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala ---
@@ -27,8 +27,12 @@ import scala.collection.Map
import scala.collection.mutable
import scala.collection.mutable.ArrayBuffer
import scala.reflect.ClassTag
+import scala.util.control.Breaks._
--- End diff --
remove this
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1025#issuecomment-48419505
Merged build finished.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r14694283
--- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
@@ -0,0 +1,311 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util.random
+
+import scala.collection.Map
+import scala.collection.mutable.{ArrayBuffer, HashMap, Map => MMap}
+
+import org.apache.commons.math3.random.RandomDataGenerator
+import org.apache.spark.Logging
+import org.apache.spark.rdd.RDD
+
+/**
+ * Auxiliary functions and data structures for the sampleByKey method in PairRDDFunctions.
+ *
+ * Essentially, when exact sample size is necessary, we make additional passes over the RDD to
+ * compute the exact threshold value to use for each stratum to guarantee exact sample size with
+ * high probability. This is achieved by maintaining a waitlist of size O(log(s)), where s is the
+ * desired sample size for each stratum.
+ *
+ * Like in simple random sampling, we generate a random value for each item from the
+ * uniform distribution [0.0, 1.0]. All items with values <= min(values of items in the waitlist)
+ * are accepted into the sample instantly. The threshold for instant accept is designed so that
+ * s - numAccepted = O(log(s)), where s is again the desired sample size. Thus, by maintaining a
+ * waitlist size = O(log(s)), we will be able to create a sample of the exact size s by adding
+ * a portion of the waitlist to the set of items that are instantly accepted. The exact threshold
+ * is computed by sorting the values in the waitlist and picking the value at (s - numAccepted).
+ *
+ * Note that since we use the same seed for the RNG when computing the thresholds and the actual
+ * sample, our computed thresholds are guaranteed to produce the desired sample size.
+ *
+ * For more theoretical background on the sampling techniques used here, please refer to
+ * http://jmlr.org/proceedings/papers/v28/meng13a.html
+ */
+
+private[spark] object StratifiedSampler extends Logging {
+
+ /**
+ * Count the number of items instantly accepted and generate the waitlist for each stratum.
+ *
+ * This is only invoked when exact sample size is required.
+ */
+ def getCounts[K, V](rdd: RDD[(K, V)],
+ withReplacement: Boolean,
+ fractions: Map[K, Double],
+ counts: Option[Map[K, Long]],
+ seed: Long): MMap[K, Stratum] = {
+ val combOp = getCombOp[K]
+ val mappedPartitionRDD = rdd.mapPartitionsWithIndex({ case (partition, iter) =>
+ val zeroU: MMap[K, Stratum] = new HashMap[K, Stratum]()
+ val rng = new RandomDataGenerator()
+ rng.reSeed(seed + partition)
+ val seqOp = getSeqOp(withReplacement, fractions, rng, counts)
+ Iterator(iter.aggregate(zeroU)(seqOp, combOp))
+ }, preservesPartitioning=true)
+ mappedPartitionRDD.reduce(combOp)
+ }
+
+ /**
+ * Returns the function used by aggregate to collect sampling statistics for each partition.
+ */
+ def getSeqOp[K, V](withReplacement: Boolean,
+ fractions: Map[K, Double],
+ rng: RandomDataGenerator,
+ counts: Option[Map[K, Long]]): (MMap[K, Stratum], (K, V)) => MMap[K, Stratum] = {
+ val delta = 5e-5
+ (result: MMap[K, Stratum], item: (K, V)) => {
+ val key = item._1
+ val fraction = fractions(key)
+ if (!result.contains(key)) {
+ result += (key -> new Stratum())
+ }
+ val stratum = result(key)
+
+ if (withReplacement) {
+ // compute acceptBound and waitListBound only if they haven't been computed already
+ // since they don't change from iteration to iteration.
+ // TODO change this to the streaming version
+ if (stratum.areBoundsEmpty) {
+ val n = counts.get(key)
+ val sampleSize = math.ceil(n * fraction).toLong
+ val lmbd1 = PoissonBounds.getLowerBound(sampleSize)
+ val minCount = PoissonBounds.getMinCount(lmbd1)
+ val lmbd2 = if (lmbd1 == 0) {
+ PoissonBounds.getUpperBound(sampleSize)
+ } else {
+ PoissonBounds.getUpperBound(sampleSize - minCount)
+ }
+ stratum.acceptBound = lmbd1 / n
+ stratum.waitListBound = lmbd2 / n
+ }
+ val acceptBound = stratum.acceptBound
+ val copiesAccepted = if (acceptBound == 0.0) 0L else rng.nextPoisson(acceptBound)
+ if (copiesAccepted > 0) {
+ stratum.incrNumAccepted(copiesAccepted)
+ }
+ val copiesWaitlisted = rng.nextPoisson(stratum.waitListBound).toInt
+ if (copiesWaitlisted > 0) {
+ stratum.addToWaitList(ArrayBuffer.fill(copiesWaitlisted)(rng.nextUniform(0.0, 1.0)))
+ }
+ } else {
+ // We use the streaming version of the algorithm for sampling without replacement to avoid
+ // using an extra pass over the RDD for computing the count.
+ // Hence, acceptBound and waitListBound change on every iteration.
+ val gamma1 = - math.log(delta) / stratum.numItems
+ val gamma2 = (2.0 / 3.0) * gamma1
+ stratum.acceptBound = math.max(0,
+ fraction + gamma2 - math.sqrt(gamma2 * gamma2 + 3 * gamma2 * fraction))
+ stratum.waitListBound = math.min(1,
+ fraction + gamma1 + math.sqrt(gamma1 * gamma1 + 2 * gamma1 * fraction))
+
+ val x = rng.nextUniform(0.0, 1.0)
+ if (x < stratum.acceptBound) {
+ stratum.incrNumAccepted()
+ } else if (x < stratum.waitListBound) {
+ stratum.addToWaitList(x)
+ }
+ }
+ stratum.incrNumItems()
+ result
+ }
+ }
+
+ /**
+ * Returns the function used combine results returned by seqOp from different partitions.
+ */
+ def getCombOp[K]: (MMap[K, Stratum], MMap[K, Stratum]) => MMap[K, Stratum] = {
+ (result1: MMap[K, Stratum], result2: MMap[K, Stratum]) => {
+ // take union of both key sets in case one partition doesn't contain all keys
+ for (key <- result1.keySet.union(result2.keySet)) {
--- End diff --
foreach is more efficient than for:
`result1.keySet.union(result2.keySet).foreach { key =>`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r14694258
--- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
@@ -0,0 +1,311 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util.random
+
+import scala.collection.Map
+import scala.collection.mutable.{ArrayBuffer, HashMap, Map => MMap}
--- End diff --
Do not import a class under `mutable` if there is one with the same name under `immutable`. Instead, import `scala.collection.mutable` and reference `Map` and `HashMap` as `mutable.Map` and `mutable.HashMap` in the code. Importing `mutable.ArrayBuffer` is fine.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1025#issuecomment-48985401
QA tests have started for PR 1025. This patch merges cleanly. <br>View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16650/consoleFull
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by dorx <gi...@git.apache.org>.
Github user dorx commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r14577949
--- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util.random
+
+import scala.collection.{Map, mutable}
+import scala.collection.mutable.ArrayBuffer
+import scala.reflect.ClassTag
+
+import org.apache.commons.math3.random.RandomDataGenerator
+import org.apache.spark.{Logging, SparkContext, TaskContext}
+import org.apache.spark.rdd.RDD
+import org.apache.spark.util.Utils
+import org.apache.spark.util.random.{PoissonBounds => PB}
+
+private[spark] object StratifiedSampler extends Logging {
+
+ /**
+ * A version of {@link #aggregate()} that passes the TaskContext to the function that does
+ * aggregation for each partition. This function avoids creating an extra depth in the RDD
+ * lineage, as opposed to using mapPartitionsWithId, which results in slightly improved run time.
+ */
+ def aggregateWithContext[U: ClassTag, T: ClassTag](zeroValue: U)
+ (rdd: RDD[T],
+ seqOp: ((TaskContext, U), T) => U,
+ combOp: (U, U) => U): U = {
+ val sc: SparkContext = rdd.sparkContext
+ // Clone the zero value since we will also be serializing it as part of tasks
+ var jobResult = Utils.clone(zeroValue, sc.env.closureSerializer.newInstance())
+ // pad seqOp and combOp with taskContext to conform to aggregate's signature in TraversableOnce
+ val paddedSeqOp = (arg1: (TaskContext, U), item: T) => (arg1._1, seqOp(arg1, item))
+ val paddedcombOp = (arg1: (TaskContext, U), arg2: (TaskContext, U)) =>
+ (arg1._1, combOp(arg1._2, arg1._2))
+ val cleanSeqOp = sc.clean(paddedSeqOp)
+ val cleanCombOp = sc.clean(paddedcombOp)
+ val aggregatePartition = (tc: TaskContext, it: Iterator[T]) =>
+ (it.aggregate(tc, zeroValue)(cleanSeqOp, cleanCombOp))._2
+ val mergeResult = (index: Int, taskResult: U) => jobResult = combOp(jobResult, taskResult)
+ sc.runJob(rdd, aggregatePartition, mergeResult)
+ jobResult
+ }
+
+ /**
+ * Returns the function used by aggregate to collect sampling statistics for each partition.
+ */
+ def getSeqOp[K, V](withReplacement: Boolean,
+ fractionByKey: (K => Double),
+ counts: Option[Map[K, Long]]): ((TaskContext, Result[K]),(K, V)) => Result[K] = {
+ val delta = 5e-5
+ (U: (TaskContext, Result[K]), item: (K, V)) => {
+ val result = U._2
+ val tc = U._1
+ val rng = result.getRand(tc.partitionId)
+ val fraction = fractionByKey(item._1)
+ val stratum = result.getEntry(item._1)
+ if (withReplacement) {
+ // compute q1 and q2 only if they haven't been computed already
+ // since they don't change from iteration to iteration.
+ // TODO change this to the streaming version
+ if (stratum.q1.isEmpty || stratum.q2.isEmpty) {
+ val n = counts.get(item._1)
+ val s = math.ceil(n * fraction).toLong
+ val lmbd1 = PB.getLowerBound(s)
+ val minCount = PB.getMinCount(lmbd1)
+ val lmbd2 = if (lmbd1 == 0) PB.getUpperBound(s) else PB.getUpperBound(s - minCount)
+ val q1 = lmbd1 / n
+ val q2 = lmbd2 / n
+ stratum.q1 = Some(q1)
+ stratum.q2 = Some(q2)
+ }
+ val x1 = if (stratum.q1.get == 0) 0L else rng.nextPoisson(stratum.q1.get)
+ if (x1 > 0) {
+ stratum.incrNumAccepted(x1)
+ }
+ val x2 = rng.nextPoisson(stratum.q2.get).toInt
+ if (x2 > 0) {
+ stratum.addToWaitList(ArrayBuffer.fill(x2)(rng.nextUniform(0.0, 1.0)))
+ }
+ } else {
+ // We use the streaming version of the algorithm for sampling without replacement.
+ // Hence, q1 and q2 change on every iteration.
--- End diff --
keeping g1 and g2 (since they're just constants) but changed q1 to q2 to more descriptive names. The expressions for q1 and q2 contains multiple references to g1 and g2. Longer names will make those two expressions really long.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by dorx <gi...@git.apache.org>.
Github user dorx commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r14907870
--- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
@@ -0,0 +1,311 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util.random
+
+import scala.collection.Map
+import scala.collection.mutable.{ArrayBuffer, HashMap, Map => MMap}
+
+import org.apache.commons.math3.random.RandomDataGenerator
+import org.apache.spark.Logging
+import org.apache.spark.rdd.RDD
+
+/**
+ * Auxiliary functions and data structures for the sampleByKey method in PairRDDFunctions.
+ *
+ * Essentially, when exact sample size is necessary, we make additional passes over the RDD to
+ * compute the exact threshold value to use for each stratum to guarantee exact sample size with
+ * high probability. This is achieved by maintaining a waitlist of size O(log(s)), where s is the
+ * desired sample size for each stratum.
+ *
+ * Like in simple random sampling, we generate a random value for each item from the
+ * uniform distribution [0.0, 1.0]. All items with values <= min(values of items in the waitlist)
+ * are accepted into the sample instantly. The threshold for instant accept is designed so that
+ * s - numAccepted = O(log(s)), where s is again the desired sample size. Thus, by maintaining a
+ * waitlist size = O(log(s)), we will be able to create a sample of the exact size s by adding
+ * a portion of the waitlist to the set of items that are instantly accepted. The exact threshold
+ * is computed by sorting the values in the waitlist and picking the value at (s - numAccepted).
+ *
+ * Note that since we use the same seed for the RNG when computing the thresholds and the actual
+ * sample, our computed thresholds are guaranteed to produce the desired sample size.
+ *
+ * For more theoretical background on the sampling techniques used here, please refer to
+ * http://jmlr.org/proceedings/papers/v28/meng13a.html
+ */
+
+private[spark] object StratifiedSampler extends Logging {
+
+ /**
+ * Count the number of items instantly accepted and generate the waitlist for each stratum.
+ *
+ * This is only invoked when exact sample size is required.
+ */
+ def getCounts[K, V](rdd: RDD[(K, V)],
+ withReplacement: Boolean,
+ fractions: Map[K, Double],
+ counts: Option[Map[K, Long]],
+ seed: Long): MMap[K, Stratum] = {
+ val combOp = getCombOp[K]
+ val mappedPartitionRDD = rdd.mapPartitionsWithIndex({ case (partition, iter) =>
+ val zeroU: MMap[K, Stratum] = new HashMap[K, Stratum]()
+ val rng = new RandomDataGenerator()
+ rng.reSeed(seed + partition)
+ val seqOp = getSeqOp(withReplacement, fractions, rng, counts)
+ Iterator(iter.aggregate(zeroU)(seqOp, combOp))
+ }, preservesPartitioning=true)
+ mappedPartitionRDD.reduce(combOp)
+ }
+
+ /**
+ * Returns the function used by aggregate to collect sampling statistics for each partition.
+ */
+ def getSeqOp[K, V](withReplacement: Boolean,
+ fractions: Map[K, Double],
+ rng: RandomDataGenerator,
+ counts: Option[Map[K, Long]]): (MMap[K, Stratum], (K, V)) => MMap[K, Stratum] = {
+ val delta = 5e-5
+ (result: MMap[K, Stratum], item: (K, V)) => {
+ val key = item._1
+ val fraction = fractions(key)
+ if (!result.contains(key)) {
+ result += (key -> new Stratum())
+ }
+ val stratum = result(key)
+
+ if (withReplacement) {
+ // compute acceptBound and waitListBound only if they haven't been computed already
+ // since they don't change from iteration to iteration.
+ // TODO change this to the streaming version
+ if (stratum.areBoundsEmpty) {
+ val n = counts.get(key)
+ val sampleSize = math.ceil(n * fraction).toLong
+ val lmbd1 = PoissonBounds.getLowerBound(sampleSize)
+ val minCount = PoissonBounds.getMinCount(lmbd1)
+ val lmbd2 = if (lmbd1 == 0) {
+ PoissonBounds.getUpperBound(sampleSize)
+ } else {
+ PoissonBounds.getUpperBound(sampleSize - minCount)
+ }
+ stratum.acceptBound = lmbd1 / n
+ stratum.waitListBound = lmbd2 / n
+ }
+ val acceptBound = stratum.acceptBound
+ val copiesAccepted = if (acceptBound == 0.0) 0L else rng.nextPoisson(acceptBound)
+ if (copiesAccepted > 0) {
+ stratum.incrNumAccepted(copiesAccepted)
+ }
+ val copiesWaitlisted = rng.nextPoisson(stratum.waitListBound).toInt
+ if (copiesWaitlisted > 0) {
+ stratum.addToWaitList(ArrayBuffer.fill(copiesWaitlisted)(rng.nextUniform(0.0, 1.0)))
+ }
+ } else {
+ // We use the streaming version of the algorithm for sampling without replacement to avoid
+ // using an extra pass over the RDD for computing the count.
+ // Hence, acceptBound and waitListBound change on every iteration.
+ val gamma1 = - math.log(delta) / stratum.numItems
+ val gamma2 = (2.0 / 3.0) * gamma1
+ stratum.acceptBound = math.max(0,
+ fraction + gamma2 - math.sqrt(gamma2 * gamma2 + 3 * gamma2 * fraction))
+ stratum.waitListBound = math.min(1,
+ fraction + gamma1 + math.sqrt(gamma1 * gamma1 + 2 * gamma1 * fraction))
+
+ val x = rng.nextUniform(0.0, 1.0)
+ if (x < stratum.acceptBound) {
+ stratum.incrNumAccepted()
+ } else if (x < stratum.waitListBound) {
+ stratum.addToWaitList(x)
+ }
+ }
+ stratum.incrNumItems()
+ result
+ }
+ }
+
+ /**
+ * Returns the function used combine results returned by seqOp from different partitions.
+ */
+ def getCombOp[K]: (MMap[K, Stratum], MMap[K, Stratum]) => MMap[K, Stratum] = {
+ (result1: MMap[K, Stratum], result2: MMap[K, Stratum]) => {
+ // take union of both key sets in case one partition doesn't contain all keys
+ for (key <- result1.keySet.union(result2.keySet)) {
+ // Use result2 to keep the combined result since r1 is usual empty
+ val entry1 = result1.get(key)
+ if (result2.contains(key)) {
+ result2(key).merge(entry1)
+ } else {
+ if (entry1.isDefined) {
+ result2 += (key -> entry1.get)
+ }
+ }
+ }
+ result2
+ }
+ }
+
+ /**
+ * Given the result returned by getCounts, determine the threshold for accepting items to
+ * generate exact sample size.
+ *
+ * To do so, we compute sampleSize = math.ceil(size * samplingRate) for each stratum and compare
+ * it to the number of items that were accepted instantly and the number of items in the waitlist
+ * for that stratum. Most of the time, numAccepted <= sampleSize <= (numAccepted + numWaitlisted),
+ * which means we need to sort the elements in the waitlist by their associated values in order
+ * to find the value T s.t. |{elements in the stratum whose associated values <= T}| = sampleSize.
+ * Note that all elements in the waitlist have values >= bound for instant accept, so a T value
+ * in the waitlist range would allow all elements that were instantly accepted on the first pass
+ * to be included in the sample.
+ */
+ def computeThresholdByKey[K](finalResult: Map[K, Stratum],
+ fractions: Map[K, Double]): Map[K, Double] = {
+ val thresholdByKey = new HashMap[K, Double]()
+ for ((key, stratum) <- finalResult) {
+ val sampleSize = math.ceil(stratum.numItems * fractions(key)).toLong
+ if (stratum.numAccepted > sampleSize) {
+ logWarning("Pre-accepted too many")
+ thresholdByKey += (key -> stratum.acceptBound)
+ } else {
+ val numWaitListAccepted = (sampleSize - stratum.numAccepted).toInt
+ if (numWaitListAccepted >= stratum.waitList.size) {
+ logWarning("WaitList too short")
+ thresholdByKey += (key -> stratum.waitListBound)
+ } else {
+ thresholdByKey += (key -> stratum.waitList.sorted.apply(numWaitListAccepted))
+ }
+ }
+ }
+ thresholdByKey
+ }
+
+ /**
+ * Return the per partition sampling function used for sampling without replacement.
+ *
+ * When exact sample size is required, we make an additional pass over the RDD to determine the
+ * exact sampling rate that guarantees sample size with high confidence.
+ *
+ * The sampling function has a unique seed per partition.
+ */
+ def getBernoulliSamplingFunction[K, V](rdd: RDD[(K, V)],
+ fractions: Map[K, Double],
+ exact: Boolean,
+ seed: Long): (Int, Iterator[(K, V)]) => Iterator[(K, V)] = {
+ var samplingRateByKey = fractions
+ if (exact) {
+ // determine threshold for each stratum and resample
+ val finalResult = getCounts(rdd, false, fractions, None, seed)
+ samplingRateByKey = computeThresholdByKey(finalResult, fractions)
+ }
+ (idx: Int, iter: Iterator[(K, V)]) => {
+ val rng = new RandomDataGenerator
+ rng.reSeed(seed + idx)
+ iter.filter(t => rng.nextUniform(0.0, 1.0) < samplingRateByKey(t._1))
+ }
+ }
+
+ /**
+ * Return the per partition sampling function used for sampling with replacement.
+ *
+ * When exact sample size is required, we make two additional passed over the RDD to determine
+ * the exact sampling rate that guarantees sample size with high confidence. The first pass
+ * counts the number of items in each stratum (group of items with the same key) in the RDD, and
+ * the second pass uses the counts to determine exact sampling rates.
+ *
+ * The sampling function has a unique seed per partition.
+ */
+ def getPoissonSamplingFunction[K, V](rdd: RDD[(K, V)],
+ fractions: Map[K, Double],
+ exact: Boolean,
+ counts: Option[Map[K, Long]],
+ seed: Long): (Int, Iterator[(K, V)]) => Iterator[(K, V)] = {
+ // TODO implement the streaming version of sampling w/ replacement that doesn't require counts
+ if (exact) {
+ val finalResult = getCounts(rdd, true, fractions, counts, seed)
+ val thresholdByKey = computeThresholdByKey(finalResult, fractions)
+ (idx: Int, iter: Iterator[(K, V)]) => {
+ val rng = new RandomDataGenerator()
+ rng.reSeed(seed + idx)
+ iter.flatMap { item =>
+ val key = item._1
+ val acceptBound = finalResult(key).acceptBound
+ val copiesAccepted = if (acceptBound == 0) 0L else rng.nextPoisson(acceptBound)
+ val copiesWailisted = rng.nextPoisson(finalResult(key).waitListBound).toInt
+ val copiesInSample = copiesAccepted +
+ (0 until copiesWailisted).count(i => rng.nextUniform(0.0, 1.0) < thresholdByKey(key))
+ if (copiesInSample > 0) {
+ Iterator.fill(copiesInSample.toInt)(item)
+ } else {
+ Iterator.empty
+ }
+ }
+ }
+ } else {
+ (idx: Int, iter: Iterator[(K, V)]) => {
+ val rng = new RandomDataGenerator()
+ rng.reSeed(seed + idx)
+ iter.flatMap { item =>
+ val count = rng.nextPoisson(fractions(item._1)).toInt
+ if (count > 0) {
+ Iterator.fill(count)(item)
+ } else {
+ Iterator.empty
+ }
+ }
+ }
+ }
+ }
+}
+
+/**
+ * Object used by seqOp to keep track of the number of items accepted and items waitlisted per
+ * stratum, as well as the bounds for accepting and waitlisting items.
+ *
+ * `[random]` here is necessary since it's in the return type signature of seqOp defined above
--- End diff --
I don't know. I can see people down the road wanting to remove it thinking it's superfluous and break compile by accident.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r14694296
--- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
@@ -0,0 +1,311 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util.random
+
+import scala.collection.Map
+import scala.collection.mutable.{ArrayBuffer, HashMap, Map => MMap}
+
+import org.apache.commons.math3.random.RandomDataGenerator
+import org.apache.spark.Logging
+import org.apache.spark.rdd.RDD
+
+/**
+ * Auxiliary functions and data structures for the sampleByKey method in PairRDDFunctions.
+ *
+ * Essentially, when exact sample size is necessary, we make additional passes over the RDD to
+ * compute the exact threshold value to use for each stratum to guarantee exact sample size with
+ * high probability. This is achieved by maintaining a waitlist of size O(log(s)), where s is the
+ * desired sample size for each stratum.
+ *
+ * Like in simple random sampling, we generate a random value for each item from the
+ * uniform distribution [0.0, 1.0]. All items with values <= min(values of items in the waitlist)
+ * are accepted into the sample instantly. The threshold for instant accept is designed so that
+ * s - numAccepted = O(log(s)), where s is again the desired sample size. Thus, by maintaining a
+ * waitlist size = O(log(s)), we will be able to create a sample of the exact size s by adding
+ * a portion of the waitlist to the set of items that are instantly accepted. The exact threshold
+ * is computed by sorting the values in the waitlist and picking the value at (s - numAccepted).
+ *
+ * Note that since we use the same seed for the RNG when computing the thresholds and the actual
+ * sample, our computed thresholds are guaranteed to produce the desired sample size.
+ *
+ * For more theoretical background on the sampling techniques used here, please refer to
+ * http://jmlr.org/proceedings/papers/v28/meng13a.html
+ */
+
+private[spark] object StratifiedSampler extends Logging {
+
+ /**
+ * Count the number of items instantly accepted and generate the waitlist for each stratum.
+ *
+ * This is only invoked when exact sample size is required.
+ */
+ def getCounts[K, V](rdd: RDD[(K, V)],
+ withReplacement: Boolean,
+ fractions: Map[K, Double],
+ counts: Option[Map[K, Long]],
+ seed: Long): MMap[K, Stratum] = {
+ val combOp = getCombOp[K]
+ val mappedPartitionRDD = rdd.mapPartitionsWithIndex({ case (partition, iter) =>
+ val zeroU: MMap[K, Stratum] = new HashMap[K, Stratum]()
+ val rng = new RandomDataGenerator()
+ rng.reSeed(seed + partition)
+ val seqOp = getSeqOp(withReplacement, fractions, rng, counts)
+ Iterator(iter.aggregate(zeroU)(seqOp, combOp))
+ }, preservesPartitioning=true)
+ mappedPartitionRDD.reduce(combOp)
+ }
+
+ /**
+ * Returns the function used by aggregate to collect sampling statistics for each partition.
+ */
+ def getSeqOp[K, V](withReplacement: Boolean,
+ fractions: Map[K, Double],
+ rng: RandomDataGenerator,
+ counts: Option[Map[K, Long]]): (MMap[K, Stratum], (K, V)) => MMap[K, Stratum] = {
+ val delta = 5e-5
+ (result: MMap[K, Stratum], item: (K, V)) => {
+ val key = item._1
+ val fraction = fractions(key)
+ if (!result.contains(key)) {
+ result += (key -> new Stratum())
+ }
+ val stratum = result(key)
+
+ if (withReplacement) {
+ // compute acceptBound and waitListBound only if they haven't been computed already
+ // since they don't change from iteration to iteration.
+ // TODO change this to the streaming version
+ if (stratum.areBoundsEmpty) {
+ val n = counts.get(key)
+ val sampleSize = math.ceil(n * fraction).toLong
+ val lmbd1 = PoissonBounds.getLowerBound(sampleSize)
+ val minCount = PoissonBounds.getMinCount(lmbd1)
+ val lmbd2 = if (lmbd1 == 0) {
+ PoissonBounds.getUpperBound(sampleSize)
+ } else {
+ PoissonBounds.getUpperBound(sampleSize - minCount)
+ }
+ stratum.acceptBound = lmbd1 / n
+ stratum.waitListBound = lmbd2 / n
+ }
+ val acceptBound = stratum.acceptBound
+ val copiesAccepted = if (acceptBound == 0.0) 0L else rng.nextPoisson(acceptBound)
+ if (copiesAccepted > 0) {
+ stratum.incrNumAccepted(copiesAccepted)
+ }
+ val copiesWaitlisted = rng.nextPoisson(stratum.waitListBound).toInt
+ if (copiesWaitlisted > 0) {
+ stratum.addToWaitList(ArrayBuffer.fill(copiesWaitlisted)(rng.nextUniform(0.0, 1.0)))
+ }
+ } else {
+ // We use the streaming version of the algorithm for sampling without replacement to avoid
+ // using an extra pass over the RDD for computing the count.
+ // Hence, acceptBound and waitListBound change on every iteration.
+ val gamma1 = - math.log(delta) / stratum.numItems
+ val gamma2 = (2.0 / 3.0) * gamma1
+ stratum.acceptBound = math.max(0,
+ fraction + gamma2 - math.sqrt(gamma2 * gamma2 + 3 * gamma2 * fraction))
+ stratum.waitListBound = math.min(1,
+ fraction + gamma1 + math.sqrt(gamma1 * gamma1 + 2 * gamma1 * fraction))
+
+ val x = rng.nextUniform(0.0, 1.0)
+ if (x < stratum.acceptBound) {
+ stratum.incrNumAccepted()
+ } else if (x < stratum.waitListBound) {
+ stratum.addToWaitList(x)
+ }
+ }
+ stratum.incrNumItems()
+ result
+ }
+ }
+
+ /**
+ * Returns the function used combine results returned by seqOp from different partitions.
+ */
+ def getCombOp[K]: (MMap[K, Stratum], MMap[K, Stratum]) => MMap[K, Stratum] = {
+ (result1: MMap[K, Stratum], result2: MMap[K, Stratum]) => {
+ // take union of both key sets in case one partition doesn't contain all keys
+ for (key <- result1.keySet.union(result2.keySet)) {
+ // Use result2 to keep the combined result since r1 is usual empty
+ val entry1 = result1.get(key)
+ if (result2.contains(key)) {
+ result2(key).merge(entry1)
+ } else {
+ if (entry1.isDefined) {
+ result2 += (key -> entry1.get)
+ }
+ }
+ }
+ result2
+ }
+ }
+
+ /**
+ * Given the result returned by getCounts, determine the threshold for accepting items to
+ * generate exact sample size.
+ *
+ * To do so, we compute sampleSize = math.ceil(size * samplingRate) for each stratum and compare
+ * it to the number of items that were accepted instantly and the number of items in the waitlist
+ * for that stratum. Most of the time, numAccepted <= sampleSize <= (numAccepted + numWaitlisted),
+ * which means we need to sort the elements in the waitlist by their associated values in order
+ * to find the value T s.t. |{elements in the stratum whose associated values <= T}| = sampleSize.
+ * Note that all elements in the waitlist have values >= bound for instant accept, so a T value
+ * in the waitlist range would allow all elements that were instantly accepted on the first pass
+ * to be included in the sample.
+ */
+ def computeThresholdByKey[K](finalResult: Map[K, Stratum],
+ fractions: Map[K, Double]): Map[K, Double] = {
+ val thresholdByKey = new HashMap[K, Double]()
+ for ((key, stratum) <- finalResult) {
+ val sampleSize = math.ceil(stratum.numItems * fractions(key)).toLong
+ if (stratum.numAccepted > sampleSize) {
+ logWarning("Pre-accepted too many")
+ thresholdByKey += (key -> stratum.acceptBound)
+ } else {
+ val numWaitListAccepted = (sampleSize - stratum.numAccepted).toInt
+ if (numWaitListAccepted >= stratum.waitList.size) {
+ logWarning("WaitList too short")
+ thresholdByKey += (key -> stratum.waitListBound)
+ } else {
+ thresholdByKey += (key -> stratum.waitList.sorted.apply(numWaitListAccepted))
+ }
+ }
+ }
+ thresholdByKey
+ }
+
+ /**
+ * Return the per partition sampling function used for sampling without replacement.
+ *
+ * When exact sample size is required, we make an additional pass over the RDD to determine the
+ * exact sampling rate that guarantees sample size with high confidence.
+ *
+ * The sampling function has a unique seed per partition.
+ */
+ def getBernoulliSamplingFunction[K, V](rdd: RDD[(K, V)],
+ fractions: Map[K, Double],
+ exact: Boolean,
+ seed: Long): (Int, Iterator[(K, V)]) => Iterator[(K, V)] = {
+ var samplingRateByKey = fractions
+ if (exact) {
+ // determine threshold for each stratum and resample
+ val finalResult = getCounts(rdd, false, fractions, None, seed)
+ samplingRateByKey = computeThresholdByKey(finalResult, fractions)
+ }
+ (idx: Int, iter: Iterator[(K, V)]) => {
+ val rng = new RandomDataGenerator
+ rng.reSeed(seed + idx)
+ iter.filter(t => rng.nextUniform(0.0, 1.0) < samplingRateByKey(t._1))
+ }
+ }
+
+ /**
+ * Return the per partition sampling function used for sampling with replacement.
+ *
+ * When exact sample size is required, we make two additional passed over the RDD to determine
+ * the exact sampling rate that guarantees sample size with high confidence. The first pass
+ * counts the number of items in each stratum (group of items with the same key) in the RDD, and
+ * the second pass uses the counts to determine exact sampling rates.
+ *
+ * The sampling function has a unique seed per partition.
+ */
+ def getPoissonSamplingFunction[K, V](rdd: RDD[(K, V)],
+ fractions: Map[K, Double],
+ exact: Boolean,
+ counts: Option[Map[K, Long]],
+ seed: Long): (Int, Iterator[(K, V)]) => Iterator[(K, V)] = {
+ // TODO implement the streaming version of sampling w/ replacement that doesn't require counts
+ if (exact) {
+ val finalResult = getCounts(rdd, true, fractions, counts, seed)
+ val thresholdByKey = computeThresholdByKey(finalResult, fractions)
+ (idx: Int, iter: Iterator[(K, V)]) => {
+ val rng = new RandomDataGenerator()
+ rng.reSeed(seed + idx)
+ iter.flatMap { item =>
+ val key = item._1
+ val acceptBound = finalResult(key).acceptBound
+ val copiesAccepted = if (acceptBound == 0) 0L else rng.nextPoisson(acceptBound)
+ val copiesWailisted = rng.nextPoisson(finalResult(key).waitListBound).toInt
+ val copiesInSample = copiesAccepted +
+ (0 until copiesWailisted).count(i => rng.nextUniform(0.0, 1.0) < thresholdByKey(key))
+ if (copiesInSample > 0) {
+ Iterator.fill(copiesInSample.toInt)(item)
+ } else {
+ Iterator.empty
+ }
+ }
+ }
+ } else {
+ (idx: Int, iter: Iterator[(K, V)]) => {
+ val rng = new RandomDataGenerator()
+ rng.reSeed(seed + idx)
+ iter.flatMap { item =>
+ val count = rng.nextPoisson(fractions(item._1)).toInt
+ if (count > 0) {
+ Iterator.fill(count)(item)
+ } else {
+ Iterator.empty
+ }
+ }
+ }
+ }
+ }
+}
+
+/**
+ * Object used by seqOp to keep track of the number of items accepted and items waitlisted per
+ * stratum, as well as the bounds for accepting and waitlisting items.
+ *
+ * `[random]` here is necessary since it's in the return type signature of seqOp defined above
+ */
+private[random] class Stratum(var numItems: Long = 0L, var numAccepted: Long = 0L)
--- End diff --
This is not a stratum but something like an `AcceptanceSummary` or `AcceptanceResult`.
The implementation could be simplified:
1) replace `private val _waitlist` and `def waitlist` by `val waitlist`
2) define `var acceptBound` and initialize it to `Double.NaN` to avoid possibly boxing/unboxing. Same for `waitlistBound`.
3) `this.incrNumItems(num)` can be replaced by `this.numItems += num`, which should be concise and sufficient. Same for `incrNumAccepted` and `addToWaitList`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by falaki <gi...@git.apache.org>.
Github user falaki commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r14688121
--- Diff: core/src/main/scala/org/apache/spark/util/random/SamplingUtils.scala ---
@@ -45,11 +50,75 @@ private[spark] object SamplingUtils {
val fraction = sampleSizeLowerBound.toDouble / total
if (withReplacement) {
val numStDev = if (sampleSizeLowerBound < 12) 9 else 5
- fraction + numStDev * math.sqrt(fraction / total)
+ math.max(1e-10, fraction + numStDev * math.sqrt(fraction / total))
} else {
val delta = 1e-4
val gamma = - math.log(delta) / total
- math.min(1, fraction + gamma + math.sqrt(gamma * gamma + 2 * gamma * fraction))
+ math.min(1,
+ math.max(1e-10, fraction + gamma + math.sqrt(gamma * gamma + 2 * gamma * fraction)))
+ }
+ }
+}
+
+/**
+ * Utility functions that help us determine bounds on adjusted sampling rate to guarantee exact
+ * sample sizes with high confidence when sampling with replacement.
+ *
+ * The algorithm for guaranteeing sample size instantly accepts items whose associated value drawn
+ * from Pois(s) is less than the lower bound and puts items whose value is between the lower and
+ * upper bound in a waitlist. The final sample is consisted of all items accepted on the fly and a
+ * portion of the waitlist needed to make the exact sample size.
+ */
+private[spark] object PoissonBounds {
+
+ val delta = 1e-4 / 3.0
+ val epsilon = 1e-15
+
+ /**
+ * Compute the threshold for accepting items on the fly. The threshold value is a fairly small
+ * number, which means if the item has an associated value < threshold, it is highly likely to
+ * be in the final sample. Hence we accept items with values less than the returned value of this
+ * function instantly.
+ *
+ * @param s sample size
+ * @return threshold for accepting items on the fly
+ */
+ def getLowerBound(s: Double): Double = {
+ var lb = math.max(0.0, s - math.sqrt(s / delta)) // Chebyshev's inequality
+ var ub = s
+ while (lb < ub - 1.0) {
+ val m = (lb + ub) / 2.0
+ val poisson = new PoissonDistribution(m, epsilon)
+ val y = poisson.inverseCumulativeProbability(1 - delta)
+ if (y > s) ub = m else lb = m
+ }
+ lb
+ }
+
+ def getMinCount(lmbd: Double): Double = {
+ if (lmbd == 0) return 0
--- End diff --
Style nit: it would be nicer to avoid 'return' and write this as an expression.
```
if (lmbd == 0) {
0
} else {
val poisson = new PoissonDistribution(lmbd, epsilon)
poisson.inverseCumulativeProbability(delta)
}
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by falaki <gi...@git.apache.org>.
Github user falaki commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r14544709
--- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util.random
+
+import scala.collection.{Map, mutable}
+import scala.collection.mutable.ArrayBuffer
+import scala.reflect.ClassTag
+
+import org.apache.commons.math3.random.RandomDataGenerator
+import org.apache.spark.{Logging, SparkContext, TaskContext}
+import org.apache.spark.rdd.RDD
+import org.apache.spark.util.Utils
+import org.apache.spark.util.random.{PoissonBounds => PB}
+
+private[spark] object StratifiedSampler extends Logging {
+
+ /**
+ * A version of {@link #aggregate()} that passes the TaskContext to the function that does
+ * aggregation for each partition. This function avoids creating an extra depth in the RDD
+ * lineage, as opposed to using mapPartitionsWithId, which results in slightly improved run time.
+ */
+ def aggregateWithContext[U: ClassTag, T: ClassTag](zeroValue: U)
+ (rdd: RDD[T],
+ seqOp: ((TaskContext, U), T) => U,
+ combOp: (U, U) => U): U = {
+ val sc: SparkContext = rdd.sparkContext
+ // Clone the zero value since we will also be serializing it as part of tasks
+ var jobResult = Utils.clone(zeroValue, sc.env.closureSerializer.newInstance())
+ // pad seqOp and combOp with taskContext to conform to aggregate's signature in TraversableOnce
+ val paddedSeqOp = (arg1: (TaskContext, U), item: T) => (arg1._1, seqOp(arg1, item))
+ val paddedcombOp = (arg1: (TaskContext, U), arg2: (TaskContext, U)) =>
+ (arg1._1, combOp(arg1._2, arg1._2))
+ val cleanSeqOp = sc.clean(paddedSeqOp)
+ val cleanCombOp = sc.clean(paddedcombOp)
+ val aggregatePartition = (tc: TaskContext, it: Iterator[T]) =>
+ (it.aggregate(tc, zeroValue)(cleanSeqOp, cleanCombOp))._2
+ val mergeResult = (index: Int, taskResult: U) => jobResult = combOp(jobResult, taskResult)
+ sc.runJob(rdd, aggregatePartition, mergeResult)
+ jobResult
+ }
+
+ /**
+ * Returns the function used by aggregate to collect sampling statistics for each partition.
+ */
+ def getSeqOp[K, V](withReplacement: Boolean,
+ fractionByKey: (K => Double),
+ counts: Option[Map[K, Long]]): ((TaskContext, Result[K]),(K, V)) => Result[K] = {
+ val delta = 5e-5
+ (U: (TaskContext, Result[K]), item: (K, V)) => {
+ val result = U._2
+ val tc = U._1
+ val rng = result.getRand(tc.partitionId)
+ val fraction = fractionByKey(item._1)
+ val stratum = result.getEntry(item._1)
+ if (withReplacement) {
+ // compute q1 and q2 only if they haven't been computed already
+ // since they don't change from iteration to iteration.
+ // TODO change this to the streaming version
+ if (stratum.q1.isEmpty || stratum.q2.isEmpty) {
+ val n = counts.get(item._1)
+ val s = math.ceil(n * fraction).toLong
+ val lmbd1 = PB.getLowerBound(s)
+ val minCount = PB.getMinCount(lmbd1)
+ val lmbd2 = if (lmbd1 == 0) PB.getUpperBound(s) else PB.getUpperBound(s - minCount)
+ val q1 = lmbd1 / n
+ val q2 = lmbd2 / n
+ stratum.q1 = Some(q1)
+ stratum.q2 = Some(q2)
+ }
+ val x1 = if (stratum.q1.get == 0) 0L else rng.nextPoisson(stratum.q1.get)
+ if (x1 > 0) {
+ stratum.incrNumAccepted(x1)
+ }
+ val x2 = rng.nextPoisson(stratum.q2.get).toInt
+ if (x2 > 0) {
+ stratum.addToWaitList(ArrayBuffer.fill(x2)(rng.nextUniform(0.0, 1.0)))
+ }
+ } else {
+ // We use the streaming version of the algorithm for sampling without replacement.
+ // Hence, q1 and q2 change on every iteration.
+ val g1 = - math.log(delta) / stratum.numItems
+ val g2 = (2.0 / 3.0) * g1
+ val q1 = math.max(0, fraction + g2 - math.sqrt((g2 * g2 + 3 * g2 * fraction)))
+ val q2 = math.min(1, fraction + g1 + math.sqrt(g1 * g1 + 2 * g1 * fraction))
+
+ val x = rng.nextUniform(0.0, 1.0)
+ if (x < q1) {
+ stratum.incrNumAccepted()
+ } else if (x < q2) {
+ stratum.addToWaitList(x)
+ }
+ stratum.q1 = Some(q1)
+ stratum.q2 = Some(q2)
+ }
+ stratum.incrNumItems()
+ result
+ }
+ }
+
+ /**
+ * Returns the function used by aggregate to combine results from different partitions, as
+ * returned by seqOp.
+ */
+ def getCombOp[K](): (Result[K], Result[K]) => Result[K] = {
+ (r1: Result[K], r2: Result[K]) => {
+ // take union of both key sets in case one partition doesn't contain all keys
+ val keyUnion = r1.resultMap.keys.toSet.union(r2.resultMap.keys.toSet)
+
+ // Use r2 to keep the combined result since r1 is usual empty
+ for (key <- keyUnion) {
+ val entry1 = r1.resultMap.get(key)
+ val entry2 = r2.resultMap.get(key)
+ if (entry2.isEmpty && entry1.isDefined) {
+ r2.resultMap += (key -> entry1.get)
+ } else if (entry1.isDefined && entry2.isDefined) {
+ entry2.get.addToWaitList(entry1.get.waitList)
+ entry2.get.incrNumAccepted(entry1.get.numAccepted)
+ entry2.get.incrNumItems(entry1.get.numItems)
+ }
+ }
+ r2
+ }
+ }
+
+ /**
+ * Given the result returned by the aggregate function, we need to determine the threshold used
+ * to accept items to generate the exact sample size.
+ */
+ def computeThresholdByKey[K](finalResult: Map[K, Stratum], fractionByKey: (K => Double)):
--- End diff --
Style nit: Break the after the first argument, so the return type of the function is not on a new line by itself.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1025#issuecomment-50383895
QA tests have started for PR 1025. This patch merges cleanly. <br>View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17300/consoleFull
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1025#issuecomment-50409833
QA tests have started for PR 1025. This patch merges cleanly. <br>View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17307/consoleFull
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1025#issuecomment-50521377
QA tests have started for PR 1025. This patch merges cleanly. <br>View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17368/consoleFull
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by falaki <gi...@git.apache.org>.
Github user falaki commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r14688398
--- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
@@ -0,0 +1,310 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util.random
+
+import scala.collection.Map
+import scala.collection.mutable.{ArrayBuffer, HashMap, Map => MMap}
+
+import org.apache.commons.math3.random.RandomDataGenerator
+import org.apache.spark.Logging
+import org.apache.spark.rdd.RDD
+
+/**
+ * Auxiliary functions and data structures for the sampleByKey method in PairRDDFunctions.
+ *
+ * Essentially, when exact sample size is necessary, we make additional passes over the RDD to
+ * compute the exact threshold value to use for each stratum to guarantee exact sample size with
+ * high probability. This is achieved by maintaining a waitlist of size O(log(s)), where s is the
+ * desired sample size for each stratum.
+ *
+ * Like in simple random sampling, we generate a random value for each item from the
+ * uniform distribution [0.0, 1.0]. All items with values <= min(values of items in the waitlist)
+ * are accepted into the sample instantly. The threshold for instant accept is designed so that
+ * s - numAccepted = O(log(s)), where s is again the desired sample size. Thus, by maintaining a
+ * waitlist size = O(log(s)), we will be able to create a sample of the exact size s by adding
+ * a portion of the waitlist to the set of items that are instantly accepted. The exact threshold
+ * is computed by sorting the values in the waitlist and picking the value at (s - numAccepted).
+ *
+ * Note that since we use the same seed for the RNG when computing the thresholds and the actual
+ * sample, our computed thresholds are guaranteed to produce the desired sample size.
+ *
+ * For more theoretical background on the sampling techniques used here, please refer to
+ * http://jmlr.org/proceedings/papers/v28/meng13a.html
+ */
+
+private[spark] object StratifiedSampler extends Logging {
+
+ /**
+ * Count the number of items instantly accepted and generate the waitlist for each stratum.
+ *
+ * This is only invoked when exact sample size is required.
+ */
+ def getCounts[K, V](rdd: RDD[(K, V)],
+ withReplacement: Boolean,
+ fractions: Map[K, Double],
+ counts: Option[Map[K, Long]],
+ seed: Long): MMap[K, Stratum] = {
+ val combOp = getCombOp[K]
+ val mappedPartitionRDD = rdd.mapPartitionsWithIndex({ case (partition, iter) =>
+ val zeroU: MMap[K, Stratum] = new HashMap[K, Stratum]()
+ val rng = new RandomDataGenerator()
+ rng.reSeed(seed + partition)
+ val seqOp = getSeqOp(withReplacement, fractions, rng, counts)
+ Iterator(iter.aggregate(zeroU)(seqOp, combOp))
+ }, preservesPartitioning=true)
+ mappedPartitionRDD.reduce(combOp)
+ }
+
+ /**
+ * Returns the function used by aggregate to collect sampling statistics for each partition.
+ */
+ def getSeqOp[K, V](withReplacement: Boolean,
+ fractions: Map[K, Double],
+ rng: RandomDataGenerator,
+ counts: Option[Map[K, Long]]): (MMap[K, Stratum], (K, V)) => MMap[K, Stratum] = {
+ val delta = 5e-5
+ (result: MMap[K, Stratum], item: (K, V)) => {
+ val key = item._1
+ val fraction = fractions(key)
+ if (!result.contains(key)) {
+ result += (key -> new Stratum())
+ }
+ val stratum = result(key)
+
+ if (withReplacement) {
+ // compute acceptBound and waitListBound only if they haven't been computed already
+ // since they don't change from iteration to iteration.
+ // TODO change this to the streaming version
+ if (stratum.areBoundsEmpty) {
+ val n = counts.get(key)
+ val sampleSize = math.ceil(n * fraction).toLong
+ val lmbd1 = PoissonBounds.getLowerBound(sampleSize)
+ val minCount = PoissonBounds.getMinCount(lmbd1)
+ val lmbd2 = if (lmbd1 == 0) {
+ PoissonBounds.getUpperBound(sampleSize)
+ } else {
+ PoissonBounds.getUpperBound(sampleSize - minCount)
+ }
+ stratum.acceptBound = lmbd1 / n
+ stratum.waitListBound = lmbd2 / n
+ }
+ val acceptBound = stratum.acceptBound
+ val copiesAccepted = if (acceptBound == 0.0) 0L else rng.nextPoisson(acceptBound)
+ if (copiesAccepted > 0) {
+ stratum.incrNumAccepted(copiesAccepted)
+ }
+ val copiesWaitlisted = rng.nextPoisson(stratum.waitListBound).toInt
+ if (copiesWaitlisted > 0) {
+ stratum.addToWaitList(ArrayBuffer.fill(copiesWaitlisted)(rng.nextUniform(0.0, 1.0)))
+ }
+ } else {
+ // We use the streaming version of the algorithm for sampling without replacement to avoid
+ // using an extra pass over the RDD for computing the count.
+ // Hence, acceptBound and waitListBound change on every iteration.
+ val g1 = - math.log(delta) / stratum.numItems // gamma1
+ val g2 = (2.0 / 3.0) * g1 // gamma 2
+ stratum.acceptBound = math.max(0, fraction + g2 - math.sqrt(g2 * g2 + 3 * g2 * fraction))
+ stratum.waitListBound = math.min(1, fraction + g1 + math.sqrt(g1 * g1 + 2 * g1 * fraction))
+
+ val x = rng.nextUniform(0.0, 1.0)
+ if (x < stratum.acceptBound) {
+ stratum.incrNumAccepted()
+ } else if (x < stratum.waitListBound) {
+ stratum.addToWaitList(x)
+ }
+ }
+ stratum.incrNumItems()
+ result
+ }
+ }
+
+ /**
+ * Returns the function used combine results returned by seqOp from different partitions.
+ */
+ def getCombOp[K]: (MMap[K, Stratum], MMap[K, Stratum]) => MMap[K, Stratum] = {
+ (r1: MMap[K, Stratum], r2: MMap[K, Stratum]) => {
+ // take union of both key sets in case one partition doesn't contain all keys
+ for (key <- r1.keySet.union(r2.keySet)) {
+ // Use r2 to keep the combined result since r1 is usual empty
+ val entry1 = r1.get(key)
+ if (r2.contains(key)) {
+ r2(key).merge(entry1)
+ } else {
+ if (entry1.isDefined) {
+ r2 += (key -> entry1.get)
+ }
+ }
+ }
+ r2
+ }
+ }
+
+ /**
+ * Given the result returned by getCounts, determine the threshold for accepting items to
+ * generate exact sample size.
+ *
+ * To do so, we compute sampleSize = math.ceil(size * samplingRate) for each stratum and compare
+ * it to the number of items that were accepted instantly and the number of items in the waitlist
+ * for that stratum. Most of the time, numAccepted <= sampleSize <= (numAccepted + numWaitlisted),
+ * which means we need to sort the elements in the waitlist by their associated values in order
+ * to find the value T s.t. |{elements in the stratum whose associated values <= T}| = sampleSize.
+ * Note that all elements in the waitlist have values >= bound for instant accept, so a T value
+ * in the waitlist range would allow all elements that were instantly accepted on the first pass
+ * to be included in the sample.
+ */
+ def computeThresholdByKey[K](finalResult: Map[K, Stratum],
+ fractions: Map[K, Double]): Map[K, Double] = {
+ val thresholdByKey = new HashMap[K, Double]()
+ for ((key, stratum) <- finalResult) {
+ val sampleSize = math.ceil(stratum.numItems * fractions(key)).toLong
+ if (stratum.numAccepted > sampleSize) {
+ logWarning("Pre-accepted too many")
+ thresholdByKey += (key -> stratum.acceptBound)
+ } else {
+ val numWaitListAccepted = (sampleSize - stratum.numAccepted).toInt
+ if (numWaitListAccepted >= stratum.waitList.size) {
+ logWarning("WaitList too short")
+ thresholdByKey += (key -> stratum.waitListBound)
+ } else {
+ thresholdByKey += (key -> stratum.waitList.sorted.apply(numWaitListAccepted))
+ }
+ }
+ }
+ thresholdByKey
+ }
+
+ /**
+ * Return the per partition sampling function used for sampling without replacement.
+ *
+ * When exact sample size is required, we make an additional pass over the RDD to determine the
+ * exact sampling rate that guarantees sample size with high confidence.
+ *
+ * The sampling function has a unique seed per partition.
+ */
+ def getBernoulliSamplingFunction[K, V](rdd: RDD[(K, V)],
+ fractions: Map[K, Double],
+ exact: Boolean,
+ seed: Long): (Int, Iterator[(K, V)]) => Iterator[(K, V)] = {
+ var samplingRateByKey = fractions
+ if (exact) {
+ // determine threshold for each stratum and resample
+ val finalResult = getCounts(rdd, false, fractions, None, seed)
+ samplingRateByKey = computeThresholdByKey(finalResult, fractions)
+ }
+ (idx: Int, iter: Iterator[(K, V)]) => {
+ val rng = new RandomDataGenerator
+ rng.reSeed(seed + idx)
+ iter.filter(t => rng.nextUniform(0.0, 1.0) < samplingRateByKey(t._1))
+ }
+ }
+
+ /**
+ * Return the per partition sampling function used for sampling with replacement.
+ *
+ * When exact sample size is required, we make two additional passed over the RDD to determine
+ * the exact sampling rate that guarantees sample size with high confidence. The first pass
+ * counts the number of items in each stratum (group of items with the same key) in the RDD, and
+ * the second pass uses the counts to determine exact sampling rates.
+ *
+ * The sampling function has a unique seed per partition.
+ */
+ def getPoissonSamplingFunction[K, V](rdd: RDD[(K, V)],
+ fractions: Map[K, Double],
+ exact: Boolean,
+ counts: Option[Map[K, Long]],
+ seed: Long): (Int, Iterator[(K, V)]) => Iterator[(K, V)] = {
+ // TODO implement the streaming version of sampling w/ replacement that doesn't require counts
+ if (exact) {
+ val finalResult = getCounts(rdd, true, fractions, counts, seed)
+ val thresholdByKey = computeThresholdByKey(finalResult, fractions)
+ (idx: Int, iter: Iterator[(K, V)]) => {
+ val rng = new RandomDataGenerator()
+ rng.reSeed(seed + idx)
+ iter.flatMap { item =>
+ val key = item._1
+ val q1 = finalResult(key).acceptBound
+ val q2 = finalResult(key).waitListBound
--- End diff --
Similarly for q1 and q2, more descriptive names can help.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by falaki <gi...@git.apache.org>.
Github user falaki commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r14542829
--- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util.random
+
+import scala.collection.{Map, mutable}
+import scala.collection.mutable.ArrayBuffer
+import scala.reflect.ClassTag
+
+import org.apache.commons.math3.random.RandomDataGenerator
+import org.apache.spark.{Logging, SparkContext, TaskContext}
+import org.apache.spark.rdd.RDD
+import org.apache.spark.util.Utils
+import org.apache.spark.util.random.{PoissonBounds => PB}
+
+private[spark] object StratifiedSampler extends Logging {
+
+ /**
+ * A version of {@link #aggregate()} that passes the TaskContext to the function that does
+ * aggregation for each partition. This function avoids creating an extra depth in the RDD
+ * lineage, as opposed to using mapPartitionsWithId, which results in slightly improved run time.
--- End diff --
Do you mean ```mapPartiionsWithIndex()```?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1025#issuecomment-47990765
Merged build finished. All automated tests passed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1025#issuecomment-45553587
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15581/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r13578391
--- Diff: core/src/main/scala/org/apache/spark/util/random/SamplingUtils.scala ---
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util.random
+
+import org.apache.commons.math3.distribution.{PoissonDistribution, NormalDistribution}
+
+private[spark] object PoissonBounds {
--- End diff --
Though private, this object needs some documentation. I understand this is a little unfair ... :p
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by falaki <gi...@git.apache.org>.
Github user falaki commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r14688363
--- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
@@ -0,0 +1,310 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util.random
+
+import scala.collection.Map
+import scala.collection.mutable.{ArrayBuffer, HashMap, Map => MMap}
+
+import org.apache.commons.math3.random.RandomDataGenerator
+import org.apache.spark.Logging
+import org.apache.spark.rdd.RDD
+
+/**
+ * Auxiliary functions and data structures for the sampleByKey method in PairRDDFunctions.
+ *
+ * Essentially, when exact sample size is necessary, we make additional passes over the RDD to
+ * compute the exact threshold value to use for each stratum to guarantee exact sample size with
+ * high probability. This is achieved by maintaining a waitlist of size O(log(s)), where s is the
+ * desired sample size for each stratum.
+ *
+ * Like in simple random sampling, we generate a random value for each item from the
+ * uniform distribution [0.0, 1.0]. All items with values <= min(values of items in the waitlist)
+ * are accepted into the sample instantly. The threshold for instant accept is designed so that
+ * s - numAccepted = O(log(s)), where s is again the desired sample size. Thus, by maintaining a
+ * waitlist size = O(log(s)), we will be able to create a sample of the exact size s by adding
+ * a portion of the waitlist to the set of items that are instantly accepted. The exact threshold
+ * is computed by sorting the values in the waitlist and picking the value at (s - numAccepted).
+ *
+ * Note that since we use the same seed for the RNG when computing the thresholds and the actual
+ * sample, our computed thresholds are guaranteed to produce the desired sample size.
+ *
+ * For more theoretical background on the sampling techniques used here, please refer to
+ * http://jmlr.org/proceedings/papers/v28/meng13a.html
+ */
+
+private[spark] object StratifiedSampler extends Logging {
+
+ /**
+ * Count the number of items instantly accepted and generate the waitlist for each stratum.
+ *
+ * This is only invoked when exact sample size is required.
+ */
+ def getCounts[K, V](rdd: RDD[(K, V)],
+ withReplacement: Boolean,
+ fractions: Map[K, Double],
+ counts: Option[Map[K, Long]],
+ seed: Long): MMap[K, Stratum] = {
+ val combOp = getCombOp[K]
+ val mappedPartitionRDD = rdd.mapPartitionsWithIndex({ case (partition, iter) =>
+ val zeroU: MMap[K, Stratum] = new HashMap[K, Stratum]()
+ val rng = new RandomDataGenerator()
+ rng.reSeed(seed + partition)
+ val seqOp = getSeqOp(withReplacement, fractions, rng, counts)
+ Iterator(iter.aggregate(zeroU)(seqOp, combOp))
+ }, preservesPartitioning=true)
+ mappedPartitionRDD.reduce(combOp)
+ }
+
+ /**
+ * Returns the function used by aggregate to collect sampling statistics for each partition.
+ */
+ def getSeqOp[K, V](withReplacement: Boolean,
+ fractions: Map[K, Double],
+ rng: RandomDataGenerator,
+ counts: Option[Map[K, Long]]): (MMap[K, Stratum], (K, V)) => MMap[K, Stratum] = {
+ val delta = 5e-5
+ (result: MMap[K, Stratum], item: (K, V)) => {
+ val key = item._1
+ val fraction = fractions(key)
+ if (!result.contains(key)) {
+ result += (key -> new Stratum())
+ }
+ val stratum = result(key)
+
+ if (withReplacement) {
+ // compute acceptBound and waitListBound only if they haven't been computed already
+ // since they don't change from iteration to iteration.
+ // TODO change this to the streaming version
+ if (stratum.areBoundsEmpty) {
+ val n = counts.get(key)
+ val sampleSize = math.ceil(n * fraction).toLong
+ val lmbd1 = PoissonBounds.getLowerBound(sampleSize)
+ val minCount = PoissonBounds.getMinCount(lmbd1)
+ val lmbd2 = if (lmbd1 == 0) {
+ PoissonBounds.getUpperBound(sampleSize)
+ } else {
+ PoissonBounds.getUpperBound(sampleSize - minCount)
+ }
+ stratum.acceptBound = lmbd1 / n
+ stratum.waitListBound = lmbd2 / n
+ }
+ val acceptBound = stratum.acceptBound
+ val copiesAccepted = if (acceptBound == 0.0) 0L else rng.nextPoisson(acceptBound)
+ if (copiesAccepted > 0) {
+ stratum.incrNumAccepted(copiesAccepted)
+ }
+ val copiesWaitlisted = rng.nextPoisson(stratum.waitListBound).toInt
+ if (copiesWaitlisted > 0) {
+ stratum.addToWaitList(ArrayBuffer.fill(copiesWaitlisted)(rng.nextUniform(0.0, 1.0)))
+ }
+ } else {
+ // We use the streaming version of the algorithm for sampling without replacement to avoid
+ // using an extra pass over the RDD for computing the count.
+ // Hence, acceptBound and waitListBound change on every iteration.
+ val g1 = - math.log(delta) / stratum.numItems // gamma1
+ val g2 = (2.0 / 3.0) * g1 // gamma 2
+ stratum.acceptBound = math.max(0, fraction + g2 - math.sqrt(g2 * g2 + 3 * g2 * fraction))
+ stratum.waitListBound = math.min(1, fraction + g1 + math.sqrt(g1 * g1 + 2 * g1 * fraction))
+
+ val x = rng.nextUniform(0.0, 1.0)
+ if (x < stratum.acceptBound) {
+ stratum.incrNumAccepted()
+ } else if (x < stratum.waitListBound) {
+ stratum.addToWaitList(x)
+ }
+ }
+ stratum.incrNumItems()
+ result
+ }
+ }
+
+ /**
+ * Returns the function used combine results returned by seqOp from different partitions.
+ */
+ def getCombOp[K]: (MMap[K, Stratum], MMap[K, Stratum]) => MMap[K, Stratum] = {
+ (r1: MMap[K, Stratum], r2: MMap[K, Stratum]) => {
--- End diff --
Minor: Similarly for r1 and r2
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1025#issuecomment-46608095
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1025#issuecomment-48414125
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by falaki <gi...@git.apache.org>.
Github user falaki commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r13677098
--- Diff: core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala ---
@@ -156,6 +161,182 @@ class PairRDDFunctions[K, V](self: RDD[(K, V)])
}
/**
+ * Return a subset of this RDD sampled by key (via stratified sampling).
+ * We guarantee a sample size = math.ceil(fraction * S_i), where S_i is the size of the ith
+ * stratum.
+ *
+ * @param withReplacement whether to sample with or without replacement
+ * @param fraction sampling rate
+ * @param seed seed for the random number generator
+ * @return RDD containing the sampled subset
+ */
+ def sampleByKey(withReplacement: Boolean,
+ fraction: Double,
+ seed: Long = Utils.random.nextLong): RDD[(K, V)]= {
+
+ class Stratum(var numItems: Long = 0L, var numAccepted: Long = 0L) extends Serializable {
+ var waitList: ArrayBuffer[Double] = new ArrayBuffer[Double]
+ var q1: Option[Double] = None
+ var q2: Option[Double] = None
+
+ def incrNumItems(by: Long = 1L) = numItems += by
+
+ def incrNumAccepted(by: Long = 1L) = numAccepted += by
+
+ def addToWaitList(elem: Double) = waitList += elem
+
+ def addToWaitList(elems: ArrayBuffer[Double]) = waitList ++= elems
+
+ override def toString() = {
+ "numItems: " + numItems + " numAccepted: " + numAccepted + " q1: " + q1 + " q2: " + q2 +
+ " waitListSize:" + waitList.size
+ }
+ }
+
+ class Result(var resultMap: Map[K, Stratum], var cachedPartitionId: Option[Int] = None)
+ extends Serializable {
+ var rand: RandomDataGenerator = new RandomDataGenerator
+
+ def getEntry(key: K, numItems: Long = 0L): Stratum = {
+ if (resultMap.get(key).isEmpty) {
+ resultMap += (key -> new Stratum(numItems))
+ }
+ resultMap.get(key).get
+ }
+
+ def getRand(partitionId: Int): RandomDataGenerator = {
+ if (cachedPartitionId.isEmpty || cachedPartitionId.get != partitionId) {
+ cachedPartitionId = Some(partitionId)
+ rand.reSeed(seed + partitionId)
+ }
+ rand
+ }
+ }
+
+ // TODO implement the streaming version of sampling w/ replacement that doesn't require counts
+ // in order to save one pass over the RDD
+ val counts = if (withReplacement) Some(this.countByKey()) else None
+
+ val seqOp = (U: (TaskContext, Result), item: (K, V)) => {
+ val delta = 5e-5
+ val result = U._2
+ val tc = U._1
+ val rng = result.getRand(tc.partitionId)
+ val stratum = result.getEntry(item._1)
+ if (withReplacement) {
+ // compute q1 and q2 only if they haven't been computed already
+ // since they don't change from iteration to iteration.
+ // TODO change this to the streaming version
+ if (stratum.q1.isEmpty || stratum.q2.isEmpty) {
+ val n = counts.get(item._1)
+ val s = math.ceil(n * fraction).toLong
+ val lmbd1 = PB.getLambda1(s)
+ val minCount = PB.getMinCount(lmbd1)
+ val lmbd2 = if (lmbd1 == 0) PB.getLambda2(s) else PB.getLambda2(s - minCount)
+ val q1 = lmbd1 / n
+ val q2 = lmbd2 / n
+ stratum.q1 = Some(q1)
+ stratum.q2 = Some(q2)
+ }
+ val x1 = if (stratum.q1.get == 0) 0L else rng.nextPoisson(stratum.q1.get)
+ if (x1 > 0) {
+ stratum.incrNumAccepted(x1)
+ }
+ val x2 = rng.nextPoisson(stratum.q2.get).toInt
+ if (x2 > 0) {
+ stratum.addToWaitList(ArrayBuffer.fill(x2)(rng.nextUniform(0.0, 1.0)))
+ }
+ } else {
+ val g1 = - math.log(delta) / stratum.numItems
+ val g2 = (2.0 / 3.0) * g1
+ val q1 = math.max(0, fraction + g2 - math.sqrt((g2 * g2 + 3 * g2 * fraction)))
+ val q2 = math.min(1, fraction + g1 + math.sqrt(g1 * g1 + 2 * g1 * fraction))
+
+ val x = rng.nextUniform(0.0, 1.0)
+ if (x < q1) {
+ stratum.incrNumAccepted()
+ } else if ( x < q2) {
+ stratum.addToWaitList(x)
+ }
+ stratum.q1 = Some(q1)
+ stratum.q2 = Some(q2)
+ }
+ stratum.incrNumItems()
+ result
+ }
+
+ val combOp = (r1: Result, r2: Result) => {
+ // take union of both key sets in case one partion doesn't contain all keys
+ val keyUnion = r1.resultMap.keys.toSet.union(r2.resultMap.keys.toSet)
+
+ // Use r2 to keep the combined result since r1 is usual empty
+ for (key <- keyUnion) {
+ val entry1 = r1.resultMap.get(key)
+ val entry2 = r2.resultMap.get(key)
+ if (entry2.isEmpty && entry1.isDefined) {
+ r2.resultMap += (key -> entry1.get)
+ } else if (entry1.isDefined && entry2.isDefined) {
+ entry2.get.addToWaitList(entry1.get.waitList)
+ entry2.get.incrNumAccepted(entry1.get.numAccepted)
+ entry2.get.incrNumItems(entry1.get.numItems)
+ }
+ }
+ r2
+ }
+
+ val zeroU = new Result(Map[K, Stratum]())
+
+ // determine threshold for each stratum and resample
+ val finalResult = self.aggregateWithContext(zeroU)(seqOp, combOp).resultMap
+ val thresholdByKey = new mutable.HashMap[K, Double]()
+ for ((key, stratum) <- finalResult) {
+ val s = math.ceil(stratum.numItems * fraction).toLong
+ breakable {
+ if (stratum.numAccepted > s) {
+ logWarning("Pre-accepted too many")
+ thresholdByKey += (key -> stratum.q1.get)
+ break
+ }
+ val numWaitListAccepted = (s - stratum.numAccepted).toInt
+ if (numWaitListAccepted >= stratum.waitList.size) {
+ logWarning("WaitList too short")
+ thresholdByKey += (key -> stratum.q2.get)
+ } else {
+ thresholdByKey += (key -> stratum.waitList.sorted.apply(numWaitListAccepted))
+ }
+ }
+ }
+
+ if (withReplacement) {
+ // Poisson sampler
+ self.mapPartitionsWithIndex((idx: Int, iter: Iterator[(K, V)]) => {
+ val random = new RandomDataGenerator()
+ random.reSeed(seed + idx)
+ iter.flatMap { t =>
+ val q1 = finalResult.get(t._1).get.q1.get
+ val q2 = finalResult.get(t._1).get.q2.get
--- End diff --
Use getOrElse to be safer.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1025#issuecomment-45558761
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15582/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1025#issuecomment-47990766
All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16313/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1025#issuecomment-45957663
Build finished.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1025#issuecomment-48131244
Merged build finished. All automated tests passed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r14907335
--- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
@@ -0,0 +1,311 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util.random
+
+import scala.collection.Map
+import scala.collection.mutable.{ArrayBuffer, HashMap, Map => MMap}
--- End diff --
`immutable.HashMap`: http://www.scala-lang.org/api/2.11.1/index.html#scala.collection.immutable.HashMap
`MMap` may be confusing because of `MultiMap`: http://www.scala-lang.org/api/2.11.1/index.html#scala.collection.immutable.MultiMap
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by falaki <gi...@git.apache.org>.
Github user falaki commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r14542796
--- Diff: core/src/main/scala/org/apache/spark/util/random/SamplingUtils.scala ---
@@ -45,11 +50,74 @@ private[spark] object SamplingUtils {
val fraction = sampleSizeLowerBound.toDouble / total
if (withReplacement) {
val numStDev = if (sampleSizeLowerBound < 12) 9 else 5
- fraction + numStDev * math.sqrt(fraction / total)
+ math.max(1e-10, fraction + numStDev * math.sqrt(fraction / total))
} else {
val delta = 1e-4
val gamma = - math.log(delta) / total
- math.min(1, fraction + gamma + math.sqrt(gamma * gamma + 2 * gamma * fraction))
+ math.min(1,
+ math.max(1e-10, fraction + gamma + math.sqrt(gamma * gamma + 2 * gamma * fraction)))
+ }
+ }
+}
+
+/**
+ * Utility functions that help us determine bounds on adjusted sampling rate to guarantee exact
+ * sample sizes with high confidence when sampling with replacement.
+ *
+ * The algorithm for guaranteeing sample size instantly accepts items whose associated value drawn
+ * from Pois(s) is less than the lower bound and puts items whose value is between the lower and
+ * upper bound in a waitlist. The final sample is consisted of all items accepted on the fly and a
+ * portion of the waitlist needed to make the exact sample size.
+ */
+private[spark] object PoissonBounds {
+
+ val delta = 1e-4 / 3.0
+
+ /**
+ * Compute the threshold for accepting items on the fly. The threshold value is a fairly small
+ * number, which means if the item has an associated value < threshold, it is highly likely to
+ * be in the final sample. Hence we accept items with values less than the returned value of this
+ * function instantly.
+ *
+ * @param s sample size
+ * @return threshold for accepting items on the fly
+ */
+ def getLowerBound(s: Double): Double = {
--- End diff --
Let's combine this function with getUpperBound() into a single getPoissonBounds() function that returns a tuple. There is good overlap between the two functions and they are used in the same place.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/1025
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by falaki <gi...@git.apache.org>.
Github user falaki commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r14544234
--- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util.random
+
+import scala.collection.{Map, mutable}
+import scala.collection.mutable.ArrayBuffer
+import scala.reflect.ClassTag
+
+import org.apache.commons.math3.random.RandomDataGenerator
+import org.apache.spark.{Logging, SparkContext, TaskContext}
+import org.apache.spark.rdd.RDD
+import org.apache.spark.util.Utils
+import org.apache.spark.util.random.{PoissonBounds => PB}
+
+private[spark] object StratifiedSampler extends Logging {
+
+ /**
+ * A version of {@link #aggregate()} that passes the TaskContext to the function that does
+ * aggregation for each partition. This function avoids creating an extra depth in the RDD
+ * lineage, as opposed to using mapPartitionsWithId, which results in slightly improved run time.
+ */
+ def aggregateWithContext[U: ClassTag, T: ClassTag](zeroValue: U)
+ (rdd: RDD[T],
+ seqOp: ((TaskContext, U), T) => U,
+ combOp: (U, U) => U): U = {
+ val sc: SparkContext = rdd.sparkContext
+ // Clone the zero value since we will also be serializing it as part of tasks
+ var jobResult = Utils.clone(zeroValue, sc.env.closureSerializer.newInstance())
+ // pad seqOp and combOp with taskContext to conform to aggregate's signature in TraversableOnce
+ val paddedSeqOp = (arg1: (TaskContext, U), item: T) => (arg1._1, seqOp(arg1, item))
+ val paddedcombOp = (arg1: (TaskContext, U), arg2: (TaskContext, U)) =>
+ (arg1._1, combOp(arg1._2, arg1._2))
+ val cleanSeqOp = sc.clean(paddedSeqOp)
+ val cleanCombOp = sc.clean(paddedcombOp)
+ val aggregatePartition = (tc: TaskContext, it: Iterator[T]) =>
+ (it.aggregate(tc, zeroValue)(cleanSeqOp, cleanCombOp))._2
+ val mergeResult = (index: Int, taskResult: U) => jobResult = combOp(jobResult, taskResult)
+ sc.runJob(rdd, aggregatePartition, mergeResult)
+ jobResult
+ }
+
+ /**
+ * Returns the function used by aggregate to collect sampling statistics for each partition.
+ */
+ def getSeqOp[K, V](withReplacement: Boolean,
+ fractionByKey: (K => Double),
+ counts: Option[Map[K, Long]]): ((TaskContext, Result[K]),(K, V)) => Result[K] = {
+ val delta = 5e-5
+ (U: (TaskContext, Result[K]), item: (K, V)) => {
+ val result = U._2
+ val tc = U._1
+ val rng = result.getRand(tc.partitionId)
+ val fraction = fractionByKey(item._1)
+ val stratum = result.getEntry(item._1)
+ if (withReplacement) {
+ // compute q1 and q2 only if they haven't been computed already
+ // since they don't change from iteration to iteration.
+ // TODO change this to the streaming version
+ if (stratum.q1.isEmpty || stratum.q2.isEmpty) {
+ val n = counts.get(item._1)
+ val s = math.ceil(n * fraction).toLong
+ val lmbd1 = PB.getLowerBound(s)
+ val minCount = PB.getMinCount(lmbd1)
+ val lmbd2 = if (lmbd1 == 0) PB.getUpperBound(s) else PB.getUpperBound(s - minCount)
+ val q1 = lmbd1 / n
+ val q2 = lmbd2 / n
+ stratum.q1 = Some(q1)
+ stratum.q2 = Some(q2)
+ }
+ val x1 = if (stratum.q1.get == 0) 0L else rng.nextPoisson(stratum.q1.get)
+ if (x1 > 0) {
+ stratum.incrNumAccepted(x1)
+ }
+ val x2 = rng.nextPoisson(stratum.q2.get).toInt
+ if (x2 > 0) {
+ stratum.addToWaitList(ArrayBuffer.fill(x2)(rng.nextUniform(0.0, 1.0)))
+ }
+ } else {
+ // We use the streaming version of the algorithm for sampling without replacement.
+ // Hence, q1 and q2 change on every iteration.
+ val g1 = - math.log(delta) / stratum.numItems
+ val g2 = (2.0 / 3.0) * g1
+ val q1 = math.max(0, fraction + g2 - math.sqrt((g2 * g2 + 3 * g2 * fraction)))
+ val q2 = math.min(1, fraction + g1 + math.sqrt(g1 * g1 + 2 * g1 * fraction))
+
+ val x = rng.nextUniform(0.0, 1.0)
+ if (x < q1) {
+ stratum.incrNumAccepted()
+ } else if (x < q2) {
+ stratum.addToWaitList(x)
+ }
+ stratum.q1 = Some(q1)
+ stratum.q2 = Some(q2)
+ }
+ stratum.incrNumItems()
+ result
+ }
+ }
+
+ /**
+ * Returns the function used by aggregate to combine results from different partitions, as
+ * returned by seqOp.
+ */
+ def getCombOp[K](): (Result[K], Result[K]) => Result[K] = {
+ (r1: Result[K], r2: Result[K]) => {
+ // take union of both key sets in case one partition doesn't contain all keys
+ val keyUnion = r1.resultMap.keys.toSet.union(r2.resultMap.keys.toSet)
+
+ // Use r2 to keep the combined result since r1 is usual empty
+ for (key <- keyUnion) {
+ val entry1 = r1.resultMap.get(key)
+ val entry2 = r2.resultMap.get(key)
+ if (entry2.isEmpty && entry1.isDefined) {
+ r2.resultMap += (key -> entry1.get)
+ } else if (entry1.isDefined && entry2.isDefined) {
+ entry2.get.addToWaitList(entry1.get.waitList)
+ entry2.get.incrNumAccepted(entry1.get.numAccepted)
+ entry2.get.incrNumItems(entry1.get.numItems)
+ }
+ }
+ r2
+ }
+ }
+
+ /**
+ * Given the result returned by the aggregate function, we need to determine the threshold used
+ * to accept items to generate the exact sample size.
+ */
+ def computeThresholdByKey[K](finalResult: Map[K, Stratum], fractionByKey: (K => Double)):
+ (K => Double) = {
+ val thresholdByKey = new mutable.HashMap[K, Double]()
+ for ((key, stratum) <- finalResult) {
+ val fraction = fractionByKey(key)
+ val s = math.ceil(stratum.numItems * fraction).toLong
+ if (stratum.numAccepted > s) {
+ logWarning("Pre-accepted too many")
+ thresholdByKey += (key -> stratum.q1.get)
+ } else {
+ val numWaitListAccepted = (s - stratum.numAccepted).toInt
+ if (numWaitListAccepted >= stratum.waitList.size) {
+ logWarning("WaitList too short")
+ thresholdByKey += (key -> stratum.q2.get)
+ } else {
+ thresholdByKey += (key -> stratum.waitList.sorted.apply(numWaitListAccepted))
+ }
+ }
+ }
+ thresholdByKey
+ }
+
+ /**
+ * Return the per partition sampling function used for sampling without replacement.
+ *
+ * When exact sample size is required, we make an additional pass over the RDD to determine the
+ * exact sampling rate that guarantees sample size with high confidence.
+ *
+ * The sampling function has a unique seed per partition.
+ */
+ def getBernoulliSamplingFunction[K, V](rdd: RDD[(K, V)],
+ fractionByKey: K => Double,
+ exact: Boolean,
+ seed: Long): (Int, Iterator[(K, V)]) => Iterator[(K, V)] = {
+ var samplingRateByKey = fractionByKey
+ if (exact) {
+ // determine threshold for each stratum and resample
+ val seqOp = StratifiedSampler.getSeqOp[K,V](false, fractionByKey, None)
+ val combOp = StratifiedSampler.getCombOp[K]()
+ val zeroU = new Result[K](Map[K, Stratum](), seed = seed)
+ val finalResult = aggregateWithContext(zeroU)(rdd, seqOp, combOp).resultMap
+ samplingRateByKey = StratifiedSampler.computeThresholdByKey(finalResult, fractionByKey)
+ }
+ (idx: Int, iter: Iterator[(K, V)]) => {
+ val random = new RandomDataGenerator
+ random.reSeed(seed + idx)
+ iter.filter(t => random.nextUniform(0.0, 1.0) < samplingRateByKey(t._1))
+ }
+ }
+
+ /**
+ * Return the per partition sampling function used for sampling with replacement.
+ *
+ * When exact sample size is required, we make two additional passed over the RDD to determine
+ * the exact sampling rate that guarantees sample size with high confidence. The first pass
+ * counts the number of items in each stratum (group of items with the same key) in the RDD, and
+ * the second pass uses the counts to determine exact sampling rates.
+ *
+ * The sampling function has a unique seed per partition.
+ */
+ def getPoissonSamplingFunction[K, V](rdd:RDD[(K, V)],
+ fractionByKey: K => Double,
+ exact: Boolean,
+ counts: Option[Map[K, Long]],
+ seed: Long): (Int, Iterator[(K, V)]) => Iterator[(K, V)] = {
+ // TODO implement the streaming version of sampling w/ replacement that doesn't require counts
+ if (exact) {
+ val seqOp = StratifiedSampler.getSeqOp[K,V](true, fractionByKey, counts)
+ val combOp = StratifiedSampler.getCombOp[K]()
+ val zeroU = new Result[K](Map[K, Stratum](), seed = seed)
+ val finalResult = aggregateWithContext(zeroU)(rdd, seqOp, combOp).resultMap
+ val thresholdByKey = StratifiedSampler.computeThresholdByKey(finalResult, fractionByKey)
+ (idx: Int, iter: Iterator[(K, V)]) => {
+ val random = new RandomDataGenerator()
+ random.reSeed(seed + idx)
+ iter.flatMap { t =>
+ val q1 = finalResult.get(t._1).get.q1.getOrElse(0.0)
+ val q2 = finalResult.get(t._1).get.q2.getOrElse(0.0)
+ val x1 = if (q1 == 0) 0L else random.nextPoisson(q1)
+ val x2 = random.nextPoisson(q2).toInt
+ val x = x1 + (0 until x2).count(i => random.nextUniform(0.0, 1.0) < thresholdByKey(t._1))
+ if (x > 0) {
+ Iterator.fill(x.toInt)(t)
+ } else {
+ Iterator.empty
+ }
+ }
+ }
+ } else {
+ (idx: Int, iter: Iterator[(K, V)]) => {
+ val random = new RandomDataGenerator()
+ random.reSeed(seed + idx)
+ iter.flatMap { t =>
+ val count = random.nextPoisson(fractionByKey(t._1)).toInt
+ if (count > 0) {
+ Iterator.fill(count)(t)
+ } else {
+ Iterator.empty
+ }
+ }
+ }
+ }
+ }
+}
+
+/**
+ * Object used by seqOp to keep track of the number of items accepted and items waitlisted per
+ * stratum, as well as the bounds for accepting and waitlisting items.
+ */
+private[random] class Stratum(var numItems: Long = 0L, var numAccepted: Long = 0L)
+ extends Serializable {
+
+ var waitList: ArrayBuffer[Double] = new ArrayBuffer[Double]
+ var q1: Option[Double] = None // upper bound for accepting item instantly
+ var q2: Option[Double] = None // upper bound for adding item to waitlist
+
+ def incrNumItems(by: Long = 1L) = numItems += by
+
+ def incrNumAccepted(by: Long = 1L) = numAccepted += by
+
+ def addToWaitList(elem: Double) = waitList += elem
+
+ def addToWaitList(elems: ArrayBuffer[Double]) = waitList ++= elems
+}
+
+/**
+ * Object used by seqOp and combOp to keep track of the sampling statistics for all strata.
+ *
+ * When used by seqOp for each partition, we also keep track of the partition ID in this object
+ * to make sure a single random number generator with a unique seed is used for each partition.
+ */
+private[random] class Result[K](var resultMap: Map[K, Stratum],
--- End diff --
Similarly, "[random]" can be removed here.
Also please change the indent of arguments in next lines to two tabs (see the style guide at: https://cwiki.apache.org/confluence/display/SPARK/Spark+Code+Style+Guide#SparkCodeStyleGuide-Indentation)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1025#issuecomment-48418912
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1025#issuecomment-45964261
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1025#issuecomment-46613651
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15922/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r14694274
--- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
@@ -0,0 +1,311 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util.random
+
+import scala.collection.Map
+import scala.collection.mutable.{ArrayBuffer, HashMap, Map => MMap}
+
+import org.apache.commons.math3.random.RandomDataGenerator
+import org.apache.spark.Logging
+import org.apache.spark.rdd.RDD
+
+/**
+ * Auxiliary functions and data structures for the sampleByKey method in PairRDDFunctions.
+ *
+ * Essentially, when exact sample size is necessary, we make additional passes over the RDD to
+ * compute the exact threshold value to use for each stratum to guarantee exact sample size with
+ * high probability. This is achieved by maintaining a waitlist of size O(log(s)), where s is the
+ * desired sample size for each stratum.
+ *
+ * Like in simple random sampling, we generate a random value for each item from the
+ * uniform distribution [0.0, 1.0]. All items with values <= min(values of items in the waitlist)
+ * are accepted into the sample instantly. The threshold for instant accept is designed so that
+ * s - numAccepted = O(log(s)), where s is again the desired sample size. Thus, by maintaining a
+ * waitlist size = O(log(s)), we will be able to create a sample of the exact size s by adding
+ * a portion of the waitlist to the set of items that are instantly accepted. The exact threshold
+ * is computed by sorting the values in the waitlist and picking the value at (s - numAccepted).
+ *
+ * Note that since we use the same seed for the RNG when computing the thresholds and the actual
+ * sample, our computed thresholds are guaranteed to produce the desired sample size.
+ *
+ * For more theoretical background on the sampling techniques used here, please refer to
+ * http://jmlr.org/proceedings/papers/v28/meng13a.html
+ */
+
+private[spark] object StratifiedSampler extends Logging {
+
+ /**
+ * Count the number of items instantly accepted and generate the waitlist for each stratum.
+ *
+ * This is only invoked when exact sample size is required.
+ */
+ def getCounts[K, V](rdd: RDD[(K, V)],
--- End diff --
The name is a little confusing, especially there is an argument named `counts`. See the comments below on the name of `Stratum`. If we change `Stratum` to `AcceptanceResult`, this could be changed to `getAcceptanceResults`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r14694237
--- Diff: core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala ---
@@ -195,6 +193,37 @@ class PairRDDFunctions[K, V](self: RDD[(K, V)])
}
/**
+ * Return a subset of this RDD sampled by key (via stratified sampling).
+ *
+ * If exact set to true, we guarantee, with high probability, a sample size =
--- End diff --
Before explaining `exact`, please add more doc explaining what `sampleByKey` does.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by dorx <gi...@git.apache.org>.
Github user dorx commented on the pull request:
https://github.com/apache/spark/pull/1025#issuecomment-50071767
Looks like there's some API changes from Xiangrui's updates. @mateiz @pwendell
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by dorx <gi...@git.apache.org>.
Github user dorx commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r14906825
--- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
@@ -0,0 +1,311 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util.random
+
+import scala.collection.Map
+import scala.collection.mutable.{ArrayBuffer, HashMap, Map => MMap}
+
+import org.apache.commons.math3.random.RandomDataGenerator
+import org.apache.spark.Logging
+import org.apache.spark.rdd.RDD
+
+/**
+ * Auxiliary functions and data structures for the sampleByKey method in PairRDDFunctions.
+ *
+ * Essentially, when exact sample size is necessary, we make additional passes over the RDD to
+ * compute the exact threshold value to use for each stratum to guarantee exact sample size with
+ * high probability. This is achieved by maintaining a waitlist of size O(log(s)), where s is the
+ * desired sample size for each stratum.
+ *
+ * Like in simple random sampling, we generate a random value for each item from the
+ * uniform distribution [0.0, 1.0]. All items with values <= min(values of items in the waitlist)
+ * are accepted into the sample instantly. The threshold for instant accept is designed so that
+ * s - numAccepted = O(log(s)), where s is again the desired sample size. Thus, by maintaining a
+ * waitlist size = O(log(s)), we will be able to create a sample of the exact size s by adding
+ * a portion of the waitlist to the set of items that are instantly accepted. The exact threshold
+ * is computed by sorting the values in the waitlist and picking the value at (s - numAccepted).
+ *
+ * Note that since we use the same seed for the RNG when computing the thresholds and the actual
+ * sample, our computed thresholds are guaranteed to produce the desired sample size.
+ *
+ * For more theoretical background on the sampling techniques used here, please refer to
+ * http://jmlr.org/proceedings/papers/v28/meng13a.html
+ */
+
+private[spark] object StratifiedSampler extends Logging {
+
+ /**
+ * Count the number of items instantly accepted and generate the waitlist for each stratum.
+ *
+ * This is only invoked when exact sample size is required.
+ */
+ def getCounts[K, V](rdd: RDD[(K, V)],
+ withReplacement: Boolean,
+ fractions: Map[K, Double],
+ counts: Option[Map[K, Long]],
+ seed: Long): MMap[K, Stratum] = {
+ val combOp = getCombOp[K]
+ val mappedPartitionRDD = rdd.mapPartitionsWithIndex({ case (partition, iter) =>
+ val zeroU: MMap[K, Stratum] = new HashMap[K, Stratum]()
+ val rng = new RandomDataGenerator()
+ rng.reSeed(seed + partition)
+ val seqOp = getSeqOp(withReplacement, fractions, rng, counts)
+ Iterator(iter.aggregate(zeroU)(seqOp, combOp))
+ }, preservesPartitioning=true)
+ mappedPartitionRDD.reduce(combOp)
+ }
+
+ /**
+ * Returns the function used by aggregate to collect sampling statistics for each partition.
+ */
+ def getSeqOp[K, V](withReplacement: Boolean,
+ fractions: Map[K, Double],
+ rng: RandomDataGenerator,
+ counts: Option[Map[K, Long]]): (MMap[K, Stratum], (K, V)) => MMap[K, Stratum] = {
+ val delta = 5e-5
+ (result: MMap[K, Stratum], item: (K, V)) => {
+ val key = item._1
+ val fraction = fractions(key)
+ if (!result.contains(key)) {
+ result += (key -> new Stratum())
+ }
+ val stratum = result(key)
+
+ if (withReplacement) {
+ // compute acceptBound and waitListBound only if they haven't been computed already
+ // since they don't change from iteration to iteration.
+ // TODO change this to the streaming version
+ if (stratum.areBoundsEmpty) {
+ val n = counts.get(key)
+ val sampleSize = math.ceil(n * fraction).toLong
+ val lmbd1 = PoissonBounds.getLowerBound(sampleSize)
+ val minCount = PoissonBounds.getMinCount(lmbd1)
--- End diff --
I'm not clear what "it" refers to.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1025#issuecomment-46391364
Merged build finished. All automated tests passed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r14907358
--- Diff: core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala ---
@@ -195,6 +193,37 @@ class PairRDDFunctions[K, V](self: RDD[(K, V)])
}
/**
+ * Return a subset of this RDD sampled by key (via stratified sampling).
+ *
+ * If exact set to true, we guarantee, with high probability, a sample size =
+ * math.ceil(fraction * S_i), where S_i is the size of the ith stratum (collection of entries
+ * that share the same key). When sampling without replacement, we need one additional pass over
+ * the RDD to guarantee sample size with a 99.99% confidence; when sampling with replacement, we
+ * need two additional passes.
+ *
+ * @param withReplacement whether to sample with or without replacement
+ * @param fractions map of specific keys to sampling rates
+ * @param seed seed for the random number generator
+ * @param exact whether sample size needs to be exactly math.ceil(fraction * size) per stratum
+ * @return RDD containing the sampled subset
+ */
+ def sampleByKey(withReplacement: Boolean,
+ fractions: Map[K, Double],
+ exact: Boolean = true,
+ seed: Long = Utils.random.nextLong): RDD[(K, V)]= {
+
+ require(fractions.forall {case(k, v) => v >= 0.0}, "Invalid sampling rates.")
--- End diff --
Sure. But we can do that in a separate PR.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1025#issuecomment-47984396
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by falaki <gi...@git.apache.org>.
Github user falaki commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r14544027
--- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util.random
+
+import scala.collection.{Map, mutable}
+import scala.collection.mutable.ArrayBuffer
+import scala.reflect.ClassTag
+
+import org.apache.commons.math3.random.RandomDataGenerator
+import org.apache.spark.{Logging, SparkContext, TaskContext}
+import org.apache.spark.rdd.RDD
+import org.apache.spark.util.Utils
+import org.apache.spark.util.random.{PoissonBounds => PB}
+
+private[spark] object StratifiedSampler extends Logging {
+
+ /**
+ * A version of {@link #aggregate()} that passes the TaskContext to the function that does
+ * aggregation for each partition. This function avoids creating an extra depth in the RDD
+ * lineage, as opposed to using mapPartitionsWithId, which results in slightly improved run time.
+ */
+ def aggregateWithContext[U: ClassTag, T: ClassTag](zeroValue: U)
+ (rdd: RDD[T],
+ seqOp: ((TaskContext, U), T) => U,
+ combOp: (U, U) => U): U = {
+ val sc: SparkContext = rdd.sparkContext
+ // Clone the zero value since we will also be serializing it as part of tasks
+ var jobResult = Utils.clone(zeroValue, sc.env.closureSerializer.newInstance())
+ // pad seqOp and combOp with taskContext to conform to aggregate's signature in TraversableOnce
+ val paddedSeqOp = (arg1: (TaskContext, U), item: T) => (arg1._1, seqOp(arg1, item))
+ val paddedcombOp = (arg1: (TaskContext, U), arg2: (TaskContext, U)) =>
+ (arg1._1, combOp(arg1._2, arg1._2))
+ val cleanSeqOp = sc.clean(paddedSeqOp)
+ val cleanCombOp = sc.clean(paddedcombOp)
+ val aggregatePartition = (tc: TaskContext, it: Iterator[T]) =>
+ (it.aggregate(tc, zeroValue)(cleanSeqOp, cleanCombOp))._2
+ val mergeResult = (index: Int, taskResult: U) => jobResult = combOp(jobResult, taskResult)
+ sc.runJob(rdd, aggregatePartition, mergeResult)
+ jobResult
+ }
+
+ /**
+ * Returns the function used by aggregate to collect sampling statistics for each partition.
+ */
+ def getSeqOp[K, V](withReplacement: Boolean,
+ fractionByKey: (K => Double),
+ counts: Option[Map[K, Long]]): ((TaskContext, Result[K]),(K, V)) => Result[K] = {
+ val delta = 5e-5
+ (U: (TaskContext, Result[K]), item: (K, V)) => {
+ val result = U._2
+ val tc = U._1
+ val rng = result.getRand(tc.partitionId)
+ val fraction = fractionByKey(item._1)
+ val stratum = result.getEntry(item._1)
+ if (withReplacement) {
+ // compute q1 and q2 only if they haven't been computed already
+ // since they don't change from iteration to iteration.
+ // TODO change this to the streaming version
+ if (stratum.q1.isEmpty || stratum.q2.isEmpty) {
+ val n = counts.get(item._1)
+ val s = math.ceil(n * fraction).toLong
+ val lmbd1 = PB.getLowerBound(s)
+ val minCount = PB.getMinCount(lmbd1)
+ val lmbd2 = if (lmbd1 == 0) PB.getUpperBound(s) else PB.getUpperBound(s - minCount)
+ val q1 = lmbd1 / n
+ val q2 = lmbd2 / n
+ stratum.q1 = Some(q1)
+ stratum.q2 = Some(q2)
+ }
+ val x1 = if (stratum.q1.get == 0) 0L else rng.nextPoisson(stratum.q1.get)
+ if (x1 > 0) {
+ stratum.incrNumAccepted(x1)
+ }
+ val x2 = rng.nextPoisson(stratum.q2.get).toInt
+ if (x2 > 0) {
+ stratum.addToWaitList(ArrayBuffer.fill(x2)(rng.nextUniform(0.0, 1.0)))
+ }
+ } else {
+ // We use the streaming version of the algorithm for sampling without replacement.
+ // Hence, q1 and q2 change on every iteration.
+ val g1 = - math.log(delta) / stratum.numItems
+ val g2 = (2.0 / 3.0) * g1
+ val q1 = math.max(0, fraction + g2 - math.sqrt((g2 * g2 + 3 * g2 * fraction)))
+ val q2 = math.min(1, fraction + g1 + math.sqrt(g1 * g1 + 2 * g1 * fraction))
+
+ val x = rng.nextUniform(0.0, 1.0)
+ if (x < q1) {
+ stratum.incrNumAccepted()
+ } else if (x < q2) {
+ stratum.addToWaitList(x)
+ }
+ stratum.q1 = Some(q1)
+ stratum.q2 = Some(q2)
+ }
+ stratum.incrNumItems()
+ result
+ }
+ }
+
+ /**
+ * Returns the function used by aggregate to combine results from different partitions, as
+ * returned by seqOp.
+ */
+ def getCombOp[K](): (Result[K], Result[K]) => Result[K] = {
+ (r1: Result[K], r2: Result[K]) => {
+ // take union of both key sets in case one partition doesn't contain all keys
+ val keyUnion = r1.resultMap.keys.toSet.union(r2.resultMap.keys.toSet)
+
+ // Use r2 to keep the combined result since r1 is usual empty
+ for (key <- keyUnion) {
+ val entry1 = r1.resultMap.get(key)
+ val entry2 = r2.resultMap.get(key)
+ if (entry2.isEmpty && entry1.isDefined) {
+ r2.resultMap += (key -> entry1.get)
+ } else if (entry1.isDefined && entry2.isDefined) {
+ entry2.get.addToWaitList(entry1.get.waitList)
+ entry2.get.incrNumAccepted(entry1.get.numAccepted)
+ entry2.get.incrNumItems(entry1.get.numItems)
+ }
+ }
+ r2
+ }
+ }
+
+ /**
+ * Given the result returned by the aggregate function, we need to determine the threshold used
+ * to accept items to generate the exact sample size.
+ */
+ def computeThresholdByKey[K](finalResult: Map[K, Stratum], fractionByKey: (K => Double)):
--- End diff --
Similarly, let's change the type of fractionByKey to Map[K, Double]
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by falaki <gi...@git.apache.org>.
Github user falaki commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r14688624
--- Diff: core/src/test/scala/org/apache/spark/rdd/PairRDDFunctionsSuite.scala ---
@@ -83,6 +83,120 @@ class PairRDDFunctionsSuite extends FunSuite with SharedSparkContext {
assert(valuesFor2.toList.sorted === List(1))
}
+ test("sampleByKey") {
+ def stratifier (fractionPositive: Double) = {
+ (x: Int) => if (x % 10 < (10 * fractionPositive).toInt) "1" else "0"
+ }
+
+ def checkSize(exact: Boolean, withReplacement: Boolean,
+ expected: Long, actual: Long, p: Double): Boolean = {
+ if (exact) {
+ return expected == actual
+ }
+ val stdev = if (withReplacement) math.sqrt(expected) else math.sqrt(expected * p * (1 - p))
+ // Very forgiving margin since we're dealing with very small sample sizes most of the time
+ math.abs(actual - expected) <= 6 * stdev
+ }
+
+ // Without replacement validation
+ def takeSampleAndValidateBernoulli(stratifiedData: RDD[(String, Int)],
+ exact: Boolean,
+ samplingRate: Double,
+ seed: Long,
+ n: Long) = {
+ val expectedSampleSize = stratifiedData.countByKey().mapValues(count =>
+ math.ceil(count * samplingRate).toInt)
+ val fractions = Map("1" -> samplingRate, "0" -> samplingRate)
+ val sample = stratifiedData.sampleByKey(false, fractions, exact, seed)
+ val sampleCounts = sample.countByKey()
+ val takeSample = sample.collect()
+ assert(sampleCounts.forall({case(k,v) =>
+ checkSize(exact, false, expectedSampleSize(k), v, samplingRate)}))
+ assert(takeSample.size === takeSample.toSet.size)
+ assert(takeSample.forall(x => 1 <= x._2 && x._2 <= n), s"elements not in [1, $n]")
+ }
+
+ // With replacement validation
+ def takeSampleAndValidatePoisson(stratifiedData: RDD[(String, Int)],
+ exact: Boolean,
+ samplingRate: Double,
+ seed: Long,
+ n: Long) = {
+ val expectedSampleSize = stratifiedData.countByKey().mapValues(count =>
+ math.ceil(count * samplingRate).toInt)
+ val fractions = Map("1" -> samplingRate, "0" -> samplingRate)
+ val sample = stratifiedData.sampleByKey(true, fractions, exact, seed)
+ val sampleCounts = sample.countByKey()
+ val takeSample = sample.collect()
+ assert(sampleCounts.forall({case(k,v) =>
+ checkSize(exact, true, expectedSampleSize(k), v, samplingRate)}))
+ val groupedByKey = takeSample.groupBy({case(k, v) => k})
--- End diff --
And here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1025#issuecomment-45571805
Build finished. All automated tests passed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1025#issuecomment-46389593
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1025#issuecomment-48419506
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16439/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r14694394
--- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
@@ -0,0 +1,311 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util.random
+
+import scala.collection.Map
+import scala.collection.mutable.{ArrayBuffer, HashMap, Map => MMap}
+
+import org.apache.commons.math3.random.RandomDataGenerator
+import org.apache.spark.Logging
+import org.apache.spark.rdd.RDD
+
+/**
+ * Auxiliary functions and data structures for the sampleByKey method in PairRDDFunctions.
+ *
+ * Essentially, when exact sample size is necessary, we make additional passes over the RDD to
+ * compute the exact threshold value to use for each stratum to guarantee exact sample size with
+ * high probability. This is achieved by maintaining a waitlist of size O(log(s)), where s is the
+ * desired sample size for each stratum.
+ *
+ * Like in simple random sampling, we generate a random value for each item from the
+ * uniform distribution [0.0, 1.0]. All items with values <= min(values of items in the waitlist)
+ * are accepted into the sample instantly. The threshold for instant accept is designed so that
+ * s - numAccepted = O(log(s)), where s is again the desired sample size. Thus, by maintaining a
+ * waitlist size = O(log(s)), we will be able to create a sample of the exact size s by adding
+ * a portion of the waitlist to the set of items that are instantly accepted. The exact threshold
+ * is computed by sorting the values in the waitlist and picking the value at (s - numAccepted).
+ *
+ * Note that since we use the same seed for the RNG when computing the thresholds and the actual
+ * sample, our computed thresholds are guaranteed to produce the desired sample size.
+ *
+ * For more theoretical background on the sampling techniques used here, please refer to
+ * http://jmlr.org/proceedings/papers/v28/meng13a.html
+ */
+
+private[spark] object StratifiedSampler extends Logging {
+
+ /**
+ * Count the number of items instantly accepted and generate the waitlist for each stratum.
+ *
+ * This is only invoked when exact sample size is required.
+ */
+ def getCounts[K, V](rdd: RDD[(K, V)],
+ withReplacement: Boolean,
+ fractions: Map[K, Double],
+ counts: Option[Map[K, Long]],
+ seed: Long): MMap[K, Stratum] = {
+ val combOp = getCombOp[K]
+ val mappedPartitionRDD = rdd.mapPartitionsWithIndex({ case (partition, iter) =>
+ val zeroU: MMap[K, Stratum] = new HashMap[K, Stratum]()
+ val rng = new RandomDataGenerator()
+ rng.reSeed(seed + partition)
+ val seqOp = getSeqOp(withReplacement, fractions, rng, counts)
+ Iterator(iter.aggregate(zeroU)(seqOp, combOp))
+ }, preservesPartitioning=true)
+ mappedPartitionRDD.reduce(combOp)
+ }
+
+ /**
+ * Returns the function used by aggregate to collect sampling statistics for each partition.
+ */
+ def getSeqOp[K, V](withReplacement: Boolean,
+ fractions: Map[K, Double],
+ rng: RandomDataGenerator,
+ counts: Option[Map[K, Long]]): (MMap[K, Stratum], (K, V)) => MMap[K, Stratum] = {
+ val delta = 5e-5
+ (result: MMap[K, Stratum], item: (K, V)) => {
+ val key = item._1
+ val fraction = fractions(key)
+ if (!result.contains(key)) {
+ result += (key -> new Stratum())
+ }
+ val stratum = result(key)
+
+ if (withReplacement) {
+ // compute acceptBound and waitListBound only if they haven't been computed already
+ // since they don't change from iteration to iteration.
+ // TODO change this to the streaming version
+ if (stratum.areBoundsEmpty) {
+ val n = counts.get(key)
+ val sampleSize = math.ceil(n * fraction).toLong
+ val lmbd1 = PoissonBounds.getLowerBound(sampleSize)
+ val minCount = PoissonBounds.getMinCount(lmbd1)
+ val lmbd2 = if (lmbd1 == 0) {
+ PoissonBounds.getUpperBound(sampleSize)
+ } else {
+ PoissonBounds.getUpperBound(sampleSize - minCount)
+ }
+ stratum.acceptBound = lmbd1 / n
+ stratum.waitListBound = lmbd2 / n
--- End diff --
Sorry, this was my fault.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by falaki <gi...@git.apache.org>.
Github user falaki commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r14544036
--- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util.random
+
+import scala.collection.{Map, mutable}
+import scala.collection.mutable.ArrayBuffer
+import scala.reflect.ClassTag
+
+import org.apache.commons.math3.random.RandomDataGenerator
+import org.apache.spark.{Logging, SparkContext, TaskContext}
+import org.apache.spark.rdd.RDD
+import org.apache.spark.util.Utils
+import org.apache.spark.util.random.{PoissonBounds => PB}
+
+private[spark] object StratifiedSampler extends Logging {
+
+ /**
+ * A version of {@link #aggregate()} that passes the TaskContext to the function that does
+ * aggregation for each partition. This function avoids creating an extra depth in the RDD
+ * lineage, as opposed to using mapPartitionsWithId, which results in slightly improved run time.
+ */
+ def aggregateWithContext[U: ClassTag, T: ClassTag](zeroValue: U)
+ (rdd: RDD[T],
+ seqOp: ((TaskContext, U), T) => U,
+ combOp: (U, U) => U): U = {
+ val sc: SparkContext = rdd.sparkContext
+ // Clone the zero value since we will also be serializing it as part of tasks
+ var jobResult = Utils.clone(zeroValue, sc.env.closureSerializer.newInstance())
+ // pad seqOp and combOp with taskContext to conform to aggregate's signature in TraversableOnce
+ val paddedSeqOp = (arg1: (TaskContext, U), item: T) => (arg1._1, seqOp(arg1, item))
+ val paddedcombOp = (arg1: (TaskContext, U), arg2: (TaskContext, U)) =>
+ (arg1._1, combOp(arg1._2, arg1._2))
+ val cleanSeqOp = sc.clean(paddedSeqOp)
+ val cleanCombOp = sc.clean(paddedcombOp)
+ val aggregatePartition = (tc: TaskContext, it: Iterator[T]) =>
+ (it.aggregate(tc, zeroValue)(cleanSeqOp, cleanCombOp))._2
+ val mergeResult = (index: Int, taskResult: U) => jobResult = combOp(jobResult, taskResult)
+ sc.runJob(rdd, aggregatePartition, mergeResult)
+ jobResult
+ }
+
+ /**
+ * Returns the function used by aggregate to collect sampling statistics for each partition.
+ */
+ def getSeqOp[K, V](withReplacement: Boolean,
+ fractionByKey: (K => Double),
+ counts: Option[Map[K, Long]]): ((TaskContext, Result[K]),(K, V)) => Result[K] = {
+ val delta = 5e-5
+ (U: (TaskContext, Result[K]), item: (K, V)) => {
+ val result = U._2
+ val tc = U._1
+ val rng = result.getRand(tc.partitionId)
+ val fraction = fractionByKey(item._1)
+ val stratum = result.getEntry(item._1)
+ if (withReplacement) {
+ // compute q1 and q2 only if they haven't been computed already
+ // since they don't change from iteration to iteration.
+ // TODO change this to the streaming version
+ if (stratum.q1.isEmpty || stratum.q2.isEmpty) {
+ val n = counts.get(item._1)
+ val s = math.ceil(n * fraction).toLong
+ val lmbd1 = PB.getLowerBound(s)
+ val minCount = PB.getMinCount(lmbd1)
+ val lmbd2 = if (lmbd1 == 0) PB.getUpperBound(s) else PB.getUpperBound(s - minCount)
+ val q1 = lmbd1 / n
+ val q2 = lmbd2 / n
+ stratum.q1 = Some(q1)
+ stratum.q2 = Some(q2)
+ }
+ val x1 = if (stratum.q1.get == 0) 0L else rng.nextPoisson(stratum.q1.get)
+ if (x1 > 0) {
+ stratum.incrNumAccepted(x1)
+ }
+ val x2 = rng.nextPoisson(stratum.q2.get).toInt
+ if (x2 > 0) {
+ stratum.addToWaitList(ArrayBuffer.fill(x2)(rng.nextUniform(0.0, 1.0)))
+ }
+ } else {
+ // We use the streaming version of the algorithm for sampling without replacement.
+ // Hence, q1 and q2 change on every iteration.
+ val g1 = - math.log(delta) / stratum.numItems
+ val g2 = (2.0 / 3.0) * g1
+ val q1 = math.max(0, fraction + g2 - math.sqrt((g2 * g2 + 3 * g2 * fraction)))
+ val q2 = math.min(1, fraction + g1 + math.sqrt(g1 * g1 + 2 * g1 * fraction))
+
+ val x = rng.nextUniform(0.0, 1.0)
+ if (x < q1) {
+ stratum.incrNumAccepted()
+ } else if (x < q2) {
+ stratum.addToWaitList(x)
+ }
+ stratum.q1 = Some(q1)
+ stratum.q2 = Some(q2)
+ }
+ stratum.incrNumItems()
+ result
+ }
+ }
+
+ /**
+ * Returns the function used by aggregate to combine results from different partitions, as
+ * returned by seqOp.
+ */
+ def getCombOp[K](): (Result[K], Result[K]) => Result[K] = {
+ (r1: Result[K], r2: Result[K]) => {
+ // take union of both key sets in case one partition doesn't contain all keys
+ val keyUnion = r1.resultMap.keys.toSet.union(r2.resultMap.keys.toSet)
+
+ // Use r2 to keep the combined result since r1 is usual empty
+ for (key <- keyUnion) {
+ val entry1 = r1.resultMap.get(key)
+ val entry2 = r2.resultMap.get(key)
+ if (entry2.isEmpty && entry1.isDefined) {
+ r2.resultMap += (key -> entry1.get)
+ } else if (entry1.isDefined && entry2.isDefined) {
+ entry2.get.addToWaitList(entry1.get.waitList)
+ entry2.get.incrNumAccepted(entry1.get.numAccepted)
+ entry2.get.incrNumItems(entry1.get.numItems)
+ }
+ }
+ r2
+ }
+ }
+
+ /**
+ * Given the result returned by the aggregate function, we need to determine the threshold used
+ * to accept items to generate the exact sample size.
+ */
+ def computeThresholdByKey[K](finalResult: Map[K, Stratum], fractionByKey: (K => Double)):
+ (K => Double) = {
+ val thresholdByKey = new mutable.HashMap[K, Double]()
+ for ((key, stratum) <- finalResult) {
+ val fraction = fractionByKey(key)
+ val s = math.ceil(stratum.numItems * fraction).toLong
+ if (stratum.numAccepted > s) {
+ logWarning("Pre-accepted too many")
+ thresholdByKey += (key -> stratum.q1.get)
+ } else {
+ val numWaitListAccepted = (s - stratum.numAccepted).toInt
+ if (numWaitListAccepted >= stratum.waitList.size) {
+ logWarning("WaitList too short")
+ thresholdByKey += (key -> stratum.q2.get)
+ } else {
+ thresholdByKey += (key -> stratum.waitList.sorted.apply(numWaitListAccepted))
+ }
+ }
+ }
+ thresholdByKey
+ }
+
+ /**
+ * Return the per partition sampling function used for sampling without replacement.
+ *
+ * When exact sample size is required, we make an additional pass over the RDD to determine the
+ * exact sampling rate that guarantees sample size with high confidence.
+ *
+ * The sampling function has a unique seed per partition.
+ */
+ def getBernoulliSamplingFunction[K, V](rdd: RDD[(K, V)],
+ fractionByKey: K => Double,
--- End diff --
Similarly, let's change the type of fractionByKey to Map[K, Double]
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1025#issuecomment-50380482
QA tests have started for PR 1025. This patch merges cleanly. <br>View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17296/consoleFull
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by dorx <gi...@git.apache.org>.
Github user dorx commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r14907202
--- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
@@ -0,0 +1,311 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util.random
+
+import scala.collection.Map
+import scala.collection.mutable.{ArrayBuffer, HashMap, Map => MMap}
+
+import org.apache.commons.math3.random.RandomDataGenerator
+import org.apache.spark.Logging
+import org.apache.spark.rdd.RDD
+
+/**
+ * Auxiliary functions and data structures for the sampleByKey method in PairRDDFunctions.
+ *
+ * Essentially, when exact sample size is necessary, we make additional passes over the RDD to
+ * compute the exact threshold value to use for each stratum to guarantee exact sample size with
+ * high probability. This is achieved by maintaining a waitlist of size O(log(s)), where s is the
+ * desired sample size for each stratum.
+ *
+ * Like in simple random sampling, we generate a random value for each item from the
+ * uniform distribution [0.0, 1.0]. All items with values <= min(values of items in the waitlist)
+ * are accepted into the sample instantly. The threshold for instant accept is designed so that
+ * s - numAccepted = O(log(s)), where s is again the desired sample size. Thus, by maintaining a
+ * waitlist size = O(log(s)), we will be able to create a sample of the exact size s by adding
+ * a portion of the waitlist to the set of items that are instantly accepted. The exact threshold
+ * is computed by sorting the values in the waitlist and picking the value at (s - numAccepted).
+ *
+ * Note that since we use the same seed for the RNG when computing the thresholds and the actual
+ * sample, our computed thresholds are guaranteed to produce the desired sample size.
+ *
+ * For more theoretical background on the sampling techniques used here, please refer to
+ * http://jmlr.org/proceedings/papers/v28/meng13a.html
+ */
+
+private[spark] object StratifiedSampler extends Logging {
+
+ /**
+ * Count the number of items instantly accepted and generate the waitlist for each stratum.
+ *
+ * This is only invoked when exact sample size is required.
+ */
+ def getCounts[K, V](rdd: RDD[(K, V)],
+ withReplacement: Boolean,
+ fractions: Map[K, Double],
+ counts: Option[Map[K, Long]],
+ seed: Long): MMap[K, Stratum] = {
+ val combOp = getCombOp[K]
+ val mappedPartitionRDD = rdd.mapPartitionsWithIndex({ case (partition, iter) =>
+ val zeroU: MMap[K, Stratum] = new HashMap[K, Stratum]()
+ val rng = new RandomDataGenerator()
+ rng.reSeed(seed + partition)
+ val seqOp = getSeqOp(withReplacement, fractions, rng, counts)
+ Iterator(iter.aggregate(zeroU)(seqOp, combOp))
+ }, preservesPartitioning=true)
+ mappedPartitionRDD.reduce(combOp)
+ }
+
+ /**
+ * Returns the function used by aggregate to collect sampling statistics for each partition.
+ */
+ def getSeqOp[K, V](withReplacement: Boolean,
+ fractions: Map[K, Double],
+ rng: RandomDataGenerator,
+ counts: Option[Map[K, Long]]): (MMap[K, Stratum], (K, V)) => MMap[K, Stratum] = {
+ val delta = 5e-5
+ (result: MMap[K, Stratum], item: (K, V)) => {
+ val key = item._1
+ val fraction = fractions(key)
+ if (!result.contains(key)) {
+ result += (key -> new Stratum())
+ }
+ val stratum = result(key)
+
+ if (withReplacement) {
+ // compute acceptBound and waitListBound only if they haven't been computed already
+ // since they don't change from iteration to iteration.
+ // TODO change this to the streaming version
+ if (stratum.areBoundsEmpty) {
+ val n = counts.get(key)
+ val sampleSize = math.ceil(n * fraction).toLong
+ val lmbd1 = PoissonBounds.getLowerBound(sampleSize)
+ val minCount = PoissonBounds.getMinCount(lmbd1)
+ val lmbd2 = if (lmbd1 == 0) {
+ PoissonBounds.getUpperBound(sampleSize)
+ } else {
+ PoissonBounds.getUpperBound(sampleSize - minCount)
+ }
+ stratum.acceptBound = lmbd1 / n
+ stratum.waitListBound = lmbd2 / n
+ }
+ val acceptBound = stratum.acceptBound
+ val copiesAccepted = if (acceptBound == 0.0) 0L else rng.nextPoisson(acceptBound)
+ if (copiesAccepted > 0) {
+ stratum.incrNumAccepted(copiesAccepted)
+ }
+ val copiesWaitlisted = rng.nextPoisson(stratum.waitListBound).toInt
+ if (copiesWaitlisted > 0) {
+ stratum.addToWaitList(ArrayBuffer.fill(copiesWaitlisted)(rng.nextUniform(0.0, 1.0)))
+ }
+ } else {
+ // We use the streaming version of the algorithm for sampling without replacement to avoid
+ // using an extra pass over the RDD for computing the count.
+ // Hence, acceptBound and waitListBound change on every iteration.
+ val gamma1 = - math.log(delta) / stratum.numItems
+ val gamma2 = (2.0 / 3.0) * gamma1
+ stratum.acceptBound = math.max(0,
+ fraction + gamma2 - math.sqrt(gamma2 * gamma2 + 3 * gamma2 * fraction))
+ stratum.waitListBound = math.min(1,
+ fraction + gamma1 + math.sqrt(gamma1 * gamma1 + 2 * gamma1 * fraction))
+
+ val x = rng.nextUniform(0.0, 1.0)
+ if (x < stratum.acceptBound) {
+ stratum.incrNumAccepted()
+ } else if (x < stratum.waitListBound) {
+ stratum.addToWaitList(x)
+ }
+ }
+ stratum.incrNumItems()
+ result
+ }
+ }
+
+ /**
+ * Returns the function used combine results returned by seqOp from different partitions.
+ */
+ def getCombOp[K]: (MMap[K, Stratum], MMap[K, Stratum]) => MMap[K, Stratum] = {
+ (result1: MMap[K, Stratum], result2: MMap[K, Stratum]) => {
+ // take union of both key sets in case one partition doesn't contain all keys
+ for (key <- result1.keySet.union(result2.keySet)) {
+ // Use result2 to keep the combined result since r1 is usual empty
+ val entry1 = result1.get(key)
+ if (result2.contains(key)) {
+ result2(key).merge(entry1)
+ } else {
+ if (entry1.isDefined) {
+ result2 += (key -> entry1.get)
+ }
+ }
+ }
+ result2
+ }
+ }
+
+ /**
+ * Given the result returned by getCounts, determine the threshold for accepting items to
+ * generate exact sample size.
+ *
+ * To do so, we compute sampleSize = math.ceil(size * samplingRate) for each stratum and compare
+ * it to the number of items that were accepted instantly and the number of items in the waitlist
+ * for that stratum. Most of the time, numAccepted <= sampleSize <= (numAccepted + numWaitlisted),
+ * which means we need to sort the elements in the waitlist by their associated values in order
+ * to find the value T s.t. |{elements in the stratum whose associated values <= T}| = sampleSize.
+ * Note that all elements in the waitlist have values >= bound for instant accept, so a T value
+ * in the waitlist range would allow all elements that were instantly accepted on the first pass
+ * to be included in the sample.
+ */
+ def computeThresholdByKey[K](finalResult: Map[K, Stratum],
+ fractions: Map[K, Double]): Map[K, Double] = {
+ val thresholdByKey = new HashMap[K, Double]()
+ for ((key, stratum) <- finalResult) {
+ val sampleSize = math.ceil(stratum.numItems * fractions(key)).toLong
+ if (stratum.numAccepted > sampleSize) {
+ logWarning("Pre-accepted too many")
+ thresholdByKey += (key -> stratum.acceptBound)
+ } else {
+ val numWaitListAccepted = (sampleSize - stratum.numAccepted).toInt
+ if (numWaitListAccepted >= stratum.waitList.size) {
+ logWarning("WaitList too short")
+ thresholdByKey += (key -> stratum.waitListBound)
+ } else {
+ thresholdByKey += (key -> stratum.waitList.sorted.apply(numWaitListAccepted))
+ }
+ }
+ }
+ thresholdByKey
+ }
+
+ /**
+ * Return the per partition sampling function used for sampling without replacement.
+ *
+ * When exact sample size is required, we make an additional pass over the RDD to determine the
+ * exact sampling rate that guarantees sample size with high confidence.
+ *
+ * The sampling function has a unique seed per partition.
+ */
+ def getBernoulliSamplingFunction[K, V](rdd: RDD[(K, V)],
+ fractions: Map[K, Double],
+ exact: Boolean,
+ seed: Long): (Int, Iterator[(K, V)]) => Iterator[(K, V)] = {
+ var samplingRateByKey = fractions
+ if (exact) {
+ // determine threshold for each stratum and resample
+ val finalResult = getCounts(rdd, false, fractions, None, seed)
+ samplingRateByKey = computeThresholdByKey(finalResult, fractions)
+ }
+ (idx: Int, iter: Iterator[(K, V)]) => {
+ val rng = new RandomDataGenerator
+ rng.reSeed(seed + idx)
+ iter.filter(t => rng.nextUniform(0.0, 1.0) < samplingRateByKey(t._1))
+ }
+ }
+
+ /**
+ * Return the per partition sampling function used for sampling with replacement.
+ *
+ * When exact sample size is required, we make two additional passed over the RDD to determine
+ * the exact sampling rate that guarantees sample size with high confidence. The first pass
+ * counts the number of items in each stratum (group of items with the same key) in the RDD, and
+ * the second pass uses the counts to determine exact sampling rates.
+ *
+ * The sampling function has a unique seed per partition.
+ */
+ def getPoissonSamplingFunction[K, V](rdd: RDD[(K, V)],
+ fractions: Map[K, Double],
+ exact: Boolean,
+ counts: Option[Map[K, Long]],
+ seed: Long): (Int, Iterator[(K, V)]) => Iterator[(K, V)] = {
+ // TODO implement the streaming version of sampling w/ replacement that doesn't require counts
+ if (exact) {
+ val finalResult = getCounts(rdd, true, fractions, counts, seed)
+ val thresholdByKey = computeThresholdByKey(finalResult, fractions)
+ (idx: Int, iter: Iterator[(K, V)]) => {
+ val rng = new RandomDataGenerator()
+ rng.reSeed(seed + idx)
+ iter.flatMap { item =>
+ val key = item._1
+ val acceptBound = finalResult(key).acceptBound
+ val copiesAccepted = if (acceptBound == 0) 0L else rng.nextPoisson(acceptBound)
+ val copiesWailisted = rng.nextPoisson(finalResult(key).waitListBound).toInt
+ val copiesInSample = copiesAccepted +
+ (0 until copiesWailisted).count(i => rng.nextUniform(0.0, 1.0) < thresholdByKey(key))
+ if (copiesInSample > 0) {
+ Iterator.fill(copiesInSample.toInt)(item)
+ } else {
+ Iterator.empty
+ }
+ }
+ }
+ } else {
+ (idx: Int, iter: Iterator[(K, V)]) => {
+ val rng = new RandomDataGenerator()
+ rng.reSeed(seed + idx)
+ iter.flatMap { item =>
+ val count = rng.nextPoisson(fractions(item._1)).toInt
+ if (count > 0) {
+ Iterator.fill(count)(item)
+ } else {
+ Iterator.empty
+ }
+ }
+ }
+ }
+ }
+}
+
+/**
+ * Object used by seqOp to keep track of the number of items accepted and items waitlisted per
+ * stratum, as well as the bounds for accepting and waitlisting items.
+ *
+ * `[random]` here is necessary since it's in the return type signature of seqOp defined above
--- End diff --
Wanna to try to compile it for me without `[random]`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by mateiz <gi...@git.apache.org>.
Github user mateiz commented on the pull request:
https://github.com/apache/spark/pull/1025#issuecomment-50109108
Well, the other "sample" functions are already approximate anyway. I kind of like this here because it conveys that it's more expensive. The other thing is that if we want the Exact one to be experimental, we can't just make it a parameter.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1025#issuecomment-45964253
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1025#issuecomment-46613650
Merged build finished.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by falaki <gi...@git.apache.org>.
Github user falaki commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r14541686
--- Diff: core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala ---
@@ -195,6 +193,40 @@ class PairRDDFunctions[K, V](self: RDD[(K, V)])
}
/**
+ * Return a subset of this RDD sampled by key (via stratified sampling).
+ *
+ * If exact set to true, we guarantee, with high probability, a sample size =
+ * math.ceil(fraction * S_i), where S_i is the size of the ith stratum (collection of entries
+ * that share the same key). When sampling without replacement, we need one additional pass over
+ * the RDD to guarantee sample size with a 99.99% confidence; when sampling with replacement, we
+ * need two additional passes.
+ *
+ * @param withReplacement whether to sample with or without replacement
+ * @param fractions map of specific keys to sampling rates
+ * @param seed seed for the random number generator
+ * @param exact whether sample size needs to be exactly math.ceil(fraction * size) per stratum
+ * @return RDD containing the sampled subset
+ */
+ def sampleByKey(withReplacement: Boolean,
+ fractions: Map[K, Double],
+ exact: Boolean = true,
+ seed: Long = Utils.random.nextLong): RDD[(K, V)]= {
+
+ require(fractions.forall({case(k, v) => v >= 0.0}), "Invalid sampling rates.")
--- End diff --
Could be just:
```forall{case(k, v) => v >= 0.0}```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1025#issuecomment-48388110
Merged build finished.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by falaki <gi...@git.apache.org>.
Github user falaki commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r14622823
--- Diff: core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala ---
@@ -195,6 +193,37 @@ class PairRDDFunctions[K, V](self: RDD[(K, V)])
}
/**
+ * Return a subset of this RDD sampled by key (via stratified sampling).
+ *
+ * If exact set to true, we guarantee, with high probability, a sample size =
+ * math.ceil(fraction * S_i), where S_i is the size of the ith stratum (collection of entries
+ * that share the same key). When sampling without replacement, we need one additional pass over
+ * the RDD to guarantee sample size with a 99.99% confidence; when sampling with replacement, we
+ * need two additional passes.
+ *
+ * @param withReplacement whether to sample with or without replacement
+ * @param fractions map of specific keys to sampling rates
+ * @param seed seed for the random number generator
+ * @param exact whether sample size needs to be exactly math.ceil(fraction * size) per stratum
+ * @return RDD containing the sampled subset
+ */
+ def sampleByKey(withReplacement: Boolean,
+ fractions: Map[K, Double],
+ exact: Boolean = true,
+ seed: Long = Utils.random.nextLong): RDD[(K, V)]= {
+
+ require(fractions.forall {case(k, v) => v >= 0.0}, "Invalid sampling rates.")
+
+ val samplingFunc = if (withReplacement) {
+ val counts = if (exact) Some(this.countByKey()) else None
+ StratifiedSampler.getPoissonSamplingFunction(self, fractions, exact, counts, seed)
+ } else {
+ StratifiedSampler.getBernoulliSamplingFunction(self, fractions, exact, seed)
--- End diff --
Remove extra indent.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by dorx <gi...@git.apache.org>.
Github user dorx commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r14688633
--- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
@@ -0,0 +1,310 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util.random
+
+import scala.collection.Map
+import scala.collection.mutable.{ArrayBuffer, HashMap, Map => MMap}
+
+import org.apache.commons.math3.random.RandomDataGenerator
+import org.apache.spark.Logging
+import org.apache.spark.rdd.RDD
+
+/**
+ * Auxiliary functions and data structures for the sampleByKey method in PairRDDFunctions.
+ *
+ * Essentially, when exact sample size is necessary, we make additional passes over the RDD to
+ * compute the exact threshold value to use for each stratum to guarantee exact sample size with
+ * high probability. This is achieved by maintaining a waitlist of size O(log(s)), where s is the
+ * desired sample size for each stratum.
+ *
+ * Like in simple random sampling, we generate a random value for each item from the
+ * uniform distribution [0.0, 1.0]. All items with values <= min(values of items in the waitlist)
+ * are accepted into the sample instantly. The threshold for instant accept is designed so that
+ * s - numAccepted = O(log(s)), where s is again the desired sample size. Thus, by maintaining a
+ * waitlist size = O(log(s)), we will be able to create a sample of the exact size s by adding
+ * a portion of the waitlist to the set of items that are instantly accepted. The exact threshold
+ * is computed by sorting the values in the waitlist and picking the value at (s - numAccepted).
+ *
+ * Note that since we use the same seed for the RNG when computing the thresholds and the actual
+ * sample, our computed thresholds are guaranteed to produce the desired sample size.
+ *
+ * For more theoretical background on the sampling techniques used here, please refer to
+ * http://jmlr.org/proceedings/papers/v28/meng13a.html
+ */
+
+private[spark] object StratifiedSampler extends Logging {
+
+ /**
+ * Count the number of items instantly accepted and generate the waitlist for each stratum.
+ *
+ * This is only invoked when exact sample size is required.
+ */
+ def getCounts[K, V](rdd: RDD[(K, V)],
+ withReplacement: Boolean,
+ fractions: Map[K, Double],
+ counts: Option[Map[K, Long]],
+ seed: Long): MMap[K, Stratum] = {
+ val combOp = getCombOp[K]
+ val mappedPartitionRDD = rdd.mapPartitionsWithIndex({ case (partition, iter) =>
+ val zeroU: MMap[K, Stratum] = new HashMap[K, Stratum]()
+ val rng = new RandomDataGenerator()
+ rng.reSeed(seed + partition)
+ val seqOp = getSeqOp(withReplacement, fractions, rng, counts)
+ Iterator(iter.aggregate(zeroU)(seqOp, combOp))
+ }, preservesPartitioning=true)
+ mappedPartitionRDD.reduce(combOp)
+ }
+
+ /**
+ * Returns the function used by aggregate to collect sampling statistics for each partition.
+ */
+ def getSeqOp[K, V](withReplacement: Boolean,
+ fractions: Map[K, Double],
+ rng: RandomDataGenerator,
+ counts: Option[Map[K, Long]]): (MMap[K, Stratum], (K, V)) => MMap[K, Stratum] = {
+ val delta = 5e-5
+ (result: MMap[K, Stratum], item: (K, V)) => {
+ val key = item._1
+ val fraction = fractions(key)
+ if (!result.contains(key)) {
+ result += (key -> new Stratum())
+ }
+ val stratum = result(key)
+
+ if (withReplacement) {
+ // compute acceptBound and waitListBound only if they haven't been computed already
+ // since they don't change from iteration to iteration.
+ // TODO change this to the streaming version
+ if (stratum.areBoundsEmpty) {
+ val n = counts.get(key)
+ val sampleSize = math.ceil(n * fraction).toLong
+ val lmbd1 = PoissonBounds.getLowerBound(sampleSize)
+ val minCount = PoissonBounds.getMinCount(lmbd1)
+ val lmbd2 = if (lmbd1 == 0) {
+ PoissonBounds.getUpperBound(sampleSize)
+ } else {
+ PoissonBounds.getUpperBound(sampleSize - minCount)
+ }
+ stratum.acceptBound = lmbd1 / n
+ stratum.waitListBound = lmbd2 / n
+ }
+ val acceptBound = stratum.acceptBound
+ val copiesAccepted = if (acceptBound == 0.0) 0L else rng.nextPoisson(acceptBound)
+ if (copiesAccepted > 0) {
+ stratum.incrNumAccepted(copiesAccepted)
+ }
+ val copiesWaitlisted = rng.nextPoisson(stratum.waitListBound).toInt
+ if (copiesWaitlisted > 0) {
+ stratum.addToWaitList(ArrayBuffer.fill(copiesWaitlisted)(rng.nextUniform(0.0, 1.0)))
+ }
+ } else {
+ // We use the streaming version of the algorithm for sampling without replacement to avoid
+ // using an extra pass over the RDD for computing the count.
+ // Hence, acceptBound and waitListBound change on every iteration.
+ val g1 = - math.log(delta) / stratum.numItems // gamma1
+ val g2 = (2.0 / 3.0) * g1 // gamma 2
--- End diff --
I was trying to make the next two lines fit on the same lines, but it doesn't actually look too bad breaking them into two lines each.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1025#issuecomment-48418905
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1025#issuecomment-48386807
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by falaki <gi...@git.apache.org>.
Github user falaki commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r14622781
--- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
@@ -0,0 +1,335 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util.random
+
+import scala.collection.Map
+import scala.collection.mutable.{ArrayBuffer, HashMap, Map => MMap}
+import scala.reflect.ClassTag
+
+import org.apache.commons.math3.random.RandomDataGenerator
+import org.apache.spark.{Logging, SparkContext, TaskContext}
+import org.apache.spark.rdd.RDD
+import org.apache.spark.util.Utils
+
+/**
+ * Auxiliary functions and data structures for the sampleByKey method in PairRDDFunctions.
+ *
+ * For more theoretical background on the sampling technqiues used here, please refer to
+ * http://jmlr.org/proceedings/papers/v28/meng13a.html
+ */
+private[spark] object StratifiedSampler extends Logging {
+
+ /**
+ * A version of {@link #aggregate()} that passes the TaskContext to the function that does
+ * aggregation for each partition. This function avoids creating an extra depth in the RDD
+ * lineage, as opposed to using mapPartitionsWithIndex, which results in slightly improved
+ * run time.
+ */
+ def aggregateWithContext[U: ClassTag, T: ClassTag](zeroValue: U)
+ (rdd: RDD[T],
+ seqOp: ((TaskContext, U), T) => U,
+ combOp: (U, U) => U): U = {
+ val sc: SparkContext = rdd.sparkContext
+ // Clone the zero value since we will also be serializing it as part of tasks
+ var jobResult = Utils.clone(zeroValue, sc.env.closureSerializer.newInstance())
+ // pad seqOp and combOp with taskContext to conform to aggregate's signature in TraversableOnce
+ val paddedSeqOp = (arg1: (TaskContext, U), item: T) => (arg1._1, seqOp(arg1, item))
+ val paddedCombOp = (arg1: (TaskContext, U), arg2: (TaskContext, U)) =>
+ (arg1._1, combOp(arg1._2, arg1._2))
+ val cleanSeqOp = sc.clean(paddedSeqOp)
+ val cleanCombOp = sc.clean(paddedCombOp)
+ val aggregatePartition = (tc: TaskContext, it: Iterator[T]) =>
+ (it.aggregate(tc, zeroValue)(cleanSeqOp, cleanCombOp))._2
+ val mergeResult = (index: Int, taskResult: U) => jobResult = combOp(jobResult, taskResult)
+ sc.runJob(rdd, aggregatePartition, mergeResult)
+ jobResult
+ }
+
+ /**
+ * Returns the function used by aggregate to collect sampling statistics for each partition.
+ */
+ def getSeqOp[K, V](withReplacement: Boolean,
+ fractions: Map[K, Double],
+ counts: Option[Map[K, Long]]): ((TaskContext, Result[K]), (K, V)) => Result[K] = {
+ val delta = 5e-5
+ (output: (TaskContext, Result[K]), item: (K, V)) => {
+ val result = output._2
+ val tc = output._1
+ val rng = result.getRand(tc.partitionId)
+ val fraction = fractions(item._1)
+ val stratum = result.getEntry(item._1)
+ if (withReplacement) {
+ // compute acceptBound and waitListBound only if they haven't been computed already
+ // since they don't change from iteration to iteration.
+ // TODO change this to the streaming version
+ if (stratum.areBoundsEmpty) {
+ val n = counts.get(item._1)
+ val sampleSize = math.ceil(n * fraction).toLong
+ val lmbd1 = PoissonBounds.getLowerBound(sampleSize)
+ val minCount = PoissonBounds.getMinCount(lmbd1)
+ val lmbd2 = if (lmbd1 == 0) {
+ PoissonBounds.getUpperBound(sampleSize)
+ } else {
+ PoissonBounds.getUpperBound(sampleSize - minCount)
+ }
+ stratum.acceptBound = lmbd1 / n
+ stratum.waitListBound = lmbd2 / n
+ }
+ val x1 = if (stratum.acceptBound == 0.0) 0L else rng.nextPoisson(stratum.acceptBound)
+ if (x1 > 0) {
+ stratum.incrNumAccepted(x1)
+ }
+ val x2 = rng.nextPoisson(stratum.waitListBound).toInt
+ if (x2 > 0) {
+ stratum.addToWaitList(ArrayBuffer.fill(x2)(rng.nextUniform(0.0, 1.0)))
+ }
+ } else {
+ // We use the streaming version of the algorithm for sampling without replacement to avoid
+ // using an extra pass over the RDD for computing the count.
+ // Hence, acceptBound and waitListBound change on every iteration.
+ val g1 = - math.log(delta) / stratum.numItems // gamma1
+ val g2 = (2.0 / 3.0) * g1 // gamma 2
+ stratum.acceptBound = math.max(0, fraction + g2 - math.sqrt((g2 * g2 + 3 * g2 * fraction)))
+ stratum.waitListBound = math.min(1, fraction + g1 + math.sqrt(g1 * g1 + 2 * g1 * fraction))
+
+ val x = rng.nextUniform(0.0, 1.0)
+ if (x < stratum.acceptBound) {
+ stratum.incrNumAccepted()
+ } else if (x < stratum.waitListBound) {
+ stratum.addToWaitList(x)
+ }
+ }
+ stratum.incrNumItems()
+ result
+ }
+ }
+
+ /**
+ * Returns the function used by aggregate to combine results from different partitions, as
+ * returned by seqOp.
+ */
+ def getCombOp[K](): (Result[K], Result[K]) => Result[K] = {
+ (r1: Result[K], r2: Result[K]) => {
+ // take union of both key sets in case one partition doesn't contain all keys
+ val keyUnion = r1.resultMap.keySet.union(r2.resultMap.keySet)
+
+ // Use r2 to keep the combined result since r1 is usual empty
+ for (key <- keyUnion) {
+ val entry1 = r1.resultMap.get(key)
+ if (r2.resultMap.contains(key)) {
+ r2.resultMap(key).merge(entry1)
+ } else {
+ r2.addEntry(key, entry1)
+ }
+ }
+ r2
+ }
+ }
+
+ /**
+ * Given the result returned by the aggregate function, determine the threshold for accepting
+ * items to generate exact sample size.
+ *
+ * To do so, we compute sampleSize = math.ceil(size * samplingRate) for each stratum and compare
+ * it to the number of items that were accepted instantly and the number of items in the waitlist
+ * for that stratum. Most of the time, numAccepted <= sampleSize <= (numAccepted + numWaitlisted),
+ * which means we need to sort the elements in the waitlist by their associated values in order
+ * to find the value T s.t. |{elements in the stratum whose associated values <= T}| = sampleSize.
+ * Note that all elements in the waitlist have values >= bound for instant accept, so a T value
+ * in the waitlist range would allow all elements that were instantly accepted on the first pass
+ * to be included in the sample.
+ */
+ def computeThresholdByKey[K](finalResult: Map[K, Stratum],
+ fractions: Map[K, Double]):
--- End diff --
Style nit: Return type of the function is in a new line. We can join these two lines.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r15534415
--- Diff: core/src/main/scala/org/apache/spark/util/random/SamplingUtils.scala ---
@@ -88,14 +91,73 @@ private[spark] object SamplingUtils {
*/
def computeFractionForSampleSize(sampleSizeLowerBound: Int, total: Long,
withReplacement: Boolean): Double = {
- val fraction = sampleSizeLowerBound.toDouble / total
if (withReplacement) {
- val numStDev = if (sampleSizeLowerBound < 12) 9 else 5
- fraction + numStDev * math.sqrt(fraction / total)
+ PoissonBounds.getUpperBound(sampleSizeLowerBound) / total
} else {
- val delta = 1e-4
- val gamma = - math.log(delta) / total
- math.min(1, fraction + gamma + math.sqrt(gamma * gamma + 2 * gamma * fraction))
+ val fraction = sampleSizeLowerBound.toDouble / total
+ BinomialBounds.getUpperBound(1e-4, total, fraction)
}
}
}
+
+/**
+ * Utility functions that help us determine bounds on adjusted sampling rate to guarantee exact
+ * sample sizes with high confidence when sampling with replacement.
+ */
+private[spark] object PoissonBounds {
+
+ /**
+ * Returns a lambda such that Pr[X > s] is very small, where X ~ Pois(lambda).
+ */
+ def getLowerBound(s: Double): Double = {
+ math.max(s - numStd(s) * math.sqrt(s), 1e-15)
+ }
+
+ /**
+ * Returns a lambda such that Pr[X < s] is very small, where X ~ Pois(lambda).
+ *
+ * @param s sample size
+ */
+ def getUpperBound(s: Double): Double = {
+ math.max(s + numStd(s) * math.sqrt(s), 1e-10)
+ }
+
+ private def numStd(s: Double): Double = {
+ // TODO: Make it tighter.
+ if (s < 6.0) {
+ 12.0
+ } else if (s < 16.0) {
+ 9.0
+ } else {
+ 6.0
+ }
+ }
+}
+
+/**
+ * Utility functions that help us determine bounds on adjusted sampling rate to guarantee exact
+ * sample size with high confidence when sampling without replacement.
+ */
+private[spark] object BinomialBounds {
+
+ val minSamplingRate = 1e-10
+
+ /**
+ * Returns a threshold `p` such that if we conduct n Bernoulli trials with success rate = `p`,
+ * it is very unlikely to have more than `fraction * n` successes.
+ */
+ def getLowerBound(delta: Double, n: Long, fraction: Double): Double = {
+ val gamma = - math.log(delta) / n * (2.0 / 3.0)
+ math.max(minSamplingRate,
--- End diff --
It should be okay to return `0` here. The `minSamplingRate` should be applied to `getUpperBound`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by falaki <gi...@git.apache.org>.
Github user falaki commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r14544843
--- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util.random
+
+import scala.collection.{Map, mutable}
+import scala.collection.mutable.ArrayBuffer
+import scala.reflect.ClassTag
+
+import org.apache.commons.math3.random.RandomDataGenerator
+import org.apache.spark.{Logging, SparkContext, TaskContext}
+import org.apache.spark.rdd.RDD
+import org.apache.spark.util.Utils
+import org.apache.spark.util.random.{PoissonBounds => PB}
+
+private[spark] object StratifiedSampler extends Logging {
+
+ /**
+ * A version of {@link #aggregate()} that passes the TaskContext to the function that does
+ * aggregation for each partition. This function avoids creating an extra depth in the RDD
+ * lineage, as opposed to using mapPartitionsWithId, which results in slightly improved run time.
+ */
+ def aggregateWithContext[U: ClassTag, T: ClassTag](zeroValue: U)
+ (rdd: RDD[T],
+ seqOp: ((TaskContext, U), T) => U,
+ combOp: (U, U) => U): U = {
+ val sc: SparkContext = rdd.sparkContext
+ // Clone the zero value since we will also be serializing it as part of tasks
+ var jobResult = Utils.clone(zeroValue, sc.env.closureSerializer.newInstance())
+ // pad seqOp and combOp with taskContext to conform to aggregate's signature in TraversableOnce
+ val paddedSeqOp = (arg1: (TaskContext, U), item: T) => (arg1._1, seqOp(arg1, item))
+ val paddedcombOp = (arg1: (TaskContext, U), arg2: (TaskContext, U)) =>
+ (arg1._1, combOp(arg1._2, arg1._2))
+ val cleanSeqOp = sc.clean(paddedSeqOp)
+ val cleanCombOp = sc.clean(paddedcombOp)
+ val aggregatePartition = (tc: TaskContext, it: Iterator[T]) =>
+ (it.aggregate(tc, zeroValue)(cleanSeqOp, cleanCombOp))._2
+ val mergeResult = (index: Int, taskResult: U) => jobResult = combOp(jobResult, taskResult)
+ sc.runJob(rdd, aggregatePartition, mergeResult)
+ jobResult
+ }
+
+ /**
+ * Returns the function used by aggregate to collect sampling statistics for each partition.
+ */
+ def getSeqOp[K, V](withReplacement: Boolean,
+ fractionByKey: (K => Double),
+ counts: Option[Map[K, Long]]): ((TaskContext, Result[K]),(K, V)) => Result[K] = {
+ val delta = 5e-5
+ (U: (TaskContext, Result[K]), item: (K, V)) => {
+ val result = U._2
+ val tc = U._1
+ val rng = result.getRand(tc.partitionId)
+ val fraction = fractionByKey(item._1)
+ val stratum = result.getEntry(item._1)
+ if (withReplacement) {
+ // compute q1 and q2 only if they haven't been computed already
+ // since they don't change from iteration to iteration.
+ // TODO change this to the streaming version
+ if (stratum.q1.isEmpty || stratum.q2.isEmpty) {
+ val n = counts.get(item._1)
+ val s = math.ceil(n * fraction).toLong
+ val lmbd1 = PB.getLowerBound(s)
+ val minCount = PB.getMinCount(lmbd1)
+ val lmbd2 = if (lmbd1 == 0) PB.getUpperBound(s) else PB.getUpperBound(s - minCount)
+ val q1 = lmbd1 / n
+ val q2 = lmbd2 / n
+ stratum.q1 = Some(q1)
+ stratum.q2 = Some(q2)
+ }
+ val x1 = if (stratum.q1.get == 0) 0L else rng.nextPoisson(stratum.q1.get)
+ if (x1 > 0) {
+ stratum.incrNumAccepted(x1)
+ }
+ val x2 = rng.nextPoisson(stratum.q2.get).toInt
+ if (x2 > 0) {
+ stratum.addToWaitList(ArrayBuffer.fill(x2)(rng.nextUniform(0.0, 1.0)))
+ }
+ } else {
+ // We use the streaming version of the algorithm for sampling without replacement.
+ // Hence, q1 and q2 change on every iteration.
+ val g1 = - math.log(delta) / stratum.numItems
+ val g2 = (2.0 / 3.0) * g1
+ val q1 = math.max(0, fraction + g2 - math.sqrt((g2 * g2 + 3 * g2 * fraction)))
+ val q2 = math.min(1, fraction + g1 + math.sqrt(g1 * g1 + 2 * g1 * fraction))
+
+ val x = rng.nextUniform(0.0, 1.0)
+ if (x < q1) {
+ stratum.incrNumAccepted()
+ } else if (x < q2) {
+ stratum.addToWaitList(x)
+ }
+ stratum.q1 = Some(q1)
+ stratum.q2 = Some(q2)
+ }
+ stratum.incrNumItems()
+ result
+ }
+ }
+
+ /**
+ * Returns the function used by aggregate to combine results from different partitions, as
+ * returned by seqOp.
+ */
+ def getCombOp[K](): (Result[K], Result[K]) => Result[K] = {
+ (r1: Result[K], r2: Result[K]) => {
+ // take union of both key sets in case one partition doesn't contain all keys
+ val keyUnion = r1.resultMap.keys.toSet.union(r2.resultMap.keys.toSet)
+
+ // Use r2 to keep the combined result since r1 is usual empty
+ for (key <- keyUnion) {
+ val entry1 = r1.resultMap.get(key)
+ val entry2 = r2.resultMap.get(key)
+ if (entry2.isEmpty && entry1.isDefined) {
+ r2.resultMap += (key -> entry1.get)
+ } else if (entry1.isDefined && entry2.isDefined) {
+ entry2.get.addToWaitList(entry1.get.waitList)
+ entry2.get.incrNumAccepted(entry1.get.numAccepted)
+ entry2.get.incrNumItems(entry1.get.numItems)
+ }
+ }
+ r2
+ }
+ }
+
+ /**
+ * Given the result returned by the aggregate function, we need to determine the threshold used
+ * to accept items to generate the exact sample size.
+ */
+ def computeThresholdByKey[K](finalResult: Map[K, Stratum], fractionByKey: (K => Double)):
+ (K => Double) = {
+ val thresholdByKey = new mutable.HashMap[K, Double]()
+ for ((key, stratum) <- finalResult) {
+ val fraction = fractionByKey(key)
+ val s = math.ceil(stratum.numItems * fraction).toLong
+ if (stratum.numAccepted > s) {
+ logWarning("Pre-accepted too many")
+ thresholdByKey += (key -> stratum.q1.get)
+ } else {
+ val numWaitListAccepted = (s - stratum.numAccepted).toInt
+ if (numWaitListAccepted >= stratum.waitList.size) {
+ logWarning("WaitList too short")
+ thresholdByKey += (key -> stratum.q2.get)
+ } else {
+ thresholdByKey += (key -> stratum.waitList.sorted.apply(numWaitListAccepted))
+ }
+ }
+ }
+ thresholdByKey
+ }
+
+ /**
+ * Return the per partition sampling function used for sampling without replacement.
+ *
+ * When exact sample size is required, we make an additional pass over the RDD to determine the
+ * exact sampling rate that guarantees sample size with high confidence.
+ *
+ * The sampling function has a unique seed per partition.
+ */
+ def getBernoulliSamplingFunction[K, V](rdd: RDD[(K, V)],
+ fractionByKey: K => Double,
+ exact: Boolean,
+ seed: Long): (Int, Iterator[(K, V)]) => Iterator[(K, V)] = {
+ var samplingRateByKey = fractionByKey
+ if (exact) {
+ // determine threshold for each stratum and resample
+ val seqOp = StratifiedSampler.getSeqOp[K,V](false, fractionByKey, None)
+ val combOp = StratifiedSampler.getCombOp[K]()
+ val zeroU = new Result[K](Map[K, Stratum](), seed = seed)
+ val finalResult = aggregateWithContext(zeroU)(rdd, seqOp, combOp).resultMap
+ samplingRateByKey = StratifiedSampler.computeThresholdByKey(finalResult, fractionByKey)
+ }
+ (idx: Int, iter: Iterator[(K, V)]) => {
+ val random = new RandomDataGenerator
+ random.reSeed(seed + idx)
+ iter.filter(t => random.nextUniform(0.0, 1.0) < samplingRateByKey(t._1))
+ }
+ }
+
+ /**
+ * Return the per partition sampling function used for sampling with replacement.
+ *
+ * When exact sample size is required, we make two additional passed over the RDD to determine
+ * the exact sampling rate that guarantees sample size with high confidence. The first pass
+ * counts the number of items in each stratum (group of items with the same key) in the RDD, and
+ * the second pass uses the counts to determine exact sampling rates.
+ *
+ * The sampling function has a unique seed per partition.
+ */
+ def getPoissonSamplingFunction[K, V](rdd:RDD[(K, V)],
+ fractionByKey: K => Double,
+ exact: Boolean,
+ counts: Option[Map[K, Long]],
+ seed: Long): (Int, Iterator[(K, V)]) => Iterator[(K, V)] = {
+ // TODO implement the streaming version of sampling w/ replacement that doesn't require counts
+ if (exact) {
+ val seqOp = StratifiedSampler.getSeqOp[K,V](true, fractionByKey, counts)
+ val combOp = StratifiedSampler.getCombOp[K]()
+ val zeroU = new Result[K](Map[K, Stratum](), seed = seed)
+ val finalResult = aggregateWithContext(zeroU)(rdd, seqOp, combOp).resultMap
+ val thresholdByKey = StratifiedSampler.computeThresholdByKey(finalResult, fractionByKey)
+ (idx: Int, iter: Iterator[(K, V)]) => {
+ val random = new RandomDataGenerator()
+ random.reSeed(seed + idx)
+ iter.flatMap { t =>
+ val q1 = finalResult.get(t._1).get.q1.getOrElse(0.0)
+ val q2 = finalResult.get(t._1).get.q2.getOrElse(0.0)
+ val x1 = if (q1 == 0) 0L else random.nextPoisson(q1)
+ val x2 = random.nextPoisson(q2).toInt
+ val x = x1 + (0 until x2).count(i => random.nextUniform(0.0, 1.0) < thresholdByKey(t._1))
+ if (x > 0) {
+ Iterator.fill(x.toInt)(t)
+ } else {
+ Iterator.empty
+ }
+ }
+ }
+ } else {
+ (idx: Int, iter: Iterator[(K, V)]) => {
+ val random = new RandomDataGenerator()
+ random.reSeed(seed + idx)
+ iter.flatMap { t =>
+ val count = random.nextPoisson(fractionByKey(t._1)).toInt
+ if (count > 0) {
+ Iterator.fill(count)(t)
+ } else {
+ Iterator.empty
+ }
+ }
+ }
+ }
+ }
+}
+
+/**
+ * Object used by seqOp to keep track of the number of items accepted and items waitlisted per
+ * stratum, as well as the bounds for accepting and waitlisting items.
+ */
+private[random] class Stratum(var numItems: Long = 0L, var numAccepted: Long = 0L)
+ extends Serializable {
+
+ var waitList: ArrayBuffer[Double] = new ArrayBuffer[Double]
+ var q1: Option[Double] = None // upper bound for accepting item instantly
+ var q2: Option[Double] = None // upper bound for adding item to waitlist
+
+ def incrNumItems(by: Long = 1L) = numItems += by
+
+ def incrNumAccepted(by: Long = 1L) = numAccepted += by
+
+ def addToWaitList(elem: Double) = waitList += elem
+
+ def addToWaitList(elems: ArrayBuffer[Double]) = waitList ++= elems
+}
+
+/**
+ * Object used by seqOp and combOp to keep track of the sampling statistics for all strata.
+ *
+ * When used by seqOp for each partition, we also keep track of the partition ID in this object
+ * to make sure a single random number generator with a unique seed is used for each partition.
+ */
+private[random] class Result[K](var resultMap: Map[K, Stratum],
+ var cachedPartitionId: Option[Int] = None,
+ val seed: Long)
+ extends Serializable {
+
+ var rand: RandomDataGenerator = new RandomDataGenerator
--- End diff --
Here you don't need to explicitly declare the type (please apply this comment to other similar places where you explicitly declare type of values or variables).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by dorx <gi...@git.apache.org>.
Github user dorx commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r14672589
--- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
@@ -0,0 +1,335 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util.random
+
+import scala.collection.Map
+import scala.collection.mutable.{ArrayBuffer, HashMap, Map => MMap}
+import scala.reflect.ClassTag
+
+import org.apache.commons.math3.random.RandomDataGenerator
+import org.apache.spark.{Logging, SparkContext, TaskContext}
+import org.apache.spark.rdd.RDD
+import org.apache.spark.util.Utils
+
+/**
+ * Auxiliary functions and data structures for the sampleByKey method in PairRDDFunctions.
+ *
+ * For more theoretical background on the sampling technqiues used here, please refer to
+ * http://jmlr.org/proceedings/papers/v28/meng13a.html
+ */
+private[spark] object StratifiedSampler extends Logging {
+
+ /**
+ * A version of {@link #aggregate()} that passes the TaskContext to the function that does
+ * aggregation for each partition. This function avoids creating an extra depth in the RDD
+ * lineage, as opposed to using mapPartitionsWithIndex, which results in slightly improved
+ * run time.
+ */
+ def aggregateWithContext[U: ClassTag, T: ClassTag](zeroValue: U)
+ (rdd: RDD[T],
+ seqOp: ((TaskContext, U), T) => U,
+ combOp: (U, U) => U): U = {
+ val sc: SparkContext = rdd.sparkContext
+ // Clone the zero value since we will also be serializing it as part of tasks
+ var jobResult = Utils.clone(zeroValue, sc.env.closureSerializer.newInstance())
+ // pad seqOp and combOp with taskContext to conform to aggregate's signature in TraversableOnce
+ val paddedSeqOp = (arg1: (TaskContext, U), item: T) => (arg1._1, seqOp(arg1, item))
+ val paddedCombOp = (arg1: (TaskContext, U), arg2: (TaskContext, U)) =>
+ (arg1._1, combOp(arg1._2, arg1._2))
+ val cleanSeqOp = sc.clean(paddedSeqOp)
+ val cleanCombOp = sc.clean(paddedCombOp)
+ val aggregatePartition = (tc: TaskContext, it: Iterator[T]) =>
+ (it.aggregate(tc, zeroValue)(cleanSeqOp, cleanCombOp))._2
+ val mergeResult = (index: Int, taskResult: U) => jobResult = combOp(jobResult, taskResult)
+ sc.runJob(rdd, aggregatePartition, mergeResult)
+ jobResult
+ }
+
+ /**
+ * Returns the function used by aggregate to collect sampling statistics for each partition.
+ */
+ def getSeqOp[K, V](withReplacement: Boolean,
+ fractions: Map[K, Double],
+ counts: Option[Map[K, Long]]): ((TaskContext, Result[K]), (K, V)) => Result[K] = {
+ val delta = 5e-5
+ (output: (TaskContext, Result[K]), item: (K, V)) => {
+ val result = output._2
+ val tc = output._1
+ val rng = result.getRand(tc.partitionId)
+ val fraction = fractions(item._1)
+ val stratum = result.getEntry(item._1)
+ if (withReplacement) {
+ // compute acceptBound and waitListBound only if they haven't been computed already
+ // since they don't change from iteration to iteration.
+ // TODO change this to the streaming version
+ if (stratum.areBoundsEmpty) {
+ val n = counts.get(item._1)
+ val sampleSize = math.ceil(n * fraction).toLong
+ val lmbd1 = PoissonBounds.getLowerBound(sampleSize)
+ val minCount = PoissonBounds.getMinCount(lmbd1)
+ val lmbd2 = if (lmbd1 == 0) {
+ PoissonBounds.getUpperBound(sampleSize)
+ } else {
+ PoissonBounds.getUpperBound(sampleSize - minCount)
+ }
+ stratum.acceptBound = lmbd1 / n
+ stratum.waitListBound = lmbd2 / n
+ }
+ val x1 = if (stratum.acceptBound == 0.0) 0L else rng.nextPoisson(stratum.acceptBound)
+ if (x1 > 0) {
+ stratum.incrNumAccepted(x1)
+ }
+ val x2 = rng.nextPoisson(stratum.waitListBound).toInt
+ if (x2 > 0) {
+ stratum.addToWaitList(ArrayBuffer.fill(x2)(rng.nextUniform(0.0, 1.0)))
+ }
+ } else {
+ // We use the streaming version of the algorithm for sampling without replacement to avoid
+ // using an extra pass over the RDD for computing the count.
+ // Hence, acceptBound and waitListBound change on every iteration.
+ val g1 = - math.log(delta) / stratum.numItems // gamma1
+ val g2 = (2.0 / 3.0) * g1 // gamma 2
+ stratum.acceptBound = math.max(0, fraction + g2 - math.sqrt((g2 * g2 + 3 * g2 * fraction)))
+ stratum.waitListBound = math.min(1, fraction + g1 + math.sqrt(g1 * g1 + 2 * g1 * fraction))
+
+ val x = rng.nextUniform(0.0, 1.0)
+ if (x < stratum.acceptBound) {
+ stratum.incrNumAccepted()
+ } else if (x < stratum.waitListBound) {
+ stratum.addToWaitList(x)
+ }
+ }
+ stratum.incrNumItems()
+ result
+ }
+ }
+
+ /**
+ * Returns the function used by aggregate to combine results from different partitions, as
+ * returned by seqOp.
+ */
+ def getCombOp[K](): (Result[K], Result[K]) => Result[K] = {
+ (r1: Result[K], r2: Result[K]) => {
+ // take union of both key sets in case one partition doesn't contain all keys
+ val keyUnion = r1.resultMap.keySet.union(r2.resultMap.keySet)
+
+ // Use r2 to keep the combined result since r1 is usual empty
+ for (key <- keyUnion) {
+ val entry1 = r1.resultMap.get(key)
+ if (r2.resultMap.contains(key)) {
+ r2.resultMap(key).merge(entry1)
+ } else {
+ r2.addEntry(key, entry1)
+ }
+ }
+ r2
+ }
+ }
+
+ /**
+ * Given the result returned by the aggregate function, determine the threshold for accepting
+ * items to generate exact sample size.
+ *
+ * To do so, we compute sampleSize = math.ceil(size * samplingRate) for each stratum and compare
+ * it to the number of items that were accepted instantly and the number of items in the waitlist
+ * for that stratum. Most of the time, numAccepted <= sampleSize <= (numAccepted + numWaitlisted),
+ * which means we need to sort the elements in the waitlist by their associated values in order
+ * to find the value T s.t. |{elements in the stratum whose associated values <= T}| = sampleSize.
+ * Note that all elements in the waitlist have values >= bound for instant accept, so a T value
+ * in the waitlist range would allow all elements that were instantly accepted on the first pass
+ * to be included in the sample.
+ */
+ def computeThresholdByKey[K](finalResult: Map[K, Stratum],
+ fractions: Map[K, Double]):
--- End diff --
Actually what's the rule of thumb here? I was under the impression that if you put your arguments on different lines the return type needs its own line, no?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1025#issuecomment-48384891
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by dorx <gi...@git.apache.org>.
Github user dorx commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r14907155
--- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
@@ -0,0 +1,311 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util.random
+
+import scala.collection.Map
+import scala.collection.mutable.{ArrayBuffer, HashMap, Map => MMap}
+
+import org.apache.commons.math3.random.RandomDataGenerator
+import org.apache.spark.Logging
+import org.apache.spark.rdd.RDD
+
+/**
+ * Auxiliary functions and data structures for the sampleByKey method in PairRDDFunctions.
+ *
+ * Essentially, when exact sample size is necessary, we make additional passes over the RDD to
+ * compute the exact threshold value to use for each stratum to guarantee exact sample size with
+ * high probability. This is achieved by maintaining a waitlist of size O(log(s)), where s is the
+ * desired sample size for each stratum.
+ *
+ * Like in simple random sampling, we generate a random value for each item from the
+ * uniform distribution [0.0, 1.0]. All items with values <= min(values of items in the waitlist)
+ * are accepted into the sample instantly. The threshold for instant accept is designed so that
+ * s - numAccepted = O(log(s)), where s is again the desired sample size. Thus, by maintaining a
+ * waitlist size = O(log(s)), we will be able to create a sample of the exact size s by adding
+ * a portion of the waitlist to the set of items that are instantly accepted. The exact threshold
+ * is computed by sorting the values in the waitlist and picking the value at (s - numAccepted).
+ *
+ * Note that since we use the same seed for the RNG when computing the thresholds and the actual
+ * sample, our computed thresholds are guaranteed to produce the desired sample size.
+ *
+ * For more theoretical background on the sampling techniques used here, please refer to
+ * http://jmlr.org/proceedings/papers/v28/meng13a.html
+ */
+
+private[spark] object StratifiedSampler extends Logging {
+
+ /**
+ * Count the number of items instantly accepted and generate the waitlist for each stratum.
+ *
+ * This is only invoked when exact sample size is required.
+ */
+ def getCounts[K, V](rdd: RDD[(K, V)],
+ withReplacement: Boolean,
+ fractions: Map[K, Double],
+ counts: Option[Map[K, Long]],
+ seed: Long): MMap[K, Stratum] = {
+ val combOp = getCombOp[K]
+ val mappedPartitionRDD = rdd.mapPartitionsWithIndex({ case (partition, iter) =>
+ val zeroU: MMap[K, Stratum] = new HashMap[K, Stratum]()
+ val rng = new RandomDataGenerator()
+ rng.reSeed(seed + partition)
+ val seqOp = getSeqOp(withReplacement, fractions, rng, counts)
+ Iterator(iter.aggregate(zeroU)(seqOp, combOp))
+ }, preservesPartitioning=true)
+ mappedPartitionRDD.reduce(combOp)
+ }
+
+ /**
+ * Returns the function used by aggregate to collect sampling statistics for each partition.
+ */
+ def getSeqOp[K, V](withReplacement: Boolean,
+ fractions: Map[K, Double],
+ rng: RandomDataGenerator,
+ counts: Option[Map[K, Long]]): (MMap[K, Stratum], (K, V)) => MMap[K, Stratum] = {
+ val delta = 5e-5
+ (result: MMap[K, Stratum], item: (K, V)) => {
+ val key = item._1
+ val fraction = fractions(key)
+ if (!result.contains(key)) {
+ result += (key -> new Stratum())
+ }
+ val stratum = result(key)
+
+ if (withReplacement) {
+ // compute acceptBound and waitListBound only if they haven't been computed already
+ // since they don't change from iteration to iteration.
+ // TODO change this to the streaming version
+ if (stratum.areBoundsEmpty) {
+ val n = counts.get(key)
+ val sampleSize = math.ceil(n * fraction).toLong
+ val lmbd1 = PoissonBounds.getLowerBound(sampleSize)
+ val minCount = PoissonBounds.getMinCount(lmbd1)
+ val lmbd2 = if (lmbd1 == 0) {
+ PoissonBounds.getUpperBound(sampleSize)
+ } else {
+ PoissonBounds.getUpperBound(sampleSize - minCount)
+ }
+ stratum.acceptBound = lmbd1 / n
+ stratum.waitListBound = lmbd2 / n
+ }
+ val acceptBound = stratum.acceptBound
+ val copiesAccepted = if (acceptBound == 0.0) 0L else rng.nextPoisson(acceptBound)
+ if (copiesAccepted > 0) {
+ stratum.incrNumAccepted(copiesAccepted)
+ }
+ val copiesWaitlisted = rng.nextPoisson(stratum.waitListBound).toInt
+ if (copiesWaitlisted > 0) {
+ stratum.addToWaitList(ArrayBuffer.fill(copiesWaitlisted)(rng.nextUniform(0.0, 1.0)))
+ }
+ } else {
+ // We use the streaming version of the algorithm for sampling without replacement to avoid
+ // using an extra pass over the RDD for computing the count.
+ // Hence, acceptBound and waitListBound change on every iteration.
+ val gamma1 = - math.log(delta) / stratum.numItems
+ val gamma2 = (2.0 / 3.0) * gamma1
+ stratum.acceptBound = math.max(0,
+ fraction + gamma2 - math.sqrt(gamma2 * gamma2 + 3 * gamma2 * fraction))
+ stratum.waitListBound = math.min(1,
+ fraction + gamma1 + math.sqrt(gamma1 * gamma1 + 2 * gamma1 * fraction))
+
+ val x = rng.nextUniform(0.0, 1.0)
+ if (x < stratum.acceptBound) {
+ stratum.incrNumAccepted()
+ } else if (x < stratum.waitListBound) {
+ stratum.addToWaitList(x)
+ }
+ }
+ stratum.incrNumItems()
+ result
+ }
+ }
+
+ /**
+ * Returns the function used combine results returned by seqOp from different partitions.
+ */
+ def getCombOp[K]: (MMap[K, Stratum], MMap[K, Stratum]) => MMap[K, Stratum] = {
+ (result1: MMap[K, Stratum], result2: MMap[K, Stratum]) => {
+ // take union of both key sets in case one partition doesn't contain all keys
+ for (key <- result1.keySet.union(result2.keySet)) {
+ // Use result2 to keep the combined result since r1 is usual empty
--- End diff --
contract per...? I'm not seeing it in the Scala docs: http://www.scala-lang.org/api/2.10.3/index.html#scala.collection.immutable.List
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r14907639
--- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
@@ -0,0 +1,311 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util.random
+
+import scala.collection.Map
+import scala.collection.mutable.{ArrayBuffer, HashMap, Map => MMap}
+
+import org.apache.commons.math3.random.RandomDataGenerator
+import org.apache.spark.Logging
+import org.apache.spark.rdd.RDD
+
+/**
+ * Auxiliary functions and data structures for the sampleByKey method in PairRDDFunctions.
+ *
+ * Essentially, when exact sample size is necessary, we make additional passes over the RDD to
+ * compute the exact threshold value to use for each stratum to guarantee exact sample size with
+ * high probability. This is achieved by maintaining a waitlist of size O(log(s)), where s is the
+ * desired sample size for each stratum.
+ *
+ * Like in simple random sampling, we generate a random value for each item from the
+ * uniform distribution [0.0, 1.0]. All items with values <= min(values of items in the waitlist)
+ * are accepted into the sample instantly. The threshold for instant accept is designed so that
+ * s - numAccepted = O(log(s)), where s is again the desired sample size. Thus, by maintaining a
+ * waitlist size = O(log(s)), we will be able to create a sample of the exact size s by adding
+ * a portion of the waitlist to the set of items that are instantly accepted. The exact threshold
+ * is computed by sorting the values in the waitlist and picking the value at (s - numAccepted).
+ *
+ * Note that since we use the same seed for the RNG when computing the thresholds and the actual
+ * sample, our computed thresholds are guaranteed to produce the desired sample size.
+ *
+ * For more theoretical background on the sampling techniques used here, please refer to
+ * http://jmlr.org/proceedings/papers/v28/meng13a.html
+ */
+
+private[spark] object StratifiedSampler extends Logging {
+
+ /**
+ * Count the number of items instantly accepted and generate the waitlist for each stratum.
+ *
+ * This is only invoked when exact sample size is required.
+ */
+ def getCounts[K, V](rdd: RDD[(K, V)],
+ withReplacement: Boolean,
+ fractions: Map[K, Double],
+ counts: Option[Map[K, Long]],
+ seed: Long): MMap[K, Stratum] = {
+ val combOp = getCombOp[K]
+ val mappedPartitionRDD = rdd.mapPartitionsWithIndex({ case (partition, iter) =>
+ val zeroU: MMap[K, Stratum] = new HashMap[K, Stratum]()
+ val rng = new RandomDataGenerator()
+ rng.reSeed(seed + partition)
+ val seqOp = getSeqOp(withReplacement, fractions, rng, counts)
+ Iterator(iter.aggregate(zeroU)(seqOp, combOp))
+ }, preservesPartitioning=true)
+ mappedPartitionRDD.reduce(combOp)
+ }
+
+ /**
+ * Returns the function used by aggregate to collect sampling statistics for each partition.
+ */
+ def getSeqOp[K, V](withReplacement: Boolean,
+ fractions: Map[K, Double],
+ rng: RandomDataGenerator,
+ counts: Option[Map[K, Long]]): (MMap[K, Stratum], (K, V)) => MMap[K, Stratum] = {
+ val delta = 5e-5
+ (result: MMap[K, Stratum], item: (K, V)) => {
+ val key = item._1
+ val fraction = fractions(key)
+ if (!result.contains(key)) {
+ result += (key -> new Stratum())
+ }
+ val stratum = result(key)
+
+ if (withReplacement) {
+ // compute acceptBound and waitListBound only if they haven't been computed already
+ // since they don't change from iteration to iteration.
+ // TODO change this to the streaming version
+ if (stratum.areBoundsEmpty) {
+ val n = counts.get(key)
+ val sampleSize = math.ceil(n * fraction).toLong
+ val lmbd1 = PoissonBounds.getLowerBound(sampleSize)
+ val minCount = PoissonBounds.getMinCount(lmbd1)
--- End diff --
Having `minCount` here. I think it is only useful in streaming environments but we didn't implement it in that way.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r14694294
--- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
@@ -0,0 +1,311 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util.random
+
+import scala.collection.Map
+import scala.collection.mutable.{ArrayBuffer, HashMap, Map => MMap}
+
+import org.apache.commons.math3.random.RandomDataGenerator
+import org.apache.spark.Logging
+import org.apache.spark.rdd.RDD
+
+/**
+ * Auxiliary functions and data structures for the sampleByKey method in PairRDDFunctions.
+ *
+ * Essentially, when exact sample size is necessary, we make additional passes over the RDD to
+ * compute the exact threshold value to use for each stratum to guarantee exact sample size with
+ * high probability. This is achieved by maintaining a waitlist of size O(log(s)), where s is the
+ * desired sample size for each stratum.
+ *
+ * Like in simple random sampling, we generate a random value for each item from the
+ * uniform distribution [0.0, 1.0]. All items with values <= min(values of items in the waitlist)
+ * are accepted into the sample instantly. The threshold for instant accept is designed so that
+ * s - numAccepted = O(log(s)), where s is again the desired sample size. Thus, by maintaining a
+ * waitlist size = O(log(s)), we will be able to create a sample of the exact size s by adding
+ * a portion of the waitlist to the set of items that are instantly accepted. The exact threshold
+ * is computed by sorting the values in the waitlist and picking the value at (s - numAccepted).
+ *
+ * Note that since we use the same seed for the RNG when computing the thresholds and the actual
+ * sample, our computed thresholds are guaranteed to produce the desired sample size.
+ *
+ * For more theoretical background on the sampling techniques used here, please refer to
+ * http://jmlr.org/proceedings/papers/v28/meng13a.html
+ */
+
+private[spark] object StratifiedSampler extends Logging {
+
+ /**
+ * Count the number of items instantly accepted and generate the waitlist for each stratum.
+ *
+ * This is only invoked when exact sample size is required.
+ */
+ def getCounts[K, V](rdd: RDD[(K, V)],
+ withReplacement: Boolean,
+ fractions: Map[K, Double],
+ counts: Option[Map[K, Long]],
+ seed: Long): MMap[K, Stratum] = {
+ val combOp = getCombOp[K]
+ val mappedPartitionRDD = rdd.mapPartitionsWithIndex({ case (partition, iter) =>
+ val zeroU: MMap[K, Stratum] = new HashMap[K, Stratum]()
+ val rng = new RandomDataGenerator()
+ rng.reSeed(seed + partition)
+ val seqOp = getSeqOp(withReplacement, fractions, rng, counts)
+ Iterator(iter.aggregate(zeroU)(seqOp, combOp))
+ }, preservesPartitioning=true)
+ mappedPartitionRDD.reduce(combOp)
+ }
+
+ /**
+ * Returns the function used by aggregate to collect sampling statistics for each partition.
+ */
+ def getSeqOp[K, V](withReplacement: Boolean,
+ fractions: Map[K, Double],
+ rng: RandomDataGenerator,
+ counts: Option[Map[K, Long]]): (MMap[K, Stratum], (K, V)) => MMap[K, Stratum] = {
+ val delta = 5e-5
+ (result: MMap[K, Stratum], item: (K, V)) => {
+ val key = item._1
+ val fraction = fractions(key)
+ if (!result.contains(key)) {
+ result += (key -> new Stratum())
+ }
+ val stratum = result(key)
+
+ if (withReplacement) {
+ // compute acceptBound and waitListBound only if they haven't been computed already
+ // since they don't change from iteration to iteration.
+ // TODO change this to the streaming version
+ if (stratum.areBoundsEmpty) {
+ val n = counts.get(key)
+ val sampleSize = math.ceil(n * fraction).toLong
+ val lmbd1 = PoissonBounds.getLowerBound(sampleSize)
+ val minCount = PoissonBounds.getMinCount(lmbd1)
+ val lmbd2 = if (lmbd1 == 0) {
+ PoissonBounds.getUpperBound(sampleSize)
+ } else {
+ PoissonBounds.getUpperBound(sampleSize - minCount)
+ }
+ stratum.acceptBound = lmbd1 / n
+ stratum.waitListBound = lmbd2 / n
+ }
+ val acceptBound = stratum.acceptBound
+ val copiesAccepted = if (acceptBound == 0.0) 0L else rng.nextPoisson(acceptBound)
+ if (copiesAccepted > 0) {
+ stratum.incrNumAccepted(copiesAccepted)
+ }
+ val copiesWaitlisted = rng.nextPoisson(stratum.waitListBound).toInt
+ if (copiesWaitlisted > 0) {
+ stratum.addToWaitList(ArrayBuffer.fill(copiesWaitlisted)(rng.nextUniform(0.0, 1.0)))
+ }
+ } else {
+ // We use the streaming version of the algorithm for sampling without replacement to avoid
+ // using an extra pass over the RDD for computing the count.
+ // Hence, acceptBound and waitListBound change on every iteration.
+ val gamma1 = - math.log(delta) / stratum.numItems
+ val gamma2 = (2.0 / 3.0) * gamma1
+ stratum.acceptBound = math.max(0,
+ fraction + gamma2 - math.sqrt(gamma2 * gamma2 + 3 * gamma2 * fraction))
+ stratum.waitListBound = math.min(1,
+ fraction + gamma1 + math.sqrt(gamma1 * gamma1 + 2 * gamma1 * fraction))
+
+ val x = rng.nextUniform(0.0, 1.0)
+ if (x < stratum.acceptBound) {
+ stratum.incrNumAccepted()
+ } else if (x < stratum.waitListBound) {
+ stratum.addToWaitList(x)
+ }
+ }
+ stratum.incrNumItems()
+ result
+ }
+ }
+
+ /**
+ * Returns the function used combine results returned by seqOp from different partitions.
+ */
+ def getCombOp[K]: (MMap[K, Stratum], MMap[K, Stratum]) => MMap[K, Stratum] = {
+ (result1: MMap[K, Stratum], result2: MMap[K, Stratum]) => {
+ // take union of both key sets in case one partition doesn't contain all keys
+ for (key <- result1.keySet.union(result2.keySet)) {
+ // Use result2 to keep the combined result since r1 is usual empty
+ val entry1 = result1.get(key)
+ if (result2.contains(key)) {
+ result2(key).merge(entry1)
+ } else {
+ if (entry1.isDefined) {
+ result2 += (key -> entry1.get)
+ }
+ }
+ }
+ result2
+ }
+ }
+
+ /**
+ * Given the result returned by getCounts, determine the threshold for accepting items to
+ * generate exact sample size.
+ *
+ * To do so, we compute sampleSize = math.ceil(size * samplingRate) for each stratum and compare
+ * it to the number of items that were accepted instantly and the number of items in the waitlist
+ * for that stratum. Most of the time, numAccepted <= sampleSize <= (numAccepted + numWaitlisted),
+ * which means we need to sort the elements in the waitlist by their associated values in order
+ * to find the value T s.t. |{elements in the stratum whose associated values <= T}| = sampleSize.
+ * Note that all elements in the waitlist have values >= bound for instant accept, so a T value
+ * in the waitlist range would allow all elements that were instantly accepted on the first pass
+ * to be included in the sample.
+ */
+ def computeThresholdByKey[K](finalResult: Map[K, Stratum],
+ fractions: Map[K, Double]): Map[K, Double] = {
+ val thresholdByKey = new HashMap[K, Double]()
+ for ((key, stratum) <- finalResult) {
+ val sampleSize = math.ceil(stratum.numItems * fractions(key)).toLong
+ if (stratum.numAccepted > sampleSize) {
+ logWarning("Pre-accepted too many")
+ thresholdByKey += (key -> stratum.acceptBound)
+ } else {
+ val numWaitListAccepted = (sampleSize - stratum.numAccepted).toInt
+ if (numWaitListAccepted >= stratum.waitList.size) {
+ logWarning("WaitList too short")
+ thresholdByKey += (key -> stratum.waitListBound)
+ } else {
+ thresholdByKey += (key -> stratum.waitList.sorted.apply(numWaitListAccepted))
+ }
+ }
+ }
+ thresholdByKey
+ }
+
+ /**
+ * Return the per partition sampling function used for sampling without replacement.
+ *
+ * When exact sample size is required, we make an additional pass over the RDD to determine the
+ * exact sampling rate that guarantees sample size with high confidence.
+ *
+ * The sampling function has a unique seed per partition.
+ */
+ def getBernoulliSamplingFunction[K, V](rdd: RDD[(K, V)],
+ fractions: Map[K, Double],
+ exact: Boolean,
+ seed: Long): (Int, Iterator[(K, V)]) => Iterator[(K, V)] = {
+ var samplingRateByKey = fractions
+ if (exact) {
+ // determine threshold for each stratum and resample
+ val finalResult = getCounts(rdd, false, fractions, None, seed)
+ samplingRateByKey = computeThresholdByKey(finalResult, fractions)
+ }
+ (idx: Int, iter: Iterator[(K, V)]) => {
+ val rng = new RandomDataGenerator
+ rng.reSeed(seed + idx)
+ iter.filter(t => rng.nextUniform(0.0, 1.0) < samplingRateByKey(t._1))
+ }
+ }
+
+ /**
+ * Return the per partition sampling function used for sampling with replacement.
+ *
+ * When exact sample size is required, we make two additional passed over the RDD to determine
+ * the exact sampling rate that guarantees sample size with high confidence. The first pass
+ * counts the number of items in each stratum (group of items with the same key) in the RDD, and
+ * the second pass uses the counts to determine exact sampling rates.
+ *
+ * The sampling function has a unique seed per partition.
+ */
+ def getPoissonSamplingFunction[K, V](rdd: RDD[(K, V)],
+ fractions: Map[K, Double],
+ exact: Boolean,
+ counts: Option[Map[K, Long]],
+ seed: Long): (Int, Iterator[(K, V)]) => Iterator[(K, V)] = {
+ // TODO implement the streaming version of sampling w/ replacement that doesn't require counts
+ if (exact) {
+ val finalResult = getCounts(rdd, true, fractions, counts, seed)
+ val thresholdByKey = computeThresholdByKey(finalResult, fractions)
+ (idx: Int, iter: Iterator[(K, V)]) => {
+ val rng = new RandomDataGenerator()
+ rng.reSeed(seed + idx)
+ iter.flatMap { item =>
+ val key = item._1
+ val acceptBound = finalResult(key).acceptBound
+ val copiesAccepted = if (acceptBound == 0) 0L else rng.nextPoisson(acceptBound)
+ val copiesWailisted = rng.nextPoisson(finalResult(key).waitListBound).toInt
+ val copiesInSample = copiesAccepted +
+ (0 until copiesWailisted).count(i => rng.nextUniform(0.0, 1.0) < thresholdByKey(key))
+ if (copiesInSample > 0) {
+ Iterator.fill(copiesInSample.toInt)(item)
+ } else {
+ Iterator.empty
+ }
+ }
+ }
+ } else {
+ (idx: Int, iter: Iterator[(K, V)]) => {
+ val rng = new RandomDataGenerator()
+ rng.reSeed(seed + idx)
+ iter.flatMap { item =>
+ val count = rng.nextPoisson(fractions(item._1)).toInt
+ if (count > 0) {
+ Iterator.fill(count)(item)
+ } else {
+ Iterator.empty
+ }
+ }
+ }
+ }
+ }
+}
+
+/**
+ * Object used by seqOp to keep track of the number of items accepted and items waitlisted per
+ * stratum, as well as the bounds for accepting and waitlisting items.
+ *
+ * `[random]` here is necessary since it's in the return type signature of seqOp defined above
--- End diff --
Not necessary.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1025#issuecomment-48422316
Merged build finished. All automated tests passed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1025#issuecomment-46389585
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1025#issuecomment-45553453
Build finished.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1025#issuecomment-46381896
Build finished. All automated tests passed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by dorx <gi...@git.apache.org>.
Github user dorx commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r14578266
--- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util.random
+
+import scala.collection.{Map, mutable}
+import scala.collection.mutable.ArrayBuffer
+import scala.reflect.ClassTag
+
+import org.apache.commons.math3.random.RandomDataGenerator
+import org.apache.spark.{Logging, SparkContext, TaskContext}
+import org.apache.spark.rdd.RDD
+import org.apache.spark.util.Utils
+import org.apache.spark.util.random.{PoissonBounds => PB}
+
+private[spark] object StratifiedSampler extends Logging {
+
+ /**
+ * A version of {@link #aggregate()} that passes the TaskContext to the function that does
+ * aggregation for each partition. This function avoids creating an extra depth in the RDD
+ * lineage, as opposed to using mapPartitionsWithId, which results in slightly improved run time.
+ */
+ def aggregateWithContext[U: ClassTag, T: ClassTag](zeroValue: U)
+ (rdd: RDD[T],
+ seqOp: ((TaskContext, U), T) => U,
+ combOp: (U, U) => U): U = {
+ val sc: SparkContext = rdd.sparkContext
+ // Clone the zero value since we will also be serializing it as part of tasks
+ var jobResult = Utils.clone(zeroValue, sc.env.closureSerializer.newInstance())
+ // pad seqOp and combOp with taskContext to conform to aggregate's signature in TraversableOnce
+ val paddedSeqOp = (arg1: (TaskContext, U), item: T) => (arg1._1, seqOp(arg1, item))
+ val paddedcombOp = (arg1: (TaskContext, U), arg2: (TaskContext, U)) =>
+ (arg1._1, combOp(arg1._2, arg1._2))
+ val cleanSeqOp = sc.clean(paddedSeqOp)
+ val cleanCombOp = sc.clean(paddedcombOp)
+ val aggregatePartition = (tc: TaskContext, it: Iterator[T]) =>
+ (it.aggregate(tc, zeroValue)(cleanSeqOp, cleanCombOp))._2
+ val mergeResult = (index: Int, taskResult: U) => jobResult = combOp(jobResult, taskResult)
+ sc.runJob(rdd, aggregatePartition, mergeResult)
+ jobResult
+ }
+
+ /**
+ * Returns the function used by aggregate to collect sampling statistics for each partition.
+ */
+ def getSeqOp[K, V](withReplacement: Boolean,
+ fractionByKey: (K => Double),
+ counts: Option[Map[K, Long]]): ((TaskContext, Result[K]),(K, V)) => Result[K] = {
+ val delta = 5e-5
+ (U: (TaskContext, Result[K]), item: (K, V)) => {
+ val result = U._2
+ val tc = U._1
+ val rng = result.getRand(tc.partitionId)
+ val fraction = fractionByKey(item._1)
+ val stratum = result.getEntry(item._1)
+ if (withReplacement) {
+ // compute q1 and q2 only if they haven't been computed already
+ // since they don't change from iteration to iteration.
+ // TODO change this to the streaming version
+ if (stratum.q1.isEmpty || stratum.q2.isEmpty) {
+ val n = counts.get(item._1)
+ val s = math.ceil(n * fraction).toLong
+ val lmbd1 = PB.getLowerBound(s)
+ val minCount = PB.getMinCount(lmbd1)
+ val lmbd2 = if (lmbd1 == 0) PB.getUpperBound(s) else PB.getUpperBound(s - minCount)
+ val q1 = lmbd1 / n
+ val q2 = lmbd2 / n
+ stratum.q1 = Some(q1)
+ stratum.q2 = Some(q2)
+ }
+ val x1 = if (stratum.q1.get == 0) 0L else rng.nextPoisson(stratum.q1.get)
+ if (x1 > 0) {
+ stratum.incrNumAccepted(x1)
+ }
+ val x2 = rng.nextPoisson(stratum.q2.get).toInt
+ if (x2 > 0) {
+ stratum.addToWaitList(ArrayBuffer.fill(x2)(rng.nextUniform(0.0, 1.0)))
+ }
+ } else {
+ // We use the streaming version of the algorithm for sampling without replacement.
+ // Hence, q1 and q2 change on every iteration.
+ val g1 = - math.log(delta) / stratum.numItems
+ val g2 = (2.0 / 3.0) * g1
+ val q1 = math.max(0, fraction + g2 - math.sqrt((g2 * g2 + 3 * g2 * fraction)))
+ val q2 = math.min(1, fraction + g1 + math.sqrt(g1 * g1 + 2 * g1 * fraction))
+
+ val x = rng.nextUniform(0.0, 1.0)
+ if (x < q1) {
+ stratum.incrNumAccepted()
+ } else if (x < q2) {
+ stratum.addToWaitList(x)
+ }
+ stratum.q1 = Some(q1)
+ stratum.q2 = Some(q2)
+ }
+ stratum.incrNumItems()
+ result
+ }
+ }
+
+ /**
+ * Returns the function used by aggregate to combine results from different partitions, as
+ * returned by seqOp.
+ */
+ def getCombOp[K](): (Result[K], Result[K]) => Result[K] = {
+ (r1: Result[K], r2: Result[K]) => {
+ // take union of both key sets in case one partition doesn't contain all keys
+ val keyUnion = r1.resultMap.keys.toSet.union(r2.resultMap.keys.toSet)
+
+ // Use r2 to keep the combined result since r1 is usual empty
+ for (key <- keyUnion) {
+ val entry1 = r1.resultMap.get(key)
+ val entry2 = r2.resultMap.get(key)
+ if (entry2.isEmpty && entry1.isDefined) {
+ r2.resultMap += (key -> entry1.get)
+ } else if (entry1.isDefined && entry2.isDefined) {
+ entry2.get.addToWaitList(entry1.get.waitList)
+ entry2.get.incrNumAccepted(entry1.get.numAccepted)
+ entry2.get.incrNumItems(entry1.get.numItems)
+ }
+ }
+ r2
+ }
+ }
+
+ /**
+ * Given the result returned by the aggregate function, we need to determine the threshold used
+ * to accept items to generate the exact sample size.
+ */
+ def computeThresholdByKey[K](finalResult: Map[K, Stratum], fractionByKey: (K => Double)):
+ (K => Double) = {
+ val thresholdByKey = new mutable.HashMap[K, Double]()
+ for ((key, stratum) <- finalResult) {
+ val fraction = fractionByKey(key)
+ val s = math.ceil(stratum.numItems * fraction).toLong
+ if (stratum.numAccepted > s) {
+ logWarning("Pre-accepted too many")
+ thresholdByKey += (key -> stratum.q1.get)
+ } else {
+ val numWaitListAccepted = (s - stratum.numAccepted).toInt
+ if (numWaitListAccepted >= stratum.waitList.size) {
+ logWarning("WaitList too short")
+ thresholdByKey += (key -> stratum.q2.get)
+ } else {
+ thresholdByKey += (key -> stratum.waitList.sorted.apply(numWaitListAccepted))
+ }
+ }
+ }
+ thresholdByKey
+ }
+
+ /**
+ * Return the per partition sampling function used for sampling without replacement.
+ *
+ * When exact sample size is required, we make an additional pass over the RDD to determine the
+ * exact sampling rate that guarantees sample size with high confidence.
+ *
+ * The sampling function has a unique seed per partition.
+ */
+ def getBernoulliSamplingFunction[K, V](rdd: RDD[(K, V)],
+ fractionByKey: K => Double,
+ exact: Boolean,
+ seed: Long): (Int, Iterator[(K, V)]) => Iterator[(K, V)] = {
+ var samplingRateByKey = fractionByKey
+ if (exact) {
+ // determine threshold for each stratum and resample
+ val seqOp = StratifiedSampler.getSeqOp[K,V](false, fractionByKey, None)
+ val combOp = StratifiedSampler.getCombOp[K]()
+ val zeroU = new Result[K](Map[K, Stratum](), seed = seed)
+ val finalResult = aggregateWithContext(zeroU)(rdd, seqOp, combOp).resultMap
+ samplingRateByKey = StratifiedSampler.computeThresholdByKey(finalResult, fractionByKey)
+ }
+ (idx: Int, iter: Iterator[(K, V)]) => {
+ val random = new RandomDataGenerator
+ random.reSeed(seed + idx)
+ iter.filter(t => random.nextUniform(0.0, 1.0) < samplingRateByKey(t._1))
+ }
+ }
+
+ /**
+ * Return the per partition sampling function used for sampling with replacement.
+ *
+ * When exact sample size is required, we make two additional passed over the RDD to determine
+ * the exact sampling rate that guarantees sample size with high confidence. The first pass
+ * counts the number of items in each stratum (group of items with the same key) in the RDD, and
+ * the second pass uses the counts to determine exact sampling rates.
+ *
+ * The sampling function has a unique seed per partition.
+ */
+ def getPoissonSamplingFunction[K, V](rdd:RDD[(K, V)],
+ fractionByKey: K => Double,
+ exact: Boolean,
+ counts: Option[Map[K, Long]],
+ seed: Long): (Int, Iterator[(K, V)]) => Iterator[(K, V)] = {
+ // TODO implement the streaming version of sampling w/ replacement that doesn't require counts
+ if (exact) {
+ val seqOp = StratifiedSampler.getSeqOp[K,V](true, fractionByKey, counts)
+ val combOp = StratifiedSampler.getCombOp[K]()
+ val zeroU = new Result[K](Map[K, Stratum](), seed = seed)
+ val finalResult = aggregateWithContext(zeroU)(rdd, seqOp, combOp).resultMap
+ val thresholdByKey = StratifiedSampler.computeThresholdByKey(finalResult, fractionByKey)
+ (idx: Int, iter: Iterator[(K, V)]) => {
+ val random = new RandomDataGenerator()
+ random.reSeed(seed + idx)
+ iter.flatMap { t =>
+ val q1 = finalResult.get(t._1).get.q1.getOrElse(0.0)
+ val q2 = finalResult.get(t._1).get.q2.getOrElse(0.0)
+ val x1 = if (q1 == 0) 0L else random.nextPoisson(q1)
+ val x2 = random.nextPoisson(q2).toInt
+ val x = x1 + (0 until x2).count(i => random.nextUniform(0.0, 1.0) < thresholdByKey(t._1))
+ if (x > 0) {
+ Iterator.fill(x.toInt)(t)
+ } else {
+ Iterator.empty
+ }
+ }
+ }
+ } else {
+ (idx: Int, iter: Iterator[(K, V)]) => {
+ val random = new RandomDataGenerator()
+ random.reSeed(seed + idx)
+ iter.flatMap { t =>
+ val count = random.nextPoisson(fractionByKey(t._1)).toInt
+ if (count > 0) {
+ Iterator.fill(count)(t)
+ } else {
+ Iterator.empty
+ }
+ }
+ }
+ }
+ }
+}
+
+/**
+ * Object used by seqOp to keep track of the number of items accepted and items waitlisted per
+ * stratum, as well as the bounds for accepting and waitlisting items.
+ */
+private[random] class Stratum(var numItems: Long = 0L, var numAccepted: Long = 0L)
+ extends Serializable {
+
+ var waitList: ArrayBuffer[Double] = new ArrayBuffer[Double]
+ var q1: Option[Double] = None // upper bound for accepting item instantly
+ var q2: Option[Double] = None // upper bound for adding item to waitlist
+
+ def incrNumItems(by: Long = 1L) = numItems += by
+
+ def incrNumAccepted(by: Long = 1L) = numAccepted += by
+
+ def addToWaitList(elem: Double) = waitList += elem
+
+ def addToWaitList(elems: ArrayBuffer[Double]) = waitList ++= elems
+}
+
+/**
+ * Object used by seqOp and combOp to keep track of the sampling statistics for all strata.
+ *
+ * When used by seqOp for each partition, we also keep track of the partition ID in this object
+ * to make sure a single random number generator with a unique seed is used for each partition.
+ */
+private[random] class Result[K](var resultMap: Map[K, Stratum],
--- End diff --
Actually if I remove the "[random]" here I get an out-of-scope error: "private class Result escapes its defining scope as part of type (org.apache.spark.util.random.Result[K], org.apache.spark.util.random.Result[K]) => org.apache.spark.util.random.Result[K]"
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by falaki <gi...@git.apache.org>.
Github user falaki commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r14542021
--- Diff: core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala ---
@@ -195,6 +193,40 @@ class PairRDDFunctions[K, V](self: RDD[(K, V)])
}
/**
+ * Return a subset of this RDD sampled by key (via stratified sampling).
+ *
+ * If exact set to true, we guarantee, with high probability, a sample size =
+ * math.ceil(fraction * S_i), where S_i is the size of the ith stratum (collection of entries
+ * that share the same key). When sampling without replacement, we need one additional pass over
+ * the RDD to guarantee sample size with a 99.99% confidence; when sampling with replacement, we
+ * need two additional passes.
+ *
+ * @param withReplacement whether to sample with or without replacement
+ * @param fractions map of specific keys to sampling rates
+ * @param seed seed for the random number generator
+ * @param exact whether sample size needs to be exactly math.ceil(fraction * size) per stratum
+ * @return RDD containing the sampled subset
+ */
+ def sampleByKey(withReplacement: Boolean,
+ fractions: Map[K, Double],
+ exact: Boolean = true,
+ seed: Long = Utils.random.nextLong): RDD[(K, V)]= {
+
+ require(fractions.forall({case(k, v) => v >= 0.0}), "Invalid sampling rates.")
+
+ if (withReplacement) {
--- End diff --
This can be written more functional (and readable) like this:
```scala
val samplingFunc = if (withReplacement) {
val counts = if (exact) Some(this.countByKey()) else None
StratifiedSampler.getPoissonSamplingFunction(self, fractions, exact, counts, seed)
} else {
StratifiedSampler.getBernoulliSamplingFunction(self, fractions, exact, seed)
}
self.mapPartitionsWithIndex(samplingFunc, preservesPartitioning = true)
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by falaki <gi...@git.apache.org>.
Github user falaki commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r14542085
--- Diff: core/src/main/scala/org/apache/spark/rdd/RDD.scala ---
@@ -350,6 +350,7 @@ abstract class RDD[T: ClassTag](
/**
* Return a sampled subset of this RDD.
+ *
--- End diff --
Let's remove this line.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1025#issuecomment-46381897
All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15858/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1025#issuecomment-45558759
Build finished.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1025#issuecomment-46518419
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by falaki <gi...@git.apache.org>.
Github user falaki commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r14544573
--- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util.random
+
+import scala.collection.{Map, mutable}
+import scala.collection.mutable.ArrayBuffer
+import scala.reflect.ClassTag
+
+import org.apache.commons.math3.random.RandomDataGenerator
+import org.apache.spark.{Logging, SparkContext, TaskContext}
+import org.apache.spark.rdd.RDD
+import org.apache.spark.util.Utils
+import org.apache.spark.util.random.{PoissonBounds => PB}
+
+private[spark] object StratifiedSampler extends Logging {
+
+ /**
+ * A version of {@link #aggregate()} that passes the TaskContext to the function that does
+ * aggregation for each partition. This function avoids creating an extra depth in the RDD
+ * lineage, as opposed to using mapPartitionsWithId, which results in slightly improved run time.
+ */
+ def aggregateWithContext[U: ClassTag, T: ClassTag](zeroValue: U)
+ (rdd: RDD[T],
+ seqOp: ((TaskContext, U), T) => U,
+ combOp: (U, U) => U): U = {
+ val sc: SparkContext = rdd.sparkContext
+ // Clone the zero value since we will also be serializing it as part of tasks
+ var jobResult = Utils.clone(zeroValue, sc.env.closureSerializer.newInstance())
+ // pad seqOp and combOp with taskContext to conform to aggregate's signature in TraversableOnce
+ val paddedSeqOp = (arg1: (TaskContext, U), item: T) => (arg1._1, seqOp(arg1, item))
+ val paddedcombOp = (arg1: (TaskContext, U), arg2: (TaskContext, U)) =>
+ (arg1._1, combOp(arg1._2, arg1._2))
+ val cleanSeqOp = sc.clean(paddedSeqOp)
+ val cleanCombOp = sc.clean(paddedcombOp)
+ val aggregatePartition = (tc: TaskContext, it: Iterator[T]) =>
+ (it.aggregate(tc, zeroValue)(cleanSeqOp, cleanCombOp))._2
+ val mergeResult = (index: Int, taskResult: U) => jobResult = combOp(jobResult, taskResult)
+ sc.runJob(rdd, aggregatePartition, mergeResult)
+ jobResult
+ }
+
+ /**
+ * Returns the function used by aggregate to collect sampling statistics for each partition.
+ */
+ def getSeqOp[K, V](withReplacement: Boolean,
+ fractionByKey: (K => Double),
+ counts: Option[Map[K, Long]]): ((TaskContext, Result[K]),(K, V)) => Result[K] = {
+ val delta = 5e-5
+ (U: (TaskContext, Result[K]), item: (K, V)) => {
+ val result = U._2
+ val tc = U._1
+ val rng = result.getRand(tc.partitionId)
+ val fraction = fractionByKey(item._1)
+ val stratum = result.getEntry(item._1)
+ if (withReplacement) {
+ // compute q1 and q2 only if they haven't been computed already
+ // since they don't change from iteration to iteration.
+ // TODO change this to the streaming version
+ if (stratum.q1.isEmpty || stratum.q2.isEmpty) {
+ val n = counts.get(item._1)
+ val s = math.ceil(n * fraction).toLong
+ val lmbd1 = PB.getLowerBound(s)
+ val minCount = PB.getMinCount(lmbd1)
+ val lmbd2 = if (lmbd1 == 0) PB.getUpperBound(s) else PB.getUpperBound(s - minCount)
+ val q1 = lmbd1 / n
+ val q2 = lmbd2 / n
+ stratum.q1 = Some(q1)
+ stratum.q2 = Some(q2)
+ }
+ val x1 = if (stratum.q1.get == 0) 0L else rng.nextPoisson(stratum.q1.get)
+ if (x1 > 0) {
+ stratum.incrNumAccepted(x1)
+ }
+ val x2 = rng.nextPoisson(stratum.q2.get).toInt
+ if (x2 > 0) {
+ stratum.addToWaitList(ArrayBuffer.fill(x2)(rng.nextUniform(0.0, 1.0)))
+ }
+ } else {
+ // We use the streaming version of the algorithm for sampling without replacement.
+ // Hence, q1 and q2 change on every iteration.
+ val g1 = - math.log(delta) / stratum.numItems
+ val g2 = (2.0 / 3.0) * g1
+ val q1 = math.max(0, fraction + g2 - math.sqrt((g2 * g2 + 3 * g2 * fraction)))
+ val q2 = math.min(1, fraction + g1 + math.sqrt(g1 * g1 + 2 * g1 * fraction))
+
+ val x = rng.nextUniform(0.0, 1.0)
+ if (x < q1) {
+ stratum.incrNumAccepted()
+ } else if (x < q2) {
+ stratum.addToWaitList(x)
+ }
+ stratum.q1 = Some(q1)
+ stratum.q2 = Some(q2)
+ }
+ stratum.incrNumItems()
+ result
+ }
+ }
+
+ /**
+ * Returns the function used by aggregate to combine results from different partitions, as
+ * returned by seqOp.
+ */
+ def getCombOp[K](): (Result[K], Result[K]) => Result[K] = {
+ (r1: Result[K], r2: Result[K]) => {
+ // take union of both key sets in case one partition doesn't contain all keys
+ val keyUnion = r1.resultMap.keys.toSet.union(r2.resultMap.keys.toSet)
+
+ // Use r2 to keep the combined result since r1 is usual empty
+ for (key <- keyUnion) {
+ val entry1 = r1.resultMap.get(key)
+ val entry2 = r2.resultMap.get(key)
+ if (entry2.isEmpty && entry1.isDefined) {
+ r2.resultMap += (key -> entry1.get)
+ } else if (entry1.isDefined && entry2.isDefined) {
+ entry2.get.addToWaitList(entry1.get.waitList)
+ entry2.get.incrNumAccepted(entry1.get.numAccepted)
+ entry2.get.incrNumItems(entry1.get.numItems)
+ }
+ }
+ r2
+ }
+ }
+
+ /**
+ * Given the result returned by the aggregate function, we need to determine the threshold used
+ * to accept items to generate the exact sample size.
+ */
+ def computeThresholdByKey[K](finalResult: Map[K, Stratum], fractionByKey: (K => Double)):
+ (K => Double) = {
+ val thresholdByKey = new mutable.HashMap[K, Double]()
+ for ((key, stratum) <- finalResult) {
+ val fraction = fractionByKey(key)
+ val s = math.ceil(stratum.numItems * fraction).toLong
+ if (stratum.numAccepted > s) {
+ logWarning("Pre-accepted too many")
+ thresholdByKey += (key -> stratum.q1.get)
+ } else {
+ val numWaitListAccepted = (s - stratum.numAccepted).toInt
+ if (numWaitListAccepted >= stratum.waitList.size) {
+ logWarning("WaitList too short")
+ thresholdByKey += (key -> stratum.q2.get)
+ } else {
+ thresholdByKey += (key -> stratum.waitList.sorted.apply(numWaitListAccepted))
+ }
+ }
+ }
+ thresholdByKey
+ }
+
+ /**
+ * Return the per partition sampling function used for sampling without replacement.
+ *
+ * When exact sample size is required, we make an additional pass over the RDD to determine the
+ * exact sampling rate that guarantees sample size with high confidence.
+ *
+ * The sampling function has a unique seed per partition.
+ */
+ def getBernoulliSamplingFunction[K, V](rdd: RDD[(K, V)],
+ fractionByKey: K => Double,
+ exact: Boolean,
+ seed: Long): (Int, Iterator[(K, V)]) => Iterator[(K, V)] = {
+ var samplingRateByKey = fractionByKey
+ if (exact) {
+ // determine threshold for each stratum and resample
+ val seqOp = StratifiedSampler.getSeqOp[K,V](false, fractionByKey, None)
+ val combOp = StratifiedSampler.getCombOp[K]()
+ val zeroU = new Result[K](Map[K, Stratum](), seed = seed)
+ val finalResult = aggregateWithContext(zeroU)(rdd, seqOp, combOp).resultMap
+ samplingRateByKey = StratifiedSampler.computeThresholdByKey(finalResult, fractionByKey)
+ }
+ (idx: Int, iter: Iterator[(K, V)]) => {
+ val random = new RandomDataGenerator
+ random.reSeed(seed + idx)
+ iter.filter(t => random.nextUniform(0.0, 1.0) < samplingRateByKey(t._1))
+ }
+ }
+
+ /**
+ * Return the per partition sampling function used for sampling with replacement.
+ *
+ * When exact sample size is required, we make two additional passed over the RDD to determine
+ * the exact sampling rate that guarantees sample size with high confidence. The first pass
+ * counts the number of items in each stratum (group of items with the same key) in the RDD, and
+ * the second pass uses the counts to determine exact sampling rates.
+ *
+ * The sampling function has a unique seed per partition.
+ */
+ def getPoissonSamplingFunction[K, V](rdd:RDD[(K, V)],
+ fractionByKey: K => Double,
+ exact: Boolean,
+ counts: Option[Map[K, Long]],
+ seed: Long): (Int, Iterator[(K, V)]) => Iterator[(K, V)] = {
+ // TODO implement the streaming version of sampling w/ replacement that doesn't require counts
+ if (exact) {
+ val seqOp = StratifiedSampler.getSeqOp[K,V](true, fractionByKey, counts)
+ val combOp = StratifiedSampler.getCombOp[K]()
+ val zeroU = new Result[K](Map[K, Stratum](), seed = seed)
+ val finalResult = aggregateWithContext(zeroU)(rdd, seqOp, combOp).resultMap
+ val thresholdByKey = StratifiedSampler.computeThresholdByKey(finalResult, fractionByKey)
+ (idx: Int, iter: Iterator[(K, V)]) => {
+ val random = new RandomDataGenerator()
+ random.reSeed(seed + idx)
+ iter.flatMap { t =>
+ val q1 = finalResult.get(t._1).get.q1.getOrElse(0.0)
+ val q2 = finalResult.get(t._1).get.q2.getOrElse(0.0)
+ val x1 = if (q1 == 0) 0L else random.nextPoisson(q1)
+ val x2 = random.nextPoisson(q2).toInt
+ val x = x1 + (0 until x2).count(i => random.nextUniform(0.0, 1.0) < thresholdByKey(t._1))
+ if (x > 0) {
+ Iterator.fill(x.toInt)(t)
+ } else {
+ Iterator.empty
+ }
+ }
+ }
+ } else {
+ (idx: Int, iter: Iterator[(K, V)]) => {
+ val random = new RandomDataGenerator()
+ random.reSeed(seed + idx)
+ iter.flatMap { t =>
+ val count = random.nextPoisson(fractionByKey(t._1)).toInt
+ if (count > 0) {
+ Iterator.fill(count)(t)
+ } else {
+ Iterator.empty
+ }
+ }
+ }
+ }
+ }
+}
+
+/**
+ * Object used by seqOp to keep track of the number of items accepted and items waitlisted per
+ * stratum, as well as the bounds for accepting and waitlisting items.
+ */
+private[random] class Stratum(var numItems: Long = 0L, var numAccepted: Long = 0L)
+ extends Serializable {
+
+ var waitList: ArrayBuffer[Double] = new ArrayBuffer[Double]
+ var q1: Option[Double] = None // upper bound for accepting item instantly
--- End diff --
We are setting q1 and q2 from outside, let's add more methods to this class for setting and getting values of q1 and q2. Let's use better names than ```updateq1()```. Even better would be if we rename q1 and q2 to more descriptive names (the comments you added help a lot by the way).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by dorx <gi...@git.apache.org>.
Github user dorx commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r14907781
--- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
@@ -0,0 +1,311 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util.random
+
+import scala.collection.Map
+import scala.collection.mutable.{ArrayBuffer, HashMap, Map => MMap}
+
+import org.apache.commons.math3.random.RandomDataGenerator
+import org.apache.spark.Logging
+import org.apache.spark.rdd.RDD
+
+/**
+ * Auxiliary functions and data structures for the sampleByKey method in PairRDDFunctions.
+ *
+ * Essentially, when exact sample size is necessary, we make additional passes over the RDD to
+ * compute the exact threshold value to use for each stratum to guarantee exact sample size with
+ * high probability. This is achieved by maintaining a waitlist of size O(log(s)), where s is the
+ * desired sample size for each stratum.
+ *
+ * Like in simple random sampling, we generate a random value for each item from the
+ * uniform distribution [0.0, 1.0]. All items with values <= min(values of items in the waitlist)
+ * are accepted into the sample instantly. The threshold for instant accept is designed so that
+ * s - numAccepted = O(log(s)), where s is again the desired sample size. Thus, by maintaining a
+ * waitlist size = O(log(s)), we will be able to create a sample of the exact size s by adding
+ * a portion of the waitlist to the set of items that are instantly accepted. The exact threshold
+ * is computed by sorting the values in the waitlist and picking the value at (s - numAccepted).
+ *
+ * Note that since we use the same seed for the RNG when computing the thresholds and the actual
+ * sample, our computed thresholds are guaranteed to produce the desired sample size.
+ *
+ * For more theoretical background on the sampling techniques used here, please refer to
+ * http://jmlr.org/proceedings/papers/v28/meng13a.html
+ */
+
+private[spark] object StratifiedSampler extends Logging {
+
+ /**
+ * Count the number of items instantly accepted and generate the waitlist for each stratum.
+ *
+ * This is only invoked when exact sample size is required.
+ */
+ def getCounts[K, V](rdd: RDD[(K, V)],
+ withReplacement: Boolean,
+ fractions: Map[K, Double],
+ counts: Option[Map[K, Long]],
+ seed: Long): MMap[K, Stratum] = {
+ val combOp = getCombOp[K]
+ val mappedPartitionRDD = rdd.mapPartitionsWithIndex({ case (partition, iter) =>
+ val zeroU: MMap[K, Stratum] = new HashMap[K, Stratum]()
+ val rng = new RandomDataGenerator()
+ rng.reSeed(seed + partition)
+ val seqOp = getSeqOp(withReplacement, fractions, rng, counts)
+ Iterator(iter.aggregate(zeroU)(seqOp, combOp))
+ }, preservesPartitioning=true)
+ mappedPartitionRDD.reduce(combOp)
+ }
+
+ /**
+ * Returns the function used by aggregate to collect sampling statistics for each partition.
+ */
+ def getSeqOp[K, V](withReplacement: Boolean,
+ fractions: Map[K, Double],
+ rng: RandomDataGenerator,
+ counts: Option[Map[K, Long]]): (MMap[K, Stratum], (K, V)) => MMap[K, Stratum] = {
+ val delta = 5e-5
+ (result: MMap[K, Stratum], item: (K, V)) => {
+ val key = item._1
+ val fraction = fractions(key)
+ if (!result.contains(key)) {
+ result += (key -> new Stratum())
+ }
+ val stratum = result(key)
+
+ if (withReplacement) {
+ // compute acceptBound and waitListBound only if they haven't been computed already
+ // since they don't change from iteration to iteration.
+ // TODO change this to the streaming version
+ if (stratum.areBoundsEmpty) {
+ val n = counts.get(key)
+ val sampleSize = math.ceil(n * fraction).toLong
+ val lmbd1 = PoissonBounds.getLowerBound(sampleSize)
+ val minCount = PoissonBounds.getMinCount(lmbd1)
+ val lmbd2 = if (lmbd1 == 0) {
+ PoissonBounds.getUpperBound(sampleSize)
+ } else {
+ PoissonBounds.getUpperBound(sampleSize - minCount)
+ }
+ stratum.acceptBound = lmbd1 / n
+ stratum.waitListBound = lmbd2 / n
+ }
+ val acceptBound = stratum.acceptBound
+ val copiesAccepted = if (acceptBound == 0.0) 0L else rng.nextPoisson(acceptBound)
+ if (copiesAccepted > 0) {
+ stratum.incrNumAccepted(copiesAccepted)
+ }
+ val copiesWaitlisted = rng.nextPoisson(stratum.waitListBound).toInt
+ if (copiesWaitlisted > 0) {
+ stratum.addToWaitList(ArrayBuffer.fill(copiesWaitlisted)(rng.nextUniform(0.0, 1.0)))
+ }
+ } else {
+ // We use the streaming version of the algorithm for sampling without replacement to avoid
+ // using an extra pass over the RDD for computing the count.
+ // Hence, acceptBound and waitListBound change on every iteration.
+ val gamma1 = - math.log(delta) / stratum.numItems
+ val gamma2 = (2.0 / 3.0) * gamma1
+ stratum.acceptBound = math.max(0,
+ fraction + gamma2 - math.sqrt(gamma2 * gamma2 + 3 * gamma2 * fraction))
+ stratum.waitListBound = math.min(1,
+ fraction + gamma1 + math.sqrt(gamma1 * gamma1 + 2 * gamma1 * fraction))
+
+ val x = rng.nextUniform(0.0, 1.0)
+ if (x < stratum.acceptBound) {
+ stratum.incrNumAccepted()
+ } else if (x < stratum.waitListBound) {
+ stratum.addToWaitList(x)
+ }
+ }
+ stratum.incrNumItems()
+ result
+ }
+ }
+
+ /**
+ * Returns the function used combine results returned by seqOp from different partitions.
+ */
+ def getCombOp[K]: (MMap[K, Stratum], MMap[K, Stratum]) => MMap[K, Stratum] = {
+ (result1: MMap[K, Stratum], result2: MMap[K, Stratum]) => {
+ // take union of both key sets in case one partition doesn't contain all keys
+ for (key <- result1.keySet.union(result2.keySet)) {
+ // Use result2 to keep the combined result since r1 is usual empty
--- End diff --
Yeah but we're not using RDD.aggregate here. We're using Scala's TraversableOnce.aggregate
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1025#issuecomment-45951336
Build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r14694378
--- Diff: core/src/test/scala/org/apache/spark/rdd/PairRDDFunctionsSuite.scala ---
@@ -83,6 +83,125 @@ class PairRDDFunctionsSuite extends FunSuite with SharedSparkContext {
assert(valuesFor2.toList.sorted === List(1))
}
+ test("sampleByKey") {
+ def stratifier (fractionPositive: Double) = {
+ (x: Int) => if (x % 10 < (10 * fractionPositive).toInt) "1" else "0"
+ }
+
+ def checkSize(exact: Boolean,
+ withReplacement: Boolean,
+ expected: Long,
+ actual: Long,
+ p: Double): Boolean = {
+ if (exact) {
+ return expected == actual
+ }
+ val stdev = if (withReplacement) math.sqrt(expected) else math.sqrt(expected * p * (1 - p))
+ // Very forgiving margin since we're dealing with very small sample sizes most of the time
+ math.abs(actual - expected) <= 6 * stdev
+ }
+
+ // Without replacement validation
+ def takeSampleAndValidateBernoulli(stratifiedData: RDD[(String, Int)],
+ exact: Boolean,
+ samplingRate: Double,
+ seed: Long,
+ n: Long) = {
+ val expectedSampleSize = stratifiedData.countByKey().mapValues(count =>
--- End diff --
better merge `.mapValues` to the line below.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by dorx <gi...@git.apache.org>.
Github user dorx commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r14907896
--- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
@@ -0,0 +1,311 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util.random
+
+import scala.collection.Map
+import scala.collection.mutable.{ArrayBuffer, HashMap, Map => MMap}
+
+import org.apache.commons.math3.random.RandomDataGenerator
+import org.apache.spark.Logging
+import org.apache.spark.rdd.RDD
+
+/**
+ * Auxiliary functions and data structures for the sampleByKey method in PairRDDFunctions.
+ *
+ * Essentially, when exact sample size is necessary, we make additional passes over the RDD to
+ * compute the exact threshold value to use for each stratum to guarantee exact sample size with
+ * high probability. This is achieved by maintaining a waitlist of size O(log(s)), where s is the
+ * desired sample size for each stratum.
+ *
+ * Like in simple random sampling, we generate a random value for each item from the
+ * uniform distribution [0.0, 1.0]. All items with values <= min(values of items in the waitlist)
+ * are accepted into the sample instantly. The threshold for instant accept is designed so that
+ * s - numAccepted = O(log(s)), where s is again the desired sample size. Thus, by maintaining a
+ * waitlist size = O(log(s)), we will be able to create a sample of the exact size s by adding
+ * a portion of the waitlist to the set of items that are instantly accepted. The exact threshold
+ * is computed by sorting the values in the waitlist and picking the value at (s - numAccepted).
+ *
+ * Note that since we use the same seed for the RNG when computing the thresholds and the actual
+ * sample, our computed thresholds are guaranteed to produce the desired sample size.
+ *
+ * For more theoretical background on the sampling techniques used here, please refer to
+ * http://jmlr.org/proceedings/papers/v28/meng13a.html
+ */
+
+private[spark] object StratifiedSampler extends Logging {
+
+ /**
+ * Count the number of items instantly accepted and generate the waitlist for each stratum.
+ *
+ * This is only invoked when exact sample size is required.
+ */
+ def getCounts[K, V](rdd: RDD[(K, V)],
+ withReplacement: Boolean,
+ fractions: Map[K, Double],
+ counts: Option[Map[K, Long]],
+ seed: Long): MMap[K, Stratum] = {
+ val combOp = getCombOp[K]
+ val mappedPartitionRDD = rdd.mapPartitionsWithIndex({ case (partition, iter) =>
+ val zeroU: MMap[K, Stratum] = new HashMap[K, Stratum]()
+ val rng = new RandomDataGenerator()
+ rng.reSeed(seed + partition)
+ val seqOp = getSeqOp(withReplacement, fractions, rng, counts)
+ Iterator(iter.aggregate(zeroU)(seqOp, combOp))
+ }, preservesPartitioning=true)
+ mappedPartitionRDD.reduce(combOp)
+ }
+
+ /**
+ * Returns the function used by aggregate to collect sampling statistics for each partition.
+ */
+ def getSeqOp[K, V](withReplacement: Boolean,
+ fractions: Map[K, Double],
+ rng: RandomDataGenerator,
+ counts: Option[Map[K, Long]]): (MMap[K, Stratum], (K, V)) => MMap[K, Stratum] = {
+ val delta = 5e-5
+ (result: MMap[K, Stratum], item: (K, V)) => {
+ val key = item._1
+ val fraction = fractions(key)
+ if (!result.contains(key)) {
+ result += (key -> new Stratum())
+ }
+ val stratum = result(key)
+
+ if (withReplacement) {
+ // compute acceptBound and waitListBound only if they haven't been computed already
+ // since they don't change from iteration to iteration.
+ // TODO change this to the streaming version
+ if (stratum.areBoundsEmpty) {
+ val n = counts.get(key)
+ val sampleSize = math.ceil(n * fraction).toLong
+ val lmbd1 = PoissonBounds.getLowerBound(sampleSize)
+ val minCount = PoissonBounds.getMinCount(lmbd1)
+ val lmbd2 = if (lmbd1 == 0) {
+ PoissonBounds.getUpperBound(sampleSize)
+ } else {
+ PoissonBounds.getUpperBound(sampleSize - minCount)
+ }
+ stratum.acceptBound = lmbd1 / n
+ stratum.waitListBound = lmbd2 / n
+ }
+ val acceptBound = stratum.acceptBound
+ val copiesAccepted = if (acceptBound == 0.0) 0L else rng.nextPoisson(acceptBound)
+ if (copiesAccepted > 0) {
+ stratum.incrNumAccepted(copiesAccepted)
+ }
+ val copiesWaitlisted = rng.nextPoisson(stratum.waitListBound).toInt
+ if (copiesWaitlisted > 0) {
+ stratum.addToWaitList(ArrayBuffer.fill(copiesWaitlisted)(rng.nextUniform(0.0, 1.0)))
+ }
+ } else {
+ // We use the streaming version of the algorithm for sampling without replacement to avoid
+ // using an extra pass over the RDD for computing the count.
+ // Hence, acceptBound and waitListBound change on every iteration.
+ val gamma1 = - math.log(delta) / stratum.numItems
+ val gamma2 = (2.0 / 3.0) * gamma1
+ stratum.acceptBound = math.max(0,
+ fraction + gamma2 - math.sqrt(gamma2 * gamma2 + 3 * gamma2 * fraction))
+ stratum.waitListBound = math.min(1,
+ fraction + gamma1 + math.sqrt(gamma1 * gamma1 + 2 * gamma1 * fraction))
+
+ val x = rng.nextUniform(0.0, 1.0)
+ if (x < stratum.acceptBound) {
+ stratum.incrNumAccepted()
+ } else if (x < stratum.waitListBound) {
+ stratum.addToWaitList(x)
+ }
+ }
+ stratum.incrNumItems()
+ result
+ }
+ }
+
+ /**
+ * Returns the function used combine results returned by seqOp from different partitions.
+ */
+ def getCombOp[K]: (MMap[K, Stratum], MMap[K, Stratum]) => MMap[K, Stratum] = {
+ (result1: MMap[K, Stratum], result2: MMap[K, Stratum]) => {
+ // take union of both key sets in case one partition doesn't contain all keys
+ for (key <- result1.keySet.union(result2.keySet)) {
+ // Use result2 to keep the combined result since r1 is usual empty
+ val entry1 = result1.get(key)
+ if (result2.contains(key)) {
+ result2(key).merge(entry1)
+ } else {
+ if (entry1.isDefined) {
+ result2 += (key -> entry1.get)
+ }
+ }
+ }
+ result2
+ }
+ }
+
+ /**
+ * Given the result returned by getCounts, determine the threshold for accepting items to
+ * generate exact sample size.
+ *
+ * To do so, we compute sampleSize = math.ceil(size * samplingRate) for each stratum and compare
+ * it to the number of items that were accepted instantly and the number of items in the waitlist
+ * for that stratum. Most of the time, numAccepted <= sampleSize <= (numAccepted + numWaitlisted),
+ * which means we need to sort the elements in the waitlist by their associated values in order
+ * to find the value T s.t. |{elements in the stratum whose associated values <= T}| = sampleSize.
+ * Note that all elements in the waitlist have values >= bound for instant accept, so a T value
+ * in the waitlist range would allow all elements that were instantly accepted on the first pass
+ * to be included in the sample.
+ */
+ def computeThresholdByKey[K](finalResult: Map[K, Stratum],
+ fractions: Map[K, Double]): Map[K, Double] = {
+ val thresholdByKey = new HashMap[K, Double]()
+ for ((key, stratum) <- finalResult) {
+ val sampleSize = math.ceil(stratum.numItems * fractions(key)).toLong
+ if (stratum.numAccepted > sampleSize) {
+ logWarning("Pre-accepted too many")
+ thresholdByKey += (key -> stratum.acceptBound)
+ } else {
+ val numWaitListAccepted = (sampleSize - stratum.numAccepted).toInt
+ if (numWaitListAccepted >= stratum.waitList.size) {
+ logWarning("WaitList too short")
+ thresholdByKey += (key -> stratum.waitListBound)
+ } else {
+ thresholdByKey += (key -> stratum.waitList.sorted.apply(numWaitListAccepted))
+ }
+ }
+ }
+ thresholdByKey
+ }
+
+ /**
+ * Return the per partition sampling function used for sampling without replacement.
+ *
+ * When exact sample size is required, we make an additional pass over the RDD to determine the
+ * exact sampling rate that guarantees sample size with high confidence.
+ *
+ * The sampling function has a unique seed per partition.
+ */
+ def getBernoulliSamplingFunction[K, V](rdd: RDD[(K, V)],
+ fractions: Map[K, Double],
+ exact: Boolean,
+ seed: Long): (Int, Iterator[(K, V)]) => Iterator[(K, V)] = {
+ var samplingRateByKey = fractions
+ if (exact) {
+ // determine threshold for each stratum and resample
+ val finalResult = getCounts(rdd, false, fractions, None, seed)
+ samplingRateByKey = computeThresholdByKey(finalResult, fractions)
+ }
+ (idx: Int, iter: Iterator[(K, V)]) => {
+ val rng = new RandomDataGenerator
+ rng.reSeed(seed + idx)
+ iter.filter(t => rng.nextUniform(0.0, 1.0) < samplingRateByKey(t._1))
+ }
+ }
+
+ /**
+ * Return the per partition sampling function used for sampling with replacement.
+ *
+ * When exact sample size is required, we make two additional passed over the RDD to determine
+ * the exact sampling rate that guarantees sample size with high confidence. The first pass
+ * counts the number of items in each stratum (group of items with the same key) in the RDD, and
+ * the second pass uses the counts to determine exact sampling rates.
+ *
+ * The sampling function has a unique seed per partition.
+ */
+ def getPoissonSamplingFunction[K, V](rdd: RDD[(K, V)],
+ fractions: Map[K, Double],
+ exact: Boolean,
+ counts: Option[Map[K, Long]],
+ seed: Long): (Int, Iterator[(K, V)]) => Iterator[(K, V)] = {
+ // TODO implement the streaming version of sampling w/ replacement that doesn't require counts
+ if (exact) {
+ val finalResult = getCounts(rdd, true, fractions, counts, seed)
+ val thresholdByKey = computeThresholdByKey(finalResult, fractions)
+ (idx: Int, iter: Iterator[(K, V)]) => {
+ val rng = new RandomDataGenerator()
+ rng.reSeed(seed + idx)
+ iter.flatMap { item =>
+ val key = item._1
+ val acceptBound = finalResult(key).acceptBound
+ val copiesAccepted = if (acceptBound == 0) 0L else rng.nextPoisson(acceptBound)
+ val copiesWailisted = rng.nextPoisson(finalResult(key).waitListBound).toInt
+ val copiesInSample = copiesAccepted +
+ (0 until copiesWailisted).count(i => rng.nextUniform(0.0, 1.0) < thresholdByKey(key))
+ if (copiesInSample > 0) {
+ Iterator.fill(copiesInSample.toInt)(item)
+ } else {
+ Iterator.empty
+ }
+ }
+ }
+ } else {
+ (idx: Int, iter: Iterator[(K, V)]) => {
+ val rng = new RandomDataGenerator()
+ rng.reSeed(seed + idx)
+ iter.flatMap { item =>
+ val count = rng.nextPoisson(fractions(item._1)).toInt
+ if (count > 0) {
+ Iterator.fill(count)(item)
+ } else {
+ Iterator.empty
+ }
+ }
+ }
+ }
+ }
+}
+
+/**
+ * Object used by seqOp to keep track of the number of items accepted and items waitlisted per
+ * stratum, as well as the bounds for accepting and waitlisting items.
+ *
+ * `[random]` here is necessary since it's in the return type signature of seqOp defined above
+ */
+private[random] class Stratum(var numItems: Long = 0L, var numAccepted: Long = 0L)
+ extends Serializable {
+
+ private val _waitList = new ArrayBuffer[Double]
+ private var _acceptBound: Option[Double] = None // upper bound for accepting item instantly
+ private var _waitListBound: Option[Double] = None // upper bound for adding item to waitlist
+
+ def incrNumItems(by: Long = 1L) = numItems += by
+
+ def incrNumAccepted(by: Long = 1L) = numAccepted += by
+
+ def addToWaitList(elem: Double) = _waitList += elem
+
+ def addToWaitList(elems: ArrayBuffer[Double]) = _waitList ++= elems
+
+ def waitList = _waitList
+
+ def acceptBound = _acceptBound.getOrElse(0.0)
+
+ def acceptBound_= (value: Double) = _acceptBound = Some(value)
+
+ def waitListBound = _waitListBound.getOrElse(0.0)
+
+ def waitListBound_= (value: Double) = _waitListBound = Some(value)
+
+ def areBoundsEmpty = _acceptBound.isEmpty || _waitListBound.isEmpty
+
+ def merge(other: Option[Stratum]): Unit = {
--- End diff --
Nope.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r14694277
--- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
@@ -0,0 +1,311 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util.random
+
+import scala.collection.Map
+import scala.collection.mutable.{ArrayBuffer, HashMap, Map => MMap}
+
+import org.apache.commons.math3.random.RandomDataGenerator
+import org.apache.spark.Logging
+import org.apache.spark.rdd.RDD
+
+/**
+ * Auxiliary functions and data structures for the sampleByKey method in PairRDDFunctions.
+ *
+ * Essentially, when exact sample size is necessary, we make additional passes over the RDD to
+ * compute the exact threshold value to use for each stratum to guarantee exact sample size with
+ * high probability. This is achieved by maintaining a waitlist of size O(log(s)), where s is the
+ * desired sample size for each stratum.
+ *
+ * Like in simple random sampling, we generate a random value for each item from the
+ * uniform distribution [0.0, 1.0]. All items with values <= min(values of items in the waitlist)
+ * are accepted into the sample instantly. The threshold for instant accept is designed so that
+ * s - numAccepted = O(log(s)), where s is again the desired sample size. Thus, by maintaining a
+ * waitlist size = O(log(s)), we will be able to create a sample of the exact size s by adding
+ * a portion of the waitlist to the set of items that are instantly accepted. The exact threshold
+ * is computed by sorting the values in the waitlist and picking the value at (s - numAccepted).
+ *
+ * Note that since we use the same seed for the RNG when computing the thresholds and the actual
+ * sample, our computed thresholds are guaranteed to produce the desired sample size.
+ *
+ * For more theoretical background on the sampling techniques used here, please refer to
+ * http://jmlr.org/proceedings/papers/v28/meng13a.html
+ */
+
+private[spark] object StratifiedSampler extends Logging {
+
+ /**
+ * Count the number of items instantly accepted and generate the waitlist for each stratum.
+ *
+ * This is only invoked when exact sample size is required.
+ */
+ def getCounts[K, V](rdd: RDD[(K, V)],
+ withReplacement: Boolean,
+ fractions: Map[K, Double],
+ counts: Option[Map[K, Long]],
+ seed: Long): MMap[K, Stratum] = {
+ val combOp = getCombOp[K]
+ val mappedPartitionRDD = rdd.mapPartitionsWithIndex({ case (partition, iter) =>
+ val zeroU: MMap[K, Stratum] = new HashMap[K, Stratum]()
+ val rng = new RandomDataGenerator()
+ rng.reSeed(seed + partition)
+ val seqOp = getSeqOp(withReplacement, fractions, rng, counts)
+ Iterator(iter.aggregate(zeroU)(seqOp, combOp))
+ }, preservesPartitioning=true)
+ mappedPartitionRDD.reduce(combOp)
+ }
+
+ /**
+ * Returns the function used by aggregate to collect sampling statistics for each partition.
+ */
+ def getSeqOp[K, V](withReplacement: Boolean,
+ fractions: Map[K, Double],
+ rng: RandomDataGenerator,
+ counts: Option[Map[K, Long]]): (MMap[K, Stratum], (K, V)) => MMap[K, Stratum] = {
+ val delta = 5e-5
+ (result: MMap[K, Stratum], item: (K, V)) => {
+ val key = item._1
+ val fraction = fractions(key)
+ if (!result.contains(key)) {
+ result += (key -> new Stratum())
+ }
+ val stratum = result(key)
+
+ if (withReplacement) {
+ // compute acceptBound and waitListBound only if they haven't been computed already
+ // since they don't change from iteration to iteration.
+ // TODO change this to the streaming version
+ if (stratum.areBoundsEmpty) {
+ val n = counts.get(key)
+ val sampleSize = math.ceil(n * fraction).toLong
+ val lmbd1 = PoissonBounds.getLowerBound(sampleSize)
+ val minCount = PoissonBounds.getMinCount(lmbd1)
--- End diff --
We already have the exact count for the stratum. Is it necessary?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r14694259
--- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
@@ -0,0 +1,311 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util.random
+
+import scala.collection.Map
+import scala.collection.mutable.{ArrayBuffer, HashMap, Map => MMap}
+
+import org.apache.commons.math3.random.RandomDataGenerator
--- End diff --
Separate 3rd party imports from spark imports.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by falaki <gi...@git.apache.org>.
Github user falaki commented on the pull request:
https://github.com/apache/spark/pull/1025#issuecomment-48417024
@dorx Thanks for updating this. I left a few style and minor naming comments. Other than those this looks good to me. @mengxr Would you please take a look at this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1025#issuecomment-45544398
Build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1025#issuecomment-46376893
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by falaki <gi...@git.apache.org>.
Github user falaki commented on the pull request:
https://github.com/apache/spark/pull/1025#issuecomment-50104259
This is the first place we introduce 'exact' to our API. We already have 'approx' in function names. I think having both of them is confusing to users.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by falaki <gi...@git.apache.org>.
Github user falaki commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r14544125
--- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util.random
+
+import scala.collection.{Map, mutable}
+import scala.collection.mutable.ArrayBuffer
+import scala.reflect.ClassTag
+
+import org.apache.commons.math3.random.RandomDataGenerator
+import org.apache.spark.{Logging, SparkContext, TaskContext}
+import org.apache.spark.rdd.RDD
+import org.apache.spark.util.Utils
+import org.apache.spark.util.random.{PoissonBounds => PB}
+
+private[spark] object StratifiedSampler extends Logging {
+
+ /**
+ * A version of {@link #aggregate()} that passes the TaskContext to the function that does
+ * aggregation for each partition. This function avoids creating an extra depth in the RDD
+ * lineage, as opposed to using mapPartitionsWithId, which results in slightly improved run time.
+ */
+ def aggregateWithContext[U: ClassTag, T: ClassTag](zeroValue: U)
+ (rdd: RDD[T],
+ seqOp: ((TaskContext, U), T) => U,
+ combOp: (U, U) => U): U = {
+ val sc: SparkContext = rdd.sparkContext
+ // Clone the zero value since we will also be serializing it as part of tasks
+ var jobResult = Utils.clone(zeroValue, sc.env.closureSerializer.newInstance())
+ // pad seqOp and combOp with taskContext to conform to aggregate's signature in TraversableOnce
+ val paddedSeqOp = (arg1: (TaskContext, U), item: T) => (arg1._1, seqOp(arg1, item))
+ val paddedcombOp = (arg1: (TaskContext, U), arg2: (TaskContext, U)) =>
+ (arg1._1, combOp(arg1._2, arg1._2))
+ val cleanSeqOp = sc.clean(paddedSeqOp)
+ val cleanCombOp = sc.clean(paddedcombOp)
+ val aggregatePartition = (tc: TaskContext, it: Iterator[T]) =>
+ (it.aggregate(tc, zeroValue)(cleanSeqOp, cleanCombOp))._2
+ val mergeResult = (index: Int, taskResult: U) => jobResult = combOp(jobResult, taskResult)
+ sc.runJob(rdd, aggregatePartition, mergeResult)
+ jobResult
+ }
+
+ /**
+ * Returns the function used by aggregate to collect sampling statistics for each partition.
+ */
+ def getSeqOp[K, V](withReplacement: Boolean,
+ fractionByKey: (K => Double),
+ counts: Option[Map[K, Long]]): ((TaskContext, Result[K]),(K, V)) => Result[K] = {
+ val delta = 5e-5
+ (U: (TaskContext, Result[K]), item: (K, V)) => {
--- End diff --
We use uppercase letters for type parameters. Let's rename 'U' to a more descriptive name.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1025#issuecomment-46609787
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1025#issuecomment-45553585
Build finished.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1025#issuecomment-46609768
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by dorx <gi...@git.apache.org>.
Github user dorx commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r14906919
--- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
@@ -0,0 +1,311 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util.random
+
+import scala.collection.Map
+import scala.collection.mutable.{ArrayBuffer, HashMap, Map => MMap}
+
+import org.apache.commons.math3.random.RandomDataGenerator
+import org.apache.spark.Logging
+import org.apache.spark.rdd.RDD
+
+/**
+ * Auxiliary functions and data structures for the sampleByKey method in PairRDDFunctions.
+ *
+ * Essentially, when exact sample size is necessary, we make additional passes over the RDD to
+ * compute the exact threshold value to use for each stratum to guarantee exact sample size with
+ * high probability. This is achieved by maintaining a waitlist of size O(log(s)), where s is the
+ * desired sample size for each stratum.
+ *
+ * Like in simple random sampling, we generate a random value for each item from the
+ * uniform distribution [0.0, 1.0]. All items with values <= min(values of items in the waitlist)
+ * are accepted into the sample instantly. The threshold for instant accept is designed so that
+ * s - numAccepted = O(log(s)), where s is again the desired sample size. Thus, by maintaining a
+ * waitlist size = O(log(s)), we will be able to create a sample of the exact size s by adding
+ * a portion of the waitlist to the set of items that are instantly accepted. The exact threshold
+ * is computed by sorting the values in the waitlist and picking the value at (s - numAccepted).
+ *
+ * Note that since we use the same seed for the RNG when computing the thresholds and the actual
+ * sample, our computed thresholds are guaranteed to produce the desired sample size.
+ *
+ * For more theoretical background on the sampling techniques used here, please refer to
+ * http://jmlr.org/proceedings/papers/v28/meng13a.html
+ */
+
+private[spark] object StratifiedSampler extends Logging {
+
+ /**
+ * Count the number of items instantly accepted and generate the waitlist for each stratum.
+ *
+ * This is only invoked when exact sample size is required.
+ */
+ def getCounts[K, V](rdd: RDD[(K, V)],
+ withReplacement: Boolean,
+ fractions: Map[K, Double],
+ counts: Option[Map[K, Long]],
+ seed: Long): MMap[K, Stratum] = {
+ val combOp = getCombOp[K]
+ val mappedPartitionRDD = rdd.mapPartitionsWithIndex({ case (partition, iter) =>
+ val zeroU: MMap[K, Stratum] = new HashMap[K, Stratum]()
+ val rng = new RandomDataGenerator()
+ rng.reSeed(seed + partition)
+ val seqOp = getSeqOp(withReplacement, fractions, rng, counts)
+ Iterator(iter.aggregate(zeroU)(seqOp, combOp))
+ }, preservesPartitioning=true)
+ mappedPartitionRDD.reduce(combOp)
+ }
+
+ /**
+ * Returns the function used by aggregate to collect sampling statistics for each partition.
+ */
+ def getSeqOp[K, V](withReplacement: Boolean,
+ fractions: Map[K, Double],
+ rng: RandomDataGenerator,
+ counts: Option[Map[K, Long]]): (MMap[K, Stratum], (K, V)) => MMap[K, Stratum] = {
+ val delta = 5e-5
+ (result: MMap[K, Stratum], item: (K, V)) => {
+ val key = item._1
+ val fraction = fractions(key)
+ if (!result.contains(key)) {
+ result += (key -> new Stratum())
+ }
+ val stratum = result(key)
+
+ if (withReplacement) {
+ // compute acceptBound and waitListBound only if they haven't been computed already
+ // since they don't change from iteration to iteration.
+ // TODO change this to the streaming version
+ if (stratum.areBoundsEmpty) {
+ val n = counts.get(key)
+ val sampleSize = math.ceil(n * fraction).toLong
+ val lmbd1 = PoissonBounds.getLowerBound(sampleSize)
+ val minCount = PoissonBounds.getMinCount(lmbd1)
+ val lmbd2 = if (lmbd1 == 0) {
+ PoissonBounds.getUpperBound(sampleSize)
+ } else {
+ PoissonBounds.getUpperBound(sampleSize - minCount)
+ }
+ stratum.acceptBound = lmbd1 / n
+ stratum.waitListBound = lmbd2 / n
+ }
+ val acceptBound = stratum.acceptBound
+ val copiesAccepted = if (acceptBound == 0.0) 0L else rng.nextPoisson(acceptBound)
+ if (copiesAccepted > 0) {
+ stratum.incrNumAccepted(copiesAccepted)
+ }
+ val copiesWaitlisted = rng.nextPoisson(stratum.waitListBound).toInt
+ if (copiesWaitlisted > 0) {
+ stratum.addToWaitList(ArrayBuffer.fill(copiesWaitlisted)(rng.nextUniform(0.0, 1.0)))
+ }
+ } else {
+ // We use the streaming version of the algorithm for sampling without replacement to avoid
+ // using an extra pass over the RDD for computing the count.
+ // Hence, acceptBound and waitListBound change on every iteration.
+ val gamma1 = - math.log(delta) / stratum.numItems
+ val gamma2 = (2.0 / 3.0) * gamma1
+ stratum.acceptBound = math.max(0,
--- End diff --
Sure. Always nice to avoid duplicate logic in different spots.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r14907579
--- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
@@ -0,0 +1,311 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util.random
+
+import scala.collection.Map
+import scala.collection.mutable.{ArrayBuffer, HashMap, Map => MMap}
+
+import org.apache.commons.math3.random.RandomDataGenerator
+import org.apache.spark.Logging
+import org.apache.spark.rdd.RDD
+
+/**
+ * Auxiliary functions and data structures for the sampleByKey method in PairRDDFunctions.
+ *
+ * Essentially, when exact sample size is necessary, we make additional passes over the RDD to
+ * compute the exact threshold value to use for each stratum to guarantee exact sample size with
+ * high probability. This is achieved by maintaining a waitlist of size O(log(s)), where s is the
+ * desired sample size for each stratum.
+ *
+ * Like in simple random sampling, we generate a random value for each item from the
+ * uniform distribution [0.0, 1.0]. All items with values <= min(values of items in the waitlist)
+ * are accepted into the sample instantly. The threshold for instant accept is designed so that
+ * s - numAccepted = O(log(s)), where s is again the desired sample size. Thus, by maintaining a
+ * waitlist size = O(log(s)), we will be able to create a sample of the exact size s by adding
+ * a portion of the waitlist to the set of items that are instantly accepted. The exact threshold
+ * is computed by sorting the values in the waitlist and picking the value at (s - numAccepted).
+ *
+ * Note that since we use the same seed for the RNG when computing the thresholds and the actual
+ * sample, our computed thresholds are guaranteed to produce the desired sample size.
+ *
+ * For more theoretical background on the sampling techniques used here, please refer to
+ * http://jmlr.org/proceedings/papers/v28/meng13a.html
+ */
+
+private[spark] object StratifiedSampler extends Logging {
--- End diff --
Sounds good.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by falaki <gi...@git.apache.org>.
Github user falaki commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r14544579
--- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util.random
+
+import scala.collection.{Map, mutable}
+import scala.collection.mutable.ArrayBuffer
+import scala.reflect.ClassTag
+
+import org.apache.commons.math3.random.RandomDataGenerator
+import org.apache.spark.{Logging, SparkContext, TaskContext}
+import org.apache.spark.rdd.RDD
+import org.apache.spark.util.Utils
+import org.apache.spark.util.random.{PoissonBounds => PB}
+
+private[spark] object StratifiedSampler extends Logging {
+
+ /**
+ * A version of {@link #aggregate()} that passes the TaskContext to the function that does
+ * aggregation for each partition. This function avoids creating an extra depth in the RDD
+ * lineage, as opposed to using mapPartitionsWithId, which results in slightly improved run time.
+ */
+ def aggregateWithContext[U: ClassTag, T: ClassTag](zeroValue: U)
+ (rdd: RDD[T],
+ seqOp: ((TaskContext, U), T) => U,
+ combOp: (U, U) => U): U = {
+ val sc: SparkContext = rdd.sparkContext
+ // Clone the zero value since we will also be serializing it as part of tasks
+ var jobResult = Utils.clone(zeroValue, sc.env.closureSerializer.newInstance())
+ // pad seqOp and combOp with taskContext to conform to aggregate's signature in TraversableOnce
+ val paddedSeqOp = (arg1: (TaskContext, U), item: T) => (arg1._1, seqOp(arg1, item))
+ val paddedcombOp = (arg1: (TaskContext, U), arg2: (TaskContext, U)) =>
+ (arg1._1, combOp(arg1._2, arg1._2))
+ val cleanSeqOp = sc.clean(paddedSeqOp)
+ val cleanCombOp = sc.clean(paddedcombOp)
+ val aggregatePartition = (tc: TaskContext, it: Iterator[T]) =>
+ (it.aggregate(tc, zeroValue)(cleanSeqOp, cleanCombOp))._2
+ val mergeResult = (index: Int, taskResult: U) => jobResult = combOp(jobResult, taskResult)
+ sc.runJob(rdd, aggregatePartition, mergeResult)
+ jobResult
+ }
+
+ /**
+ * Returns the function used by aggregate to collect sampling statistics for each partition.
+ */
+ def getSeqOp[K, V](withReplacement: Boolean,
+ fractionByKey: (K => Double),
+ counts: Option[Map[K, Long]]): ((TaskContext, Result[K]),(K, V)) => Result[K] = {
+ val delta = 5e-5
+ (U: (TaskContext, Result[K]), item: (K, V)) => {
+ val result = U._2
+ val tc = U._1
+ val rng = result.getRand(tc.partitionId)
+ val fraction = fractionByKey(item._1)
+ val stratum = result.getEntry(item._1)
+ if (withReplacement) {
+ // compute q1 and q2 only if they haven't been computed already
+ // since they don't change from iteration to iteration.
+ // TODO change this to the streaming version
+ if (stratum.q1.isEmpty || stratum.q2.isEmpty) {
+ val n = counts.get(item._1)
+ val s = math.ceil(n * fraction).toLong
+ val lmbd1 = PB.getLowerBound(s)
+ val minCount = PB.getMinCount(lmbd1)
+ val lmbd2 = if (lmbd1 == 0) PB.getUpperBound(s) else PB.getUpperBound(s - minCount)
+ val q1 = lmbd1 / n
+ val q2 = lmbd2 / n
+ stratum.q1 = Some(q1)
+ stratum.q2 = Some(q2)
+ }
+ val x1 = if (stratum.q1.get == 0) 0L else rng.nextPoisson(stratum.q1.get)
+ if (x1 > 0) {
+ stratum.incrNumAccepted(x1)
+ }
+ val x2 = rng.nextPoisson(stratum.q2.get).toInt
+ if (x2 > 0) {
+ stratum.addToWaitList(ArrayBuffer.fill(x2)(rng.nextUniform(0.0, 1.0)))
+ }
+ } else {
+ // We use the streaming version of the algorithm for sampling without replacement.
+ // Hence, q1 and q2 change on every iteration.
+ val g1 = - math.log(delta) / stratum.numItems
+ val g2 = (2.0 / 3.0) * g1
+ val q1 = math.max(0, fraction + g2 - math.sqrt((g2 * g2 + 3 * g2 * fraction)))
+ val q2 = math.min(1, fraction + g1 + math.sqrt(g1 * g1 + 2 * g1 * fraction))
+
+ val x = rng.nextUniform(0.0, 1.0)
+ if (x < q1) {
+ stratum.incrNumAccepted()
+ } else if (x < q2) {
+ stratum.addToWaitList(x)
+ }
+ stratum.q1 = Some(q1)
+ stratum.q2 = Some(q2)
+ }
+ stratum.incrNumItems()
+ result
+ }
+ }
+
+ /**
+ * Returns the function used by aggregate to combine results from different partitions, as
+ * returned by seqOp.
+ */
+ def getCombOp[K](): (Result[K], Result[K]) => Result[K] = {
+ (r1: Result[K], r2: Result[K]) => {
+ // take union of both key sets in case one partition doesn't contain all keys
+ val keyUnion = r1.resultMap.keys.toSet.union(r2.resultMap.keys.toSet)
+
+ // Use r2 to keep the combined result since r1 is usual empty
+ for (key <- keyUnion) {
+ val entry1 = r1.resultMap.get(key)
+ val entry2 = r2.resultMap.get(key)
+ if (entry2.isEmpty && entry1.isDefined) {
+ r2.resultMap += (key -> entry1.get)
+ } else if (entry1.isDefined && entry2.isDefined) {
+ entry2.get.addToWaitList(entry1.get.waitList)
+ entry2.get.incrNumAccepted(entry1.get.numAccepted)
+ entry2.get.incrNumItems(entry1.get.numItems)
+ }
+ }
+ r2
+ }
+ }
+
+ /**
+ * Given the result returned by the aggregate function, we need to determine the threshold used
+ * to accept items to generate the exact sample size.
+ */
+ def computeThresholdByKey[K](finalResult: Map[K, Stratum], fractionByKey: (K => Double)):
+ (K => Double) = {
+ val thresholdByKey = new mutable.HashMap[K, Double]()
+ for ((key, stratum) <- finalResult) {
+ val fraction = fractionByKey(key)
+ val s = math.ceil(stratum.numItems * fraction).toLong
+ if (stratum.numAccepted > s) {
+ logWarning("Pre-accepted too many")
+ thresholdByKey += (key -> stratum.q1.get)
+ } else {
+ val numWaitListAccepted = (s - stratum.numAccepted).toInt
+ if (numWaitListAccepted >= stratum.waitList.size) {
+ logWarning("WaitList too short")
+ thresholdByKey += (key -> stratum.q2.get)
+ } else {
+ thresholdByKey += (key -> stratum.waitList.sorted.apply(numWaitListAccepted))
+ }
+ }
+ }
+ thresholdByKey
+ }
+
+ /**
+ * Return the per partition sampling function used for sampling without replacement.
+ *
+ * When exact sample size is required, we make an additional pass over the RDD to determine the
+ * exact sampling rate that guarantees sample size with high confidence.
+ *
+ * The sampling function has a unique seed per partition.
+ */
+ def getBernoulliSamplingFunction[K, V](rdd: RDD[(K, V)],
+ fractionByKey: K => Double,
+ exact: Boolean,
+ seed: Long): (Int, Iterator[(K, V)]) => Iterator[(K, V)] = {
+ var samplingRateByKey = fractionByKey
+ if (exact) {
+ // determine threshold for each stratum and resample
+ val seqOp = StratifiedSampler.getSeqOp[K,V](false, fractionByKey, None)
+ val combOp = StratifiedSampler.getCombOp[K]()
+ val zeroU = new Result[K](Map[K, Stratum](), seed = seed)
+ val finalResult = aggregateWithContext(zeroU)(rdd, seqOp, combOp).resultMap
+ samplingRateByKey = StratifiedSampler.computeThresholdByKey(finalResult, fractionByKey)
+ }
+ (idx: Int, iter: Iterator[(K, V)]) => {
+ val random = new RandomDataGenerator
+ random.reSeed(seed + idx)
+ iter.filter(t => random.nextUniform(0.0, 1.0) < samplingRateByKey(t._1))
+ }
+ }
+
+ /**
+ * Return the per partition sampling function used for sampling with replacement.
+ *
+ * When exact sample size is required, we make two additional passed over the RDD to determine
+ * the exact sampling rate that guarantees sample size with high confidence. The first pass
+ * counts the number of items in each stratum (group of items with the same key) in the RDD, and
+ * the second pass uses the counts to determine exact sampling rates.
+ *
+ * The sampling function has a unique seed per partition.
+ */
+ def getPoissonSamplingFunction[K, V](rdd:RDD[(K, V)],
+ fractionByKey: K => Double,
+ exact: Boolean,
+ counts: Option[Map[K, Long]],
+ seed: Long): (Int, Iterator[(K, V)]) => Iterator[(K, V)] = {
+ // TODO implement the streaming version of sampling w/ replacement that doesn't require counts
+ if (exact) {
+ val seqOp = StratifiedSampler.getSeqOp[K,V](true, fractionByKey, counts)
+ val combOp = StratifiedSampler.getCombOp[K]()
+ val zeroU = new Result[K](Map[K, Stratum](), seed = seed)
+ val finalResult = aggregateWithContext(zeroU)(rdd, seqOp, combOp).resultMap
+ val thresholdByKey = StratifiedSampler.computeThresholdByKey(finalResult, fractionByKey)
+ (idx: Int, iter: Iterator[(K, V)]) => {
+ val random = new RandomDataGenerator()
+ random.reSeed(seed + idx)
+ iter.flatMap { t =>
+ val q1 = finalResult.get(t._1).get.q1.getOrElse(0.0)
+ val q2 = finalResult.get(t._1).get.q2.getOrElse(0.0)
+ val x1 = if (q1 == 0) 0L else random.nextPoisson(q1)
+ val x2 = random.nextPoisson(q2).toInt
+ val x = x1 + (0 until x2).count(i => random.nextUniform(0.0, 1.0) < thresholdByKey(t._1))
+ if (x > 0) {
+ Iterator.fill(x.toInt)(t)
+ } else {
+ Iterator.empty
+ }
+ }
+ }
+ } else {
+ (idx: Int, iter: Iterator[(K, V)]) => {
+ val random = new RandomDataGenerator()
+ random.reSeed(seed + idx)
+ iter.flatMap { t =>
+ val count = random.nextPoisson(fractionByKey(t._1)).toInt
+ if (count > 0) {
+ Iterator.fill(count)(t)
+ } else {
+ Iterator.empty
+ }
+ }
+ }
+ }
+ }
+}
+
+/**
+ * Object used by seqOp to keep track of the number of items accepted and items waitlisted per
+ * stratum, as well as the bounds for accepting and waitlisting items.
+ */
+private[random] class Stratum(var numItems: Long = 0L, var numAccepted: Long = 0L)
+ extends Serializable {
+
+ var waitList: ArrayBuffer[Double] = new ArrayBuffer[Double]
--- End diff --
This does not need to be a variable. ArrayBuffer is immutable and that seems enough.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1025#issuecomment-47990759
Merged build finished. All automated tests passed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by falaki <gi...@git.apache.org>.
Github user falaki commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r14543987
--- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util.random
+
+import scala.collection.{Map, mutable}
+import scala.collection.mutable.ArrayBuffer
+import scala.reflect.ClassTag
+
+import org.apache.commons.math3.random.RandomDataGenerator
+import org.apache.spark.{Logging, SparkContext, TaskContext}
+import org.apache.spark.rdd.RDD
+import org.apache.spark.util.Utils
+import org.apache.spark.util.random.{PoissonBounds => PB}
+
+private[spark] object StratifiedSampler extends Logging {
+
+ /**
+ * A version of {@link #aggregate()} that passes the TaskContext to the function that does
+ * aggregation for each partition. This function avoids creating an extra depth in the RDD
+ * lineage, as opposed to using mapPartitionsWithId, which results in slightly improved run time.
+ */
+ def aggregateWithContext[U: ClassTag, T: ClassTag](zeroValue: U)
+ (rdd: RDD[T],
+ seqOp: ((TaskContext, U), T) => U,
+ combOp: (U, U) => U): U = {
+ val sc: SparkContext = rdd.sparkContext
+ // Clone the zero value since we will also be serializing it as part of tasks
+ var jobResult = Utils.clone(zeroValue, sc.env.closureSerializer.newInstance())
+ // pad seqOp and combOp with taskContext to conform to aggregate's signature in TraversableOnce
+ val paddedSeqOp = (arg1: (TaskContext, U), item: T) => (arg1._1, seqOp(arg1, item))
+ val paddedcombOp = (arg1: (TaskContext, U), arg2: (TaskContext, U)) =>
+ (arg1._1, combOp(arg1._2, arg1._2))
+ val cleanSeqOp = sc.clean(paddedSeqOp)
+ val cleanCombOp = sc.clean(paddedcombOp)
+ val aggregatePartition = (tc: TaskContext, it: Iterator[T]) =>
+ (it.aggregate(tc, zeroValue)(cleanSeqOp, cleanCombOp))._2
+ val mergeResult = (index: Int, taskResult: U) => jobResult = combOp(jobResult, taskResult)
+ sc.runJob(rdd, aggregatePartition, mergeResult)
+ jobResult
+ }
+
+ /**
+ * Returns the function used by aggregate to collect sampling statistics for each partition.
+ */
+ def getSeqOp[K, V](withReplacement: Boolean,
+ fractionByKey: (K => Double),
--- End diff --
Let's change the type of fractionByKey to Map[K, Double]
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r14694298
--- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
@@ -0,0 +1,311 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util.random
+
+import scala.collection.Map
+import scala.collection.mutable.{ArrayBuffer, HashMap, Map => MMap}
+
+import org.apache.commons.math3.random.RandomDataGenerator
+import org.apache.spark.Logging
+import org.apache.spark.rdd.RDD
+
+/**
+ * Auxiliary functions and data structures for the sampleByKey method in PairRDDFunctions.
+ *
+ * Essentially, when exact sample size is necessary, we make additional passes over the RDD to
+ * compute the exact threshold value to use for each stratum to guarantee exact sample size with
+ * high probability. This is achieved by maintaining a waitlist of size O(log(s)), where s is the
+ * desired sample size for each stratum.
+ *
+ * Like in simple random sampling, we generate a random value for each item from the
+ * uniform distribution [0.0, 1.0]. All items with values <= min(values of items in the waitlist)
+ * are accepted into the sample instantly. The threshold for instant accept is designed so that
+ * s - numAccepted = O(log(s)), where s is again the desired sample size. Thus, by maintaining a
+ * waitlist size = O(log(s)), we will be able to create a sample of the exact size s by adding
+ * a portion of the waitlist to the set of items that are instantly accepted. The exact threshold
+ * is computed by sorting the values in the waitlist and picking the value at (s - numAccepted).
+ *
+ * Note that since we use the same seed for the RNG when computing the thresholds and the actual
+ * sample, our computed thresholds are guaranteed to produce the desired sample size.
+ *
+ * For more theoretical background on the sampling techniques used here, please refer to
+ * http://jmlr.org/proceedings/papers/v28/meng13a.html
+ */
+
+private[spark] object StratifiedSampler extends Logging {
+
+ /**
+ * Count the number of items instantly accepted and generate the waitlist for each stratum.
+ *
+ * This is only invoked when exact sample size is required.
+ */
+ def getCounts[K, V](rdd: RDD[(K, V)],
+ withReplacement: Boolean,
+ fractions: Map[K, Double],
+ counts: Option[Map[K, Long]],
+ seed: Long): MMap[K, Stratum] = {
+ val combOp = getCombOp[K]
+ val mappedPartitionRDD = rdd.mapPartitionsWithIndex({ case (partition, iter) =>
+ val zeroU: MMap[K, Stratum] = new HashMap[K, Stratum]()
+ val rng = new RandomDataGenerator()
+ rng.reSeed(seed + partition)
+ val seqOp = getSeqOp(withReplacement, fractions, rng, counts)
+ Iterator(iter.aggregate(zeroU)(seqOp, combOp))
+ }, preservesPartitioning=true)
+ mappedPartitionRDD.reduce(combOp)
+ }
+
+ /**
+ * Returns the function used by aggregate to collect sampling statistics for each partition.
+ */
+ def getSeqOp[K, V](withReplacement: Boolean,
+ fractions: Map[K, Double],
+ rng: RandomDataGenerator,
+ counts: Option[Map[K, Long]]): (MMap[K, Stratum], (K, V)) => MMap[K, Stratum] = {
+ val delta = 5e-5
+ (result: MMap[K, Stratum], item: (K, V)) => {
+ val key = item._1
+ val fraction = fractions(key)
+ if (!result.contains(key)) {
+ result += (key -> new Stratum())
+ }
+ val stratum = result(key)
+
+ if (withReplacement) {
+ // compute acceptBound and waitListBound only if they haven't been computed already
+ // since they don't change from iteration to iteration.
+ // TODO change this to the streaming version
+ if (stratum.areBoundsEmpty) {
+ val n = counts.get(key)
+ val sampleSize = math.ceil(n * fraction).toLong
+ val lmbd1 = PoissonBounds.getLowerBound(sampleSize)
+ val minCount = PoissonBounds.getMinCount(lmbd1)
+ val lmbd2 = if (lmbd1 == 0) {
+ PoissonBounds.getUpperBound(sampleSize)
+ } else {
+ PoissonBounds.getUpperBound(sampleSize - minCount)
+ }
+ stratum.acceptBound = lmbd1 / n
+ stratum.waitListBound = lmbd2 / n
+ }
+ val acceptBound = stratum.acceptBound
+ val copiesAccepted = if (acceptBound == 0.0) 0L else rng.nextPoisson(acceptBound)
+ if (copiesAccepted > 0) {
+ stratum.incrNumAccepted(copiesAccepted)
+ }
+ val copiesWaitlisted = rng.nextPoisson(stratum.waitListBound).toInt
+ if (copiesWaitlisted > 0) {
+ stratum.addToWaitList(ArrayBuffer.fill(copiesWaitlisted)(rng.nextUniform(0.0, 1.0)))
+ }
+ } else {
+ // We use the streaming version of the algorithm for sampling without replacement to avoid
+ // using an extra pass over the RDD for computing the count.
+ // Hence, acceptBound and waitListBound change on every iteration.
+ val gamma1 = - math.log(delta) / stratum.numItems
+ val gamma2 = (2.0 / 3.0) * gamma1
+ stratum.acceptBound = math.max(0,
+ fraction + gamma2 - math.sqrt(gamma2 * gamma2 + 3 * gamma2 * fraction))
+ stratum.waitListBound = math.min(1,
+ fraction + gamma1 + math.sqrt(gamma1 * gamma1 + 2 * gamma1 * fraction))
+
+ val x = rng.nextUniform(0.0, 1.0)
+ if (x < stratum.acceptBound) {
+ stratum.incrNumAccepted()
+ } else if (x < stratum.waitListBound) {
+ stratum.addToWaitList(x)
+ }
+ }
+ stratum.incrNumItems()
+ result
+ }
+ }
+
+ /**
+ * Returns the function used combine results returned by seqOp from different partitions.
+ */
+ def getCombOp[K]: (MMap[K, Stratum], MMap[K, Stratum]) => MMap[K, Stratum] = {
+ (result1: MMap[K, Stratum], result2: MMap[K, Stratum]) => {
+ // take union of both key sets in case one partition doesn't contain all keys
+ for (key <- result1.keySet.union(result2.keySet)) {
+ // Use result2 to keep the combined result since r1 is usual empty
+ val entry1 = result1.get(key)
+ if (result2.contains(key)) {
+ result2(key).merge(entry1)
+ } else {
+ if (entry1.isDefined) {
+ result2 += (key -> entry1.get)
+ }
+ }
+ }
+ result2
+ }
+ }
+
+ /**
+ * Given the result returned by getCounts, determine the threshold for accepting items to
+ * generate exact sample size.
+ *
+ * To do so, we compute sampleSize = math.ceil(size * samplingRate) for each stratum and compare
+ * it to the number of items that were accepted instantly and the number of items in the waitlist
+ * for that stratum. Most of the time, numAccepted <= sampleSize <= (numAccepted + numWaitlisted),
+ * which means we need to sort the elements in the waitlist by their associated values in order
+ * to find the value T s.t. |{elements in the stratum whose associated values <= T}| = sampleSize.
+ * Note that all elements in the waitlist have values >= bound for instant accept, so a T value
+ * in the waitlist range would allow all elements that were instantly accepted on the first pass
+ * to be included in the sample.
+ */
+ def computeThresholdByKey[K](finalResult: Map[K, Stratum],
+ fractions: Map[K, Double]): Map[K, Double] = {
+ val thresholdByKey = new HashMap[K, Double]()
+ for ((key, stratum) <- finalResult) {
+ val sampleSize = math.ceil(stratum.numItems * fractions(key)).toLong
+ if (stratum.numAccepted > sampleSize) {
+ logWarning("Pre-accepted too many")
+ thresholdByKey += (key -> stratum.acceptBound)
+ } else {
+ val numWaitListAccepted = (sampleSize - stratum.numAccepted).toInt
+ if (numWaitListAccepted >= stratum.waitList.size) {
+ logWarning("WaitList too short")
+ thresholdByKey += (key -> stratum.waitListBound)
+ } else {
+ thresholdByKey += (key -> stratum.waitList.sorted.apply(numWaitListAccepted))
+ }
+ }
+ }
+ thresholdByKey
+ }
+
+ /**
+ * Return the per partition sampling function used for sampling without replacement.
+ *
+ * When exact sample size is required, we make an additional pass over the RDD to determine the
+ * exact sampling rate that guarantees sample size with high confidence.
+ *
+ * The sampling function has a unique seed per partition.
+ */
+ def getBernoulliSamplingFunction[K, V](rdd: RDD[(K, V)],
+ fractions: Map[K, Double],
+ exact: Boolean,
+ seed: Long): (Int, Iterator[(K, V)]) => Iterator[(K, V)] = {
+ var samplingRateByKey = fractions
+ if (exact) {
+ // determine threshold for each stratum and resample
+ val finalResult = getCounts(rdd, false, fractions, None, seed)
+ samplingRateByKey = computeThresholdByKey(finalResult, fractions)
+ }
+ (idx: Int, iter: Iterator[(K, V)]) => {
+ val rng = new RandomDataGenerator
+ rng.reSeed(seed + idx)
+ iter.filter(t => rng.nextUniform(0.0, 1.0) < samplingRateByKey(t._1))
+ }
+ }
+
+ /**
+ * Return the per partition sampling function used for sampling with replacement.
+ *
+ * When exact sample size is required, we make two additional passed over the RDD to determine
+ * the exact sampling rate that guarantees sample size with high confidence. The first pass
+ * counts the number of items in each stratum (group of items with the same key) in the RDD, and
+ * the second pass uses the counts to determine exact sampling rates.
+ *
+ * The sampling function has a unique seed per partition.
+ */
+ def getPoissonSamplingFunction[K, V](rdd: RDD[(K, V)],
+ fractions: Map[K, Double],
+ exact: Boolean,
+ counts: Option[Map[K, Long]],
+ seed: Long): (Int, Iterator[(K, V)]) => Iterator[(K, V)] = {
+ // TODO implement the streaming version of sampling w/ replacement that doesn't require counts
+ if (exact) {
+ val finalResult = getCounts(rdd, true, fractions, counts, seed)
+ val thresholdByKey = computeThresholdByKey(finalResult, fractions)
+ (idx: Int, iter: Iterator[(K, V)]) => {
+ val rng = new RandomDataGenerator()
+ rng.reSeed(seed + idx)
+ iter.flatMap { item =>
+ val key = item._1
+ val acceptBound = finalResult(key).acceptBound
+ val copiesAccepted = if (acceptBound == 0) 0L else rng.nextPoisson(acceptBound)
+ val copiesWailisted = rng.nextPoisson(finalResult(key).waitListBound).toInt
+ val copiesInSample = copiesAccepted +
+ (0 until copiesWailisted).count(i => rng.nextUniform(0.0, 1.0) < thresholdByKey(key))
+ if (copiesInSample > 0) {
+ Iterator.fill(copiesInSample.toInt)(item)
+ } else {
+ Iterator.empty
+ }
+ }
+ }
+ } else {
+ (idx: Int, iter: Iterator[(K, V)]) => {
+ val rng = new RandomDataGenerator()
+ rng.reSeed(seed + idx)
+ iter.flatMap { item =>
+ val count = rng.nextPoisson(fractions(item._1)).toInt
+ if (count > 0) {
+ Iterator.fill(count)(item)
+ } else {
+ Iterator.empty
+ }
+ }
+ }
+ }
+ }
+}
+
+/**
+ * Object used by seqOp to keep track of the number of items accepted and items waitlisted per
+ * stratum, as well as the bounds for accepting and waitlisting items.
+ *
+ * `[random]` here is necessary since it's in the return type signature of seqOp defined above
+ */
+private[random] class Stratum(var numItems: Long = 0L, var numAccepted: Long = 0L)
+ extends Serializable {
+
+ private val _waitList = new ArrayBuffer[Double]
+ private var _acceptBound: Option[Double] = None // upper bound for accepting item instantly
+ private var _waitListBound: Option[Double] = None // upper bound for adding item to waitlist
+
+ def incrNumItems(by: Long = 1L) = numItems += by
+
+ def incrNumAccepted(by: Long = 1L) = numAccepted += by
+
+ def addToWaitList(elem: Double) = _waitList += elem
+
+ def addToWaitList(elems: ArrayBuffer[Double]) = _waitList ++= elems
+
+ def waitList = _waitList
+
+ def acceptBound = _acceptBound.getOrElse(0.0)
+
+ def acceptBound_= (value: Double) = _acceptBound = Some(value)
+
+ def waitListBound = _waitListBound.getOrElse(0.0)
+
+ def waitListBound_= (value: Double) = _waitListBound = Some(value)
+
+ def areBoundsEmpty = _acceptBound.isEmpty || _waitListBound.isEmpty
+
+ def merge(other: Option[Stratum]): Unit = {
--- End diff --
Any chance of merging a `None`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by falaki <gi...@git.apache.org>.
Github user falaki commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r14688585
--- Diff: core/src/test/scala/org/apache/spark/rdd/PairRDDFunctionsSuite.scala ---
@@ -83,6 +83,120 @@ class PairRDDFunctionsSuite extends FunSuite with SharedSparkContext {
assert(valuesFor2.toList.sorted === List(1))
}
+ test("sampleByKey") {
+ def stratifier (fractionPositive: Double) = {
+ (x: Int) => if (x % 10 < (10 * fractionPositive).toInt) "1" else "0"
+ }
+
+ def checkSize(exact: Boolean, withReplacement: Boolean,
+ expected: Long, actual: Long, p: Double): Boolean = {
+ if (exact) {
+ return expected == actual
+ }
+ val stdev = if (withReplacement) math.sqrt(expected) else math.sqrt(expected * p * (1 - p))
+ // Very forgiving margin since we're dealing with very small sample sizes most of the time
+ math.abs(actual - expected) <= 6 * stdev
+ }
+
+ // Without replacement validation
+ def takeSampleAndValidateBernoulli(stratifiedData: RDD[(String, Int)],
+ exact: Boolean,
+ samplingRate: Double,
+ seed: Long,
+ n: Long) = {
+ val expectedSampleSize = stratifiedData.countByKey().mapValues(count =>
+ math.ceil(count * samplingRate).toInt)
+ val fractions = Map("1" -> samplingRate, "0" -> samplingRate)
+ val sample = stratifiedData.sampleByKey(false, fractions, exact, seed)
+ val sampleCounts = sample.countByKey()
+ val takeSample = sample.collect()
+ assert(sampleCounts.forall({case(k,v) =>
--- End diff --
Parentheses are superfluous. Square bracket is enough:
```sampleCounts.forall { case(k, v) => ```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r14694278
--- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
@@ -0,0 +1,311 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util.random
+
+import scala.collection.Map
+import scala.collection.mutable.{ArrayBuffer, HashMap, Map => MMap}
+
+import org.apache.commons.math3.random.RandomDataGenerator
+import org.apache.spark.Logging
+import org.apache.spark.rdd.RDD
+
+/**
+ * Auxiliary functions and data structures for the sampleByKey method in PairRDDFunctions.
+ *
+ * Essentially, when exact sample size is necessary, we make additional passes over the RDD to
+ * compute the exact threshold value to use for each stratum to guarantee exact sample size with
+ * high probability. This is achieved by maintaining a waitlist of size O(log(s)), where s is the
+ * desired sample size for each stratum.
+ *
+ * Like in simple random sampling, we generate a random value for each item from the
+ * uniform distribution [0.0, 1.0]. All items with values <= min(values of items in the waitlist)
+ * are accepted into the sample instantly. The threshold for instant accept is designed so that
+ * s - numAccepted = O(log(s)), where s is again the desired sample size. Thus, by maintaining a
+ * waitlist size = O(log(s)), we will be able to create a sample of the exact size s by adding
+ * a portion of the waitlist to the set of items that are instantly accepted. The exact threshold
+ * is computed by sorting the values in the waitlist and picking the value at (s - numAccepted).
+ *
+ * Note that since we use the same seed for the RNG when computing the thresholds and the actual
+ * sample, our computed thresholds are guaranteed to produce the desired sample size.
+ *
+ * For more theoretical background on the sampling techniques used here, please refer to
+ * http://jmlr.org/proceedings/papers/v28/meng13a.html
+ */
+
+private[spark] object StratifiedSampler extends Logging {
+
+ /**
+ * Count the number of items instantly accepted and generate the waitlist for each stratum.
+ *
+ * This is only invoked when exact sample size is required.
+ */
+ def getCounts[K, V](rdd: RDD[(K, V)],
+ withReplacement: Boolean,
+ fractions: Map[K, Double],
+ counts: Option[Map[K, Long]],
+ seed: Long): MMap[K, Stratum] = {
+ val combOp = getCombOp[K]
+ val mappedPartitionRDD = rdd.mapPartitionsWithIndex({ case (partition, iter) =>
+ val zeroU: MMap[K, Stratum] = new HashMap[K, Stratum]()
+ val rng = new RandomDataGenerator()
+ rng.reSeed(seed + partition)
+ val seqOp = getSeqOp(withReplacement, fractions, rng, counts)
+ Iterator(iter.aggregate(zeroU)(seqOp, combOp))
+ }, preservesPartitioning=true)
+ mappedPartitionRDD.reduce(combOp)
+ }
+
+ /**
+ * Returns the function used by aggregate to collect sampling statistics for each partition.
+ */
+ def getSeqOp[K, V](withReplacement: Boolean,
+ fractions: Map[K, Double],
+ rng: RandomDataGenerator,
+ counts: Option[Map[K, Long]]): (MMap[K, Stratum], (K, V)) => MMap[K, Stratum] = {
+ val delta = 5e-5
+ (result: MMap[K, Stratum], item: (K, V)) => {
+ val key = item._1
+ val fraction = fractions(key)
+ if (!result.contains(key)) {
+ result += (key -> new Stratum())
+ }
+ val stratum = result(key)
+
+ if (withReplacement) {
+ // compute acceptBound and waitListBound only if they haven't been computed already
+ // since they don't change from iteration to iteration.
+ // TODO change this to the streaming version
+ if (stratum.areBoundsEmpty) {
+ val n = counts.get(key)
+ val sampleSize = math.ceil(n * fraction).toLong
+ val lmbd1 = PoissonBounds.getLowerBound(sampleSize)
+ val minCount = PoissonBounds.getMinCount(lmbd1)
+ val lmbd2 = if (lmbd1 == 0) {
+ PoissonBounds.getUpperBound(sampleSize)
+ } else {
+ PoissonBounds.getUpperBound(sampleSize - minCount)
+ }
+ stratum.acceptBound = lmbd1 / n
+ stratum.waitListBound = lmbd2 / n
--- End diff --
Should be `(lmbd2 - lmbd1) / n`. Otherwise, we waitlist too many. Please check the waitlist size for some large n.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r14694386
--- Diff: core/src/test/scala/org/apache/spark/rdd/PairRDDFunctionsSuite.scala ---
@@ -83,6 +83,125 @@ class PairRDDFunctionsSuite extends FunSuite with SharedSparkContext {
assert(valuesFor2.toList.sorted === List(1))
}
+ test("sampleByKey") {
+ def stratifier (fractionPositive: Double) = {
+ (x: Int) => if (x % 10 < (10 * fractionPositive).toInt) "1" else "0"
+ }
+
+ def checkSize(exact: Boolean,
+ withReplacement: Boolean,
+ expected: Long,
+ actual: Long,
+ p: Double): Boolean = {
+ if (exact) {
+ return expected == actual
+ }
+ val stdev = if (withReplacement) math.sqrt(expected) else math.sqrt(expected * p * (1 - p))
+ // Very forgiving margin since we're dealing with very small sample sizes most of the time
+ math.abs(actual - expected) <= 6 * stdev
+ }
+
+ // Without replacement validation
+ def takeSampleAndValidateBernoulli(stratifiedData: RDD[(String, Int)],
+ exact: Boolean,
+ samplingRate: Double,
+ seed: Long,
+ n: Long) = {
+ val expectedSampleSize = stratifiedData.countByKey().mapValues(count =>
+ math.ceil(count * samplingRate).toInt)
+ val fractions = Map("1" -> samplingRate, "0" -> samplingRate)
+ val sample = stratifiedData.sampleByKey(false, fractions, exact, seed)
+ val sampleCounts = sample.countByKey()
+ val takeSample = sample.collect()
+ assert(sampleCounts.forall {case(k,v) =>
+ checkSize(exact, false, expectedSampleSize(k), v, samplingRate)})
+ assert(takeSample.size === takeSample.toSet.size)
+ assert(takeSample.forall(x => 1 <= x._2 && x._2 <= n), s"elements not in [1, $n]")
+ }
+
+ // With replacement validation
+ def takeSampleAndValidatePoisson(stratifiedData: RDD[(String, Int)],
+ exact: Boolean,
+ samplingRate: Double,
+ seed: Long,
+ n: Long) = {
+ val expectedSampleSize = stratifiedData.countByKey().mapValues(count =>
+ math.ceil(count * samplingRate).toInt)
+ val fractions = Map("1" -> samplingRate, "0" -> samplingRate)
+ val sample = stratifiedData.sampleByKey(true, fractions, exact, seed)
+ val sampleCounts = sample.countByKey()
+ val takeSample = sample.collect()
+ assert(sampleCounts.forall {case(k,v) =>
+ checkSize(exact, true, expectedSampleSize(k), v, samplingRate)})
+ val groupedByKey = takeSample.groupBy {case(k, v) => k}
+ for ((key, v) <- groupedByKey) {
+ if (expectedSampleSize(key) >= 100 && samplingRate >= 0.1) {
+ // sample large enough for there to be repeats with high likelihood
+ assert(v.toSet.size < expectedSampleSize(key))
+ } else {
+ if (exact) {
+ assert(v.toSet.size <= expectedSampleSize(key))
+ } else {
+ assert(checkSize(false, true, expectedSampleSize(key), v.toSet.size, samplingRate))
+ }
+ }
+ }
+ assert(takeSample.forall(x => 1 <= x._2 && x._2 <= n), s"elements not in [1, $n]")
+ }
+
+ def checkAllCombos(stratifiedData: RDD[(String, Int)],
+ samplingRate: Double,
+ seed: Long,
+ n: Long) = {
+ takeSampleAndValidateBernoulli(stratifiedData, true, samplingRate, seed, n)
+ takeSampleAndValidateBernoulli(stratifiedData, false, samplingRate, seed, n)
+ takeSampleAndValidatePoisson(stratifiedData, true, samplingRate, seed, n)
+ takeSampleAndValidatePoisson(stratifiedData, false, samplingRate, seed, n)
+ }
+
+ val defaultSeed = 42L
+
+ //vary RDD size
--- End diff --
add a space after `//`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1025#issuecomment-50414198
QA results for PR 1025:<br>- This patch PASSES unit tests.<br>- This patch merges cleanly<br>- This patch adds no public classes<br><br>For more information see test ouptut:<br>https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17307/consoleFull
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1025#issuecomment-48989787
QA results for PR 1025:<br>- This patch PASSES unit tests.<br>- This patch merges cleanly<br>- This patch adds no public classes<br><br>For more information see test ouptut:<br>https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16650/consoleFull
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1025#issuecomment-46644692
Merged build finished.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r14500845
--- Diff: core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala ---
@@ -46,6 +48,8 @@ import org.apache.spark.Partitioner.defaultPartitioner
import org.apache.spark.SparkContext._
import org.apache.spark.partial.{BoundedDouble, PartialResult}
import org.apache.spark.serializer.Serializer
+import org.apache.spark.util.Utils
+import org.apache.spark.util.random.{Stratum, Result, StratifiedSampler, PoissonBounds => PB}
--- End diff --
Re-organize the imports (using an intellij plugin: http://plugins.jetbrains.com/plugin/7350?pr=)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by dorx <gi...@git.apache.org>.
Github user dorx commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r14688550
--- Diff: core/src/main/scala/org/apache/spark/util/random/SamplingUtils.scala ---
@@ -45,11 +50,75 @@ private[spark] object SamplingUtils {
val fraction = sampleSizeLowerBound.toDouble / total
if (withReplacement) {
val numStDev = if (sampleSizeLowerBound < 12) 9 else 5
- fraction + numStDev * math.sqrt(fraction / total)
+ math.max(1e-10, fraction + numStDev * math.sqrt(fraction / total))
} else {
val delta = 1e-4
val gamma = - math.log(delta) / total
- math.min(1, fraction + gamma + math.sqrt(gamma * gamma + 2 * gamma * fraction))
+ math.min(1,
+ math.max(1e-10, fraction + gamma + math.sqrt(gamma * gamma + 2 * gamma * fraction)))
+ }
+ }
+}
+
+/**
+ * Utility functions that help us determine bounds on adjusted sampling rate to guarantee exact
+ * sample sizes with high confidence when sampling with replacement.
+ *
+ * The algorithm for guaranteeing sample size instantly accepts items whose associated value drawn
+ * from Pois(s) is less than the lower bound and puts items whose value is between the lower and
+ * upper bound in a waitlist. The final sample is consisted of all items accepted on the fly and a
+ * portion of the waitlist needed to make the exact sample size.
+ */
+private[spark] object PoissonBounds {
+
+ val delta = 1e-4 / 3.0
+ val epsilon = 1e-15
+
+ /**
+ * Compute the threshold for accepting items on the fly. The threshold value is a fairly small
+ * number, which means if the item has an associated value < threshold, it is highly likely to
+ * be in the final sample. Hence we accept items with values less than the returned value of this
+ * function instantly.
+ *
+ * @param s sample size
+ * @return threshold for accepting items on the fly
+ */
+ def getLowerBound(s: Double): Double = {
+ var lb = math.max(0.0, s - math.sqrt(s / delta)) // Chebyshev's inequality
+ var ub = s
+ while (lb < ub - 1.0) {
+ val m = (lb + ub) / 2.0
+ val poisson = new PoissonDistribution(m, epsilon)
+ val y = poisson.inverseCumulativeProbability(1 - delta)
+ if (y > s) ub = m else lb = m
+ }
+ lb
+ }
+
+ def getMinCount(lmbd: Double): Double = {
+ if (lmbd == 0) return 0
--- End diff --
agreed. this was a copypasta i didn't look too hard at.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r13578489
--- Diff: core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala ---
@@ -156,6 +161,182 @@ class PairRDDFunctions[K, V](self: RDD[(K, V)])
}
/**
+ * Return a subset of this RDD sampled by key (via stratified sampling).
+ * We guarantee a sample size = math.ceil(fraction * S_i), where S_i is the size of the ith
--- End diff --
Also, it would be good to put a comment saying that this method requires two passes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1025#issuecomment-46391365
All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15864/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1025#issuecomment-45553302
Build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the pull request:
https://github.com/apache/spark/pull/1025#issuecomment-46643277
Jenkins, retest this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1025#issuecomment-46643453
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by dorx <gi...@git.apache.org>.
Github user dorx commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r14906680
--- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
@@ -0,0 +1,311 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util.random
+
+import scala.collection.Map
+import scala.collection.mutable.{ArrayBuffer, HashMap, Map => MMap}
--- End diff --
There isn't a immutable equivalent of `HashMap`. Also, using `mutable.Map` instead of `MMap` (short for mutable.Map) might push a couple lines over the 100 char limit.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1025#issuecomment-47990764
All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16315/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1025#issuecomment-48386184
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16414/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r14694289
--- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
@@ -0,0 +1,311 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util.random
+
+import scala.collection.Map
+import scala.collection.mutable.{ArrayBuffer, HashMap, Map => MMap}
+
+import org.apache.commons.math3.random.RandomDataGenerator
+import org.apache.spark.Logging
+import org.apache.spark.rdd.RDD
+
+/**
+ * Auxiliary functions and data structures for the sampleByKey method in PairRDDFunctions.
+ *
+ * Essentially, when exact sample size is necessary, we make additional passes over the RDD to
+ * compute the exact threshold value to use for each stratum to guarantee exact sample size with
+ * high probability. This is achieved by maintaining a waitlist of size O(log(s)), where s is the
+ * desired sample size for each stratum.
+ *
+ * Like in simple random sampling, we generate a random value for each item from the
+ * uniform distribution [0.0, 1.0]. All items with values <= min(values of items in the waitlist)
+ * are accepted into the sample instantly. The threshold for instant accept is designed so that
+ * s - numAccepted = O(log(s)), where s is again the desired sample size. Thus, by maintaining a
+ * waitlist size = O(log(s)), we will be able to create a sample of the exact size s by adding
+ * a portion of the waitlist to the set of items that are instantly accepted. The exact threshold
+ * is computed by sorting the values in the waitlist and picking the value at (s - numAccepted).
+ *
+ * Note that since we use the same seed for the RNG when computing the thresholds and the actual
+ * sample, our computed thresholds are guaranteed to produce the desired sample size.
+ *
+ * For more theoretical background on the sampling techniques used here, please refer to
+ * http://jmlr.org/proceedings/papers/v28/meng13a.html
+ */
+
+private[spark] object StratifiedSampler extends Logging {
+
+ /**
+ * Count the number of items instantly accepted and generate the waitlist for each stratum.
+ *
+ * This is only invoked when exact sample size is required.
+ */
+ def getCounts[K, V](rdd: RDD[(K, V)],
+ withReplacement: Boolean,
+ fractions: Map[K, Double],
+ counts: Option[Map[K, Long]],
+ seed: Long): MMap[K, Stratum] = {
+ val combOp = getCombOp[K]
+ val mappedPartitionRDD = rdd.mapPartitionsWithIndex({ case (partition, iter) =>
+ val zeroU: MMap[K, Stratum] = new HashMap[K, Stratum]()
+ val rng = new RandomDataGenerator()
+ rng.reSeed(seed + partition)
+ val seqOp = getSeqOp(withReplacement, fractions, rng, counts)
+ Iterator(iter.aggregate(zeroU)(seqOp, combOp))
+ }, preservesPartitioning=true)
+ mappedPartitionRDD.reduce(combOp)
+ }
+
+ /**
+ * Returns the function used by aggregate to collect sampling statistics for each partition.
+ */
+ def getSeqOp[K, V](withReplacement: Boolean,
+ fractions: Map[K, Double],
+ rng: RandomDataGenerator,
+ counts: Option[Map[K, Long]]): (MMap[K, Stratum], (K, V)) => MMap[K, Stratum] = {
+ val delta = 5e-5
+ (result: MMap[K, Stratum], item: (K, V)) => {
+ val key = item._1
+ val fraction = fractions(key)
+ if (!result.contains(key)) {
+ result += (key -> new Stratum())
+ }
+ val stratum = result(key)
+
+ if (withReplacement) {
+ // compute acceptBound and waitListBound only if they haven't been computed already
+ // since they don't change from iteration to iteration.
+ // TODO change this to the streaming version
+ if (stratum.areBoundsEmpty) {
+ val n = counts.get(key)
+ val sampleSize = math.ceil(n * fraction).toLong
+ val lmbd1 = PoissonBounds.getLowerBound(sampleSize)
+ val minCount = PoissonBounds.getMinCount(lmbd1)
+ val lmbd2 = if (lmbd1 == 0) {
+ PoissonBounds.getUpperBound(sampleSize)
+ } else {
+ PoissonBounds.getUpperBound(sampleSize - minCount)
+ }
+ stratum.acceptBound = lmbd1 / n
+ stratum.waitListBound = lmbd2 / n
+ }
+ val acceptBound = stratum.acceptBound
+ val copiesAccepted = if (acceptBound == 0.0) 0L else rng.nextPoisson(acceptBound)
+ if (copiesAccepted > 0) {
+ stratum.incrNumAccepted(copiesAccepted)
+ }
+ val copiesWaitlisted = rng.nextPoisson(stratum.waitListBound).toInt
+ if (copiesWaitlisted > 0) {
+ stratum.addToWaitList(ArrayBuffer.fill(copiesWaitlisted)(rng.nextUniform(0.0, 1.0)))
+ }
+ } else {
+ // We use the streaming version of the algorithm for sampling without replacement to avoid
+ // using an extra pass over the RDD for computing the count.
+ // Hence, acceptBound and waitListBound change on every iteration.
+ val gamma1 = - math.log(delta) / stratum.numItems
+ val gamma2 = (2.0 / 3.0) * gamma1
+ stratum.acceptBound = math.max(0,
+ fraction + gamma2 - math.sqrt(gamma2 * gamma2 + 3 * gamma2 * fraction))
+ stratum.waitListBound = math.min(1,
+ fraction + gamma1 + math.sqrt(gamma1 * gamma1 + 2 * gamma1 * fraction))
+
+ val x = rng.nextUniform(0.0, 1.0)
+ if (x < stratum.acceptBound) {
+ stratum.incrNumAccepted()
+ } else if (x < stratum.waitListBound) {
+ stratum.addToWaitList(x)
+ }
+ }
+ stratum.incrNumItems()
+ result
+ }
+ }
+
+ /**
+ * Returns the function used combine results returned by seqOp from different partitions.
+ */
+ def getCombOp[K]: (MMap[K, Stratum], MMap[K, Stratum]) => MMap[K, Stratum] = {
+ (result1: MMap[K, Stratum], result2: MMap[K, Stratum]) => {
+ // take union of both key sets in case one partition doesn't contain all keys
+ for (key <- result1.keySet.union(result2.keySet)) {
+ // Use result2 to keep the combined result since r1 is usual empty
+ val entry1 = result1.get(key)
+ if (result2.contains(key)) {
+ result2(key).merge(entry1)
+ } else {
+ if (entry1.isDefined) {
+ result2 += (key -> entry1.get)
+ }
+ }
+ }
+ result2
+ }
+ }
+
+ /**
+ * Given the result returned by getCounts, determine the threshold for accepting items to
+ * generate exact sample size.
+ *
+ * To do so, we compute sampleSize = math.ceil(size * samplingRate) for each stratum and compare
+ * it to the number of items that were accepted instantly and the number of items in the waitlist
+ * for that stratum. Most of the time, numAccepted <= sampleSize <= (numAccepted + numWaitlisted),
+ * which means we need to sort the elements in the waitlist by their associated values in order
+ * to find the value T s.t. |{elements in the stratum whose associated values <= T}| = sampleSize.
+ * Note that all elements in the waitlist have values >= bound for instant accept, so a T value
+ * in the waitlist range would allow all elements that were instantly accepted on the first pass
+ * to be included in the sample.
+ */
+ def computeThresholdByKey[K](finalResult: Map[K, Stratum],
+ fractions: Map[K, Double]): Map[K, Double] = {
+ val thresholdByKey = new HashMap[K, Double]()
+ for ((key, stratum) <- finalResult) {
+ val sampleSize = math.ceil(stratum.numItems * fractions(key)).toLong
+ if (stratum.numAccepted > sampleSize) {
+ logWarning("Pre-accepted too many")
+ thresholdByKey += (key -> stratum.acceptBound)
+ } else {
+ val numWaitListAccepted = (sampleSize - stratum.numAccepted).toInt
+ if (numWaitListAccepted >= stratum.waitList.size) {
+ logWarning("WaitList too short")
+ thresholdByKey += (key -> stratum.waitListBound)
+ } else {
+ thresholdByKey += (key -> stratum.waitList.sorted.apply(numWaitListAccepted))
+ }
+ }
+ }
+ thresholdByKey
+ }
+
+ /**
+ * Return the per partition sampling function used for sampling without replacement.
+ *
+ * When exact sample size is required, we make an additional pass over the RDD to determine the
+ * exact sampling rate that guarantees sample size with high confidence.
+ *
+ * The sampling function has a unique seed per partition.
+ */
+ def getBernoulliSamplingFunction[K, V](rdd: RDD[(K, V)],
+ fractions: Map[K, Double],
+ exact: Boolean,
+ seed: Long): (Int, Iterator[(K, V)]) => Iterator[(K, V)] = {
+ var samplingRateByKey = fractions
+ if (exact) {
+ // determine threshold for each stratum and resample
+ val finalResult = getCounts(rdd, false, fractions, None, seed)
+ samplingRateByKey = computeThresholdByKey(finalResult, fractions)
+ }
+ (idx: Int, iter: Iterator[(K, V)]) => {
+ val rng = new RandomDataGenerator
+ rng.reSeed(seed + idx)
+ iter.filter(t => rng.nextUniform(0.0, 1.0) < samplingRateByKey(t._1))
+ }
+ }
+
+ /**
+ * Return the per partition sampling function used for sampling with replacement.
+ *
+ * When exact sample size is required, we make two additional passed over the RDD to determine
+ * the exact sampling rate that guarantees sample size with high confidence. The first pass
+ * counts the number of items in each stratum (group of items with the same key) in the RDD, and
+ * the second pass uses the counts to determine exact sampling rates.
+ *
+ * The sampling function has a unique seed per partition.
+ */
+ def getPoissonSamplingFunction[K, V](rdd: RDD[(K, V)],
+ fractions: Map[K, Double],
+ exact: Boolean,
+ counts: Option[Map[K, Long]],
--- End diff --
Remove counts and compute it inside the function. Since this is only needed for Poisson and when `exact == true`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r14694233
--- Diff: core/src/main/scala/org/apache/spark/api/java/JavaPairRDD.scala ---
@@ -130,6 +130,38 @@ class JavaPairRDD[K, V](val rdd: RDD[(K, V)])
new JavaPairRDD[K, V](rdd.sample(withReplacement, fraction, seed))
/**
+ * Return a subset of this RDD sampled by key (via stratified sampling).
--- End diff --
Needs more doc here. And for the overloaded methods below, please write down the default values, especially when `exact` is omitted.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r14694262
--- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
@@ -0,0 +1,311 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util.random
+
+import scala.collection.Map
+import scala.collection.mutable.{ArrayBuffer, HashMap, Map => MMap}
+
+import org.apache.commons.math3.random.RandomDataGenerator
+import org.apache.spark.Logging
+import org.apache.spark.rdd.RDD
+
+/**
+ * Auxiliary functions and data structures for the sampleByKey method in PairRDDFunctions.
+ *
+ * Essentially, when exact sample size is necessary, we make additional passes over the RDD to
+ * compute the exact threshold value to use for each stratum to guarantee exact sample size with
+ * high probability. This is achieved by maintaining a waitlist of size O(log(s)), where s is the
--- End diff --
Should be `O(sqrt(s))` for sampling without replacement.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1025#issuecomment-45547151
Build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1025#issuecomment-48419784
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1025#issuecomment-46520000
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15892/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by falaki <gi...@git.apache.org>.
Github user falaki commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r14543734
--- Diff: core/src/main/scala/org/apache/spark/util/random/StratifiedSampler.scala ---
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util.random
+
+import scala.collection.{Map, mutable}
+import scala.collection.mutable.ArrayBuffer
+import scala.reflect.ClassTag
+
+import org.apache.commons.math3.random.RandomDataGenerator
+import org.apache.spark.{Logging, SparkContext, TaskContext}
+import org.apache.spark.rdd.RDD
+import org.apache.spark.util.Utils
+import org.apache.spark.util.random.{PoissonBounds => PB}
+
+private[spark] object StratifiedSampler extends Logging {
+
+ /**
+ * A version of {@link #aggregate()} that passes the TaskContext to the function that does
+ * aggregation for each partition. This function avoids creating an extra depth in the RDD
+ * lineage, as opposed to using mapPartitionsWithId, which results in slightly improved run time.
+ */
+ def aggregateWithContext[U: ClassTag, T: ClassTag](zeroValue: U)
+ (rdd: RDD[T],
+ seqOp: ((TaskContext, U), T) => U,
+ combOp: (U, U) => U): U = {
+ val sc: SparkContext = rdd.sparkContext
+ // Clone the zero value since we will also be serializing it as part of tasks
+ var jobResult = Utils.clone(zeroValue, sc.env.closureSerializer.newInstance())
+ // pad seqOp and combOp with taskContext to conform to aggregate's signature in TraversableOnce
+ val paddedSeqOp = (arg1: (TaskContext, U), item: T) => (arg1._1, seqOp(arg1, item))
+ val paddedcombOp = (arg1: (TaskContext, U), arg2: (TaskContext, U)) =>
+ (arg1._1, combOp(arg1._2, arg1._2))
+ val cleanSeqOp = sc.clean(paddedSeqOp)
+ val cleanCombOp = sc.clean(paddedcombOp)
+ val aggregatePartition = (tc: TaskContext, it: Iterator[T]) =>
+ (it.aggregate(tc, zeroValue)(cleanSeqOp, cleanCombOp))._2
+ val mergeResult = (index: Int, taskResult: U) => jobResult = combOp(jobResult, taskResult)
+ sc.runJob(rdd, aggregatePartition, mergeResult)
+ jobResult
+ }
+
+ /**
+ * Returns the function used by aggregate to collect sampling statistics for each partition.
+ */
+ def getSeqOp[K, V](withReplacement: Boolean,
+ fractionByKey: (K => Double),
+ counts: Option[Map[K, Long]]): ((TaskContext, Result[K]),(K, V)) => Result[K] = {
+ val delta = 5e-5
+ (U: (TaskContext, Result[K]), item: (K, V)) => {
+ val result = U._2
+ val tc = U._1
+ val rng = result.getRand(tc.partitionId)
+ val fraction = fractionByKey(item._1)
+ val stratum = result.getEntry(item._1)
+ if (withReplacement) {
+ // compute q1 and q2 only if they haven't been computed already
+ // since they don't change from iteration to iteration.
+ // TODO change this to the streaming version
+ if (stratum.q1.isEmpty || stratum.q2.isEmpty) {
+ val n = counts.get(item._1)
+ val s = math.ceil(n * fraction).toLong
+ val lmbd1 = PB.getLowerBound(s)
+ val minCount = PB.getMinCount(lmbd1)
+ val lmbd2 = if (lmbd1 == 0) PB.getUpperBound(s) else PB.getUpperBound(s - minCount)
+ val q1 = lmbd1 / n
+ val q2 = lmbd2 / n
+ stratum.q1 = Some(q1)
+ stratum.q2 = Some(q2)
+ }
+ val x1 = if (stratum.q1.get == 0) 0L else rng.nextPoisson(stratum.q1.get)
+ if (x1 > 0) {
+ stratum.incrNumAccepted(x1)
+ }
+ val x2 = rng.nextPoisson(stratum.q2.get).toInt
+ if (x2 > 0) {
+ stratum.addToWaitList(ArrayBuffer.fill(x2)(rng.nextUniform(0.0, 1.0)))
+ }
+ } else {
+ // We use the streaming version of the algorithm for sampling without replacement.
+ // Hence, q1 and q2 change on every iteration.
+ val g1 = - math.log(delta) / stratum.numItems
+ val g2 = (2.0 / 3.0) * g1
+ val q1 = math.max(0, fraction + g2 - math.sqrt((g2 * g2 + 3 * g2 * fraction)))
+ val q2 = math.min(1, fraction + g1 + math.sqrt(g1 * g1 + 2 * g1 * fraction))
+
+ val x = rng.nextUniform(0.0, 1.0)
+ if (x < q1) {
+ stratum.incrNumAccepted()
+ } else if (x < q2) {
+ stratum.addToWaitList(x)
+ }
+ stratum.q1 = Some(q1)
+ stratum.q2 = Some(q2)
+ }
+ stratum.incrNumItems()
+ result
+ }
+ }
+
+ /**
+ * Returns the function used by aggregate to combine results from different partitions, as
+ * returned by seqOp.
+ */
+ def getCombOp[K](): (Result[K], Result[K]) => Result[K] = {
+ (r1: Result[K], r2: Result[K]) => {
+ // take union of both key sets in case one partition doesn't contain all keys
+ val keyUnion = r1.resultMap.keys.toSet.union(r2.resultMap.keys.toSet)
+
+ // Use r2 to keep the combined result since r1 is usual empty
+ for (key <- keyUnion) {
+ val entry1 = r1.resultMap.get(key)
+ val entry2 = r2.resultMap.get(key)
+ if (entry2.isEmpty && entry1.isDefined) {
+ r2.resultMap += (key -> entry1.get)
+ } else if (entry1.isDefined && entry2.isDefined) {
+ entry2.get.addToWaitList(entry1.get.waitList)
+ entry2.get.incrNumAccepted(entry1.get.numAccepted)
+ entry2.get.incrNumItems(entry1.get.numItems)
+ }
+ }
+ r2
+ }
+ }
+
+ /**
+ * Given the result returned by the aggregate function, we need to determine the threshold used
+ * to accept items to generate the exact sample size.
+ */
+ def computeThresholdByKey[K](finalResult: Map[K, Stratum], fractionByKey: (K => Double)):
+ (K => Double) = {
+ val thresholdByKey = new mutable.HashMap[K, Double]()
+ for ((key, stratum) <- finalResult) {
+ val fraction = fractionByKey(key)
+ val s = math.ceil(stratum.numItems * fraction).toLong
+ if (stratum.numAccepted > s) {
+ logWarning("Pre-accepted too many")
+ thresholdByKey += (key -> stratum.q1.get)
+ } else {
+ val numWaitListAccepted = (s - stratum.numAccepted).toInt
+ if (numWaitListAccepted >= stratum.waitList.size) {
+ logWarning("WaitList too short")
+ thresholdByKey += (key -> stratum.q2.get)
+ } else {
+ thresholdByKey += (key -> stratum.waitList.sorted.apply(numWaitListAccepted))
+ }
+ }
+ }
+ thresholdByKey
+ }
+
+ /**
+ * Return the per partition sampling function used for sampling without replacement.
+ *
+ * When exact sample size is required, we make an additional pass over the RDD to determine the
+ * exact sampling rate that guarantees sample size with high confidence.
+ *
+ * The sampling function has a unique seed per partition.
+ */
+ def getBernoulliSamplingFunction[K, V](rdd: RDD[(K, V)],
+ fractionByKey: K => Double,
+ exact: Boolean,
+ seed: Long): (Int, Iterator[(K, V)]) => Iterator[(K, V)] = {
+ var samplingRateByKey = fractionByKey
+ if (exact) {
+ // determine threshold for each stratum and resample
+ val seqOp = StratifiedSampler.getSeqOp[K,V](false, fractionByKey, None)
+ val combOp = StratifiedSampler.getCombOp[K]()
+ val zeroU = new Result[K](Map[K, Stratum](), seed = seed)
+ val finalResult = aggregateWithContext(zeroU)(rdd, seqOp, combOp).resultMap
+ samplingRateByKey = StratifiedSampler.computeThresholdByKey(finalResult, fractionByKey)
+ }
+ (idx: Int, iter: Iterator[(K, V)]) => {
+ val random = new RandomDataGenerator
+ random.reSeed(seed + idx)
+ iter.filter(t => random.nextUniform(0.0, 1.0) < samplingRateByKey(t._1))
+ }
+ }
+
+ /**
+ * Return the per partition sampling function used for sampling with replacement.
+ *
+ * When exact sample size is required, we make two additional passed over the RDD to determine
+ * the exact sampling rate that guarantees sample size with high confidence. The first pass
+ * counts the number of items in each stratum (group of items with the same key) in the RDD, and
+ * the second pass uses the counts to determine exact sampling rates.
+ *
+ * The sampling function has a unique seed per partition.
+ */
+ def getPoissonSamplingFunction[K, V](rdd:RDD[(K, V)],
--- End diff --
Nit: add a space after colon
```rdd: Rdd[(K, V)]```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by falaki <gi...@git.apache.org>.
Github user falaki commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r14688702
--- Diff: core/src/test/scala/org/apache/spark/rdd/PairRDDFunctionsSuite.scala ---
@@ -83,6 +83,120 @@ class PairRDDFunctionsSuite extends FunSuite with SharedSparkContext {
assert(valuesFor2.toList.sorted === List(1))
}
+ test("sampleByKey") {
+ def stratifier (fractionPositive: Double) = {
+ (x: Int) => if (x % 10 < (10 * fractionPositive).toInt) "1" else "0"
+ }
+
+ def checkSize(exact: Boolean, withReplacement: Boolean,
+ expected: Long, actual: Long, p: Double): Boolean = {
+ if (exact) {
+ return expected == actual
+ }
+ val stdev = if (withReplacement) math.sqrt(expected) else math.sqrt(expected * p * (1 - p))
+ // Very forgiving margin since we're dealing with very small sample sizes most of the time
+ math.abs(actual - expected) <= 6 * stdev
+ }
+
+ // Without replacement validation
+ def takeSampleAndValidateBernoulli(stratifiedData: RDD[(String, Int)],
+ exact: Boolean,
+ samplingRate: Double,
+ seed: Long,
+ n: Long) = {
+ val expectedSampleSize = stratifiedData.countByKey().mapValues(count =>
+ math.ceil(count * samplingRate).toInt)
+ val fractions = Map("1" -> samplingRate, "0" -> samplingRate)
+ val sample = stratifiedData.sampleByKey(false, fractions, exact, seed)
+ val sampleCounts = sample.countByKey()
+ val takeSample = sample.collect()
+ assert(sampleCounts.forall({case(k,v) =>
+ checkSize(exact, false, expectedSampleSize(k), v, samplingRate)}))
+ assert(takeSample.size === takeSample.toSet.size)
+ assert(takeSample.forall(x => 1 <= x._2 && x._2 <= n), s"elements not in [1, $n]")
+ }
+
+ // With replacement validation
+ def takeSampleAndValidatePoisson(stratifiedData: RDD[(String, Int)],
+ exact: Boolean,
+ samplingRate: Double,
+ seed: Long,
+ n: Long) = {
+ val expectedSampleSize = stratifiedData.countByKey().mapValues(count =>
+ math.ceil(count * samplingRate).toInt)
+ val fractions = Map("1" -> samplingRate, "0" -> samplingRate)
+ val sample = stratifiedData.sampleByKey(true, fractions, exact, seed)
+ val sampleCounts = sample.countByKey()
+ val takeSample = sample.collect()
+ assert(sampleCounts.forall({case(k,v) =>
+ checkSize(exact, true, expectedSampleSize(k), v, samplingRate)}))
+ val groupedByKey = takeSample.groupBy({case(k, v) => k})
+ for ((key, v) <- groupedByKey) {
+ if (expectedSampleSize(key) >= 100 && samplingRate >= 0.1) {
+ // sample large enough for there to be repeats with high likelihood
+ assert(v.toSet.size < expectedSampleSize(key))
+ } else {
+ if (exact) {
+ assert(v.toSet.size <= expectedSampleSize(key))
+ } else {
+ assert(checkSize(false, true, expectedSampleSize(key), v.toSet.size, samplingRate))
+ }
+ }
+ }
+ assert(takeSample.forall(x => 1 <= x._2 && x._2 <= n), s"elements not in [1, $n]")
+ }
+
+ def checkAllCombos(stratifiedData: RDD[(String, Int)], samplingRate: Double,
--- End diff --
Style nit: When you break lines between parameters of a function, do it for ever one of them.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1025#issuecomment-50527473
QA results for PR 1025:<br>- This patch PASSES unit tests.<br>- This patch merges cleanly<br>- This patch adds no public classes<br><br>For more information see test ouptut:<br>https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17368/consoleFull
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1025#issuecomment-45951353
Build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by dorx <gi...@git.apache.org>.
Github user dorx commented on the pull request:
https://github.com/apache/spark/pull/1025#issuecomment-46389694
@falaki @mengxr Barring the API related issues, everything else should be in the "final" state. commons-math3 added as a Spark dependency (okay'ed by Matei). mvn clean install compiled.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1025#issuecomment-46377057
Merged build finished.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by dorx <gi...@git.apache.org>.
Github user dorx commented on the pull request:
https://github.com/apache/spark/pull/1025#issuecomment-45666934
@falaki
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1025#issuecomment-48384908
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r14694383
--- Diff: core/src/test/scala/org/apache/spark/rdd/PairRDDFunctionsSuite.scala ---
@@ -83,6 +83,125 @@ class PairRDDFunctionsSuite extends FunSuite with SharedSparkContext {
assert(valuesFor2.toList.sorted === List(1))
}
+ test("sampleByKey") {
+ def stratifier (fractionPositive: Double) = {
+ (x: Int) => if (x % 10 < (10 * fractionPositive).toInt) "1" else "0"
+ }
+
+ def checkSize(exact: Boolean,
+ withReplacement: Boolean,
+ expected: Long,
+ actual: Long,
+ p: Double): Boolean = {
+ if (exact) {
+ return expected == actual
+ }
+ val stdev = if (withReplacement) math.sqrt(expected) else math.sqrt(expected * p * (1 - p))
+ // Very forgiving margin since we're dealing with very small sample sizes most of the time
+ math.abs(actual - expected) <= 6 * stdev
+ }
+
+ // Without replacement validation
+ def takeSampleAndValidateBernoulli(stratifiedData: RDD[(String, Int)],
+ exact: Boolean,
+ samplingRate: Double,
+ seed: Long,
+ n: Long) = {
+ val expectedSampleSize = stratifiedData.countByKey().mapValues(count =>
+ math.ceil(count * samplingRate).toInt)
+ val fractions = Map("1" -> samplingRate, "0" -> samplingRate)
+ val sample = stratifiedData.sampleByKey(false, fractions, exact, seed)
+ val sampleCounts = sample.countByKey()
+ val takeSample = sample.collect()
+ assert(sampleCounts.forall {case(k,v) =>
+ checkSize(exact, false, expectedSampleSize(k), v, samplingRate)})
+ assert(takeSample.size === takeSample.toSet.size)
+ assert(takeSample.forall(x => 1 <= x._2 && x._2 <= n), s"elements not in [1, $n]")
+ }
+
+ // With replacement validation
+ def takeSampleAndValidatePoisson(stratifiedData: RDD[(String, Int)],
+ exact: Boolean,
+ samplingRate: Double,
+ seed: Long,
+ n: Long) = {
+ val expectedSampleSize = stratifiedData.countByKey().mapValues(count =>
+ math.ceil(count * samplingRate).toInt)
+ val fractions = Map("1" -> samplingRate, "0" -> samplingRate)
+ val sample = stratifiedData.sampleByKey(true, fractions, exact, seed)
+ val sampleCounts = sample.countByKey()
+ val takeSample = sample.collect()
+ assert(sampleCounts.forall {case(k,v) =>
+ checkSize(exact, true, expectedSampleSize(k), v, samplingRate)})
+ val groupedByKey = takeSample.groupBy {case(k, v) => k}
--- End diff --
`groupBy(_._1)`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by dorx <gi...@git.apache.org>.
Github user dorx commented on the pull request:
https://github.com/apache/spark/pull/1025#issuecomment-48386518
Jenkins, retest this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r14694254
--- Diff: core/src/main/scala/org/apache/spark/util/random/SamplingUtils.scala ---
@@ -45,11 +50,78 @@ private[spark] object SamplingUtils {
val fraction = sampleSizeLowerBound.toDouble / total
if (withReplacement) {
val numStDev = if (sampleSizeLowerBound < 12) 9 else 5
- fraction + numStDev * math.sqrt(fraction / total)
+ math.max(1e-10, fraction + numStDev * math.sqrt(fraction / total))
} else {
val delta = 1e-4
val gamma = - math.log(delta) / total
- math.min(1, fraction + gamma + math.sqrt(gamma * gamma + 2 * gamma * fraction))
+ math.min(1,
+ math.max(1e-10, fraction + gamma + math.sqrt(gamma * gamma + 2 * gamma * fraction)))
+ }
+ }
+}
+
+/**
+ * Utility functions that help us determine bounds on adjusted sampling rate to guarantee exact
+ * sample sizes with high confidence when sampling with replacement.
+ *
+ * The algorithm for guaranteeing sample size instantly accepts items whose associated value drawn
+ * from Pois(s) is less than the lower bound and puts items whose value is between the lower and
+ * upper bound in a waitlist. The final sample is consisted of all items accepted on the fly and a
+ * portion of the waitlist needed to make the exact sample size.
+ */
+private[spark] object PoissonBounds {
+
+ val delta = 1e-4 / 3.0
+ val epsilon = 1e-15
+
+ /**
+ * Compute the threshold for accepting items on the fly. The threshold value is a fairly small
+ * number, which means if the item has an associated value < threshold, it is highly likely to
+ * be in the final sample. Hence we accept items with values less than the returned value of this
+ * function instantly.
+ *
+ * @param s sample size
+ * @return threshold for accepting items on the fly
+ */
+ def getLowerBound(s: Double): Double = {
+ var lb = math.max(0.0, s - math.sqrt(s / delta)) // Chebyshev's inequality
+ var ub = s
+ while (lb < ub - 1.0) {
+ val m = (lb + ub) / 2.0
+ val poisson = new PoissonDistribution(m, epsilon)
+ val y = poisson.inverseCumulativeProbability(1 - delta)
+ if (y > s) ub = m else lb = m
+ }
+ lb
+ }
+
+ def getMinCount(lmbd: Double): Double = {
--- End diff --
doc?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---