sample is not a member of org.apache.spark.streaming.dstream.DStream

[Josh J <jo...@gmail.com>]

Hi,

I'm trying to using sampling with Spark Streaming. I imported the following

import org.apache.spark.{SparkConf, SparkContext}
import org.apache.spark.SparkContext._


I then call sample


val streamtoread = KafkaUtils.createStream(ssc, zkQuorum, group,
topicMap,StorageLevel.MEMORY_AND_DISK).map(_._2)

streamtoread.sample(withReplacement = true, fraction = fraction)


How do I use the sample
<http://spark.apache.org/docs/latest/programming-guide.html#transformations>()
method with Spark Streaming?


Thanks,

Josh

Re: sample is not a member of org.apache.spark.streaming.dstream.DStream

[Sean Owen <so...@cloudera.com>]

The method you're referring to is a method of RDD, not DStream. If you
want to do something with a sample of each RDD in the DStream, then
call

streamtoread.foreachRDD { rdd =>
  val sampled = rdd.sample(...)
  ...
}

On Sun, Dec 28, 2014 at 10:44 PM, Josh J <jo...@gmail.com> wrote:
> Hi,
>
> I'm trying to using sampling with Spark Streaming. I imported the following
>
> import org.apache.spark.{SparkConf, SparkContext}
> import org.apache.spark.SparkContext._
>
>
> I then call sample
>
>
> val streamtoread = KafkaUtils.createStream(ssc, zkQuorum, group,
> topicMap,StorageLevel.MEMORY_AND_DISK).map(_._2)
>
> streamtoread.sample(withReplacement = true, fraction = fraction)
>
>
> How do I use the sample() method with Spark Streaming?
>
>
> Thanks,
>
> Josh

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org