[jira] [Commented] (SPARK-31140) Support Quick sample in RDD

["L. C. Hsieh (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-31140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17059887#comment-17059887 ] 

L. C. Hsieh commented on SPARK-31140:
-------------------------------------

What does it mean to "filter the raw partition"?

> Support Quick sample in RDD
> ---------------------------
>
>                 Key: SPARK-31140
>                 URL: https://issues.apache.org/jira/browse/SPARK-31140
>             Project: Spark
>          Issue Type: Improvement
>          Components: Spark Core
>    Affects Versions: 3.0.0
>            Reporter: deshanxiao
>            Priority: Minor
>
> RDD.sample use the function of *filter* to pick up the data we need. It means that if the raw data is very huge, we must spend too much time reading it. We can filter the raw partition to speed up the processing of sample.
> {code:java}
>   override def compute(splitIn: Partition, context: TaskContext): Iterator[U] = {
>     val split = splitIn.asInstanceOf[PartitionwiseSampledRDDPartition]
>     val thisSampler = sampler.clone
>     thisSampler.setSeed(split.seed)
>     thisSampler.sample(firstParent[T].iterator(split.prev, context))
>   }
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org