[jira] [Updated] (SPARK-31140) Support Quick sample in RDD

["deshanxiao (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-31140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

deshanxiao updated SPARK-31140:
-------------------------------
    Description: 
RDD.sample use the function of *filter* to pick up the data we need. It means that if the raw data is very huge, we must cost too much time to read it. We can filter the raw partition to speed up the processing of sample.


{code:java}
  override def compute(splitIn: Partition, context: TaskContext): Iterator[U] = {
    val split = splitIn.asInstanceOf[PartitionwiseSampledRDDPartition]
    val thisSampler = sampler.clone
    thisSampler.setSeed(split.seed)
    thisSampler.sample(firstParent[T].iterator(split.prev, context))
  }
{code}


  was:
RDD.sample use *filter* to read the raw data. It means that if the raw data is very huge, we must cost too much time to read it. We can filter the raw partition to speed up the processing of sample.


{code:java}
  override def compute(splitIn: Partition, context: TaskContext): Iterator[U] = {
    val split = splitIn.asInstanceOf[PartitionwiseSampledRDDPartition]
    val thisSampler = sampler.clone
    thisSampler.setSeed(split.seed)
    thisSampler.sample(firstParent[T].iterator(split.prev, context))
  }
{code}



> Support Quick sample in RDD
> ---------------------------
>
>                 Key: SPARK-31140
>                 URL: https://issues.apache.org/jira/browse/SPARK-31140
>             Project: Spark
>          Issue Type: Improvement
>          Components: Spark Core
>    Affects Versions: 3.0.0
>            Reporter: deshanxiao
>            Priority: Minor
>
> RDD.sample use the function of *filter* to pick up the data we need. It means that if the raw data is very huge, we must cost too much time to read it. We can filter the raw partition to speed up the processing of sample.
> {code:java}
>   override def compute(splitIn: Partition, context: TaskContext): Iterator[U] = {
>     val split = splitIn.asInstanceOf[PartitionwiseSampledRDDPartition]
>     val thisSampler = sampler.clone
>     thisSampler.setSeed(split.seed)
>     thisSampler.sample(firstParent[T].iterator(split.prev, context))
>   }
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org