What's the logic in RangePartitioner.rangeBounds method of Apache Spark

[ihainan <ih...@gmail.com>]

*Firstly so sorry for my poor English.*

I was reading the source code of Apache Spark 1.4.1 and I really got stuck
at the logic of RangePartitioner.rangeBounds method. The code is shown
below.



So can anyone please explain me that:

1. What is "3.0 *" for in the code line of "val sampleSizePerPartition =
math.ceil(3.0 * sampleSize / rdd.partitions.size).toInt"? Why choose 3.0
rather than other values?

2. Why "fraction * n > sampleSizePerPartition" means that a partition
contains much more than the average number of items. Can you give an example
that we need to re-sample the partition?

Thanks a lot!



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/What-s-the-logic-in-RangePartitioner-rangeBounds-method-of-Apache-Spark-tp24296.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org