You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by WangJianfei <wa...@otcaix.iscas.ac.cn> on 2016/09/16 06:54:58 UTC
What's the meaning when the partitions is zero?
class HashPartitioner(partitions: Int) extends Partitioner {
require(partitions >= 0, s"Number of partitions ($partitions) cannot be
negative.")
the soruce code require(partitions >=0) ,but I don't know why it makes sense
when the partitions is 0.
--
View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/What-s-the-meaning-when-the-partitions-is-zero-tp18957.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: What's the meaning when the partitions is zero?
Posted by Mridul Muralidharan <mr...@gmail.com>.
When numPartitions is 0, there is no data in the rdd: so getPartition is
never invoked.
- Mridul
On Friday, September 16, 2016, WangJianfei <wa...@otcaix.iscas.ac.cn>
wrote:
> if so, we will get exception when the numPartitions is 0.
> def getPartition(key: Any): Int = key match {
> case null => 0
> // case None => 0
> case _ => Utils.nonNegativeMod(key.hashCode, numPartitions)
> }
>
>
>
> --
> View this message in context: http://apache-spark-
> developers-list.1001551.n3.nabble.com/What-s-the-meaning-
> when-the-partitions-is-zero-tp18957p18967.html
> Sent from the Apache Spark Developers List mailing list archive at
> Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org <javascript:;>
>
>
Re: What's the meaning when the partitions is zero?
Posted by WangJianfei <wa...@otcaix.iscas.ac.cn>.
if so, we will get exception when the numPartitions is 0.
def getPartition(key: Any): Int = key match {
case null => 0
// case None => 0
case _ => Utils.nonNegativeMod(key.hashCode, numPartitions)
}
--
View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/What-s-the-meaning-when-the-partitions-is-zero-tp18957p18967.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: What's the meaning when the partitions is zero?
Posted by Reynold Xin <rx...@databricks.com>.
They are valid, especially in partition pruning.
On Friday, September 16, 2016, Sean Owen <so...@cloudera.com> wrote:
> There are almost no cases in which you'd want a zero-partition RDD.
> The only one I can think of is an empty RDD, where the number of
> partitions is irrelevant. Still, I would not be surprised if other
> parts of the code assume at least 1 partition.
>
> Maybe this check could be tightened. It would be interesting to see if
> the tests catch any scenario where a 0-partition RDD is created, and
> why.
>
> On Fri, Sep 16, 2016 at 7:54 AM, WangJianfei
> <wangjianfei15@otcaix.iscas.ac.cn <javascript:;>> wrote:
> > class HashPartitioner(partitions: Int) extends Partitioner {
> > require(partitions >= 0, s"Number of partitions ($partitions) cannot be
> > negative.")
> >
> > the soruce code require(partitions >=0) ,but I don't know why it makes
> sense
> > when the partitions is 0.
> >
> >
> >
> > --
> > View this message in context: http://apache-spark-
> developers-list.1001551.n3.nabble.com/What-s-the-meaning-
> when-the-partitions-is-zero-tp18957.html
> > Sent from the Apache Spark Developers List mailing list archive at
> Nabble.com.
> >
> > ---------------------------------------------------------------------
> > To unsubscribe e-mail: dev-unsubscribe@spark.apache.org <javascript:;>
> >
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org <javascript:;>
>
>
Re: What's the meaning when the partitions is zero?
Posted by Sean Owen <so...@cloudera.com>.
There are almost no cases in which you'd want a zero-partition RDD.
The only one I can think of is an empty RDD, where the number of
partitions is irrelevant. Still, I would not be surprised if other
parts of the code assume at least 1 partition.
Maybe this check could be tightened. It would be interesting to see if
the tests catch any scenario where a 0-partition RDD is created, and
why.
On Fri, Sep 16, 2016 at 7:54 AM, WangJianfei
<wa...@otcaix.iscas.ac.cn> wrote:
> class HashPartitioner(partitions: Int) extends Partitioner {
> require(partitions >= 0, s"Number of partitions ($partitions) cannot be
> negative.")
>
> the soruce code require(partitions >=0) ,but I don't know why it makes sense
> when the partitions is 0.
>
>
>
> --
> View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/What-s-the-meaning-when-the-partitions-is-zero-tp18957.html
> Sent from the Apache Spark Developers List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org