[GitHub] spark pull request: Making RDD.isEmpty robust to empty partitions ...

[tbertelsen <gi...@git.apache.org>]

Github user tbertelsen commented on a diff in the pull request:

    https://github.com/apache/spark/pull/4534#discussion_r24521457
  
    --- Diff: core/src/main/scala/org/apache/spark/rdd/RDD.scala ---
    @@ -1253,9 +1253,9 @@ abstract class RDD[T: ClassTag](
     
       /**
        * @return true if and only if the RDD contains no elements at all. Note that an RDD
    -   *         may be empty even when it has at least 1 partition.
    +   *         may be empty even when it has 1 or more partitions.
        */
    -  def isEmpty(): Boolean = partitions.length == 0 || take(1).length == 0
    +  def isEmpty(): Boolean = partitions.length == 0 || mapPartitions(it => Iterator(!it.hasNext)).reduce(_&&_)
    --- End diff --
    
    You might have a point, that this is actually a bug in `take()`.
    
    `sc.parallelize(Seq(1,2,3),1).take(1337)` works fine but and returns Array[Int] = Array(1, 2, 3)`
    
    `sc.parallelize(Seq(),1).take(1)` fails on the other hand.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org