[jira] [Commented] (SPARK-10723) Add RDD.reduceOption method

["Apache Spark (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-10723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14900184#comment-14900184 ] 

Apache Spark commented on SPARK-10723:
--------------------------------------

User 'Attsun1031' has created a pull request for this issue:
https://github.com/apache/spark/pull/8845

> Add RDD.reduceOption method
> ---------------------------
>
>                 Key: SPARK-10723
>                 URL: https://issues.apache.org/jira/browse/SPARK-10723
>             Project: Spark
>          Issue Type: Improvement
>          Components: PySpark, Spark Core
>            Reporter: Tatsuya Atsumi
>            Priority: Minor
>
> h2. Problem
> RDD.reduce throws exception if the RDD is empty.
> It is appropriate behavior if RDD is expected to be not empty, but if it is not sure until runtime that the RDD is empty or not, it needs to wrap with try-catch to call reduce safely. 
> Example Code
> {code}
> // This RDD may be empty or not
> val rdd: RDD[Int] = originalRdd.filter(_ > 10)
> val reduced: Option[Int] = try {
>   Some(rdd.reduce(_ + _))
> } catch {
>   // if rdd is empty return None.
>   case e:UnsupportedOperationException => None
> }
> {code}
> h2. Improvement idea
> Scala’s List has reduceOption method, which returns None if List is empty.
> If RDD has reduceOption API like Scala’s List, it will become easy to handle above case.
> Example Code
> {code}
> val reduced: Option[Int] = originalRdd.filter(_ > 10).reduceOption(_ + _)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org