You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Yann Luppo <Ya...@LiveNation.com> on 2013/10/23 23:01:55 UTC

JavaPairRDD unpersist

Hi,

I've noticed that in 0.8.0 JavaRDD got a new "unpersist" method. I was wondering if this method was going to be ported eventually to JavaPairRDD.
Is this somewhere on the road map? Or should we just change all of our JavaPairRDD to JavaRDDs w/ tuples for now?

Thanks,
Yann

Re: JavaPairRDD unpersist

Posted by Yann Luppo <Ya...@LiveNation.com>.

Perfect! Thank you Josh.

From: Josh Rosen <ro...@gmail.com>>
Reply-To: "user@spark.incubator.apache.org<ma...@spark.incubator.apache.org>" <us...@spark.incubator.apache.org>>
Date: Wednesday, October 23, 2013 2:19 PM
To: "user@spark.incubator.apache.org<ma...@spark.incubator.apache.org>" <us...@spark.incubator.apache.org>>
Subject: Re: JavaPairRDD unpersist

This is a bug; all three Java*RDD classes should have had unpersist() methods added to them, instead of just JavaRDD (https://github.com/apache/incubator-spark/commit/4a318774088f829fe54c3ef0b5f565a845631b4e).  I'll submit a pull request to fix this.  In 0.8, you can access the underlying Scala RDD and call unpersist() on that:

javaPairRDD.rdd().unpersist()

If you want to know the details of why we need to add unpersist() to each Java*RDD class, rather than adding it once in JavaRDDLike, check out the "implementation traits" section of https://cwiki.apache.org/confluence/display/SPARK/Java+API+Internals

- Josh

On Wed, Oct 23, 2013 at 2:01 PM, Yann Luppo <Ya...@livenation.com>> wrote:
Hi,

I've noticed that in 0.8.0 JavaRDD got a new "unpersist" method. I was wondering if this method was going to be ported eventually to JavaPairRDD.
Is this somewhere on the road map? Or should we just change all of our JavaPairRDD to JavaRDDs w/ tuples for now?

Thanks,
Yann

Re: JavaPairRDD unpersist

Posted by Josh Rosen <ro...@gmail.com>.

This is a bug; all three Java*RDD classes should have had unpersist()
methods added to them, instead of just JavaRDD (
https://github.com/apache/incubator-spark/commit/4a318774088f829fe54c3ef0b5f565a845631b4e).
 I'll submit a pull request to fix this.  In 0.8, you can access the
underlying Scala RDD and call unpersist() on that:

javaPairRDD.rdd().unpersist()

If you want to know the details of why we need to add unpersist() to each
Java*RDD class, rather than adding it once in JavaRDDLike, check out the
"implementation traits" section of
https://cwiki.apache.org/confluence/display/SPARK/Java+API+Internals

- Josh

On Wed, Oct 23, 2013 at 2:01 PM, Yann Luppo <Ya...@livenation.com>wrote:

>  Hi,
>
>  I've noticed that in 0.8.0 JavaRDD got a new "unpersist" method. I was
> wondering if this method was going to be ported eventually to JavaPairRDD.
> Is this somewhere on the road map? Or should we just change all of our
> JavaPairRDD to JavaRDDs w/ tuples for now?
>
>  Thanks,
> Yann
>