You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Derrick Burns (JIRA)" <ji...@apache.org> on 2015/02/25 09:34:05 UTC

[jira] [Updated] (SPARK-6003) Spark should offer a "sync" method that guarantees that RDDs are eagerly evaluated and persisted

     [ https://issues.apache.org/jira/browse/SPARK-6003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Derrick Burns updated SPARK-6003:
---------------------------------
    Summary: Spark should offer a "sync" method that guarantees that RDDs are eagerly evaluated and persisted  (was: Spark should offer a "sync" method that guarantees that RDDs are eagerly evaluted and persisted)

> Spark should offer a "sync" method that guarantees that RDDs are eagerly evaluated and persisted
> ------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-6003
>                 URL: https://issues.apache.org/jira/browse/SPARK-6003
>             Project: Spark
>          Issue Type: Improvement
>          Components: Spark Core
>    Affects Versions: 1.2.1
>            Reporter: Derrick Burns
>            Priority: Minor
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> This may already exist, but I could not find it.
> One of the challenges in developing RELIABLE Spark application is dealing with the elegant lazy evaluation semantics of RDD transformations.  It would be useful to have a action with no output whose side-effect is to ensure that the RDD is eagerly evaluated and persisted according the whatever persistence level is set for the RDD.  
> Calling RDD.count() or any other action might do the trick -- and indeed I have tried this -- however, in can be the case that RDD.count() does NOT persist the data. 
> For example,  MappedRDD(x:RDD).count() === x.count(), so it is possible to implement count without persisting the result of MappedRDD(x).  Without looking at the code, one cannot know whether an operation is eagerly evaluated AND persisted or not.   Having a standard Spark primitive that both eagerly evaluated and RDD and persisted it according to its current persistence level would be very useful.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org