[jira] [Resolved] (SPARK-1074) JavaPairRDD as Object File

["Sean Owen (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-1074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sean Owen resolved SPARK-1074.
------------------------------
    Resolution: Not a Problem

Am I right in thinking that if you want to save a JavaPairRDD to HDFS, you have key-value pairs, and so you want to use JavaPairRDD.saveAsNewAPIHadoopFile, and SparkContext.sequenceFile to read it? This works. objectFile doesn't seem like the right approach anyway.

> JavaPairRDD as Object File
> --------------------------
>
>                 Key: SPARK-1074
>                 URL: https://issues.apache.org/jira/browse/SPARK-1074
>             Project: Spark
>          Issue Type: Bug
>          Components: Input/Output, Java API
>    Affects Versions: 0.9.0
>            Reporter: Kevin Mader
>            Priority: Minor
>
> So I can perform a save command on a JavaPairRDD
> {code:java}
> static public void HSave(JavaPairRDD<D3int, int[]> baseImg,String path) {
> 		final String outpath=(new File(path)).getAbsolutePath();
> 		baseImg.saveAsObjectFile(outpath);
> }
> {code}
> When I use the objectFile command from the JavaSparkContext 
> {code:java}
> static public  ReadObjectFile(JavaSparkContext jsc, final String path) {
> 	JavaPairRDD<D3int, int[]> newImage=(JavaPairRDD<D3int,int[]>) jsc.objectFile(path);
> }
> {code}
> I get an error cannot cast from JavaRDD to JavaPairRDD. Is there a way to get back to JavaPairRDD or will I need to map my data to a JavaRDD, save, load, then remap the JavaRDD back to the JavaPairRDD



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org