You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by rroxanaioana <rr...@gmail.com> on 2015/05/21 17:16:18 UTC
saveAsTextFile() part- files are missing
Hello!
I just started with Spark. I have an application which counts words in a
file (1 MB file).
The file is stored locally. I loaded the file using native code and then
created the RDD from it.
JavaRDD<String> rddFromFile = context.parallelize(myFile,
2);
JavaRDD<String> words = rddFromFile.flatMap(...);
JavaPairRDD<String, Integer> pairs = words.mapToPair(...);
JavaPairRDD<String, Integer> counter = pairs.reduceByKey(..);
counter.saveAsTextFile("file:///root/output");
context.close();
I have one master and 2 slaves. I run the program from the master node.
The output directory is created on the master node and on the 2 nodes. On
the master node I have only one file _SUCCES (empty) and on the nodes I have
_temporary file. I printed the counter at the console, the result seems ok.
What am I doing wrong?
Thank you!
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/saveAsTextFile-part-files-are-missing-tp22974.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: saveAsTextFile() part- files are missing
Posted by Tomasz Fruboes <To...@fuw.edu.pl>.
Hi,
it looks you are writing to a local filesystem. Could you try writing
to a location visible by all nodes (master and workers), e.g. nfs share?
HTH,
Tomasz
W dniu 21.05.2015 o 17:16, rroxanaioana pisze:
> Hello!
> I just started with Spark. I have an application which counts words in a
> file (1 MB file).
> The file is stored locally. I loaded the file using native code and then
> created the RDD from it.
>
> JavaRDD<String> rddFromFile = context.parallelize(myFile,
> 2);
> JavaRDD<String> words = rddFromFile.flatMap(...);
> JavaPairRDD<String, Integer> pairs = words.mapToPair(...);
> JavaPairRDD<String, Integer> counter = pairs.reduceByKey(..);
>
> counter.saveAsTextFile("file:///root/output");
> context.close();
>
> I have one master and 2 slaves. I run the program from the master node.
> The output directory is created on the master node and on the 2 nodes. On
> the master node I have only one file _SUCCES (empty) and on the nodes I have
> _temporary file. I printed the counter at the console, the result seems ok.
> What am I doing wrong?
> Thank you!
>
>
>
>
>
> --
> View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/saveAsTextFile-part-files-are-missing-tp22974.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org