You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by firemonk9 <dh...@gmail.com> on 2014/11/26 19:15:19 UTC
RDD saveAsObjectFile write to local file and HDFS
When I am running spark locally, RDD saveAsObjectFile writes the file to
local file system (ex : path /data/temp.txt)
and
when I am running spark on YARN cluster, RDD saveAsObjectFile writes the
file to hdfs. (ex : path /data/temp.txt )
Is there a way to explictly mention local file system instead of hdfs when
running on YARN cluster.
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/RDD-saveAsObjectFile-write-to-local-file-and-HDFS-tp19898.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: RDD saveAsObjectFile write to local file and HDFS
Posted by Daniel Haviv <da...@gmail.com>.
Prepend file:// to the path
Daniel
> On 26 בנוב׳ 2014, at 20:15, firemonk9 <dh...@gmail.com> wrote:
>
> When I am running spark locally, RDD saveAsObjectFile writes the file to
> local file system (ex : path /data/temp.txt)
> and
> when I am running spark on YARN cluster, RDD saveAsObjectFile writes the
> file to hdfs. (ex : path /data/temp.txt )
>
> Is there a way to explictly mention local file system instead of hdfs when
> running on YARN cluster.
>
>
>
> --
> View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/RDD-saveAsObjectFile-write-to-local-file-and-HDFS-tp19898.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
Re: RDD saveAsObjectFile write to local file and HDFS
Posted by Du Li <li...@yahoo-inc.com.INVALID>.
Add ³file://³ in front of your path.
On 11/26/14, 10:15 AM, "firemonk9" <dh...@gmail.com> wrote:
>When I am running spark locally, RDD saveAsObjectFile writes the file to
>local file system (ex : path /data/temp.txt)
>and
>when I am running spark on YARN cluster, RDD saveAsObjectFile writes the
>file to hdfs. (ex : path /data/temp.txt )
>
>Is there a way to explictly mention local file system instead of hdfs when
>running on YARN cluster.
>
>
>
>--
>View this message in context:
>http://apache-spark-user-list.1001560.n3.nabble.com/RDD-saveAsObjectFile-w
>rite-to-local-file-and-HDFS-tp19898.html
>Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>For additional commands, e-mail: user-help@spark.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org