You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by EH <ea...@gmail.com> on 2014/11/21 15:28:15 UTC
Setup Remote HDFS for Spark
Hi,
Are there any way that I can setup a remote HDFS for Spark (more specific,
for Spark Streaming checkpoints)? The reason I'm asking is that our Spark
and HDFS do not run on the same machines. I've been looked around but still
no clue so far.
Thanks,
EH
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Setup-Remote-HDFS-for-Spark-tp19481.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Setup Remote HDFS for Spark
Posted by Akhil Das <ak...@sigmoidanalytics.com>.
Yes, it is possible to have both spark and hdfs running on the same
cluster. We have a lot of clusters running without any issues. And yes, it
is possible to hook spark up with remote hdfs. You might feel a bit lag if
they are on different networks.
Thanks
Best Regards
On Fri, Nov 21, 2014 at 8:43 PM, EH <ea...@gmail.com> wrote:
> Unfortunately whether it is possible to have both Spark and HDFS running on
> the same machine is not under our control. :( Right now we have Spark and
> HDFS running in different machines. In this case, is it still possible to
> hook up a remote HDFS with Spark so that we can use Spark Streaming
> checkpoints? Thank you for your help.
>
> Best,
> EH
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Setup-Remote-HDFS-for-Spark-tp19481p19485.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>
Re: Setup Remote HDFS for Spark
Posted by Tobias Pfeiffer <tg...@preferred.jp>.
Hi,
On Sat, Nov 22, 2014 at 12:13 AM, EH <ea...@gmail.com> wrote:
> Unfortunately whether it is possible to have both Spark and HDFS running on
> the same machine is not under our control. :( Right now we have Spark and
> HDFS running in different machines. In this case, is it still possible to
> hook up a remote HDFS with Spark so that we can use Spark Streaming
> checkpoints? Thank you for your help.
>
In my case, after I copied my cluster's core-site.xml, yarn-site.xml, and
hdfs-site.xml to my CLASSPATH, I could access YARN and HDFS remotely
without any problems (read: modulo network/firewall issues), which I was
pretty surprised by myself.
Tobias
Re: Setup Remote HDFS for Spark
Posted by EH <ea...@gmail.com>.
Unfortunately whether it is possible to have both Spark and HDFS running on
the same machine is not under our control. :( Right now we have Spark and
HDFS running in different machines. In this case, is it still possible to
hook up a remote HDFS with Spark so that we can use Spark Streaming
checkpoints? Thank you for your help.
Best,
EH
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Setup-Remote-HDFS-for-Spark-tp19481p19485.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Setup Remote HDFS for Spark
Posted by Akhil Das <ak...@sigmoidanalytics.com>.
Having them on the same network will give you better performance. What
problems are you facing? Can you elaborate more about the versions? (Spark
and hadoop), Only problem that could happen is the OS's OOM Killer might be
killing your HDFS processes due to low memory and such. But you can check
the logs to get more clear picture about whats happening.
Thanks
Best Regards
On Fri, Nov 21, 2014 at 7:58 PM, EH <ea...@gmail.com> wrote:
> Hi,
>
> Are there any way that I can setup a remote HDFS for Spark (more specific,
> for Spark Streaming checkpoints)? The reason I'm asking is that our Spark
> and HDFS do not run on the same machines. I've been looked around but
> still
> no clue so far.
>
> Thanks,
> EH
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Setup-Remote-HDFS-for-Spark-tp19481.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>