You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Petr Novak <os...@gmail.com> on 2014/09/24 13:35:14 UTC
Does Spark Driver works with HDFS in HA mode
Hello,
if our Hadoop cluster is configured with HA and "fs.defaultFS" points to a
namespace instead of a namenode hostname - hdfs://<namespace_name>/ - then
our Spark job fails with exception. Is there anything to configure or it is
not implemented?
Exception in thread "main" org.apache.spark.SparkException: Job aborted due
to stage failure: Task 0 in stage 1.0 failed 4 times, most recent failure:
Lost task 0.3 in stage 1.0 (TID 4, <hostanme>):
java.lang.IllegalArgumentException: java.net.UnknownHostException:
*<namespace_name>*
Many thanks,
P.
Re: Does Spark Driver works with HDFS in HA mode
Posted by Petr Novak <os...@gmail.com>.
Thank you. HADOOP_CONF_DIR has been missing.
On Wed, Sep 24, 2014 at 4:48 PM, Matt Narrell <ma...@gmail.com>
wrote:
> Yes, this works. Make sure you have HADOOP_CONF_DIR set on your Spark
> machines
>
> mn
>
> On Sep 24, 2014, at 5:35 AM, Petr Novak <os...@gmail.com> wrote:
>
> Hello,
> if our Hadoop cluster is configured with HA and "fs.defaultFS" points to a
> namespace instead of a namenode hostname - hdfs://<namespace_name>/ - then
> our Spark job fails with exception. Is there anything to configure or it is
> not implemented?
>
>
> Exception in thread "main" org.apache.spark.SparkException: Job aborted
> due to stage failure: Task 0 in stage 1.0 failed 4 times, most recent
> failure: Lost task 0.3 in stage 1.0 (TID 4, <hostanme>):
>
>
> java.lang.IllegalArgumentException: java.net.UnknownHostException:
> *<namespace_name>*
>
>
> Many thanks,
> P.
>
>
>
Re: Does Spark Driver works with HDFS in HA mode
Posted by Matt Narrell <ma...@gmail.com>.
Yes, this works. Make sure you have HADOOP_CONF_DIR set on your Spark machines
mn
On Sep 24, 2014, at 5:35 AM, Petr Novak <os...@gmail.com> wrote:
> Hello,
> if our Hadoop cluster is configured with HA and "fs.defaultFS" points to a namespace instead of a namenode hostname - hdfs://<namespace_name>/ - then our Spark job fails with exception. Is there anything to configure or it is not implemented?
>
>
> Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 1.0 failed 4 times, most recent failure: Lost task 0.3 in stage 1.0 (TID 4, <hostanme>):
>
>
>
> java.lang.IllegalArgumentException: java.net.UnknownHostException: <namespace_name>
>
>
>
> Many thanks,
> P.
>