You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by ameyc <am...@gmail.com> on 2014/10/31 23:44:27 UTC
hadoop_conf_dir when running spark on yarn
How do i setup hadoop_conf_dir correctly when I'm running my spark job on
Yarn? My Yarn environment has the correct hadoop_conf_dir settings by the
configuration that I pull from sc.hadoopConfiguration() is incorrect.
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/hadoop-conf-dir-when-running-spark-on-yarn-tp17872.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: hadoop_conf_dir when running spark on yarn
Posted by Amey Chaugule <am...@gmail.com>.
I'm using org.apache.spark.deploy.yarn.Client object to run my spark job. I
guess this is what spark-submit wraps really.
- Amey
On Mon, Nov 3, 2014 at 5:25 PM, Tobias Pfeiffer <tg...@preferred.jp> wrote:
> Hi,
>
> On Mon, Nov 3, 2014 at 1:29 PM, Amey Chaugule <am...@gmail.com> wrote:
>
>> I thought that only applied when you're trying to run a job using
>> spark-submit or in the shell...
>>
>
> And how are you starting your Yarn job, if not via spark-submit?
>
> Tobias
>
>
--
Amey S. Chaugule
B.S Computer Engineering and Mathematics '11,
University of Illinois at Urbana-Champaign
Re: hadoop_conf_dir when running spark on yarn
Posted by Tobias Pfeiffer <tg...@preferred.jp>.
Hi,
On Mon, Nov 3, 2014 at 1:29 PM, Amey Chaugule <am...@gmail.com> wrote:
> I thought that only applied when you're trying to run a job using
> spark-submit or in the shell...
>
And how are you starting your Yarn job, if not via spark-submit?
Tobias
Re: hadoop_conf_dir when running spark on yarn
Posted by Amey Chaugule <am...@gmail.com>.
I thought that only applied when you're trying to run a job using
spark-submit or in the shell...
On Sun, Nov 2, 2014 at 8:47 AM, Akhil Das <ak...@sigmoidanalytics.com>
wrote:
> You can set HADOOP_CONF_DIR inside the spark-env.sh file
>
> Thanks
> Best Regards
>
> On Sat, Nov 1, 2014 at 4:14 AM, ameyc <am...@gmail.com> wrote:
>
>> How do i setup hadoop_conf_dir correctly when I'm running my spark job on
>> Yarn? My Yarn environment has the correct hadoop_conf_dir settings by the
>> configuration that I pull from sc.hadoopConfiguration() is incorrect.
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-spark-user-list.1001560.n3.nabble.com/hadoop-conf-dir-when-running-spark-on-yarn-tp17872.html
>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> For additional commands, e-mail: user-help@spark.apache.org
>>
>>
>
--
Amey S. Chaugule
B.S Computer Engineering and Mathematics '11,
University of Illinois at Urbana-Champaign
Re: hadoop_conf_dir when running spark on yarn
Posted by Akhil Das <ak...@sigmoidanalytics.com>.
You can set HADOOP_CONF_DIR inside the spark-env.sh file
Thanks
Best Regards
On Sat, Nov 1, 2014 at 4:14 AM, ameyc <am...@gmail.com> wrote:
> How do i setup hadoop_conf_dir correctly when I'm running my spark job on
> Yarn? My Yarn environment has the correct hadoop_conf_dir settings by the
> configuration that I pull from sc.hadoopConfiguration() is incorrect.
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/hadoop-conf-dir-when-running-spark-on-yarn-tp17872.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>