You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Archit Thakur <ar...@gmail.com> on 2014/07/30 08:37:48 UTC
Logging in Spark through YARN.
Hi,
I want to manage logging of containers when I run Spark through YARN. I
checked there is a environment variable exposed to custom log4j.properties.
Setting SPARK_LOG4J_CONF to "/dir/log4j.properties" should ideally make
containers use
"/dir/log4j.properties" file for logging. This doesn't seem working.
Also, there is a log4j.properties file present in the Spark assembly and I
guess it is picking that only.
Quick Google gave me this: https://issues.apache.org/jira/browse/SPARK-2007
Is this functnality broken/not working yet?
Or I m mistaken somewhere?
Thanks.
Archit.
Re: Logging in Spark through YARN.
Posted by Vipul Pandey <vi...@gmail.com>.
Archit, Are you able to get it to work with 1.0.0?
I tried the --files suggestion from Marcelo and it just changed logging for my client and the appmaster and executors were still the same.
~Vipul
On Jul 30, 2014, at 9:59 PM, Archit Thakur <ar...@gmail.com> wrote:
> Hi Marcelo,
>
> Thanks for your quick comment. This doesn't seem working in 1.0.0. release.
>
> -Archit Thakur.
>
>
> On Thu, Jul 31, 2014 at 3:18 AM, Marcelo Vanzin <va...@cloudera.com> wrote:
> Hi Archit,
>
> Are you using spark-submit? If so, can you try adding the following to
> its command line:
>
> --files /dir/log4j.properties
>
> I tested that locally with the master branch and it works, but I don't
> have a 1.0.x install readily available for me to test at the moment.
> Handling of SPARK_LOG4J_CONF should also be fixed on master, but I
> don't remember what was its status in the 1.0 branch...
>
>
> On Tue, Jul 29, 2014 at 11:37 PM, Archit Thakur
> <ar...@gmail.com> wrote:
> > Hi,
> >
> > I want to manage logging of containers when I run Spark through YARN. I
> > checked there is a environment variable exposed to custom log4j.properties.
> >
> > Setting SPARK_LOG4J_CONF to "/dir/log4j.properties" should ideally make
> > containers use
> > "/dir/log4j.properties" file for logging. This doesn't seem working.
> >
> > Also, there is a log4j.properties file present in the Spark assembly and I
> > guess it is picking that only.
> >
> > Quick Google gave me this: https://issues.apache.org/jira/browse/SPARK-2007
> >
> > Is this functnality broken/not working yet?
> > Or I m mistaken somewhere?
> >
> > Thanks.
> > Archit.
>
>
>
> --
> Marcelo
>
Re: Logging in Spark through YARN.
Posted by Archit Thakur <ar...@gmail.com>.
Hi Marcelo,
Thanks for your quick comment. This doesn't seem working in 1.0.0. release.
-Archit Thakur.
On Thu, Jul 31, 2014 at 3:18 AM, Marcelo Vanzin <va...@cloudera.com> wrote:
> Hi Archit,
>
> Are you using spark-submit? If so, can you try adding the following to
> its command line:
>
> --files /dir/log4j.properties
>
> I tested that locally with the master branch and it works, but I don't
> have a 1.0.x install readily available for me to test at the moment.
> Handling of SPARK_LOG4J_CONF should also be fixed on master, but I
> don't remember what was its status in the 1.0 branch...
>
>
> On Tue, Jul 29, 2014 at 11:37 PM, Archit Thakur
> <ar...@gmail.com> wrote:
> > Hi,
> >
> > I want to manage logging of containers when I run Spark through YARN. I
> > checked there is a environment variable exposed to custom
> log4j.properties.
> >
> > Setting SPARK_LOG4J_CONF to "/dir/log4j.properties" should ideally make
> > containers use
> > "/dir/log4j.properties" file for logging. This doesn't seem working.
> >
> > Also, there is a log4j.properties file present in the Spark assembly and
> I
> > guess it is picking that only.
> >
> > Quick Google gave me this:
> https://issues.apache.org/jira/browse/SPARK-2007
> >
> > Is this functnality broken/not working yet?
> > Or I m mistaken somewhere?
> >
> > Thanks.
> > Archit.
>
>
>
> --
> Marcelo
>
Re: Logging in Spark through YARN.
Posted by Marcelo Vanzin <va...@cloudera.com>.
Hi Archit,
Are you using spark-submit? If so, can you try adding the following to
its command line:
--files /dir/log4j.properties
I tested that locally with the master branch and it works, but I don't
have a 1.0.x install readily available for me to test at the moment.
Handling of SPARK_LOG4J_CONF should also be fixed on master, but I
don't remember what was its status in the 1.0 branch...
On Tue, Jul 29, 2014 at 11:37 PM, Archit Thakur
<ar...@gmail.com> wrote:
> Hi,
>
> I want to manage logging of containers when I run Spark through YARN. I
> checked there is a environment variable exposed to custom log4j.properties.
>
> Setting SPARK_LOG4J_CONF to "/dir/log4j.properties" should ideally make
> containers use
> "/dir/log4j.properties" file for logging. This doesn't seem working.
>
> Also, there is a log4j.properties file present in the Spark assembly and I
> guess it is picking that only.
>
> Quick Google gave me this: https://issues.apache.org/jira/browse/SPARK-2007
>
> Is this functnality broken/not working yet?
> Or I m mistaken somewhere?
>
> Thanks.
> Archit.
--
Marcelo