How to log using log4j to local file system inside a Spark application that runs on YARN?

[Emre Sevinc <em...@gmail.com>]

Hello,

I'm building an Apache Spark Streaming application and cannot make it log
to a file on the local filesystem *when running it on YARN*. How can
achieve this?

I've set log4.properties file so that it can successfully write to a log
file in /tmp directory on the local file system (shown below partially):

 log4j.appender.file=org.apache.log4j.FileAppender
 log4j.appender.file.File=/tmp/application.log
 log4j.appender.file.append=false
 log4j.appender.file.layout=org.apache.log4j.PatternLayout
 log4j.appender.file.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss}
%-5p %c{1}:%L - %m%n

When I run my Spark application locally by using the following command:

 spark-submit --class myModule.myClass --master local[2] --deploy-mode
client myApp.jar

It runs fine and I can see that log messages are written to
/tmp/application.log on my local file system.

But when I run the same application via YARN, e.g.

 spark-submit --class myModule.myClass --master yarn-client  --name
"myModule" --total-executor-cores 1 --executor-memory 1g myApp.jar

or

 spark-submit --class myModule.myClass --master yarn-cluster  --name
"myModule" --total-executor-cores 1 --executor-memory 1g myApp.jar

I cannot see any /tmp/application.log on the local file system of the
machine that runs YARN.

What am I missing?

-- 
Emre Sevinç

Re: How to log using log4j to local file system inside a Spark application that runs on YARN?

[Emre Sevinc <em...@gmail.com>]

Marcelo and Burak,

Thank you very much for your explanations. Now I'm able to see my logs.

On Wed, Feb 11, 2015 at 7:52 PM, Marcelo Vanzin <va...@cloudera.com> wrote:

> For Yarn, you need to upload your log4j.properties separately from
> your app's jar, because of some internal issues that are too boring to
> explain here. :-)
>
> Basically:
>
>   spark-submit --master yarn --files log4j.properties blah blah blah
>
> Having to keep it outside your app jar is sub-optimal, and I think
> there's a bug filed to fix this, but so far no one has really spent
> time looking at it.
>
>
> On Wed, Feb 11, 2015 at 4:29 AM, Emre Sevinc <em...@gmail.com>
> wrote:
> > Hello,
> >
> > I'm building an Apache Spark Streaming application and cannot make it
> log to
> > a file on the local filesystem when running it on YARN. How can achieve
> > this?
> >
> > I've set log4.properties file so that it can successfully write to a log
> > file in /tmp directory on the local file system (shown below partially):
> >
> >  log4j.appender.file=org.apache.log4j.FileAppender
> >  log4j.appender.file.File=/tmp/application.log
> >  log4j.appender.file.append=false
> >  log4j.appender.file.layout=org.apache.log4j.PatternLayout
> >  log4j.appender.file.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss}
> %-5p
> > %c{1}:%L - %m%n
> >
> > When I run my Spark application locally by using the following command:
> >
> >  spark-submit --class myModule.myClass --master local[2] --deploy-mode
> > client myApp.jar
> >
> > It runs fine and I can see that log messages are written to
> > /tmp/application.log on my local file system.
> >
> > But when I run the same application via YARN, e.g.
> >
> >  spark-submit --class myModule.myClass --master yarn-client  --name
> > "myModule" --total-executor-cores 1 --executor-memory 1g myApp.jar
> >
> > or
> >
> >  spark-submit --class myModule.myClass --master yarn-cluster  --name
> > "myModule" --total-executor-cores 1 --executor-memory 1g myApp.jar
> >
> > I cannot see any /tmp/application.log on the local file system of the
> > machine that runs YARN.
> >
> > What am I missing?
> >
> >
> > --
> > Emre Sevinç
>
>
>
> --
> Marcelo
>



-- 
Emre Sevinc

Re: How to log using log4j to local file system inside a Spark application that runs on YARN?

[Marcelo Vanzin <va...@cloudera.com>]

For Yarn, you need to upload your log4j.properties separately from
your app's jar, because of some internal issues that are too boring to
explain here. :-)

Basically:

  spark-submit --master yarn --files log4j.properties blah blah blah

Having to keep it outside your app jar is sub-optimal, and I think
there's a bug filed to fix this, but so far no one has really spent
time looking at it.


On Wed, Feb 11, 2015 at 4:29 AM, Emre Sevinc <em...@gmail.com> wrote:
> Hello,
>
> I'm building an Apache Spark Streaming application and cannot make it log to
> a file on the local filesystem when running it on YARN. How can achieve
> this?
>
> I've set log4.properties file so that it can successfully write to a log
> file in /tmp directory on the local file system (shown below partially):
>
>  log4j.appender.file=org.apache.log4j.FileAppender
>  log4j.appender.file.File=/tmp/application.log
>  log4j.appender.file.append=false
>  log4j.appender.file.layout=org.apache.log4j.PatternLayout
>  log4j.appender.file.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} %-5p
> %c{1}:%L - %m%n
>
> When I run my Spark application locally by using the following command:
>
>  spark-submit --class myModule.myClass --master local[2] --deploy-mode
> client myApp.jar
>
> It runs fine and I can see that log messages are written to
> /tmp/application.log on my local file system.
>
> But when I run the same application via YARN, e.g.
>
>  spark-submit --class myModule.myClass --master yarn-client  --name
> "myModule" --total-executor-cores 1 --executor-memory 1g myApp.jar
>
> or
>
>  spark-submit --class myModule.myClass --master yarn-cluster  --name
> "myModule" --total-executor-cores 1 --executor-memory 1g myApp.jar
>
> I cannot see any /tmp/application.log on the local file system of the
> machine that runs YARN.
>
> What am I missing?
>
>
> --
> Emre Sevinç



-- 
Marcelo

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org