How to configure log directory of Tez AM??

[Rohit Kochar <mn...@gmail.com>]

I am unable to locate the tez AM logs.
From the resource manager logs i see this :

 Command to launch container container_1391074890224_0001_01_000001 : $JAVA_HOME/bin/java -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=DEBUG,CLA  -Xmx1024m -Dsun.nio.ch.bugLevel='' org.apache.tez.dag.app.DAGAppMaster --session 1><LOG_DIR>/stdout 2><LOG_DIR>/stderr


And there after i tried setting this LOG_DIR environment variable in yarn-env.sh but to no avail.

How do i configure the log dir and where is the default location of the logs??

Thanks 
Rohit

Re: How to configure log directory of Tez AM??

[Siddharth Seth <ss...@apache.org>]

Are you looking for the logs generated by the InputFormat while computing
splits ?, or while actually processing a split ?

For split generation, the logs will show up in the AM logs only if you've
setup your DAG to generate splits in the AM (they should just show up
unless there's some complicated log4j configuration). Otherwise they should
show up on the client.

For actual split processing - the logs will be part of the task container
logs, not the AM container logs.

- Sid


On Thu, Jan 30, 2014 at 4:37 AM, Rohit Kochar <mn...@gmail.com> wrote:

> Sorry my bad.
> Please ignore the previous mail.
>
> For some reason i think the container logs doesn't contain all the logs
> being emitted by the Tez AM.
> Actually in my case all the logs statement being published by tez
> libraries are present but the logs being written by my InputFormat code are
> not present hence the confusion.
>
> Later to remove the doubt of my logger being not intialized or something
> similar, i inserted certain System.out.println() statments in my
> InputFormat Code expecting these lines to be printed in the stdout file but
> no luck.
>
> I am still investigating it and will update here  if i find something
> useful for others.
>
> Thanks
> Rohit
>
>
> On 30-Jan-2014, at 3:43 pm, Rohit Kochar <mn...@gmail.com> wrote:
>
> > I am unable to locate the tez AM logs.
> > From the resource manager logs i see this :
> >
> > Command to launch container container_1391074890224_0001_01_000001 :
> $JAVA_HOME/bin/java -Dlog4j.configuration=tez-container-log4j.properties
> -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=DEBUG,CLA
>  -Xmx1024m -Dsun.nio.ch.bugLevel='' org.apache.tez.dag.app.DAGAppMaster
> --session 1><LOG_DIR>/stdout 2><LOG_DIR>/stderr
> >
> >
> > And there after i tried setting this LOG_DIR environment variable in
> yarn-env.sh but to no avail.
> >
> > How do i configure the log dir and where is the default location of the
> logs??
> >
> > Thanks
> > Rohit
>
>

Re: How to configure log directory of Tez AM??

[Rohit Kochar <mn...@gmail.com>]

Sorry my bad.
Please ignore the previous mail.

For some reason i think the container logs doesn’t contain all the logs being emitted by the Tez AM.
Actually in my case all the logs statement being published by tez libraries are present but the logs being written by my InputFormat code are not present hence the confusion.

Later to remove the doubt of my logger being not intialized or something similar, i inserted certain System.out.println() statments in my InputFormat Code expecting these lines to be printed in the stdout file but no luck.

I am still investigating it and will update here  if i find something useful for others.

Thanks
Rohit


On 30-Jan-2014, at 3:43 pm, Rohit Kochar <mn...@gmail.com> wrote:

> I am unable to locate the tez AM logs.
> From the resource manager logs i see this :
> 
> Command to launch container container_1391074890224_0001_01_000001 : $JAVA_HOME/bin/java -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=DEBUG,CLA  -Xmx1024m -Dsun.nio.ch.bugLevel='' org.apache.tez.dag.app.DAGAppMaster --session 1><LOG_DIR>/stdout 2><LOG_DIR>/stderr
> 
> 
> And there after i tried setting this LOG_DIR environment variable in yarn-env.sh but to no avail.
> 
> How do i configure the log dir and where is the default location of the logs??
> 
> Thanks 
> Rohit