You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Debasish Das <de...@gmail.com> on 2014/10/07 20:42:35 UTC

Local tests logging to log4j

Hi,

I have added some changes to ALS tests and I am re-running tests as:

mvn -Dhadoop.version=2.3.0-cdh5.1.0 -Phadoop-2.3 -Pyarn
-DwildcardSuites=org.apache.spark.mllib.recommendation.ALSSuite test

I have some INFO logs in the code which I want to see on my console. They
work fine if I add println.

I copied conf/log4j.properties.template to conf/log4j.properties

The options are:

log4j.rootCategory=INFO, console

log4j.appender.console=org.apache.log4j.ConsoleAppender

log4j.appender.console.target=System.err

I still don't see the INFO msgs on the console.

Any idea if I am setting up my log4j properties correctly ?

Thanks.

Deb

Re: Local tests logging to log4j

Posted by Debasish Das <de...@gmail.com>.
Thanks Sean...trying them out...

On Tue, Oct 7, 2014 at 12:24 PM, Sean Owen <so...@cloudera.com> wrote:

> What has worked for me is to bundle log4j.properties in the root of
> the application's .jar file, since log4j will look for it there, and
> configuring log4j will turn off Spark's default log4j configuration.
>
> I don't think conf/log4j.properties is going to do anything by itself,
> but -Dlog4j.configuration=/path/to/file should cause it read a config
> file on the file system.
>
> But for messing with a local build of Spark, just edit
> core/src/main/resources/org/apache/spark/log4j-defaults.properties and
> rebuild.
>
> Yes I think your syntax is OK; here's some of mine where I turn off a
> bunch of INFO messages:
>
> log4j.rootLogger=INFO, stdout
> log4j.appender.stdout=org.apache.log4j.ConsoleAppender
> log4j.appender.stdout.Target=System.out
> log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
> log4j.appender.stdout.layout.ConversionPattern=%d{ISO8601} %-5p %c{1}:%L
> %m%n
> log4j.logger.org.apache.hadoop=WARN
> log4j.logger.org.apache.kafka=WARN
> log4j.logger.kafka=WARN
> log4j.logger.akka=WARN
> log4j.logger.org.apache.spark=WARN
> log4j.logger.org.apache.spark.storage.BlockManager=ERROR
> log4j.logger.org.apache.zookeeper=WARN
> log4j.logger.org.eclipse.jetty=WARN
> log4j.logger.org.I0Itec.zkclient=WARN
>
> On Tue, Oct 7, 2014 at 7:42 PM, Debasish Das <de...@gmail.com>
> wrote:
> > Hi,
> >
> > I have added some changes to ALS tests and I am re-running tests as:
> >
> > mvn -Dhadoop.version=2.3.0-cdh5.1.0 -Phadoop-2.3 -Pyarn
> > -DwildcardSuites=org.apache.spark.mllib.recommendation.ALSSuite test
> >
> > I have some INFO logs in the code which I want to see on my console. They
> > work fine if I add println.
> >
> > I copied conf/log4j.properties.template to conf/log4j.properties
> >
> > The options are:
> >
> > log4j.rootCategory=INFO, console
> >
> > log4j.appender.console=org.apache.log4j.ConsoleAppender
> >
> > log4j.appender.console.target=System.err
> >
> > I still don't see the INFO msgs on the console.
> >
> > Any idea if I am setting up my log4j properties correctly ?
> >
> > Thanks.
> >
> > Deb
>

Re: Local tests logging to log4j

Posted by Sean Owen <so...@cloudera.com>.
What has worked for me is to bundle log4j.properties in the root of
the application's .jar file, since log4j will look for it there, and
configuring log4j will turn off Spark's default log4j configuration.

I don't think conf/log4j.properties is going to do anything by itself,
but -Dlog4j.configuration=/path/to/file should cause it read a config
file on the file system.

But for messing with a local build of Spark, just edit
core/src/main/resources/org/apache/spark/log4j-defaults.properties and
rebuild.

Yes I think your syntax is OK; here's some of mine where I turn off a
bunch of INFO messages:

log4j.rootLogger=INFO, stdout
log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.Target=System.out
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=%d{ISO8601} %-5p %c{1}:%L %m%n
log4j.logger.org.apache.hadoop=WARN
log4j.logger.org.apache.kafka=WARN
log4j.logger.kafka=WARN
log4j.logger.akka=WARN
log4j.logger.org.apache.spark=WARN
log4j.logger.org.apache.spark.storage.BlockManager=ERROR
log4j.logger.org.apache.zookeeper=WARN
log4j.logger.org.eclipse.jetty=WARN
log4j.logger.org.I0Itec.zkclient=WARN

On Tue, Oct 7, 2014 at 7:42 PM, Debasish Das <de...@gmail.com> wrote:
> Hi,
>
> I have added some changes to ALS tests and I am re-running tests as:
>
> mvn -Dhadoop.version=2.3.0-cdh5.1.0 -Phadoop-2.3 -Pyarn
> -DwildcardSuites=org.apache.spark.mllib.recommendation.ALSSuite test
>
> I have some INFO logs in the code which I want to see on my console. They
> work fine if I add println.
>
> I copied conf/log4j.properties.template to conf/log4j.properties
>
> The options are:
>
> log4j.rootCategory=INFO, console
>
> log4j.appender.console=org.apache.log4j.ConsoleAppender
>
> log4j.appender.console.target=System.err
>
> I still don't see the INFO msgs on the console.
>
> Any idea if I am setting up my log4j properties correctly ?
>
> Thanks.
>
> Deb

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org