You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Jeff Jones <jj...@adaptivebiotech.com> on 2015/10/06 05:19:26 UTC
How can I disable logging when running local[*]?
I’ve written an application that hosts the Spark driver in-process using “local[*]”. I’ve turned off logging in my conf/log4j.properties file. I’ve also tried putting the following code prior to creating my SparkContext. These were coupled together from various posts I’ve. None of these steps have worked. I’m still getting a ton of logging to the console. Anything else I can try?
Thanks,
Jeff
private def disableLogging(): Unit = {
import org.apache.log4j.PropertyConfigurator
PropertyConfigurator.configure("conf/log4j.properties")
Logger.getRootLogger().setLevel(Level.OFF)
Logger.getLogger("org").setLevel(Level.OFF)
Logger.getLogger("akka").setLevel(Level.OFF)
}
This message (and any attachments) is intended only for the designated recipient(s). It
may contain confidential or proprietary information, or have other limitations on use as
indicated by the sender. If you are not a designated recipient, you may not review, use,
copy or distribute this message. If you received this in error, please notify the sender by
reply e-mail and delete this message.
Re: How can I disable logging when running local[*]?
Posted by Dean Wampler <de...@gmail.com>.
Try putting the following in the SBT build file, e.g., ./build.sbt:
// Fork when running, so you can ...
fork := true
// ... add a JVM option to use when forking a JVM for 'run'
javaOptions += "-Dlog4j.configuration=file:/<path-to-log4j.properties>"
dean
Dean Wampler, Ph.D.
Author: Programming Scala, 2nd Edition
<http://shop.oreilly.com/product/0636920033073.do> (O'Reilly)
Typesafe <http://typesafe.com>
@deanwampler <http://twitter.com/deanwampler>
http://polyglotprogramming.com
On Wed, Oct 7, 2015 at 10:39 AM, Alex Kozlov <al...@gmail.com> wrote:
> Hmm, clearly the parameter is not passed to the program. This should be
> an activator issue. I wonder how do you specify the other parameters, like
> driver memory, num cores, etc.? Just out of curiosity, can you run a
> program:
>
> import org.apache.spark.SparkConf
> val out=new SparkConf(true).get("spark.driver.extraJavaOptions")
>
> in your env and see what the output is?
>
> Also, make sure spark-defaults.conf is on your classpath.
>
> On Tue, Oct 6, 2015 at 11:19 AM, Jeff Jones <jj...@adaptivebiotech.com>
> wrote:
>
>> Here’s an example. I echoed JAVA_OPTS so that you can see what I’ve got.
>> Then I call ‘activator run’ in the project directory.
>>
>>
>> jjones-mac:analyzer-perf jjones$ echo $JAVA_OPTS
>>
>> -Xmx4g -Xmx4g
>> -Dlog4j.configuration=file:/Users/jjones/src/adaptive/adaptiveobjects/analyzer-perf/conf/log4j.properties
>>
>> jjones-mac:analyzer-perf jjones$ activator run
>>
>> [info] Loading project definition from
>> /Users/jjones/src/adaptive/adaptiveobjects/analyzer-perf/project
>>
>> [info] Set current project to analyzer-perf (in build
>> file:/Users/jjones/src/adaptive/adaptiveobjects/analyzer-perf/)
>>
>> [info] Running com.adaptive.analyzer.perf.AnalyzerPerf
>>
>> 11:15:24.066 [run-main-0] INFO org.apache.spark.SparkContext - Running
>> Spark version 1.4.1
>>
>> 11:15:24.150 [run-main-0] DEBUG o.a.h.m.lib.MutableMetricsFactory - field
>> org.apache.hadoop.metrics2.lib.MutableRate
>> org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginSuccess
>> with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=,
>> always=false, sampleName=Ops, type=DEFAULT, value=[Rate of successful
>> kerberos logins and latency (milliseconds)], valueName=Time)
>>
>> 11:15:24.156 [run-main-0] DEBUG o.a.h.m.lib.MutableMetricsFactory - field
>> org.apache.hadoop.metrics2.lib.MutableRate
>> org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginFailure
>> with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=,
>> always=false, sampleName=Ops, type=DEFAULT, value=[Rate of failed kerberos
>> logins and latency (milliseconds)], valueName=Time)
>>
>> As I mentioned below but repeated for completeness, I also have this in
>> my code.
>>
>> import org.apache.log4j.PropertyConfigurator
>>
>> PropertyConfigurator.configure("conf/log4j.properties")
>> Logger.getRootLogger().setLevel(Level.OFF)
>> Logger.getLogger("org").setLevel(Level.OFF)
>> Logger.getLogger("akka").setLevel(Level.OFF)
>>
>> And here’s my log4j.properties (note, I’ve also tried setting the level
>> to OFF):
>>
>> # Set everything to be logged to the console
>>
>> log4j.rootCategory=WARN
>>
>> log4j.appender.console=org.apache.log4j.ConsoleAppender
>>
>> log4j.appender.console.layout=org.apache.log4j.PatternLayout
>>
>> log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p
>> %c{1}: %m%n
>>
>>
>> # Change this to set Spark log level
>>
>> log4j.logger.org.apache.spark=WARN
>>
>>
>> # Silence akka remoting
>>
>> log4j.logger.Remoting=WARN
>>
>>
>> # Ignore messages below warning level from Jetty, because it's a bit
>> verbose
>>
>> log4j.logger.org.eclipse.jetty=WARN
>>
>>
>> spark.log.threshold=OFF
>>
>> spark.root.logger=OFF,DRFA
>>
>>
>> From: Alex Kozlov
>> Date: Tuesday, October 6, 2015 at 10:50 AM
>>
>> To: Jeff Jones
>> Cc: "user@spark.apache.org"
>> Subject: Re: How can I disable logging when running local[*]?
>>
>> Try
>>
>> JAVA_OPTS='-Dlog4j.configuration=file:/<path-to-log4j.properties>'
>>
>> Internally, this is just spark.driver.extraJavaOptions, which you should
>> be able to set in conf/spark-defaults.conf
>>
>> Can you provide more details how you invoke the driver?
>>
>> On Tue, Oct 6, 2015 at 9:48 AM, Jeff Jones <jj...@adaptivebiotech.com>
>> wrote:
>>
>>> Thanks. Any chance you know how to pass this to a Scala app that is run
>>> via TypeSafe activator?
>>>
>>> I tried putting it $JAVA_OPTS but I get:
>>>
>>> Unrecognized option: --driver-java-options
>>>
>>> Error: Could not create the Java Virtual Machine.
>>>
>>> Error: A fatal exception has occurred. Program will exit.
>>>
>>>
>>> I tried a bunch of different quoting but nothing produced a good result.
>>> I also tried passing it directly to activator using –jvm but it still
>>> produces the same results with verbose logging. Is there a way I can tell
>>> if it’s picking up my file?
>>>
>>>
>>>
>>> From: Alex Kozlov
>>> Date: Monday, October 5, 2015 at 8:34 PM
>>> To: Jeff Jones
>>> Cc: "user@spark.apache.org"
>>> Subject: Re: How can I disable logging when running local[*]?
>>>
>>> Did you try “--driver-java-options
>>> '-Dlog4j.configuration=file:/<path-to-log4j.properties>'” and setting the
>>> log4j.rootLogger=FATAL,console?
>>>
>>> On Mon, Oct 5, 2015 at 8:19 PM, Jeff Jones <jj...@adaptivebiotech.com>
>>> wrote:
>>>
>>>> I’ve written an application that hosts the Spark driver in-process
>>>> using “local[*]”. I’ve turned off logging in my conf/log4j.properties file.
>>>> I’ve also tried putting the following code prior to creating my
>>>> SparkContext. These were coupled together from various posts I’ve. None of
>>>> these steps have worked. I’m still getting a ton of logging to the console.
>>>> Anything else I can try?
>>>>
>>>> Thanks,
>>>> Jeff
>>>>
>>>> private def disableLogging(): Unit = {
>>>> import org.apache.log4j.PropertyConfigurator
>>>>
>>>> PropertyConfigurator.configure("conf/log4j.properties")
>>>> Logger.getRootLogger().setLevel(Level.OFF)
>>>> Logger.getLogger("org").setLevel(Level.OFF)
>>>> Logger.getLogger("akka").setLevel(Level.OFF)
>>>> }
>>>>
>>>>
>>>>
>>>> This message (and any attachments) is intended only for the designated
>>>> recipient(s). It
>>>> may contain confidential or proprietary information, or have other
>>>> limitations on use as
>>>> indicated by the sender. If you are not a designated recipient, you may
>>>> not review, use,
>>>> copy or distribute this message. If you received this in error, please
>>>> notify the sender by
>>>> reply e-mail and delete this message.
>>>>
>>>
>>>
>>>
>>> --
>>> Alex Kozlov
>>> (408) 507-4987
>>> (408) 830-9982 fax
>>> (650) 887-2135 efax
>>> alexvk@gmail.com
>>>
>>>
>>> This message (and any attachments) is intended only for the designated
>>> recipient(s). It
>>> may contain confidential or proprietary information, or have other
>>> limitations on use as
>>> indicated by the sender. If you are not a designated recipient, you may
>>> not review, use,
>>> copy or distribute this message. If you received this in error, please
>>> notify the sender by
>>> reply e-mail and delete this message.
>>>
>>
>>
>>
>> This message (and any attachments) is intended only for the designated
>> recipient(s). It
>> may contain confidential or proprietary information, or have other
>> limitations on use as
>> indicated by the sender. If you are not a designated recipient, you may
>> not review, use,
>> copy or distribute this message. If you received this in error, please
>> notify the sender by
>> reply e-mail and delete this message.
>>
>
>
>
> --
> Alex Kozlov
> (408) 507-4987
> (408) 830-9982 fax
> (650) 887-2135 efax
> alexvk@gmail.com
>
Re: How can I disable logging when running local[*]?
Posted by Alex Kozlov <al...@gmail.com>.
Hmm, clearly the parameter is not passed to the program. This should be an
activator issue. I wonder how do you specify the other parameters, like
driver memory, num cores, etc.? Just out of curiosity, can you run a
program:
import org.apache.spark.SparkConf
val out=new SparkConf(true).get("spark.driver.extraJavaOptions")
in your env and see what the output is?
Also, make sure spark-defaults.conf is on your classpath.
On Tue, Oct 6, 2015 at 11:19 AM, Jeff Jones <jj...@adaptivebiotech.com>
wrote:
> Here’s an example. I echoed JAVA_OPTS so that you can see what I’ve got.
> Then I call ‘activator run’ in the project directory.
>
>
> jjones-mac:analyzer-perf jjones$ echo $JAVA_OPTS
>
> -Xmx4g -Xmx4g
> -Dlog4j.configuration=file:/Users/jjones/src/adaptive/adaptiveobjects/analyzer-perf/conf/log4j.properties
>
> jjones-mac:analyzer-perf jjones$ activator run
>
> [info] Loading project definition from
> /Users/jjones/src/adaptive/adaptiveobjects/analyzer-perf/project
>
> [info] Set current project to analyzer-perf (in build
> file:/Users/jjones/src/adaptive/adaptiveobjects/analyzer-perf/)
>
> [info] Running com.adaptive.analyzer.perf.AnalyzerPerf
>
> 11:15:24.066 [run-main-0] INFO org.apache.spark.SparkContext - Running
> Spark version 1.4.1
>
> 11:15:24.150 [run-main-0] DEBUG o.a.h.m.lib.MutableMetricsFactory - field
> org.apache.hadoop.metrics2.lib.MutableRate
> org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginSuccess
> with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=,
> always=false, sampleName=Ops, type=DEFAULT, value=[Rate of successful
> kerberos logins and latency (milliseconds)], valueName=Time)
>
> 11:15:24.156 [run-main-0] DEBUG o.a.h.m.lib.MutableMetricsFactory - field
> org.apache.hadoop.metrics2.lib.MutableRate
> org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginFailure
> with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=,
> always=false, sampleName=Ops, type=DEFAULT, value=[Rate of failed kerberos
> logins and latency (milliseconds)], valueName=Time)
>
> As I mentioned below but repeated for completeness, I also have this in my
> code.
>
> import org.apache.log4j.PropertyConfigurator
>
> PropertyConfigurator.configure("conf/log4j.properties")
> Logger.getRootLogger().setLevel(Level.OFF)
> Logger.getLogger("org").setLevel(Level.OFF)
> Logger.getLogger("akka").setLevel(Level.OFF)
>
> And here’s my log4j.properties (note, I’ve also tried setting the level to
> OFF):
>
> # Set everything to be logged to the console
>
> log4j.rootCategory=WARN
>
> log4j.appender.console=org.apache.log4j.ConsoleAppender
>
> log4j.appender.console.layout=org.apache.log4j.PatternLayout
>
> log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p
> %c{1}: %m%n
>
>
> # Change this to set Spark log level
>
> log4j.logger.org.apache.spark=WARN
>
>
> # Silence akka remoting
>
> log4j.logger.Remoting=WARN
>
>
> # Ignore messages below warning level from Jetty, because it's a bit
> verbose
>
> log4j.logger.org.eclipse.jetty=WARN
>
>
> spark.log.threshold=OFF
>
> spark.root.logger=OFF,DRFA
>
>
> From: Alex Kozlov
> Date: Tuesday, October 6, 2015 at 10:50 AM
>
> To: Jeff Jones
> Cc: "user@spark.apache.org"
> Subject: Re: How can I disable logging when running local[*]?
>
> Try
>
> JAVA_OPTS='-Dlog4j.configuration=file:/<path-to-log4j.properties>'
>
> Internally, this is just spark.driver.extraJavaOptions, which you should
> be able to set in conf/spark-defaults.conf
>
> Can you provide more details how you invoke the driver?
>
> On Tue, Oct 6, 2015 at 9:48 AM, Jeff Jones <jj...@adaptivebiotech.com>
> wrote:
>
>> Thanks. Any chance you know how to pass this to a Scala app that is run
>> via TypeSafe activator?
>>
>> I tried putting it $JAVA_OPTS but I get:
>>
>> Unrecognized option: --driver-java-options
>>
>> Error: Could not create the Java Virtual Machine.
>>
>> Error: A fatal exception has occurred. Program will exit.
>>
>>
>> I tried a bunch of different quoting but nothing produced a good result.
>> I also tried passing it directly to activator using –jvm but it still
>> produces the same results with verbose logging. Is there a way I can tell
>> if it’s picking up my file?
>>
>>
>>
>> From: Alex Kozlov
>> Date: Monday, October 5, 2015 at 8:34 PM
>> To: Jeff Jones
>> Cc: "user@spark.apache.org"
>> Subject: Re: How can I disable logging when running local[*]?
>>
>> Did you try “--driver-java-options
>> '-Dlog4j.configuration=file:/<path-to-log4j.properties>'” and setting the
>> log4j.rootLogger=FATAL,console?
>>
>> On Mon, Oct 5, 2015 at 8:19 PM, Jeff Jones <jj...@adaptivebiotech.com>
>> wrote:
>>
>>> I’ve written an application that hosts the Spark driver in-process using
>>> “local[*]”. I’ve turned off logging in my conf/log4j.properties file. I’ve
>>> also tried putting the following code prior to creating my SparkContext.
>>> These were coupled together from various posts I’ve. None of these steps
>>> have worked. I’m still getting a ton of logging to the console. Anything
>>> else I can try?
>>>
>>> Thanks,
>>> Jeff
>>>
>>> private def disableLogging(): Unit = {
>>> import org.apache.log4j.PropertyConfigurator
>>>
>>> PropertyConfigurator.configure("conf/log4j.properties")
>>> Logger.getRootLogger().setLevel(Level.OFF)
>>> Logger.getLogger("org").setLevel(Level.OFF)
>>> Logger.getLogger("akka").setLevel(Level.OFF)
>>> }
>>>
>>>
>>>
>>> This message (and any attachments) is intended only for the designated
>>> recipient(s). It
>>> may contain confidential or proprietary information, or have other
>>> limitations on use as
>>> indicated by the sender. If you are not a designated recipient, you may
>>> not review, use,
>>> copy or distribute this message. If you received this in error, please
>>> notify the sender by
>>> reply e-mail and delete this message.
>>>
>>
>>
>>
>> --
>> Alex Kozlov
>> (408) 507-4987
>> (408) 830-9982 fax
>> (650) 887-2135 efax
>> alexvk@gmail.com
>>
>>
>> This message (and any attachments) is intended only for the designated
>> recipient(s). It
>> may contain confidential or proprietary information, or have other
>> limitations on use as
>> indicated by the sender. If you are not a designated recipient, you may
>> not review, use,
>> copy or distribute this message. If you received this in error, please
>> notify the sender by
>> reply e-mail and delete this message.
>>
>
>
>
> This message (and any attachments) is intended only for the designated
> recipient(s). It
> may contain confidential or proprietary information, or have other
> limitations on use as
> indicated by the sender. If you are not a designated recipient, you may
> not review, use,
> copy or distribute this message. If you received this in error, please
> notify the sender by
> reply e-mail and delete this message.
>
--
Alex Kozlov
(408) 507-4987
(408) 830-9982 fax
(650) 887-2135 efax
alexvk@gmail.com
Re: How can I disable logging when running local[*]?
Posted by Jeff Jones <jj...@adaptivebiotech.com>.
Here’s an example. I echoed JAVA_OPTS so that you can see what I’ve got. Then I call ‘activator run’ in the project directory.
jjones-mac:analyzer-perf jjones$ echo $JAVA_OPTS
-Xmx4g -Xmx4g -Dlog4j.configuration=file:/Users/jjones/src/adaptive/adaptiveobjects/analyzer-perf/conf/log4j.properties
jjones-mac:analyzer-perf jjones$ activator run
[info] Loading project definition from /Users/jjones/src/adaptive/adaptiveobjects/analyzer-perf/project
[info] Set current project to analyzer-perf (in build file:/Users/jjones/src/adaptive/adaptiveobjects/analyzer-perf/)
[info] Running com.adaptive.analyzer.perf.AnalyzerPerf
11:15:24.066 [run-main-0] INFO org.apache.spark.SparkContext - Running Spark version 1.4.1
11:15:24.150 [run-main-0] DEBUG o.a.h.m.lib.MutableMetricsFactory - field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginSuccess with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, always=false, sampleName=Ops, type=DEFAULT, value=[Rate of successful kerberos logins and latency (milliseconds)], valueName=Time)
11:15:24.156 [run-main-0] DEBUG o.a.h.m.lib.MutableMetricsFactory - field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginFailure with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, always=false, sampleName=Ops, type=DEFAULT, value=[Rate of failed kerberos logins and latency (milliseconds)], valueName=Time)
As I mentioned below but repeated for completeness, I also have this in my code.
import org.apache.log4j.PropertyConfigurator
PropertyConfigurator.configure("conf/log4j.properties")
Logger.getRootLogger().setLevel(Level.OFF)
Logger.getLogger("org").setLevel(Level.OFF)
Logger.getLogger("akka").setLevel(Level.OFF)
And here’s my log4j.properties (note, I’ve also tried setting the level to OFF):
# Set everything to be logged to the console
log4j.rootCategory=WARN
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n
# Change this to set Spark log level
log4j.logger.org.apache.spark=WARN
# Silence akka remoting
log4j.logger.Remoting=WARN
# Ignore messages below warning level from Jetty, because it's a bit verbose
log4j.logger.org.eclipse.jetty=WARN
spark.log.threshold=OFF
spark.root.logger=OFF,DRFA
From: Alex Kozlov
Date: Tuesday, October 6, 2015 at 10:50 AM
To: Jeff Jones
Cc: "user@spark.apache.org<ma...@spark.apache.org>"
Subject: Re: How can I disable logging when running local[*]?
Try
JAVA_OPTS='-Dlog4j.configuration=file:/<path-to-log4j.properties>'
Internally, this is just spark.driver.extraJavaOptions, which you should be able to set in conf/spark-defaults.conf
Can you provide more details how you invoke the driver?
On Tue, Oct 6, 2015 at 9:48 AM, Jeff Jones <jj...@adaptivebiotech.com>> wrote:
Thanks. Any chance you know how to pass this to a Scala app that is run via TypeSafe activator?
I tried putting it $JAVA_OPTS but I get:
Unrecognized option: --driver-java-options
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
I tried a bunch of different quoting but nothing produced a good result. I also tried passing it directly to activator using –jvm but it still produces the same results with verbose logging. Is there a way I can tell if it’s picking up my file?
From: Alex Kozlov
Date: Monday, October 5, 2015 at 8:34 PM
To: Jeff Jones
Cc: "user@spark.apache.org<ma...@spark.apache.org>"
Subject: Re: How can I disable logging when running local[*]?
Did you try “--driver-java-options '-Dlog4j.configuration=file:/<path-to-log4j.properties>'” and setting the log4j.rootLogger=FATAL,console?
On Mon, Oct 5, 2015 at 8:19 PM, Jeff Jones <jj...@adaptivebiotech.com>> wrote:
I’ve written an application that hosts the Spark driver in-process using “local[*]”. I’ve turned off logging in my conf/log4j.properties file. I’ve also tried putting the following code prior to creating my SparkContext. These were coupled together from various posts I’ve. None of these steps have worked. I’m still getting a ton of logging to the console. Anything else I can try?
Thanks,
Jeff
private def disableLogging(): Unit = {
import org.apache.log4j.PropertyConfigurator
PropertyConfigurator.configure("conf/log4j.properties")
Logger.getRootLogger().setLevel(Level.OFF)
Logger.getLogger("org").setLevel(Level.OFF)
Logger.getLogger("akka").setLevel(Level.OFF)
}
This message (and any attachments) is intended only for the designated recipient(s). It
may contain confidential or proprietary information, or have other limitations on use as
indicated by the sender. If you are not a designated recipient, you may not review, use,
copy or distribute this message. If you received this in error, please notify the sender by
reply e-mail and delete this message.
--
Alex Kozlov
(408) 507-4987<tel:%28408%29%20507-4987>
(408) 830-9982<tel:%28408%29%20830-9982> fax
(650) 887-2135<tel:%28650%29%20887-2135> efax
alexvk@gmail.com<ma...@gmail.com>
This message (and any attachments) is intended only for the designated recipient(s). It
may contain confidential or proprietary information, or have other limitations on use as
indicated by the sender. If you are not a designated recipient, you may not review, use,
copy or distribute this message. If you received this in error, please notify the sender by
reply e-mail and delete this message.
This message (and any attachments) is intended only for the designated recipient(s). It
may contain confidential or proprietary information, or have other limitations on use as
indicated by the sender. If you are not a designated recipient, you may not review, use,
copy or distribute this message. If you received this in error, please notify the sender by
reply e-mail and delete this message.
Re: How can I disable logging when running local[*]?
Posted by Alexander Pivovarov <ap...@gmail.com>.
The easiest way to control logging in spark shell is to run Logger.setLevel
commands at the beginning of your program
e.g.
org.apache.log4j.Logger.getLogger("com.amazon").setLevel(org.apache.log4j.Level.WARN)
org.apache.log4j.Logger.getLogger("com.amazonaws").setLevel(org.apache.log4j.Level.WARN)
org.apache.log4j.Logger.getLogger("amazon.emr").setLevel(org.apache.log4j.Level.WARN)
org.apache.log4j.Logger.getLogger("akka").setLevel(org.apache.log4j.Level.WARN)
On Tue, Oct 6, 2015 at 10:50 AM, Alex Kozlov <al...@gmail.com> wrote:
> Try
>
> JAVA_OPTS='-Dlog4j.configuration=file:/<path-to-log4j.properties>'
>
> Internally, this is just spark.driver.extraJavaOptions, which you should
> be able to set in conf/spark-defaults.conf
>
> Can you provide more details how you invoke the driver?
>
> On Tue, Oct 6, 2015 at 9:48 AM, Jeff Jones <jj...@adaptivebiotech.com>
> wrote:
>
>> Thanks. Any chance you know how to pass this to a Scala app that is run
>> via TypeSafe activator?
>>
>> I tried putting it $JAVA_OPTS but I get:
>>
>> Unrecognized option: --driver-java-options
>>
>> Error: Could not create the Java Virtual Machine.
>>
>> Error: A fatal exception has occurred. Program will exit.
>>
>>
>> I tried a bunch of different quoting but nothing produced a good result.
>> I also tried passing it directly to activator using –jvm but it still
>> produces the same results with verbose logging. Is there a way I can tell
>> if it’s picking up my file?
>>
>>
>>
>> From: Alex Kozlov
>> Date: Monday, October 5, 2015 at 8:34 PM
>> To: Jeff Jones
>> Cc: "user@spark.apache.org"
>> Subject: Re: How can I disable logging when running local[*]?
>>
>> Did you try “--driver-java-options
>> '-Dlog4j.configuration=file:/<path-to-log4j.properties>'” and setting the
>> log4j.rootLogger=FATAL,console?
>>
>> On Mon, Oct 5, 2015 at 8:19 PM, Jeff Jones <jj...@adaptivebiotech.com>
>> wrote:
>>
>>> I’ve written an application that hosts the Spark driver in-process using
>>> “local[*]”. I’ve turned off logging in my conf/log4j.properties file. I’ve
>>> also tried putting the following code prior to creating my SparkContext.
>>> These were coupled together from various posts I’ve. None of these steps
>>> have worked. I’m still getting a ton of logging to the console. Anything
>>> else I can try?
>>>
>>> Thanks,
>>> Jeff
>>>
>>> private def disableLogging(): Unit = {
>>> import org.apache.log4j.PropertyConfigurator
>>>
>>> PropertyConfigurator.configure("conf/log4j.properties")
>>> Logger.getRootLogger().setLevel(Level.OFF)
>>> Logger.getLogger("org").setLevel(Level.OFF)
>>> Logger.getLogger("akka").setLevel(Level.OFF)
>>> }
>>>
>>>
>>>
>>> This message (and any attachments) is intended only for the designated
>>> recipient(s). It
>>> may contain confidential or proprietary information, or have other
>>> limitations on use as
>>> indicated by the sender. If you are not a designated recipient, you may
>>> not review, use,
>>> copy or distribute this message. If you received this in error, please
>>> notify the sender by
>>> reply e-mail and delete this message.
>>>
>>
>>
>>
>> --
>> Alex Kozlov
>> (408) 507-4987
>> (408) 830-9982 fax
>> (650) 887-2135 efax
>> alexvk@gmail.com
>>
>>
>> This message (and any attachments) is intended only for the designated
>> recipient(s). It
>> may contain confidential or proprietary information, or have other
>> limitations on use as
>> indicated by the sender. If you are not a designated recipient, you may
>> not review, use,
>> copy or distribute this message. If you received this in error, please
>> notify the sender by
>> reply e-mail and delete this message.
>>
>
>
Re: How can I disable logging when running local[*]?
Posted by Alex Kozlov <al...@gmail.com>.
Try
JAVA_OPTS='-Dlog4j.configuration=file:/<path-to-log4j.properties>'
Internally, this is just spark.driver.extraJavaOptions, which you should be
able to set in conf/spark-defaults.conf
Can you provide more details how you invoke the driver?
On Tue, Oct 6, 2015 at 9:48 AM, Jeff Jones <jj...@adaptivebiotech.com>
wrote:
> Thanks. Any chance you know how to pass this to a Scala app that is run
> via TypeSafe activator?
>
> I tried putting it $JAVA_OPTS but I get:
>
> Unrecognized option: --driver-java-options
>
> Error: Could not create the Java Virtual Machine.
>
> Error: A fatal exception has occurred. Program will exit.
>
>
> I tried a bunch of different quoting but nothing produced a good result. I
> also tried passing it directly to activator using –jvm but it still
> produces the same results with verbose logging. Is there a way I can tell
> if it’s picking up my file?
>
>
>
> From: Alex Kozlov
> Date: Monday, October 5, 2015 at 8:34 PM
> To: Jeff Jones
> Cc: "user@spark.apache.org"
> Subject: Re: How can I disable logging when running local[*]?
>
> Did you try “--driver-java-options
> '-Dlog4j.configuration=file:/<path-to-log4j.properties>'” and setting the
> log4j.rootLogger=FATAL,console?
>
> On Mon, Oct 5, 2015 at 8:19 PM, Jeff Jones <jj...@adaptivebiotech.com>
> wrote:
>
>> I’ve written an application that hosts the Spark driver in-process using
>> “local[*]”. I’ve turned off logging in my conf/log4j.properties file. I’ve
>> also tried putting the following code prior to creating my SparkContext.
>> These were coupled together from various posts I’ve. None of these steps
>> have worked. I’m still getting a ton of logging to the console. Anything
>> else I can try?
>>
>> Thanks,
>> Jeff
>>
>> private def disableLogging(): Unit = {
>> import org.apache.log4j.PropertyConfigurator
>>
>> PropertyConfigurator.configure("conf/log4j.properties")
>> Logger.getRootLogger().setLevel(Level.OFF)
>> Logger.getLogger("org").setLevel(Level.OFF)
>> Logger.getLogger("akka").setLevel(Level.OFF)
>> }
>>
>>
>>
>> This message (and any attachments) is intended only for the designated
>> recipient(s). It
>> may contain confidential or proprietary information, or have other
>> limitations on use as
>> indicated by the sender. If you are not a designated recipient, you may
>> not review, use,
>> copy or distribute this message. If you received this in error, please
>> notify the sender by
>> reply e-mail and delete this message.
>>
>
>
>
> --
> Alex Kozlov
> (408) 507-4987
> (408) 830-9982 fax
> (650) 887-2135 efax
> alexvk@gmail.com
>
>
> This message (and any attachments) is intended only for the designated
> recipient(s). It
> may contain confidential or proprietary information, or have other
> limitations on use as
> indicated by the sender. If you are not a designated recipient, you may
> not review, use,
> copy or distribute this message. If you received this in error, please
> notify the sender by
> reply e-mail and delete this message.
>
Re: How can I disable logging when running local[*]?
Posted by Jeff Jones <jj...@adaptivebiotech.com>.
Thanks. Any chance you know how to pass this to a Scala app that is run via TypeSafe activator?
I tried putting it $JAVA_OPTS but I get:
Unrecognized option: --driver-java-options
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
I tried a bunch of different quoting but nothing produced a good result. I also tried passing it directly to activator using –jvm but it still produces the same results with verbose logging. Is there a way I can tell if it’s picking up my file?
From: Alex Kozlov
Date: Monday, October 5, 2015 at 8:34 PM
To: Jeff Jones
Cc: "user@spark.apache.org<ma...@spark.apache.org>"
Subject: Re: How can I disable logging when running local[*]?
Did you try “--driver-java-options '-Dlog4j.configuration=file:/<path-to-log4j.properties>'” and setting the log4j.rootLogger=FATAL,console?
On Mon, Oct 5, 2015 at 8:19 PM, Jeff Jones <jj...@adaptivebiotech.com>> wrote:
I’ve written an application that hosts the Spark driver in-process using “local[*]”. I’ve turned off logging in my conf/log4j.properties file. I’ve also tried putting the following code prior to creating my SparkContext. These were coupled together from various posts I’ve. None of these steps have worked. I’m still getting a ton of logging to the console. Anything else I can try?
Thanks,
Jeff
private def disableLogging(): Unit = {
import org.apache.log4j.PropertyConfigurator
PropertyConfigurator.configure("conf/log4j.properties")
Logger.getRootLogger().setLevel(Level.OFF)
Logger.getLogger("org").setLevel(Level.OFF)
Logger.getLogger("akka").setLevel(Level.OFF)
}
This message (and any attachments) is intended only for the designated recipient(s). It
may contain confidential or proprietary information, or have other limitations on use as
indicated by the sender. If you are not a designated recipient, you may not review, use,
copy or distribute this message. If you received this in error, please notify the sender by
reply e-mail and delete this message.
--
Alex Kozlov
(408) 507-4987
(408) 830-9982 fax
(650) 887-2135 efax
alexvk@gmail.com<ma...@gmail.com>
This message (and any attachments) is intended only for the designated recipient(s). It
may contain confidential or proprietary information, or have other limitations on use as
indicated by the sender. If you are not a designated recipient, you may not review, use,
copy or distribute this message. If you received this in error, please notify the sender by
reply e-mail and delete this message.
Re: How can I disable logging when running local[*]?
Posted by Alex Kozlov <al...@gmail.com>.
Did you try “--driver-java-options
'-Dlog4j.configuration=file:/<path-to-log4j.properties>'” and setting the
log4j.rootLogger=FATAL,console?
On Mon, Oct 5, 2015 at 8:19 PM, Jeff Jones <jj...@adaptivebiotech.com>
wrote:
> I’ve written an application that hosts the Spark driver in-process using
> “local[*]”. I’ve turned off logging in my conf/log4j.properties file. I’ve
> also tried putting the following code prior to creating my SparkContext.
> These were coupled together from various posts I’ve. None of these steps
> have worked. I’m still getting a ton of logging to the console. Anything
> else I can try?
>
> Thanks,
> Jeff
>
> private def disableLogging(): Unit = {
> import org.apache.log4j.PropertyConfigurator
>
> PropertyConfigurator.configure("conf/log4j.properties")
> Logger.getRootLogger().setLevel(Level.OFF)
> Logger.getLogger("org").setLevel(Level.OFF)
> Logger.getLogger("akka").setLevel(Level.OFF)
> }
>
>
>
> This message (and any attachments) is intended only for the designated
> recipient(s). It
> may contain confidential or proprietary information, or have other
> limitations on use as
> indicated by the sender. If you are not a designated recipient, you may
> not review, use,
> copy or distribute this message. If you received this in error, please
> notify the sender by
> reply e-mail and delete this message.
>
--
Alex Kozlov
(408) 507-4987
(408) 830-9982 fax
(650) 887-2135 efax
alexvk@gmail.com