You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Prabhu Joseph <pr...@gmail.com> on 2016/02/27 21:40:27 UTC
Spark log4j fully qualified class name
Hi All,
When i change the spark log4j.properties conversion pattern to know the
fully qualified class name, all the logs has the FQCN as
org.apache.spark.Logging. The actual fully qualified class name is
overwritten.
log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p
*%C:* %m%n
16/02/27 15:34:40 INFO org.apache.spark.Logging$class: Successfully started
service 'sparkDriver' on port 49482.
16/02/27 15:34:40 DEBUG org.apache.spark.Logging$class: Using serializer:
class org.apache.spark.serializer.JavaSerializer
16/02/27 15:34:40 INFO org.apache.spark.Logging$class: Registering
MapOutputTracker
16/02/27 15:34:40 INFO org.apache.spark.Logging$class: Registering
BlockManagerMaster
16/02/27 15:34:40 ERROR org.apache.spark.Logging$class: Failed to create
dir in maprfs:///var/mapr/local/prabhuFS2/spark. Ignoring this directory.
16/02/27 15:34:40 ERROR org.apache.spark.Logging$class: Failed to create
any local dir.
16/02/27 15:34:40 INFO org.apache.spark.Logging$class: Shutdown hook called
16/02/27 15:34:40 INFO org.apache.spark.Logging$class: Deleting directory
/tmp/spark-5544c349-0393-4bd0-8aab-c20331a9a1cf
Thanks,
Prabhu Joseph
Re: Spark log4j fully qualified class name
Posted by Steve Loughran <st...@hortonworks.com>.
On 27 Feb 2016, at 20:40, Prabhu Joseph <pr...@gmail.com>> wrote:
Hi All,
When i change the spark log4j.properties conversion pattern to know the fully qualified class name, all the logs has the FQCN as org.apache.spark.Logging. The actual fully qualified class name is overwritten.
log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %C: %m%n
FWIW, my test patter is:
log4j.appender.stdout.layout.ConversionPattern=%d{ISO8601} [%t] %-5p %c (%F:%M(%L)) - %m%n
Gives me things like
2016-02-23 18:54:24,645 [EventPoster] DEBUG org.apache.spark.deploy.history.yarn.YarnHistoryService (Logging.scala:logDebug(58)) - entity successfully posted
2016-02-23 18:54:24,645 [EventPoster] DEBUG org.apache.hadoop.yarn.client.api.impl.FileSystemTimelineWriter (FileSystemTimelineWriter.java:flush(286)) - Flushing cache
2016-02-23 18:54:24,646 [EventPoster] DEBUG org.apache.spark.deploy.history.yarn.YarnHistoryService (Logging.scala:logDebug(58)) - Queue stopped
2016-02-23 18:54:24,646 [EventPoster] DEBUG org.apache.spark.deploy.history.yarn.YarnHistoryService (Logging.scala:logDebug(58)) - Queue shutdown, time limit= 18:54
2016-02-23 18:54:24,646 [EventPoster] INFO org.apache.spark.deploy.history.yarn.YarnHistoryService (Logging.scala:logInfo(54)) - Stopping dequeue service, final queue size is 0; outstanding events to post count: 0
2016-02-23 18:54:24,646 [EventPoster] DEBUG org.apache.hadoop.yarn.client.api.impl.FileSystemTimelineWriter (FileSystemTimelineWriter.java:close(276)) - Closing cache
2016-02-23 18:54:24,648 [ScalaTest-main-running-IncompleteApplicationSuite] INFO org.apache.spark.deploy.history.yarn.testtools.YarnTestUtils (Logging.scala:logInfo(54)) - awaitServiceThreadStopped
Getting that thread in there is invaluable when there are lots of threads about, having the line number in lets you track down where the code is. Here you can see that there was a background thread [EventPoster], alongside the test thread itself
As ted points out, this crawls in production, as it creates an exception on every log, so that the stack trace can be examined —you would only use it in production in an emergency. For test runs though, wonderful.
Re: Spark log4j fully qualified class name
Posted by Prabhu Joseph <pr...@gmail.com>.
Ted,
That was a useful information. But i want it to use in development
cluster to learn the internals through the logs. On Using %C, actual FQCN
is not printed instead org.apache.spark.Logging overwrites. Is it a
intended change in Spark.
On Sun, Feb 28, 2016 at 3:46 AM, Ted Yu <yu...@gmail.com> wrote:
> Looking at
>
> https://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/PatternLayout.html
>
> *WARNING* Generating the caller class information is slow. Thus, use
> should be avoided unless execution speed is not an issue.
>
> On Sat, Feb 27, 2016 at 12:40 PM, Prabhu Joseph <
> prabhujose.gates@gmail.com> wrote:
>
>> Hi All,
>>
>> When i change the spark log4j.properties conversion pattern to know
>> the fully qualified class name, all the logs has the FQCN as
>> org.apache.spark.Logging. The actual fully qualified class name is
>> overwritten.
>>
>> log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p
>> *%C:* %m%n
>>
>>
>> 16/02/27 15:34:40 INFO org.apache.spark.Logging$class: Successfully
>> started service 'sparkDriver' on port 49482.
>> 16/02/27 15:34:40 DEBUG org.apache.spark.Logging$class: Using serializer:
>> class org.apache.spark.serializer.JavaSerializer
>> 16/02/27 15:34:40 INFO org.apache.spark.Logging$class: Registering
>> MapOutputTracker
>> 16/02/27 15:34:40 INFO org.apache.spark.Logging$class: Registering
>> BlockManagerMaster
>> 16/02/27 15:34:40 ERROR org.apache.spark.Logging$class: Failed to create
>> dir in maprfs:///var/mapr/local/prabhuFS2/spark. Ignoring this directory.
>> 16/02/27 15:34:40 ERROR org.apache.spark.Logging$class: Failed to create
>> any local dir.
>> 16/02/27 15:34:40 INFO org.apache.spark.Logging$class: Shutdown hook
>> called
>> 16/02/27 15:34:40 INFO org.apache.spark.Logging$class: Deleting directory
>> /tmp/spark-5544c349-0393-4bd0-8aab-c20331a9a1cf
>>
>>
>> Thanks,
>> Prabhu Joseph
>>
>>
>
Re: Spark log4j fully qualified class name
Posted by Ted Yu <yu...@gmail.com>.
Looking at
https://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/PatternLayout.html
*WARNING* Generating the caller class information is slow. Thus, use should
be avoided unless execution speed is not an issue.
On Sat, Feb 27, 2016 at 12:40 PM, Prabhu Joseph <pr...@gmail.com>
wrote:
> Hi All,
>
> When i change the spark log4j.properties conversion pattern to know
> the fully qualified class name, all the logs has the FQCN as
> org.apache.spark.Logging. The actual fully qualified class name is
> overwritten.
>
> log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p
> *%C:* %m%n
>
>
> 16/02/27 15:34:40 INFO org.apache.spark.Logging$class: Successfully
> started service 'sparkDriver' on port 49482.
> 16/02/27 15:34:40 DEBUG org.apache.spark.Logging$class: Using serializer:
> class org.apache.spark.serializer.JavaSerializer
> 16/02/27 15:34:40 INFO org.apache.spark.Logging$class: Registering
> MapOutputTracker
> 16/02/27 15:34:40 INFO org.apache.spark.Logging$class: Registering
> BlockManagerMaster
> 16/02/27 15:34:40 ERROR org.apache.spark.Logging$class: Failed to create
> dir in maprfs:///var/mapr/local/prabhuFS2/spark. Ignoring this directory.
> 16/02/27 15:34:40 ERROR org.apache.spark.Logging$class: Failed to create
> any local dir.
> 16/02/27 15:34:40 INFO org.apache.spark.Logging$class: Shutdown hook called
> 16/02/27 15:34:40 INFO org.apache.spark.Logging$class: Deleting directory
> /tmp/spark-5544c349-0393-4bd0-8aab-c20331a9a1cf
>
>
> Thanks,
> Prabhu Joseph
>
>