You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@storm.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2015/11/17 19:33:10 UTC
[jira] [Commented] (STORM-1215) Use Async Loggers to avoid locking
and logging overhead
[ https://issues.apache.org/jira/browse/STORM-1215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15009200#comment-15009200 ]
ASF GitHub Bot commented on STORM-1215:
---------------------------------------
GitHub user kishorvpatil opened a pull request:
https://github.com/apache/storm/pull/888
[STORM-1215] Use Asynchronous logger for improving performance.
The changes to loggers for all daemons and workers.
- Turn off immediateFlush.
- Enable async logging.
- Turn off Immediate Fail for log events on syslog logger to avoid making blocking call without timeout.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/kishorvpatil/incubator-storm STORM-1215
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/storm/pull/888.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #888
----
commit 601efa5cad6604bf922f7f1ef843e3f8ae9bf1f2
Author: Kishor Patil <kp...@yahoo-inc.com>
Date: 2015-11-17T16:14:56Z
All daemons should Async Logging to avoid locking issues
commit 1660da0e36b6f5647bc21f9813f3eca1c3c53afa
Author: Kishor Patil <kp...@yahoo-inc.com>
Date: 2015-11-17T17:23:38Z
Make worker log appenders async for performance
----
> Use Async Loggers to avoid locking and logging overhead
> --------------------------------------------------------
>
> Key: STORM-1215
> URL: https://issues.apache.org/jira/browse/STORM-1215
> Project: Apache Storm
> Issue Type: Improvement
> Components: storm-core
> Reporter: Kishor Patil
> Assignee: Kishor Patil
>
> The loggers are synchronous with immediateFlush to disk, making some of the daemons slow down. In some other cases, nimbus is slow too with submit-lock.
> Making loggers asynchronous with no necessity to write to disk on every logger event would improve cpu resource usage for logging.
> {code}
> "pool-7-thread-986" #1025 prio=5 os_prio=0 tid=0x00007f0f9628c800 nid=0x1b84 runnable [0x00007f0f0fa2a000]
> java.lang.Thread.State: RUNNABLE
> at java.io.FileOutputStream.writeBytes(Native Method)
> at java.io.FileOutputStream.write(FileOutputStream.java:326)
> at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
> at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
> - locked <0x00000003c00ae520> (a java.io.BufferedOutputStream)
> at java.io.PrintStream.write(PrintStream.java:482)
> - locked <0x00000003c00ae500> (a java.io.PrintStream)
> at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
> at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:291)
> at sun.nio.cs.StreamEncoder.flushBuffer(StreamEncoder.java:104)
> - locked <0x00000003c00ae640> (a java.io.OutputStreamWriter)
> at java.io.OutputStreamWriter.flushBuffer(OutputStreamWriter.java:185)
> at java.io.PrintStream.write(PrintStream.java:527)
> - locked <0x00000003c00ae500> (a java.io.PrintStream)
> at java.io.PrintStream.print(PrintStream.java:669)
> at java.io.PrintStream.println(PrintStream.java:806)
> - locked <0x00000003c00ae500> (a java.io.PrintStream)
> at org.apache.logging.log4j.status.StatusConsoleListener.log(StatusConsoleListener.java:81)
> at org.apache.logging.log4j.status.StatusLogger.logMessage(StatusLogger.java:218)
> at org.apache.logging.log4j.spi.AbstractLogger.logMessage(AbstractLogger.java:727)
> at org.apache.logging.log4j.spi.AbstractLogger.logIfEnabled(AbstractLogger.java:716)
> at org.apache.logging.log4j.spi.AbstractLogger.error(AbstractLogger.java:344)
> at org.apache.logging.log4j.core.appender.DefaultErrorHandler.error(DefaultErrorHandler.java:59)
> at org.apache.logging.log4j.core.appender.AbstractAppender.error(AbstractAppender.java:86)
> at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.append(AbstractOutputStreamAppender.java:116)
> at org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:99)
> at org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(LoggerConfig.java:430)
> at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:409)
> at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:367)
> at org.apache.logging.log4j.core.Logger.logMessage(Logger.java:112)
> at org.apache.logging.log4j.spi.AbstractLogger.logMessage(AbstractLogger.java:727)
> at org.apache.logging.log4j.spi.AbstractLogger.logIfEnabled(AbstractLogger.java:716)
> at org.apache.logging.slf4j.Log4jLogger.info(Log4jLogger.java:198)
> at clojure.tools.logging$eval1$fn__7.invoke(NO_SOURCE_FILE:0)
> at clojure.tools.logging.impl$fn__28$G__8__39.invoke(impl.clj:16)
> at clojure.tools.logging$log_STAR_.invoke(logging.clj:59)
> at backtype.storm.daemon.nimbus$mk_assignments.doInvoke(nimbus.clj:781)
> at clojure.lang.RestFn.invoke(RestFn.java:410)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)