You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@brooklyn.apache.org by sj...@apache.org on 2014/12/03 17:39:54 UTC
[1/2] incubator-brooklyn git commit: Logging brooklyn-persister
thread Jclouds messages in a separate file
Repository: incubator-brooklyn
Updated Branches:
refs/heads/master 5de642ecf -> 6db032864
Logging brooklyn-persister thread Jclouds messages in a separate file
When a server with persistence to an object store enabled manages lots of
entities the Jclouds log is overwhelmed by messages logged from the
brooklyn-persister thread (in one case logging at a rate of ~ half a
million lines every four minutes).
The appender added in this commit alleviates - a little! - this logging
overload.
Anecdotal measurements of the ratio between the persister log and jclouds
log sizes, with EmptySoftwareProcess entities:
0 entities: persister log 0.01 times size of jclouds log
1 entity: 0.36
5 entities: ~1.13
10 entities: 1.75
50 entities: 8.6
Project: http://git-wip-us.apache.org/repos/asf/incubator-brooklyn/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-brooklyn/commit/501412e1
Tree: http://git-wip-us.apache.org/repos/asf/incubator-brooklyn/tree/501412e1
Diff: http://git-wip-us.apache.org/repos/asf/incubator-brooklyn/diff/501412e1
Branch: refs/heads/master
Commit: 501412e1db726c22dec83cd2927177296e918879
Parents: 4638c77
Author: Sam Corbett <sa...@cloudsoftcorp.com>
Authored: Fri Nov 28 18:02:37 2014 +0000
Committer: Sam Corbett <sa...@cloudsoftcorp.com>
Committed: Fri Nov 28 18:02:37 2014 +0000
----------------------------------------------------------------------
.../BrooklynMementoPersisterToObjectStore.java | 1 +
docs/dev/tips/logging.md | 4 ++
usage/logback-includes/pom.xml | 11 ++++
.../JcloudsPersistenceThreadDiscriminator.java | 65 ++++++++++++++++++++
.../brooklyn/logback-appender-jclouds.xml | 39 +++++++-----
.../brooklyn/logback-logger-debug-jclouds.xml | 22 ++++---
6 files changed, 119 insertions(+), 23 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-brooklyn/blob/501412e1/core/src/main/java/brooklyn/entity/rebind/persister/BrooklynMementoPersisterToObjectStore.java
----------------------------------------------------------------------
diff --git a/core/src/main/java/brooklyn/entity/rebind/persister/BrooklynMementoPersisterToObjectStore.java b/core/src/main/java/brooklyn/entity/rebind/persister/BrooklynMementoPersisterToObjectStore.java
index 98fff36..c94a588 100644
--- a/core/src/main/java/brooklyn/entity/rebind/persister/BrooklynMementoPersisterToObjectStore.java
+++ b/core/src/main/java/brooklyn/entity/rebind/persister/BrooklynMementoPersisterToObjectStore.java
@@ -132,6 +132,7 @@ public class BrooklynMementoPersisterToObjectStore implements BrooklynMementoPer
executor = MoreExecutors.listeningDecorator(Executors.newFixedThreadPool(maxThreadPoolSize, new ThreadFactory() {
@Override public Thread newThread(Runnable r) {
+ // Note: Thread name referenced in logback-includes' ThreadNameDiscriminator
return new Thread(r, "brooklyn-persister");
}}));
}
http://git-wip-us.apache.org/repos/asf/incubator-brooklyn/blob/501412e1/docs/dev/tips/logging.md
----------------------------------------------------------------------
diff --git a/docs/dev/tips/logging.md b/docs/dev/tips/logging.md
index 927fd0b..03db9d1 100644
--- a/docs/dev/tips/logging.md
+++ b/docs/dev/tips/logging.md
@@ -66,6 +66,10 @@ For example:
configuration file, except it should be wrapped in ``<included>`` XML tags rather
than ``<configuration>`` XML tags (because it is included from the ``logback.xml``
which comes with ``brooklyn-logback-xml``.)
+* To redirect all jclouds logging to a separate file include ``brooklyn/logback-logger-debug-jclouds.xml``.
+ This redirects all logging from ``org.jclouds`` and ``jclouds`` to one of two files: anything
+ logged from Brooklyn's persistence thread will end up in a `persistence.log`, everything else
+ will end up in ``jclouds.log``.
You should **not** supply your own ``logback.xml`` if you are using ``brooklyn-logback-xml``.
If you do, logback will detect multiple files with that name and will scream at you.
http://git-wip-us.apache.org/repos/asf/incubator-brooklyn/blob/501412e1/usage/logback-includes/pom.xml
----------------------------------------------------------------------
diff --git a/usage/logback-includes/pom.xml b/usage/logback-includes/pom.xml
index aca98e2..eea13c3 100644
--- a/usage/logback-includes/pom.xml
+++ b/usage/logback-includes/pom.xml
@@ -38,4 +38,15 @@
<relativePath>../../pom.xml</relativePath>
</parent>
+ <dependencies>
+ <dependency>
+ <groupId>ch.qos.logback</groupId>
+ <artifactId>logback-classic</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>ch.qos.logback</groupId>
+ <artifactId>logback-core</artifactId>
+ </dependency>
+ </dependencies>
+
</project>
http://git-wip-us.apache.org/repos/asf/incubator-brooklyn/blob/501412e1/usage/logback-includes/src/main/java/brooklyn/logging/JcloudsPersistenceThreadDiscriminator.java
----------------------------------------------------------------------
diff --git a/usage/logback-includes/src/main/java/brooklyn/logging/JcloudsPersistenceThreadDiscriminator.java b/usage/logback-includes/src/main/java/brooklyn/logging/JcloudsPersistenceThreadDiscriminator.java
new file mode 100644
index 0000000..a085177
--- /dev/null
+++ b/usage/logback-includes/src/main/java/brooklyn/logging/JcloudsPersistenceThreadDiscriminator.java
@@ -0,0 +1,65 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package brooklyn.logging;
+
+import ch.qos.logback.classic.spi.ILoggingEvent;
+import ch.qos.logback.core.sift.Discriminator;
+
+/**
+ * Discriminates logging events per whether the current thread is named "brooklyn-persister" or not.
+ * <p>
+ * Use a {@link ch.qos.logback.classic.sift.SiftingAppender SiftingAppender} and refer to the
+ * <code>jcloudsPersistSwitch</code> property. The property's value will be either "jclouds-persister",
+ * for messages logged from the persistence thread, and "jclouds" otherwise.
+ */
+public class JcloudsPersistenceThreadDiscriminator implements Discriminator<ILoggingEvent> {
+
+ private static final String PERSISTENCE_THREAD = "brooklyn-persister";
+ private static final String KEY = "jcloudsPersistSwitch";
+
+ private boolean isStarted;
+
+ @Override
+ public String getDiscriminatingValue(ILoggingEvent o) {
+ return Thread.currentThread().getName().startsWith(PERSISTENCE_THREAD)
+ ? "persistence"
+ : "jclouds";
+ }
+
+ @Override
+ public String getKey() {
+ return KEY;
+ }
+
+ @Override
+ public void start() {
+ isStarted = true;
+ }
+
+ @Override
+ public void stop() {
+ isStarted = false;
+ }
+
+ @Override
+ public boolean isStarted() {
+ return isStarted;
+ }
+}
+
http://git-wip-us.apache.org/repos/asf/incubator-brooklyn/blob/501412e1/usage/logback-includes/src/main/resources/brooklyn/logback-appender-jclouds.xml
----------------------------------------------------------------------
diff --git a/usage/logback-includes/src/main/resources/brooklyn/logback-appender-jclouds.xml b/usage/logback-includes/src/main/resources/brooklyn/logback-appender-jclouds.xml
index 7a54304..5a76665 100644
--- a/usage/logback-includes/src/main/resources/brooklyn/logback-appender-jclouds.xml
+++ b/usage/logback-includes/src/main/resources/brooklyn/logback-appender-jclouds.xml
@@ -19,24 +19,31 @@
-->
<included>
- <!-- create a separate jclouds log file -->
- <appender name="JCLOUDS-FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
- <file>${logging.dir:-./}${logging.basename:-brooklyn}.jclouds.log</file>
- <append>true</append>
- <encoder>
- <pattern>%d %-5level %logger{30} [%thread{15}]: %msg%n</pattern>
- </encoder>
+ <!--
+ Create one rotating log file for each discriminator given by JcloudsPersistenceThreadDiscriminator.
+ -->
+ <appender name="JCLOUDS-SIFT" class="ch.qos.logback.classic.sift.SiftingAppender">
+ <discriminator class="brooklyn.logging.JcloudsPersistenceThreadDiscriminator" />
+ <sift>
+ <appender name="JCLOUDS-FILE-${jcloudsPersistSwitch}" class="ch.qos.logback.core.rolling.RollingFileAppender">
+ <file>${logging.dir:-./}${logging.basename:-brooklyn}.${jcloudsPersistSwitch}.log</file>
+ <append>true</append>
+ <encoder>
+ <pattern>%d %-5level %logger{30} [%thread{15}]: %msg%n</pattern>
+ </encoder>
- <!-- Truncate log at 100 MB, max history of 10 -->
- <rollingPolicy class="ch.qos.logback.core.rolling.FixedWindowRollingPolicy">
- <fileNamePattern>${logging.dir:-./}${logging.basename:-brooklyn}.jclouds-%i.log.zip</fileNamePattern>
- <minIndex>1</minIndex>
- <maxIndex>10</maxIndex>
- </rollingPolicy>
+ <!-- Truncate log at 100 MB, max history of 10 -->
+ <rollingPolicy class="ch.qos.logback.core.rolling.FixedWindowRollingPolicy">
+ <fileNamePattern>${logging.dir:-./}${logging.basename:-brooklyn}.${jcloudsPersistSwitch}-%i.log.zip</fileNamePattern>
+ <minIndex>1</minIndex>
+ <maxIndex>10</maxIndex>
+ </rollingPolicy>
- <triggeringPolicy class="ch.qos.logback.core.rolling.SizeBasedTriggeringPolicy">
- <maxFileSize>100MB</maxFileSize>
- </triggeringPolicy>
+ <triggeringPolicy class="ch.qos.logback.core.rolling.SizeBasedTriggeringPolicy">
+ <maxFileSize>100MB</maxFileSize>
+ </triggeringPolicy>
+ </appender>
+ </sift>
</appender>
</included>
http://git-wip-us.apache.org/repos/asf/incubator-brooklyn/blob/501412e1/usage/logback-includes/src/main/resources/brooklyn/logback-logger-debug-jclouds.xml
----------------------------------------------------------------------
diff --git a/usage/logback-includes/src/main/resources/brooklyn/logback-logger-debug-jclouds.xml b/usage/logback-includes/src/main/resources/brooklyn/logback-logger-debug-jclouds.xml
index 4c80547..a92f400 100644
--- a/usage/logback-includes/src/main/resources/brooklyn/logback-logger-debug-jclouds.xml
+++ b/usage/logback-includes/src/main/resources/brooklyn/logback-logger-debug-jclouds.xml
@@ -20,20 +20,28 @@
<included>
<!--
- including this file after all other debug configuration will move all of the jclouds
- logging messages to a separate file, named 'brooklyn-application.jclouds.log'
+ Including this file after all other debug configuration will move all of the jclouds
+ logging messages to two files, named 'brooklyn-application.jclouds.log' and
+ 'brooklyn-application.jclouds-persister.log'. The latter will contain all messages
+ logged by jclouds when called from Brooklyn's persistence thread, the former will
+ contain all other messages logged by jclouds.
- -->
+ If you have configured persistence to an object store and your server will manage more
+ than ~ 10 entities then it is recommended to use this appender over
+ logback-logger-debug-jcloud.xml as otherwise your log file will be swamped by
+ persistence-related messages.
- <!-- include jclouds log file appender -->
+ -->
+
+ <!-- Include jclouds log file appender -->
<include resource="brooklyn/logback-appender-jclouds.xml"/>
- <!-- send jclouds logging categories to the file -->
+ <!-- Send jclouds logging categories to the file -->
<logger name="org.jclouds" additivity="false">
- <appender-ref ref="JCLOUDS-FILE" />
+ <appender-ref ref="JCLOUDS-SIFT" />
</logger>
<logger name="jclouds" additivity="false">
- <appender-ref ref="JCLOUDS-FILE" />
+ <appender-ref ref="JCLOUDS-SIFT" />
</logger>
</included>
[2/2] incubator-brooklyn git commit: This closes #283
Posted by sj...@apache.org.
This closes #283
Project: http://git-wip-us.apache.org/repos/asf/incubator-brooklyn/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-brooklyn/commit/6db03286
Tree: http://git-wip-us.apache.org/repos/asf/incubator-brooklyn/tree/6db03286
Diff: http://git-wip-us.apache.org/repos/asf/incubator-brooklyn/diff/6db03286
Branch: refs/heads/master
Commit: 6db032864c9117b9f6c854c5bfb5a625c3c44b96
Parents: 5de642e 501412e
Author: Sam Corbett <sa...@cloudsoftcorp.com>
Authored: Wed Dec 3 17:38:51 2014 +0100
Committer: Sam Corbett <sa...@cloudsoftcorp.com>
Committed: Wed Dec 3 17:38:51 2014 +0100
----------------------------------------------------------------------
.../BrooklynMementoPersisterToObjectStore.java | 1 +
docs/dev/tips/logging.md | 4 ++
usage/logback-includes/pom.xml | 11 ++++
.../JcloudsPersistenceThreadDiscriminator.java | 65 ++++++++++++++++++++
.../brooklyn/logback-appender-jclouds.xml | 39 +++++++-----
.../brooklyn/logback-logger-debug-jclouds.xml | 22 ++++---
6 files changed, 119 insertions(+), 23 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-brooklyn/blob/6db03286/core/src/main/java/brooklyn/entity/rebind/persister/BrooklynMementoPersisterToObjectStore.java
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-brooklyn/blob/6db03286/usage/logback-includes/pom.xml
----------------------------------------------------------------------