You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2015/12/10 02:49:11 UTC
[jira] [Commented] (KAFKA-2927) System tests: reduce storage
footprint of collected logs
[ https://issues.apache.org/jira/browse/KAFKA-2927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15049845#comment-15049845 ]
ASF GitHub Bot commented on KAFKA-2927:
---------------------------------------
GitHub user granders opened a pull request:
https://github.com/apache/kafka/pull/657
KAFKA-2927: reduce system test storage footprint
Split kafka logging into two levels - DEBUG and INFO, and do not collect DEBUG by default.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/confluentinc/kafka KAFKA-2927-reduce-log-footprint
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/kafka/pull/657.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #657
----
commit 0dc3a1a367083f57f3cb6d8e1cd82571598d7108
Author: Geoff Anderson <ge...@confluent.io>
Date: 2015-12-10T01:09:59Z
Split kafka logging into two levels - DEBUG and INFO, and do not collect DEBUG by default
----
> System tests: reduce storage footprint of collected logs
> --------------------------------------------------------
>
> Key: KAFKA-2927
> URL: https://issues.apache.org/jira/browse/KAFKA-2927
> Project: Kafka
> Issue Type: Bug
> Reporter: Geoff Anderson
> Assignee: Geoff Anderson
>
> Looking at recent night test runs (testing.confluent.io/kafka), the storage requirements for log output from the various services has increased significantly, up to 7-10G for a single test run, up from hundreds of MB
> Current breakdown:
> 23M Benchmark
> 3.2M ClientCompatibilityTest
> 613M ConnectDistributedTest
> 1.1M ConnectRestApiTest
> 1.5M ConnectStandaloneFileTest
> 2.0M ConsoleConsumerTest
> 440K KafkaVersionTest
> 744K Log4jAppenderTest
> 49M QuotaTest
> 3.0G ReplicationTest
> 1.2G TestMirrorMakerService
> 185M TestUpgrade
> 372K TestVerifiableProducer
> 2.3G VerifiableConsumerTest
> The biggest contributors in these test suites:
> ReplicationTest:
> verifiable_producer.log (currently TRACE level)
> VerifiableConsumerTest:
> kafka server.log
> TestMirrorMakerService:
> verifiable_producer.log
> ConnectDistributedTest:
> kafka server.log
> The worst offenders are therefore
> verifiable_producer.log which is logging at TRACE level, and kafka server.log which is logging at debug level
> One solution is to:
> 1) Update the log4j configs to log separately to both an INFO level file, and another file for DEBUG at least for the worst offenders.
> 2) Don't collect these DEBUG (and below) logs by default; only mark for collection during failure
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)