You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Jaikiran Pai <ja...@gmail.com> on 2016/12/13 09:32:59 UTC
Improve default Kafka logger settings to prevent extremely high disk
space usage issue?
We happened to run into a disk space usage issue with Kafka 0.10.0.1
(the version we are using) on one of our production setups this morning.
Turns out (log4j) logging from Kafka ended up using 81G and more of disk
space. Looking at the files, I see the controller.log itself is 30G and
more (for a single day). Looking at the default log4j.properties that's
shipped in Kafka, it uses the DailyRollingFileAppender which is one of
the things that contributes to this issue. I see that there's already a
patch and JIRA to fix this
https://issues.apache.org/jira/browse/KAFKA-2394. It's been marked for
0.11 because there wasn't a clear decision when to ship it.
Given that we have been going through 0.10.x releases these days and the
0.11 release looking a bit away, is there any chance, this specific JIRA
can make it to 0.10.x? I personally don't see any compatibility issues
that it will introduce when it comes to *functionality/features* of
Kafka itself, so I am not sure if it's that big a change to wait all the
way till 0.11. Furthermore, since the default shipped setting can cause
issues like the one I noted, I think it probably would be fine to
include it in one of the 0.10.x releases. Of course, we ourselves can
setup the logging config on our setup to use a size based rolling file
config instead of the one shipped by default, but if this is something
that can make it to 0.10.x, we would like to avoid doing this
customization ourselves.
That's one part of the issue. The other is, I see this in the default
shipped log4j.properties:
log4j.logger.kafka.controller=*TRACE,* controllerAppender
log4j.additivity.kafka.controller=false
log4j.logger.state.change.logger=*TRACE*, stateChangeAppender
log4j.additivity.state.change.logger=false
Is it intentional to have this at TRACE level for the default shipped
config instead of having something like INFO or maybe DEBUG?
-Jaikiran
Re: Improve default Kafka logger settings to prevent extremely high
disk space usage issue?
Posted by Jaikiran Pai <ja...@gmail.com>.
On Tuesday 13 December 2016 03:29 PM, Ismael Juma wrote:
> The log config settings for the controller and state change logger have
> been that way since they were introduced. They're generally useful when
> investigating issues with the controller. Looks like this is too noisy in
> some scenarios though. It may be worth filing a JIRA with specifics of your
> use case to see if something can be done to improve that.
For now, we decided to use a custom log4j config. I'll get back to this
and check if there are specific log messages that cause this or if it's
just regular logs at TRACE level which contribute to this, before filing
a JIRA.
-Jaikiran
>
> Ismael
>
> On Tue, Dec 13, 2016 at 9:32 AM, Jaikiran Pai <ja...@gmail.com>
> wrote:
>
>> We happened to run into a disk space usage issue with Kafka 0.10.0.1 (the
>> version we are using) on one of our production setups this morning. Turns
>> out (log4j) logging from Kafka ended up using 81G and more of disk space.
>> Looking at the files, I see the controller.log itself is 30G and more (for
>> a single day). Looking at the default log4j.properties that's shipped in
>> Kafka, it uses the DailyRollingFileAppender which is one of the things that
>> contributes to this issue. I see that there's already a patch and JIRA to
>> fix this https://issues.apache.org/jira/browse/KAFKA-2394. It's been
>> marked for 0.11 because there wasn't a clear decision when to ship it.
>>
>> Given that we have been going through 0.10.x releases these days and the
>> 0.11 release looking a bit away, is there any chance, this specific JIRA
>> can make it to 0.10.x? I personally don't see any compatibility issues that
>> it will introduce when it comes to *functionality/features* of Kafka
>> itself, so I am not sure if it's that big a change to wait all the way till
>> 0.11. Furthermore, since the default shipped setting can cause issues like
>> the one I noted, I think it probably would be fine to include it in one of
>> the 0.10.x releases. Of course, we ourselves can setup the logging config
>> on our setup to use a size based rolling file config instead of the one
>> shipped by default, but if this is something that can make it to 0.10.x, we
>> would like to avoid doing this customization ourselves.
>>
>> That's one part of the issue. The other is, I see this in the default
>> shipped log4j.properties:
>>
>>
>> log4j.logger.kafka.controller=*TRACE,* controllerAppender
>> log4j.additivity.kafka.controller=false
>>
>> log4j.logger.state.change.logger=*TRACE*, stateChangeAppender
>> log4j.additivity.state.change.logger=false
>>
>>
>> Is it intentional to have this at TRACE level for the default shipped
>> config instead of having something like INFO or maybe DEBUG?
>>
>>
>> -Jaikiran
>>
Re: Improve default Kafka logger settings to prevent extremely high
disk space usage issue?
Posted by Ismael Juma <is...@juma.me.uk>.
Hi Jaikiran,
Thanks for raising this. The compatibility issue is that the file name
pattern for the rolled files are different. Because there's a simple
workaround, it was deemed that it could wait until 0.11.0. This is actually
not far off if you consider that we have 4 active KIPs that are proposing
message format changes (a message format change implies a major version
bump).
The log config settings for the controller and state change logger have
been that way since they were introduced. They're generally useful when
investigating issues with the controller. Looks like this is too noisy in
some scenarios though. It may be worth filing a JIRA with specifics of your
use case to see if something can be done to improve that.
Ismael
On Tue, Dec 13, 2016 at 9:32 AM, Jaikiran Pai <ja...@gmail.com>
wrote:
> We happened to run into a disk space usage issue with Kafka 0.10.0.1 (the
> version we are using) on one of our production setups this morning. Turns
> out (log4j) logging from Kafka ended up using 81G and more of disk space.
> Looking at the files, I see the controller.log itself is 30G and more (for
> a single day). Looking at the default log4j.properties that's shipped in
> Kafka, it uses the DailyRollingFileAppender which is one of the things that
> contributes to this issue. I see that there's already a patch and JIRA to
> fix this https://issues.apache.org/jira/browse/KAFKA-2394. It's been
> marked for 0.11 because there wasn't a clear decision when to ship it.
>
> Given that we have been going through 0.10.x releases these days and the
> 0.11 release looking a bit away, is there any chance, this specific JIRA
> can make it to 0.10.x? I personally don't see any compatibility issues that
> it will introduce when it comes to *functionality/features* of Kafka
> itself, so I am not sure if it's that big a change to wait all the way till
> 0.11. Furthermore, since the default shipped setting can cause issues like
> the one I noted, I think it probably would be fine to include it in one of
> the 0.10.x releases. Of course, we ourselves can setup the logging config
> on our setup to use a size based rolling file config instead of the one
> shipped by default, but if this is something that can make it to 0.10.x, we
> would like to avoid doing this customization ourselves.
>
> That's one part of the issue. The other is, I see this in the default
> shipped log4j.properties:
>
>
> log4j.logger.kafka.controller=*TRACE,* controllerAppender
> log4j.additivity.kafka.controller=false
>
> log4j.logger.state.change.logger=*TRACE*, stateChangeAppender
> log4j.additivity.state.change.logger=false
>
>
> Is it intentional to have this at TRACE level for the default shipped
> config instead of having something like INFO or maybe DEBUG?
>
>
> -Jaikiran
>
Re: Improve default Kafka logger settings to prevent extremely high
disk space usage issue?
Posted by Jaikiran Pai <ja...@gmail.com>.
On Tuesday 13 December 2016 03:02 PM, Jaikiran Pai wrote:
>
> log4j.logger.kafka.controller=*TRACE,* controllerAppender
> log4j.additivity.kafka.controller=false
>
> log4j.logger.state.change.logger=*TRACE*, stateChangeAppender
> log4j.additivity.state.change.logger=false
>
>
> Is it intentional to have this at TRACE level for the default shipped
> config instead of having something like INFO or maybe DEBUG?
>
>
> -Jaikiran
Having now looked at one of the log messages in that controller.log, I
don't think DEBUG level would be a good default either since it logs
messages which dump all topic names and the broker ids associated with
it very often at DEBUG level. Something like:
[2016-12-13 09:34:53,869] DEBUG [Controller 0]: preferred replicas by
broker Map.....
(note, I couldn't paste the exact log message, which was large, because
there wasn't a easy way to remove references to our application specific
names from it, for the topic names).
That single log message itself is huge every time since we have a large
number of topics which get created dynamically for our application use
case.
-Jaikiran