You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by "Guozhang Wang (JIRA)" <ji...@apache.org> on 2014/10/02 00:11:35 UTC
[jira] [Commented] (KAFKA-1590) Binarize trace level request
logging along with debug level text logging
[ https://issues.apache.org/jira/browse/KAFKA-1590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14155647#comment-14155647 ]
Guozhang Wang commented on KAFKA-1590:
--------------------------------------
[~jkreps] I agree that this would not be the default config.
By default: the request log would be written in text.
When this knob turns on: the request log would be written only as summary in text (i.e. today's DEBUG level logging), and the detailed log will be recorded as kafka-request-details.gzip/[any other compression types?].
I personally do not know what the % savings would be with the proposed binary / compression format compared the the text format. Would be great if we can get some numbers with a preliminary patch.
[~Abhishek Sharma] do you think it is possible that we write a simple cross-platform tool to parse the request-details binary files?
> Binarize trace level request logging along with debug level text logging
> ------------------------------------------------------------------------
>
> Key: KAFKA-1590
> URL: https://issues.apache.org/jira/browse/KAFKA-1590
> Project: Kafka
> Issue Type: Bug
> Reporter: Guozhang Wang
> Assignee: Abhishek Sharma
> Labels: newbie
> Fix For: 0.9.0
>
> Attachments: Jira-1590-Log4j Binary Appender.patch
>
>
> With trace level logging, the request handling logs can grow very fast depending on the client behavior (e.g. consumer with 0 maxWait and hence keep sending fetch requests). Previously we have changed it to debug level which only provides a summary of the requests, omitting request details. However this does not work perfectly since summaries are not sufficient for trouble-shooting, and turning on trace level upon issues will be too late.
> The proposed solution here, is to default to debug level logging with trace level logging printed as binary format at the same time. The generated binary files can then be further compressed / rolled out. When needed, we will then decompress / parse the trace logs into texts.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)