You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Edward Capriolo <ed...@gmail.com> on 2014/08/28 21:49:53 UTC
Request log is 300MB/hour
At a certain hour I have seen a huge up tick in requests.
log4j.appender.requestAppender=org.apache.log4j.DailyRollingFileAppender
log4j.appender.requestAppender.DatePattern='.'yyyy-MM-dd-HH
log4j.appender.requestAppender.File=logs/kafka-request.log
log4j.appender.requestAppender.layout=org.apache.log4j.PatternLayout
log4j.appender.requestAppender.layout.ConversionPattern=[%d] %p %m (%c)%n
[2014-08-28 18:00:00,031] TRACE Completed request:Name: FetchRequest;
Version: 0; CorrelationId: 10672153; ClientId: xxxx; ReplicaId: -1;
MaxWait: 100 ms; MinBytes: 1 bytes; RequestInfo: [events,0] ->
PartitionFetchInfo(41358968,1048576) from client
/10.9.61.6:55168;totalTime:100,queueTime:0,localTime:0,remoteTime:100,sendTime:0
(kafka.request.logger)
Everything seems normal, we have a topic with 100 partitions a good flow of
data an 10 active producers. I am going ot lower the log level but does
anything thing this chatter is indicative of a problem?
Re: Request log is 300MB/hour
Posted by Guozhang Wang <wa...@gmail.com>.
Hello Edward,
You can try to increase the maxWait and minBytes (currently they are 100 ms
and 1 byte from your logs) so that the consumer will not frequently try to
pull data if there is little.
Guozhang
On Thu, Aug 28, 2014 at 12:49 PM, Edward Capriolo <ed...@gmail.com>
wrote:
> At a certain hour I have seen a huge up tick in requests.
>
> log4j.appender.requestAppender=org.apache.log4j.DailyRollingFileAppender
> log4j.appender.requestAppender.DatePattern='.'yyyy-MM-dd-HH
> log4j.appender.requestAppender.File=logs/kafka-request.log
> log4j.appender.requestAppender.layout=org.apache.log4j.PatternLayout
> log4j.appender.requestAppender.layout.ConversionPattern=[%d] %p %m (%c)%n
>
>
> [2014-08-28 18:00:00,031] TRACE Completed request:Name: FetchRequest;
> Version: 0; CorrelationId: 10672153; ClientId: xxxx; ReplicaId: -1;
> MaxWait: 100 ms; MinBytes: 1 bytes; RequestInfo: [events,0] ->
> PartitionFetchInfo(41358968,1048576) from client
> /10.9.61.6:55168
> ;totalTime:100,queueTime:0,localTime:0,remoteTime:100,sendTime:0
> (kafka.request.logger)
>
> Everything seems normal, we have a topic with 100 partitions a good flow of
> data an 10 active producers. I am going ot lower the log level but does
> anything thing this chatter is indicative of a problem?
>
--
-- Guozhang
Re: Request log is 300MB/hour
Posted by David Corley <da...@gmail.com>.
Edward, I believe the request log was set to TRACE by default in older
versions of Kafka, but has changed to WARN in newer versions. We had the
same problem as you, and lowered our log level to WARN with no apparent
issues
On Thu, Aug 28, 2014 at 8:49 PM, Edward Capriolo <ed...@gmail.com>
wrote:
> At a certain hour I have seen a huge up tick in requests.
>
> log4j.appender.requestAppender=org.apache.log4j.DailyRollingFileAppender
> log4j.appender.requestAppender.DatePattern='.'yyyy-MM-dd-HH
> log4j.appender.requestAppender.File=logs/kafka-request.log
> log4j.appender.requestAppender.layout=org.apache.log4j.PatternLayout
> log4j.appender.requestAppender.layout.ConversionPattern=[%d] %p %m (%c)%n
>
>
> [2014-08-28 18:00:00,031] TRACE Completed request:Name: FetchRequest;
> Version: 0; CorrelationId: 10672153; ClientId: xxxx; ReplicaId: -1;
> MaxWait: 100 ms; MinBytes: 1 bytes; RequestInfo: [events,0] ->
> PartitionFetchInfo(41358968,1048576) from client
> /10.9.61.6:55168
> ;totalTime:100,queueTime:0,localTime:0,remoteTime:100,sendTime:0
> (kafka.request.logger)
>
> Everything seems normal, we have a topic with 100 partitions a good flow of
> data an 10 active producers. I am going ot lower the log level but does
> anything thing this chatter is indicative of a problem?
>