Re: Issue with log4j Kafka Appender.

[Madhavi Sreerangam <ma...@gmail.com>]

I have added the configurations suggested, now my log4j appender file looks
as follows

log4j.appender.KAFKA=kafka.producer.KafkaLog4jAppender

log4j.appender.KAFKA.BrokerList=iakafka301p.dev.ch3.s.com:9092,
iakafka302p.dev.ch3.s.com:9092,iakafka303p.dev.ch3.s.com:9092

log4j.appender.KAFKA.Topic=dev-1.0_audit

log4j.appender.KAFKA.requiredNumAcks=0

log4j.appender.KAFKA.layout=org.apache.log4j.PatternLayout

log4j.appender.KAFKA.layout.ConversionPattern=%m-%d

log4j.appender.KAFKA.syncSend=false


but still I could not see messages on queue without sleep time. I have
enabled the log4 debug and following is the debug trace

2015-07-01 10:49:59,238 [main] INFO  clients.producer.ProducerConfig  -
ProducerConfig values:

value.serializer = class
org.apache.kafka.common.serialization.ByteArraySerializer

key.serializer = class
org.apache.kafka.common.serialization.ByteArraySerializer

block.on.buffer.full = true

retry.backoff.ms = 100

buffer.memory = 33554432

batch.size = 16384

metrics.sample.window.ms = 30000

metadata.max.age.ms = 300000

receive.buffer.bytes = 32768

timeout.ms = 30000

max.in.flight.requests.per.connection = 5

bootstrap.servers = [iakafka301p.dev.ch3.s.com:9092,
iakafka302p.dev.ch3.s.com:9092, iakafka303p.dev.ch3.s.com:9092]

metric.reporters = []

client.id =

compression.type = none

retries = 0

max.request.size = 1048576

send.buffer.bytes = 131072

acks = 1

reconnect.backoff.ms = 10

linger.ms = 0

metrics.num.samples = 2

metadata.fetch.timeout.ms = 60000


2015-07-01 10:49:59,515 [main] INFO  kafka.utils.VerifiableProperties  -
Verifying properties

2015-07-01 10:49:59,550 [main] INFO  kafka.utils.VerifiableProperties  -
Property metadata.broker.list is overridden to
iakafka301p.dev.ch3.s.com:9092,iakafka302p.dev.ch3.s.com:9092,
iakafka303p.dev.ch3.s.com:9092

2015-07-01 10:49:59,550 [main] INFO  kafka.utils.VerifiableProperties  -
Property request.required.acks is overridden to 1

2015-07-01 10:49:59,550 [main] INFO  kafka.utils.VerifiableProperties  -
Property serializer.class is overridden to kafka.serializer.StringEncoder


I could not find much documentation also. Can anyone help me to find the
proper documentation/code base for this issue.

On Thu, Jun 18, 2015 at 6:14 AM, Manikumar Reddy <ku...@nmsworks.co.in>
wrote:

> You can enable producer  debug log and verify. In 0.8.2.0, you can set
>  compressionType
> , requiredNumAcks,  syncSend producer config properties to log4j.xml. Trunk
> build can take additional retries property .
>
>
> Manikumar
>
> On Thu, Jun 18, 2015 at 1:14 AM, Madhavi Sreerangam <
> madhavi.sreerangam@gmail.com> wrote:
>
> > I have configured my log4j with Kafka Appender.(Kafka version 0.8.2.0)
> > Following are entries from my log4j file
> >
> > log4j.appender.KAFKA=kafka.producer.KafkaLog4jAppender
> > log4j.appender.KAFKA.BrokerList=iakafka301p.dev.ch3.s.com:9092,
> > iakafka302p.dev.ch3.s.com:9092,iakafka303p.dev.ch3.s.com:9092
> > log4j.appender.KAFKA.Topic=dev-1.0_audit
> > log4j.appender.KAFKA.Serializer=kafka.test.AppenderStringSerializer
> > log4j.appender.KAFKA.layout=org.apache.log4j.PatternLayout
> > log4j.appender.KAFKA.layout.ConversionPattern=%m-%d
> >
> > Kafka is configured with 3 servers, 3 partitions and 3 replicas.
> > I have created a test method to publish the messages to kafka topic as
> > follows
> >
> >     private void testKAFKAlog(int noOfMessages){
> >         for(int i=0; i < noOfMessages; i++){
> >             KAFKA_LOG.info("Test Message: " + i);
> >         }
> >     }
> > I could not see any messages published into the topic. Then I have
> modified
> > the test method to introduce some wait between the requests as follows
> >
> >     private void testKAFKAlog(int noOfMessages){
> >         for(int i=0; i < noOfMessages; i++){
> >             try {
> >                 Thread.sleep(10);
> >             } catch (InterruptedException e) {
> >                 e.printStackTrace();
> >             }
> >             KAFKA_LOG.info("Test Message: " + i);
> >         }
> >     }
> >
> > Then all the messages started publishing. I did this exercise couple of
> > times with and without sleep between the requests. Messages got published
> > only when there is sleep in between the requests.
> > Does any one help me here, what is wrong with the configurations I am
> > using. (I can't afford 10ms wait for each message, as my application logs
> > few Million messages for each run).
> > Is there any way that I can override the default ProducerConfig for log4j
> > kafka appender.
> >
>