You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Madhavi Sreerangam <ma...@gmail.com> on 2015/06/17 21:44:34 UTC
Issue with log4j Kafka Appender.
I have configured my log4j with Kafka Appender.(Kafka version 0.8.2.0)
Following are entries from my log4j file
log4j.appender.KAFKA=kafka.producer.KafkaLog4jAppender
log4j.appender.KAFKA.BrokerList=iakafka301p.dev.ch3.s.com:9092,
iakafka302p.dev.ch3.s.com:9092,iakafka303p.dev.ch3.s.com:9092
log4j.appender.KAFKA.Topic=dev-1.0_audit
log4j.appender.KAFKA.Serializer=kafka.test.AppenderStringSerializer
log4j.appender.KAFKA.layout=org.apache.log4j.PatternLayout
log4j.appender.KAFKA.layout.ConversionPattern=%m-%d
Kafka is configured with 3 servers, 3 partitions and 3 replicas.
I have created a test method to publish the messages to kafka topic as
follows
private void testKAFKAlog(int noOfMessages){
for(int i=0; i < noOfMessages; i++){
KAFKA_LOG.info("Test Message: " + i);
}
}
I could not see any messages published into the topic. Then I have modified
the test method to introduce some wait between the requests as follows
private void testKAFKAlog(int noOfMessages){
for(int i=0; i < noOfMessages; i++){
try {
Thread.sleep(10);
} catch (InterruptedException e) {
e.printStackTrace();
}
KAFKA_LOG.info("Test Message: " + i);
}
}
Then all the messages started publishing. I did this exercise couple of
times with and without sleep between the requests. Messages got published
only when there is sleep in between the requests.
Does any one help me here, what is wrong with the configurations I am
using. (I can't afford 10ms wait for each message, as my application logs
few Million messages for each run).
Is there any way that I can override the default ProducerConfig for log4j
kafka appender.
Re: Issue with log4j Kafka Appender.
Posted by Madhavi Sreerangam <ma...@gmail.com>.
I have added the configurations suggested, now my log4j appender file looks
as follows
log4j.appender.KAFKA=kafka.producer.KafkaLog4jAppender
log4j.appender.KAFKA.BrokerList=iakafka301p.dev.ch3.s.com:9092,
iakafka302p.dev.ch3.s.com:9092,iakafka303p.dev.ch3.s.com:9092
log4j.appender.KAFKA.Topic=dev-1.0_audit
log4j.appender.KAFKA.requiredNumAcks=0
log4j.appender.KAFKA.layout=org.apache.log4j.PatternLayout
log4j.appender.KAFKA.layout.ConversionPattern=%m-%d
log4j.appender.KAFKA.syncSend=false
but still I could not see messages on queue without sleep time. I have
enabled the log4 debug and following is the debug trace
2015-07-01 10:49:59,238 [main] INFO clients.producer.ProducerConfig -
ProducerConfig values:
value.serializer = class
org.apache.kafka.common.serialization.ByteArraySerializer
key.serializer = class
org.apache.kafka.common.serialization.ByteArraySerializer
block.on.buffer.full = true
retry.backoff.ms = 100
buffer.memory = 33554432
batch.size = 16384
metrics.sample.window.ms = 30000
metadata.max.age.ms = 300000
receive.buffer.bytes = 32768
timeout.ms = 30000
max.in.flight.requests.per.connection = 5
bootstrap.servers = [iakafka301p.dev.ch3.s.com:9092,
iakafka302p.dev.ch3.s.com:9092, iakafka303p.dev.ch3.s.com:9092]
metric.reporters = []
client.id =
compression.type = none
retries = 0
max.request.size = 1048576
send.buffer.bytes = 131072
acks = 1
reconnect.backoff.ms = 10
linger.ms = 0
metrics.num.samples = 2
metadata.fetch.timeout.ms = 60000
2015-07-01 10:49:59,515 [main] INFO kafka.utils.VerifiableProperties -
Verifying properties
2015-07-01 10:49:59,550 [main] INFO kafka.utils.VerifiableProperties -
Property metadata.broker.list is overridden to
iakafka301p.dev.ch3.s.com:9092,iakafka302p.dev.ch3.s.com:9092,
iakafka303p.dev.ch3.s.com:9092
2015-07-01 10:49:59,550 [main] INFO kafka.utils.VerifiableProperties -
Property request.required.acks is overridden to 1
2015-07-01 10:49:59,550 [main] INFO kafka.utils.VerifiableProperties -
Property serializer.class is overridden to kafka.serializer.StringEncoder
I could not find much documentation also. Can anyone help me to find the
proper documentation/code base for this issue.
On Thu, Jun 18, 2015 at 6:14 AM, Manikumar Reddy <ku...@nmsworks.co.in>
wrote:
> You can enable producer debug log and verify. In 0.8.2.0, you can set
> compressionType
> , requiredNumAcks, syncSend producer config properties to log4j.xml. Trunk
> build can take additional retries property .
>
>
> Manikumar
>
> On Thu, Jun 18, 2015 at 1:14 AM, Madhavi Sreerangam <
> madhavi.sreerangam@gmail.com> wrote:
>
> > I have configured my log4j with Kafka Appender.(Kafka version 0.8.2.0)
> > Following are entries from my log4j file
> >
> > log4j.appender.KAFKA=kafka.producer.KafkaLog4jAppender
> > log4j.appender.KAFKA.BrokerList=iakafka301p.dev.ch3.s.com:9092,
> > iakafka302p.dev.ch3.s.com:9092,iakafka303p.dev.ch3.s.com:9092
> > log4j.appender.KAFKA.Topic=dev-1.0_audit
> > log4j.appender.KAFKA.Serializer=kafka.test.AppenderStringSerializer
> > log4j.appender.KAFKA.layout=org.apache.log4j.PatternLayout
> > log4j.appender.KAFKA.layout.ConversionPattern=%m-%d
> >
> > Kafka is configured with 3 servers, 3 partitions and 3 replicas.
> > I have created a test method to publish the messages to kafka topic as
> > follows
> >
> > private void testKAFKAlog(int noOfMessages){
> > for(int i=0; i < noOfMessages; i++){
> > KAFKA_LOG.info("Test Message: " + i);
> > }
> > }
> > I could not see any messages published into the topic. Then I have
> modified
> > the test method to introduce some wait between the requests as follows
> >
> > private void testKAFKAlog(int noOfMessages){
> > for(int i=0; i < noOfMessages; i++){
> > try {
> > Thread.sleep(10);
> > } catch (InterruptedException e) {
> > e.printStackTrace();
> > }
> > KAFKA_LOG.info("Test Message: " + i);
> > }
> > }
> >
> > Then all the messages started publishing. I did this exercise couple of
> > times with and without sleep between the requests. Messages got published
> > only when there is sleep in between the requests.
> > Does any one help me here, what is wrong with the configurations I am
> > using. (I can't afford 10ms wait for each message, as my application logs
> > few Million messages for each run).
> > Is there any way that I can override the default ProducerConfig for log4j
> > kafka appender.
> >
>
Re: Issue with log4j Kafka Appender.
Posted by Manikumar Reddy <ku...@nmsworks.co.in>.
You can enable producer debug log and verify. In 0.8.2.0, you can set
compressionType
, requiredNumAcks, syncSend producer config properties to log4j.xml. Trunk
build can take additional retries property .
Manikumar
On Thu, Jun 18, 2015 at 1:14 AM, Madhavi Sreerangam <
madhavi.sreerangam@gmail.com> wrote:
> I have configured my log4j with Kafka Appender.(Kafka version 0.8.2.0)
> Following are entries from my log4j file
>
> log4j.appender.KAFKA=kafka.producer.KafkaLog4jAppender
> log4j.appender.KAFKA.BrokerList=iakafka301p.dev.ch3.s.com:9092,
> iakafka302p.dev.ch3.s.com:9092,iakafka303p.dev.ch3.s.com:9092
> log4j.appender.KAFKA.Topic=dev-1.0_audit
> log4j.appender.KAFKA.Serializer=kafka.test.AppenderStringSerializer
> log4j.appender.KAFKA.layout=org.apache.log4j.PatternLayout
> log4j.appender.KAFKA.layout.ConversionPattern=%m-%d
>
> Kafka is configured with 3 servers, 3 partitions and 3 replicas.
> I have created a test method to publish the messages to kafka topic as
> follows
>
> private void testKAFKAlog(int noOfMessages){
> for(int i=0; i < noOfMessages; i++){
> KAFKA_LOG.info("Test Message: " + i);
> }
> }
> I could not see any messages published into the topic. Then I have modified
> the test method to introduce some wait between the requests as follows
>
> private void testKAFKAlog(int noOfMessages){
> for(int i=0; i < noOfMessages; i++){
> try {
> Thread.sleep(10);
> } catch (InterruptedException e) {
> e.printStackTrace();
> }
> KAFKA_LOG.info("Test Message: " + i);
> }
> }
>
> Then all the messages started publishing. I did this exercise couple of
> times with and without sleep between the requests. Messages got published
> only when there is sleep in between the requests.
> Does any one help me here, what is wrong with the configurations I am
> using. (I can't afford 10ms wait for each message, as my application logs
> few Million messages for each run).
> Is there any way that I can override the default ProducerConfig for log4j
> kafka appender.
>