You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Madhukar Bharti <bh...@gmail.com> on 2015/03/31 07:44:37 UTC
Async producer using Sync producer for send
Hi All,
I am using *async *producer to send the data. When I checked the log it is
showing as below:
[2015-03-31 11:09:55,915] INFO Verifying properties
(kafka.utils.VerifiableProperties)
[2015-03-31 11:09:55,946] INFO Property key.serializer.class is overridden
to kafka.serializer.StringEncoder (kafka.utils.VerifiableProperties)
[2015-03-31 11:09:55,947] INFO Property metadata.broker.list is overridden
to 172.20.6.201:9092,172.20.6.25:9092,172.20.8.62:9092
(kafka.utils.VerifiableProperties)
[2015-03-31 11:09:55,947] INFO Property producer.type is overridden to
async (kafka.utils.VerifiableProperties)
[2015-03-31 11:09:55,947] INFO Property queue.buffering.max.ms is
overridden to 300 (kafka.utils.VerifiableProperties)
[2015-03-31 11:09:55,947] INFO Property queue.enqueue.timeout.ms is
overridden to 50 (kafka.utils.VerifiableProperties)
[2015-03-31 11:09:55,947] INFO Property request.required.acks is overridden
to 1 (kafka.utils.VerifiableProperties)
[2015-03-31 11:09:55,948] INFO Property send.buffer.bytes is overridden to
4048 (kafka.utils.VerifiableProperties)
[2015-03-31 11:09:55,948] INFO Property serializer.class is overridden to
kafka.serializer.StringEncoder (kafka.utils.VerifiableProperties)
Thread Number0
Sent: This is message 0, Topic::TestMQ
Sent: This is message 1, Topic::TestMQ
Sent: This is message 2, Topic::TestMQ
Sent: This is message 3, Topic::TestMQ
Sent: This is message 4, Topic::TestMQ
Sent: This is message 5, Topic::TestMQ
Sent: This is message 6, Topic::TestMQ
Sent: This is message 7, Topic::TestMQ
Sent: This is message 8, Topic::TestMQ
Sent: This is message 9, Topic::TestMQ
[2015-03-31 11:09:56,395] INFO Fetching metadata from broker
id:1,host:172.20.6.25,port:9092 with correlation id 0 for 1 topic(s)
Set(TestMQ) (kafka.client.ClientUtils$)
[2015-03-31 11:09:56,404] INFO Connected to 172.20.6.25:9092 for producing
(kafka.producer.SyncProducer)
[2015-03-31 11:09:56,438] INFO Disconnecting from 172.20.6.25:9092
(kafka.producer.SyncProducer)
[2015-03-31 11:09:56,479] INFO Connected to 172.20.6.25:9092 for producing
(kafka.producer.SyncProducer)
[2015-03-31 11:09:56,573] INFO Connected to 172.20.6.201:9092 for producing
(kafka.producer.SyncProducer)
[2015-03-31 11:09:56,591] INFO Connected to 172.20.8.62:9092 for producing
(kafka.producer.SyncProducer)
My doubt is why it is using "*kafka.producer.SyncProducer*"? I am using
kafka 0.8.1.1.
Thanks in advance!
Re: Async producer using Sync producer for send
Posted by Madhukar Bharti <bh...@gmail.com>.
Thanks Jiangjie,
I too have thought the same after looking the code. Thanks a lot for
clearing my doubt!
On Tue, Mar 31, 2015 at 11:45 AM, Jiangjie Qin <jq...@linkedin.com.invalid>
wrote:
> The async send() put the message into a message queue then returns. When
> the messages are pulled out of the queue by the sender thread, it still
> uses SyncProducer to send ProducerRequests to brokers.
>
> Jiangjie (Becket) Qin
>
> On 3/30/15, 10:44 PM, "Madhukar Bharti" <bh...@gmail.com> wrote:
>
> >Hi All,
> >
> >I am using *async *producer to send the data. When I checked the log it is
> >showing as below:
> >
> >
> >[2015-03-31 11:09:55,915] INFO Verifying properties
> >(kafka.utils.VerifiableProperties)
> >[2015-03-31 11:09:55,946] INFO Property key.serializer.class is overridden
> >to kafka.serializer.StringEncoder (kafka.utils.VerifiableProperties)
> >[2015-03-31 11:09:55,947] INFO Property metadata.broker.list is overridden
> >to 172.20.6.201:9092,172.20.6.25:9092,172.20.8.62:9092
> >(kafka.utils.VerifiableProperties)
> >[2015-03-31 11:09:55,947] INFO Property producer.type is overridden to
> >async (kafka.utils.VerifiableProperties)
> >[2015-03-31 11:09:55,947] INFO Property queue.buffering.max.ms is
> >overridden to 300 (kafka.utils.VerifiableProperties)
> >[2015-03-31 11:09:55,947] INFO Property queue.enqueue.timeout.ms is
> >overridden to 50 (kafka.utils.VerifiableProperties)
> >[2015-03-31 11:09:55,947] INFO Property request.required.acks is
> >overridden
> >to 1 (kafka.utils.VerifiableProperties)
> >[2015-03-31 11:09:55,948] INFO Property send.buffer.bytes is overridden to
> >4048 (kafka.utils.VerifiableProperties)
> >[2015-03-31 11:09:55,948] INFO Property serializer.class is overridden to
> >kafka.serializer.StringEncoder (kafka.utils.VerifiableProperties)
> >Thread Number0
> >Sent: This is message 0, Topic::TestMQ
> >Sent: This is message 1, Topic::TestMQ
> >Sent: This is message 2, Topic::TestMQ
> >Sent: This is message 3, Topic::TestMQ
> >Sent: This is message 4, Topic::TestMQ
> >Sent: This is message 5, Topic::TestMQ
> >Sent: This is message 6, Topic::TestMQ
> >Sent: This is message 7, Topic::TestMQ
> >Sent: This is message 8, Topic::TestMQ
> >Sent: This is message 9, Topic::TestMQ
> >[2015-03-31 11:09:56,395] INFO Fetching metadata from broker
> >id:1,host:172.20.6.25,port:9092 with correlation id 0 for 1 topic(s)
> >Set(TestMQ) (kafka.client.ClientUtils$)
> >[2015-03-31 11:09:56,404] INFO Connected to 172.20.6.25:9092 for
> producing
> >(kafka.producer.SyncProducer)
> >[2015-03-31 11:09:56,438] INFO Disconnecting from 172.20.6.25:9092
> >(kafka.producer.SyncProducer)
> >[2015-03-31 11:09:56,479] INFO Connected to 172.20.6.25:9092 for
> producing
> >(kafka.producer.SyncProducer)
> >[2015-03-31 11:09:56,573] INFO Connected to 172.20.6.201:9092 for
> >producing
> >(kafka.producer.SyncProducer)
> >[2015-03-31 11:09:56,591] INFO Connected to 172.20.8.62:9092 for
> producing
> >(kafka.producer.SyncProducer)
> >
> >
> >
> >My doubt is why it is using "*kafka.producer.SyncProducer*"? I am using
> >kafka 0.8.1.1.
> >
> >
> >Thanks in advance!
>
>
Re: Async producer using Sync producer for send
Posted by Jiangjie Qin <jq...@linkedin.com.INVALID>.
The async send() put the message into a message queue then returns. When
the messages are pulled out of the queue by the sender thread, it still
uses SyncProducer to send ProducerRequests to brokers.
Jiangjie (Becket) Qin
On 3/30/15, 10:44 PM, "Madhukar Bharti" <bh...@gmail.com> wrote:
>Hi All,
>
>I am using *async *producer to send the data. When I checked the log it is
>showing as below:
>
>
>[2015-03-31 11:09:55,915] INFO Verifying properties
>(kafka.utils.VerifiableProperties)
>[2015-03-31 11:09:55,946] INFO Property key.serializer.class is overridden
>to kafka.serializer.StringEncoder (kafka.utils.VerifiableProperties)
>[2015-03-31 11:09:55,947] INFO Property metadata.broker.list is overridden
>to 172.20.6.201:9092,172.20.6.25:9092,172.20.8.62:9092
>(kafka.utils.VerifiableProperties)
>[2015-03-31 11:09:55,947] INFO Property producer.type is overridden to
>async (kafka.utils.VerifiableProperties)
>[2015-03-31 11:09:55,947] INFO Property queue.buffering.max.ms is
>overridden to 300 (kafka.utils.VerifiableProperties)
>[2015-03-31 11:09:55,947] INFO Property queue.enqueue.timeout.ms is
>overridden to 50 (kafka.utils.VerifiableProperties)
>[2015-03-31 11:09:55,947] INFO Property request.required.acks is
>overridden
>to 1 (kafka.utils.VerifiableProperties)
>[2015-03-31 11:09:55,948] INFO Property send.buffer.bytes is overridden to
>4048 (kafka.utils.VerifiableProperties)
>[2015-03-31 11:09:55,948] INFO Property serializer.class is overridden to
>kafka.serializer.StringEncoder (kafka.utils.VerifiableProperties)
>Thread Number0
>Sent: This is message 0, Topic::TestMQ
>Sent: This is message 1, Topic::TestMQ
>Sent: This is message 2, Topic::TestMQ
>Sent: This is message 3, Topic::TestMQ
>Sent: This is message 4, Topic::TestMQ
>Sent: This is message 5, Topic::TestMQ
>Sent: This is message 6, Topic::TestMQ
>Sent: This is message 7, Topic::TestMQ
>Sent: This is message 8, Topic::TestMQ
>Sent: This is message 9, Topic::TestMQ
>[2015-03-31 11:09:56,395] INFO Fetching metadata from broker
>id:1,host:172.20.6.25,port:9092 with correlation id 0 for 1 topic(s)
>Set(TestMQ) (kafka.client.ClientUtils$)
>[2015-03-31 11:09:56,404] INFO Connected to 172.20.6.25:9092 for producing
>(kafka.producer.SyncProducer)
>[2015-03-31 11:09:56,438] INFO Disconnecting from 172.20.6.25:9092
>(kafka.producer.SyncProducer)
>[2015-03-31 11:09:56,479] INFO Connected to 172.20.6.25:9092 for producing
>(kafka.producer.SyncProducer)
>[2015-03-31 11:09:56,573] INFO Connected to 172.20.6.201:9092 for
>producing
>(kafka.producer.SyncProducer)
>[2015-03-31 11:09:56,591] INFO Connected to 172.20.8.62:9092 for producing
>(kafka.producer.SyncProducer)
>
>
>
>My doubt is why it is using "*kafka.producer.SyncProducer*"? I am using
>kafka 0.8.1.1.
>
>
>Thanks in advance!