You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Yu Yang <yu...@gmail.com> on 2015/03/09 08:17:26 UTC
Does Kafka 0.8.2 producer has a lower throughput in sync-mode,
comparing with 0.8.1.x?
Hi,
Kafka 0.8.1.1 allows us to send a list of messages in sync mode:
public void send(List<KeyedMessage<K,V> messages);
I did not find a counter-part of this api in the new producer that is
introduced in kafka 0.8.2. It seems that we can use the following method
to do sync send in kafka 0.8.2:
producer.send(new ProducerRecord(...))).get();
My understanding is that we can only send one message at a time in
sync-mode. This will limit the throughput of kafka producer. Is there a way
to send a batch of messages in sync mode using kafka 0.8.2 producer? Is
there any study on the throughput of Kafka 0.8.2 producer, comparing with
kafka 0.8.1.1?
Thanks!
Regards,
Yu
Re: Does Kafka 0.8.2 producer has a lower throughput in sync-mode,
comparing with 0.8.1.x?
Posted by Jiangjie Qin <jq...@linkedin.com.INVALID>.
Hi Yang,
In the code suggested by Manikumar, yes, it is possible message 3 still
got sent even message 2 failed.
There is no single line code for send a batch of message synchronously
now, but after KAFKA-1660 is checked in, you may be able to achieve this
by doing the following:
Set a callback for each send, in the callback, if exception occurs, call
producer.close(-1) - this will prevent further messages to be sent.
Then you can the following code:
For(record: records)
Producer.send(record, callback)
Producer.flush()
Jiangjie (Becket) Qin
On 3/9/15, 8:49 AM, "Yu Yang" <yu...@gmail.com> wrote:
>If a send request in the middle of the list fails, will all send requests
>that follows it fail? Or only the messages that are put in the same batch
>by the underneath transportation layer fail?
>
>On Mon, Mar 9, 2015 at 1:31 AM, Manikumar Reddy <ku...@nmsworks.co.in>
>wrote:
>
>> 1. We can send list of messages and wait on the returned futures
>>
>> List responses = new ArrayList();
>> for(input: recordBatch)
>> responses.add(producer.send(input));
>> for(response: responses)
>> response.get
>>
>> 2. messages will be send in the submission order.
>>
>> On Mon, Mar 9, 2015 at 1:56 PM, Manikumar Reddy <ku...@nmsworks.co.in>
>> wrote:
>>
>> > 1 .
>> >
>> >
>> >
>> > On Mon, Mar 9, 2015 at 1:03 PM, Yu Yang <yu...@gmail.com> wrote:
>> >
>> >> The confluent blog
>> >> <
>> http://blog.confluent.io/2014/12/02/whats-coming-in-apache-kafka-0-8-2/>
>> >> mentions
>> >> that the the batching is done whenever possible now. "The sync
>>producer,
>> >> under load, can get performance as good as the async producer. "
>>Does
>> it
>> >> mean that kafka 0.8.2 guarantees that the sequence of
>> >> broker-received-message is the same as the async-call sequence by the
>> >> producer? If I have code as follows to send messages to the same
>>topic
>> >> partition:
>> >>
>> >> producer.send(msg1);
>> >> prodcuer.send(msg2);
>> >> producer.send(msg3);
>> >>
>> >> If all three calls succeed, is it possible that broker received msg3
>> >> before
>> >> msg2?
>> >>
>> >>
>> >>
>> >>
>> >> On Mon, Mar 9, 2015 at 12:17 AM, Yu Yang <yu...@gmail.com> wrote:
>> >>
>> >> > Hi,
>> >> >
>> >> > Kafka 0.8.1.1 allows us to send a list of messages in sync mode:
>> >> >
>> >> > public void send(List<KeyedMessage<K,V> messages);
>> >> >
>> >> > I did not find a counter-part of this api in the new producer that
>>is
>> >> > introduced in kafka 0.8.2. It seems that we can use the following
>> >> method
>> >> > to do sync send in kafka 0.8.2:
>> >> >
>> >> > producer.send(new ProducerRecord(...))).get();
>> >> >
>> >> > My understanding is that we can only send one message at a time in
>> >> > sync-mode. This will limit the throughput of kafka producer. Is
>>there
>> a
>> >> way
>> >> > to send a batch of messages in sync mode using kafka 0.8.2
>>producer?
>> Is
>> >> > there any study on the throughput of Kafka 0.8.2 producer,
>>comparing
>> >> with
>> >> > kafka 0.8.1.1?
>> >> >
>> >> > Thanks!
>> >> >
>> >> > Regards,
>> >> > Yu
>> >> >
>> >> >
>> >> >
>> >>
>> >
>> >
>>
Re: Does Kafka 0.8.2 producer has a lower throughput in sync-mode,
comparing with 0.8.1.x?
Posted by Yu Yang <yu...@gmail.com>.
If a send request in the middle of the list fails, will all send requests
that follows it fail? Or only the messages that are put in the same batch
by the underneath transportation layer fail?
On Mon, Mar 9, 2015 at 1:31 AM, Manikumar Reddy <ku...@nmsworks.co.in>
wrote:
> 1. We can send list of messages and wait on the returned futures
>
> List responses = new ArrayList();
> for(input: recordBatch)
> responses.add(producer.send(input));
> for(response: responses)
> response.get
>
> 2. messages will be send in the submission order.
>
> On Mon, Mar 9, 2015 at 1:56 PM, Manikumar Reddy <ku...@nmsworks.co.in>
> wrote:
>
> > 1 .
> >
> >
> >
> > On Mon, Mar 9, 2015 at 1:03 PM, Yu Yang <yu...@gmail.com> wrote:
> >
> >> The confluent blog
> >> <
> http://blog.confluent.io/2014/12/02/whats-coming-in-apache-kafka-0-8-2/>
> >> mentions
> >> that the the batching is done whenever possible now. "The sync producer,
> >> under load, can get performance as good as the async producer. " Does
> it
> >> mean that kafka 0.8.2 guarantees that the sequence of
> >> broker-received-message is the same as the async-call sequence by the
> >> producer? If I have code as follows to send messages to the same topic
> >> partition:
> >>
> >> producer.send(msg1);
> >> prodcuer.send(msg2);
> >> producer.send(msg3);
> >>
> >> If all three calls succeed, is it possible that broker received msg3
> >> before
> >> msg2?
> >>
> >>
> >>
> >>
> >> On Mon, Mar 9, 2015 at 12:17 AM, Yu Yang <yu...@gmail.com> wrote:
> >>
> >> > Hi,
> >> >
> >> > Kafka 0.8.1.1 allows us to send a list of messages in sync mode:
> >> >
> >> > public void send(List<KeyedMessage<K,V> messages);
> >> >
> >> > I did not find a counter-part of this api in the new producer that is
> >> > introduced in kafka 0.8.2. It seems that we can use the following
> >> method
> >> > to do sync send in kafka 0.8.2:
> >> >
> >> > producer.send(new ProducerRecord(...))).get();
> >> >
> >> > My understanding is that we can only send one message at a time in
> >> > sync-mode. This will limit the throughput of kafka producer. Is there
> a
> >> way
> >> > to send a batch of messages in sync mode using kafka 0.8.2 producer?
> Is
> >> > there any study on the throughput of Kafka 0.8.2 producer, comparing
> >> with
> >> > kafka 0.8.1.1?
> >> >
> >> > Thanks!
> >> >
> >> > Regards,
> >> > Yu
> >> >
> >> >
> >> >
> >>
> >
> >
>
Re: Does Kafka 0.8.2 producer has a lower throughput in sync-mode,
comparing with 0.8.1.x?
Posted by Manikumar Reddy <ku...@nmsworks.co.in>.
1. We can send list of messages and wait on the returned futures
List responses = new ArrayList();
for(input: recordBatch)
responses.add(producer.send(input));
for(response: responses)
response.get
2. messages will be send in the submission order.
On Mon, Mar 9, 2015 at 1:56 PM, Manikumar Reddy <ku...@nmsworks.co.in>
wrote:
> 1 .
>
>
>
> On Mon, Mar 9, 2015 at 1:03 PM, Yu Yang <yu...@gmail.com> wrote:
>
>> The confluent blog
>> <http://blog.confluent.io/2014/12/02/whats-coming-in-apache-kafka-0-8-2/>
>> mentions
>> that the the batching is done whenever possible now. "The sync producer,
>> under load, can get performance as good as the async producer. " Does it
>> mean that kafka 0.8.2 guarantees that the sequence of
>> broker-received-message is the same as the async-call sequence by the
>> producer? If I have code as follows to send messages to the same topic
>> partition:
>>
>> producer.send(msg1);
>> prodcuer.send(msg2);
>> producer.send(msg3);
>>
>> If all three calls succeed, is it possible that broker received msg3
>> before
>> msg2?
>>
>>
>>
>>
>> On Mon, Mar 9, 2015 at 12:17 AM, Yu Yang <yu...@gmail.com> wrote:
>>
>> > Hi,
>> >
>> > Kafka 0.8.1.1 allows us to send a list of messages in sync mode:
>> >
>> > public void send(List<KeyedMessage<K,V> messages);
>> >
>> > I did not find a counter-part of this api in the new producer that is
>> > introduced in kafka 0.8.2. It seems that we can use the following
>> method
>> > to do sync send in kafka 0.8.2:
>> >
>> > producer.send(new ProducerRecord(...))).get();
>> >
>> > My understanding is that we can only send one message at a time in
>> > sync-mode. This will limit the throughput of kafka producer. Is there a
>> way
>> > to send a batch of messages in sync mode using kafka 0.8.2 producer? Is
>> > there any study on the throughput of Kafka 0.8.2 producer, comparing
>> with
>> > kafka 0.8.1.1?
>> >
>> > Thanks!
>> >
>> > Regards,
>> > Yu
>> >
>> >
>> >
>>
>
>
Re: Does Kafka 0.8.2 producer has a lower throughput in sync-mode,
comparing with 0.8.1.x?
Posted by Manikumar Reddy <ku...@nmsworks.co.in>.
1 .
On Mon, Mar 9, 2015 at 1:03 PM, Yu Yang <yu...@gmail.com> wrote:
> The confluent blog
> <http://blog.confluent.io/2014/12/02/whats-coming-in-apache-kafka-0-8-2/>
> mentions
> that the the batching is done whenever possible now. "The sync producer,
> under load, can get performance as good as the async producer. " Does it
> mean that kafka 0.8.2 guarantees that the sequence of
> broker-received-message is the same as the async-call sequence by the
> producer? If I have code as follows to send messages to the same topic
> partition:
>
> producer.send(msg1);
> prodcuer.send(msg2);
> producer.send(msg3);
>
> If all three calls succeed, is it possible that broker received msg3 before
> msg2?
>
>
>
>
> On Mon, Mar 9, 2015 at 12:17 AM, Yu Yang <yu...@gmail.com> wrote:
>
> > Hi,
> >
> > Kafka 0.8.1.1 allows us to send a list of messages in sync mode:
> >
> > public void send(List<KeyedMessage<K,V> messages);
> >
> > I did not find a counter-part of this api in the new producer that is
> > introduced in kafka 0.8.2. It seems that we can use the following method
> > to do sync send in kafka 0.8.2:
> >
> > producer.send(new ProducerRecord(...))).get();
> >
> > My understanding is that we can only send one message at a time in
> > sync-mode. This will limit the throughput of kafka producer. Is there a
> way
> > to send a batch of messages in sync mode using kafka 0.8.2 producer? Is
> > there any study on the throughput of Kafka 0.8.2 producer, comparing with
> > kafka 0.8.1.1?
> >
> > Thanks!
> >
> > Regards,
> > Yu
> >
> >
> >
>
Re: Does Kafka 0.8.2 producer has a lower throughput in sync-mode,
comparing with 0.8.1.x?
Posted by Yu Yang <yu...@gmail.com>.
The confluent blog
<http://blog.confluent.io/2014/12/02/whats-coming-in-apache-kafka-0-8-2/>
mentions
that the the batching is done whenever possible now. "The sync producer,
under load, can get performance as good as the async producer. " Does it
mean that kafka 0.8.2 guarantees that the sequence of
broker-received-message is the same as the async-call sequence by the
producer? If I have code as follows to send messages to the same topic
partition:
producer.send(msg1);
prodcuer.send(msg2);
producer.send(msg3);
If all three calls succeed, is it possible that broker received msg3 before
msg2?
On Mon, Mar 9, 2015 at 12:17 AM, Yu Yang <yu...@gmail.com> wrote:
> Hi,
>
> Kafka 0.8.1.1 allows us to send a list of messages in sync mode:
>
> public void send(List<KeyedMessage<K,V> messages);
>
> I did not find a counter-part of this api in the new producer that is
> introduced in kafka 0.8.2. It seems that we can use the following method
> to do sync send in kafka 0.8.2:
>
> producer.send(new ProducerRecord(...))).get();
>
> My understanding is that we can only send one message at a time in
> sync-mode. This will limit the throughput of kafka producer. Is there a way
> to send a batch of messages in sync mode using kafka 0.8.2 producer? Is
> there any study on the throughput of Kafka 0.8.2 producer, comparing with
> kafka 0.8.1.1?
>
> Thanks!
>
> Regards,
> Yu
>
>
>