You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Pavel Sapozhnikov <pa...@gmail.com> on 2018/05/20 18:00:20 UTC
Best practices
Hello
What are the best practices when it comes to publishing a message into
kafka. When sending a message into Kafka is it possible to know if that
message has successfully been published? If not, what is the best practice
to know when messages are not getting published?
Second question.
If I have two kafka brokers and very simplistic one kafka topic replicated
on both. Do I need to send to both brokers. What are the best practices for
that.
Re: Best practices
Posted by "Matthias J. Sax" <ma...@confluent.io>.
If you specify some bootstrap.servers, after connecting the producer
will learn about all available brokers automatically, by fetching
cluster metadata from the first broker it connects. Thus, in practice it
is usually sufficient to specify 3 to 5 brokers (in case one is down,
the producer can "bootstrap" itself connecting to any other broker in
the list initially).
Also note, that producer do no write to an arbitrary broker: for each
partition, there is a dedicates leader and the producer sends all write
to the leader (as explained in the blog post is shared in my last reply
--- please read it:
https://www.confluent.io/blog/hands-free-kafka-replication-a-lesson-in-operational-simplicity/)
For best practices, it's recommended to configure producer retries via
config parameter `retries` (default is zero). For this case, if a write
fails, the producer will retry the write internally (potentially to a
different broker in case the leader changed). Only after all retries are
exhausted, you would get a callback that indicates the failed write.
-Matthias
On 5/21/18 5:40 AM, Pavel Sapozhnikov wrote:
> If a process failed to write a message into one broker what is the best
> practice to write to same topic on different broker? Is there one? I should
> be able to get a list of brokers programmatically from zk path /brokers/ids
> ?
>
> On Sun, May 20, 2018, 3:21 PM Matthias J. Sax <ma...@confluent.io> wrote:
>
>> You can register a callback for each sent record to learn about
>> successful write or fail:
>>
>>> producer.send(record, callback);
>>
>> For replication, you don't need to send twice. If the replication factor
>> is configured broker side, the broker take care of replication
>> automatically.
>>
>> You can also configure when you want to be informed about successful
>> write: before or after replication.
>>
>> This blog post should help:
>>
>> https://www.confluent.io/blog/hands-free-kafka-replication-a-lesson-in-operational-simplicity/
>>
>>
>>
>> -Matthias
>>
>> On 5/20/18 11:00 AM, Pavel Sapozhnikov wrote:
>>> Hello
>>>
>>> What are the best practices when it comes to publishing a message into
>>> kafka. When sending a message into Kafka is it possible to know if that
>>> message has successfully been published? If not, what is the best
>> practice
>>> to know when messages are not getting published?
>>>
>>> Second question.
>>>
>>> If I have two kafka brokers and very simplistic one kafka topic
>> replicated
>>> on both. Do I need to send to both brokers. What are the best practices
>> for
>>> that.
>>>
>>
>>
>
Re: Best practices
Posted by Pavel Sapozhnikov <pa...@gmail.com>.
If a process failed to write a message into one broker what is the best
practice to write to same topic on different broker? Is there one? I should
be able to get a list of brokers programmatically from zk path /brokers/ids
?
On Sun, May 20, 2018, 3:21 PM Matthias J. Sax <ma...@confluent.io> wrote:
> You can register a callback for each sent record to learn about
> successful write or fail:
>
> > producer.send(record, callback);
>
> For replication, you don't need to send twice. If the replication factor
> is configured broker side, the broker take care of replication
> automatically.
>
> You can also configure when you want to be informed about successful
> write: before or after replication.
>
> This blog post should help:
>
> https://www.confluent.io/blog/hands-free-kafka-replication-a-lesson-in-operational-simplicity/
>
>
>
> -Matthias
>
> On 5/20/18 11:00 AM, Pavel Sapozhnikov wrote:
> > Hello
> >
> > What are the best practices when it comes to publishing a message into
> > kafka. When sending a message into Kafka is it possible to know if that
> > message has successfully been published? If not, what is the best
> practice
> > to know when messages are not getting published?
> >
> > Second question.
> >
> > If I have two kafka brokers and very simplistic one kafka topic
> replicated
> > on both. Do I need to send to both brokers. What are the best practices
> for
> > that.
> >
>
>
Re: Best practices
Posted by "Matthias J. Sax" <ma...@confluent.io>.
You can register a callback for each sent record to learn about
successful write or fail:
> producer.send(record, callback);
For replication, you don't need to send twice. If the replication factor
is configured broker side, the broker take care of replication
automatically.
You can also configure when you want to be informed about successful
write: before or after replication.
This blog post should help:
https://www.confluent.io/blog/hands-free-kafka-replication-a-lesson-in-operational-simplicity/
-Matthias
On 5/20/18 11:00 AM, Pavel Sapozhnikov wrote:
> Hello
>
> What are the best practices when it comes to publishing a message into
> kafka. When sending a message into Kafka is it possible to know if that
> message has successfully been published? If not, what is the best practice
> to know when messages are not getting published?
>
> Second question.
>
> If I have two kafka brokers and very simplistic one kafka topic replicated
> on both. Do I need to send to both brokers. What are the best practices for
> that.
>