You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Demian Berjman <db...@despegar.com> on 2013/11/28 15:37:33 UTC

kafka producer - retry messages

Hi.

Anyone has build a retry system (durable) for the producer in case the
kakfa cluster is down? We have certain messages that must be sent because
there are after a transaction that cannot be undo.

We could set the property "message.send.max.retries", but if the producer
goes down, we lost the messages.

Thanks,

Re: kafka producer - retry messages

Posted by Steve Morin <st...@stevemorin.com>.
I would say follow Sparkngin and you'll be able to use the  "Log
persistence if the log producer connection is down". functionality when
it's complete.


On Thu, Nov 28, 2013 at 8:36 AM, Demian Berjman <db...@despegar.com>wrote:

> Steve, our use case is very simple. There are many reasons for a cluster to
> go down. If that the case, what we do with the producers? Hopefully it will
> be a time window of a couple of hours. If your concern are the queued
> messages, we have only a few thousands per day.
>
> Thanks,
>
>
> On Thu, Nov 28, 2013 at 1:12 PM, Steve Morin <st...@stevemorin.com> wrote:
>
> > What I mean by that is that your looking to have the Kafka cluster able
> to
> > be down for like 5 minutes or upto a day.  The problem is estimating how
> > long it will take to recover.
> >
> > Is this work your doing for a consulting project?  Or are you doing
> > something on behalf of an employer.  Basically would like to know more
> > about the use-case.  You can email me directly at steve@stevemorin.comso
> > we don't clog the message board.
> >
> >
> > On Thu, Nov 28, 2013 at 8:05 AM, Demian Berjman <dberjman@despegar.com
> > >wrote:
> >
> > > Steve, thanks for the response! I don't understand what you mean by
> "what
> > > kind of window?"
> > >
> > > I am looking for something like, i think you did it in Sparkngin: "Log
> > > persistence if the log producer connection is down".
> > >
> >
>

Re: kafka producer - retry messages

Posted by Demian Berjman <db...@despegar.com>.
Steve, our use case is very simple. There are many reasons for a cluster to
go down. If that the case, what we do with the producers? Hopefully it will
be a time window of a couple of hours. If your concern are the queued
messages, we have only a few thousands per day.

Thanks,


On Thu, Nov 28, 2013 at 1:12 PM, Steve Morin <st...@stevemorin.com> wrote:

> What I mean by that is that your looking to have the Kafka cluster able to
> be down for like 5 minutes or upto a day.  The problem is estimating how
> long it will take to recover.
>
> Is this work your doing for a consulting project?  Or are you doing
> something on behalf of an employer.  Basically would like to know more
> about the use-case.  You can email me directly at steve@stevemorin.com so
> we don't clog the message board.
>
>
> On Thu, Nov 28, 2013 at 8:05 AM, Demian Berjman <dberjman@despegar.com
> >wrote:
>
> > Steve, thanks for the response! I don't understand what you mean by "what
> > kind of window?"
> >
> > I am looking for something like, i think you did it in Sparkngin: "Log
> > persistence if the log producer connection is down".
> >
>

Re: kafka producer - retry messages

Posted by Steve Morin <st...@stevemorin.com>.
What I mean by that is that your looking to have the Kafka cluster able to
be down for like 5 minutes or upto a day.  The problem is estimating how
long it will take to recover.

Is this work your doing for a consulting project?  Or are you doing
something on behalf of an employer.  Basically would like to know more
about the use-case.  You can email me directly at steve@stevemorin.com so
we don't clog the message board.


On Thu, Nov 28, 2013 at 8:05 AM, Demian Berjman <db...@despegar.com>wrote:

> Steve, thanks for the response! I don't understand what you mean by "what
> kind of window?"
>
> I am looking for something like, i think you did it in Sparkngin: "Log
> persistence if the log producer connection is down".
>

Re: kafka producer - retry messages

Posted by Demian Berjman <db...@despegar.com>.
Steve, thanks for the response! I don't understand what you mean by "what
kind of window?"

I am looking for something like, i think you did it in Sparkngin: "Log
persistence if the log producer connection is down".

Re: kafka producer - retry messages

Posted by Steve Morin <st...@stevemorin.com>.
Demian,
  I have been looking at building that into Sparkngin (
https://github.com/DemandCube/Sparkngin)  What kind of window are you
looking for?
-Steve



On Thu, Nov 28, 2013 at 7:23 AM, Demian Berjman <db...@despegar.com>wrote:

> Joe, i meant that all the kafka cluster is down, even the replicas of that
> topic.
>
> Thanks,
>
>
> On Thu, Nov 28, 2013 at 12:10 PM, Joe Stein <jo...@stealth.ly> wrote:
>
> > Have you tried the replication feature added to 0.8
> > http://kafka.apache.org/documentation.html#replication
> >
> >
> > /*******************************************
> >  Joe Stein
> >  Founder, Principal Consultant
> >  Big Data Open Source Security LLC
> >  http://www.stealth.ly
> >  Twitter: @allthingshadoop
> > ********************************************/
> >
> >
> > On Nov 28, 2013, at 9:37 AM, Demian Berjman <db...@despegar.com>
> wrote:
> >
> > > Hi.
> > >
> > > Anyone has build a retry system (durable) for the producer in case the
> > > kakfa cluster is down? We have certain messages that must be sent
> because
> > > there are after a transaction that cannot be undo.
> > >
> > > We could set the property "message.send.max.retries", but if the
> producer
> > > goes down, we lost the messages.
> > >
> > > Thanks,
> >
>

Re: kafka producer - retry messages

Posted by Demian Berjman <db...@despegar.com>.
Joe, i meant that all the kafka cluster is down, even the replicas of that
topic.

Thanks,


On Thu, Nov 28, 2013 at 12:10 PM, Joe Stein <jo...@stealth.ly> wrote:

> Have you tried the replication feature added to 0.8
> http://kafka.apache.org/documentation.html#replication
>
>
> /*******************************************
>  Joe Stein
>  Founder, Principal Consultant
>  Big Data Open Source Security LLC
>  http://www.stealth.ly
>  Twitter: @allthingshadoop
> ********************************************/
>
>
> On Nov 28, 2013, at 9:37 AM, Demian Berjman <db...@despegar.com> wrote:
>
> > Hi.
> >
> > Anyone has build a retry system (durable) for the producer in case the
> > kakfa cluster is down? We have certain messages that must be sent because
> > there are after a transaction that cannot be undo.
> >
> > We could set the property "message.send.max.retries", but if the producer
> > goes down, we lost the messages.
> >
> > Thanks,
>

Re: kafka producer - retry messages

Posted by Joe Stein <jo...@stealth.ly>.
Have you tried the replication feature added to 0.8 http://kafka.apache.org/documentation.html#replication


/*******************************************
 Joe Stein
 Founder, Principal Consultant
 Big Data Open Source Security LLC
 http://www.stealth.ly
 Twitter: @allthingshadoop
********************************************/


On Nov 28, 2013, at 9:37 AM, Demian Berjman <db...@despegar.com> wrote:

> Hi.
> 
> Anyone has build a retry system (durable) for the producer in case the
> kakfa cluster is down? We have certain messages that must be sent because
> there are after a transaction that cannot be undo.
> 
> We could set the property "message.send.max.retries", but if the producer
> goes down, we lost the messages.
> 
> Thanks,

Re: kafka producer - retry messages

Posted by Philip O'Toole <ph...@loggly.com>.
Not exactly. But we do pay an enormous amount of attention to our producer-Kafka-consumer subsystems. They are certainly mission-critical for us. 

Philip

> On Nov 28, 2013, at 4:12 PM, Steve Morin <st...@gmail.com> wrote:
> 
> Philip,
>  Do you do that at loggly?
> 
> Otis,
>  How was your retry code structured?  Have you open sourced it?
> 
>> On Nov 28, 2013, at 16:08, Philip O'Toole <ph...@loggly.com> wrote:
>> 
>> By FS I guess you mean file system. 
>> 
>> In that case, if one is that concerned, why not run a single Kafka broker on the same machine, and connect to it over localhost? And disable ZK mode too, perhaps. 
>> 
>> I may be missing something, but I never fully understand why people try really hard to build a stream-to-disk backup approach, when they might be able to couple tightly to Kafka, which, well, just streams to disk. 
>> 
>> Philip
>> 
>>> On Nov 28, 2013, at 3:58 PM, Otis Gospodnetic <ot...@gmail.com> wrote:
>>> 
>>> Hi,
>>> 
>>> We've done this at Sematext, where we use Kafka in all 3 products/services
>>> you see in my signature.  When we fail to push a message into Kafka we
>>> store it in the FS and from there we can process it later.
>>> 
>>> Otis
>>> --
>>> Performance Monitoring * Log Analytics * Search Analytics
>>> Solr & Elasticsearch Support * http://sematext.com/
>>> 
>>> 
>>> On Thu, Nov 28, 2013 at 9:37 AM, Demian Berjman <db...@despegar.com>wrote:
>>> 
>>>> Hi.
>>>> 
>>>> Anyone has build a retry system (durable) for the producer in case the
>>>> kakfa cluster is down? We have certain messages that must be sent because
>>>> there are after a transaction that cannot be undo.
>>>> 
>>>> We could set the property "message.send.max.retries", but if the producer
>>>> goes down, we lost the messages.
>>>> 
>>>> Thanks,
>>>> 

Re: kafka producer - retry messages

Posted by Steve Morin <st...@gmail.com>.
Philip,
  Do you do that at loggly?

Otis,
  How was your retry code structured?  Have you open sourced it?

> On Nov 28, 2013, at 16:08, Philip O'Toole <ph...@loggly.com> wrote:
> 
> By FS I guess you mean file system. 
> 
> In that case, if one is that concerned, why not run a single Kafka broker on the same machine, and connect to it over localhost? And disable ZK mode too, perhaps. 
> 
> I may be missing something, but I never fully understand why people try really hard to build a stream-to-disk backup approach, when they might be able to couple tightly to Kafka, which, well, just streams to disk. 
> 
> Philip
> 
>> On Nov 28, 2013, at 3:58 PM, Otis Gospodnetic <ot...@gmail.com> wrote:
>> 
>> Hi,
>> 
>> We've done this at Sematext, where we use Kafka in all 3 products/services
>> you see in my signature.  When we fail to push a message into Kafka we
>> store it in the FS and from there we can process it later.
>> 
>> Otis
>> --
>> Performance Monitoring * Log Analytics * Search Analytics
>> Solr & Elasticsearch Support * http://sematext.com/
>> 
>> 
>> On Thu, Nov 28, 2013 at 9:37 AM, Demian Berjman <db...@despegar.com>wrote:
>> 
>>> Hi.
>>> 
>>> Anyone has build a retry system (durable) for the producer in case the
>>> kakfa cluster is down? We have certain messages that must be sent because
>>> there are after a transaction that cannot be undo.
>>> 
>>> We could set the property "message.send.max.retries", but if the producer
>>> goes down, we lost the messages.
>>> 
>>> Thanks,
>>> 

Re: kafka producer - retry messages

Posted by Philip O'Toole <ph...@loggly.com>.
There are many options. Another simple consumer could read from it to a second broker. 

Philip

> On Nov 28, 2013, at 4:18 PM, Steve Morin <st...@gmail.com> wrote:
> 
> Philip,
>  How would do you mirror this to a main Kafka instance?
> -Steve
> 
>> On Nov 28, 2013, at 16:14, Philip O'Toole <ph...@loggly.com> wrote:
>> 
>> I should add in our custom producers we buffer in RAM if required, so Kafka can be restarted etc. But I would never code streaming to disk now. I would just run a Kafka instance on the same node. 
>> 
>> Philip
>> 
>>> On Nov 28, 2013, at 4:08 PM, Philip O'Toole <ph...@loggly.com> wrote:
>>> 
>>> By FS I guess you mean file system. 
>>> 
>>> In that case, if one is that concerned, why not run a single Kafka broker on the same machine, and connect to it over localhost? And disable ZK mode too, perhaps. 
>>> 
>>> I may be missing something, but I never fully understand why people try really hard to build a stream-to-disk backup approach, when they might be able to couple tightly to Kafka, which, well, just streams to disk. 
>>> Philip
>>> 
>>>> On Nov 28, 2013, at 3:58 PM, Otis Gospodnetic <ot...@gmail.com> wrote:
>>>> 
>>>> Hi,
>>>> 
>>>> We've done this at Sematext, where we use Kafka in all 3 products/services
>>>> you see in my signature.  When we fail to push a message into Kafka we
>>>> store it in the FS and from there we can process it later.
>>>> 
>>>> Otis
>>>> --
>>>> Performance Monitoring * Log Analytics * Search Analytics
>>>> Solr & Elasticsearch Support * http://sematext.com/
>>>> 
>>>> 
>>>> On Thu, Nov 28, 2013 at 9:37 AM, Demian Berjman <db...@despegar.com>wrote:
>>>> 
>>>>> Hi.
>>>>> 
>>>>> Anyone has build a retry system (durable) for the producer in case the
>>>>> kakfa cluster is down? We have certain messages that must be sent because
>>>>> there are after a transaction that cannot be undo.
>>>>> 
>>>>> We could set the property "message.send.max.retries", but if the producer
>>>>> goes down, we lost the messages.
>>>>> 
>>>>> Thanks,
>>>>> 

Re: kafka producer - retry messages

Posted by Steve Morin <st...@gmail.com>.
Philip,
  How would do you mirror this to a main Kafka instance?
-Steve

> On Nov 28, 2013, at 16:14, Philip O'Toole <ph...@loggly.com> wrote:
> 
> I should add in our custom producers we buffer in RAM if required, so Kafka can be restarted etc. But I would never code streaming to disk now. I would just run a Kafka instance on the same node. 
> 
> Philip
> 
>> On Nov 28, 2013, at 4:08 PM, Philip O'Toole <ph...@loggly.com> wrote:
>> 
>> By FS I guess you mean file system. 
>> 
>> In that case, if one is that concerned, why not run a single Kafka broker on the same machine, and connect to it over localhost? And disable ZK mode too, perhaps. 
>> 
>> I may be missing something, but I never fully understand why people try really hard to build a stream-to-disk backup approach, when they might be able to couple tightly to Kafka, which, well, just streams to disk. 
>> 
>> Philip
>> 
>>> On Nov 28, 2013, at 3:58 PM, Otis Gospodnetic <ot...@gmail.com> wrote:
>>> 
>>> Hi,
>>> 
>>> We've done this at Sematext, where we use Kafka in all 3 products/services
>>> you see in my signature.  When we fail to push a message into Kafka we
>>> store it in the FS and from there we can process it later.
>>> 
>>> Otis
>>> --
>>> Performance Monitoring * Log Analytics * Search Analytics
>>> Solr & Elasticsearch Support * http://sematext.com/
>>> 
>>> 
>>> On Thu, Nov 28, 2013 at 9:37 AM, Demian Berjman <db...@despegar.com>wrote:
>>> 
>>>> Hi.
>>>> 
>>>> Anyone has build a retry system (durable) for the producer in case the
>>>> kakfa cluster is down? We have certain messages that must be sent because
>>>> there are after a transaction that cannot be undo.
>>>> 
>>>> We could set the property "message.send.max.retries", but if the producer
>>>> goes down, we lost the messages.
>>>> 
>>>> Thanks,
>>>> 

Re: kafka producer - retry messages

Posted by Philip O'Toole <ph...@loggly.com>.
I should add in our custom producers we buffer in RAM if required, so Kafka can be restarted etc. But I would never code streaming to disk now. I would just run a Kafka instance on the same node. 

Philip

On Nov 28, 2013, at 4:08 PM, Philip O'Toole <ph...@loggly.com> wrote:

> By FS I guess you mean file system. 
> 
> In that case, if one is that concerned, why not run a single Kafka broker on the same machine, and connect to it over localhost? And disable ZK mode too, perhaps. 
> 
> I may be missing something, but I never fully understand why people try really hard to build a stream-to-disk backup approach, when they might be able to couple tightly to Kafka, which, well, just streams to disk. 
> 
> Philip
> 
> On Nov 28, 2013, at 3:58 PM, Otis Gospodnetic <ot...@gmail.com> wrote:
> 
>> Hi,
>> 
>> We've done this at Sematext, where we use Kafka in all 3 products/services
>> you see in my signature.  When we fail to push a message into Kafka we
>> store it in the FS and from there we can process it later.
>> 
>> Otis
>> --
>> Performance Monitoring * Log Analytics * Search Analytics
>> Solr & Elasticsearch Support * http://sematext.com/
>> 
>> 
>> On Thu, Nov 28, 2013 at 9:37 AM, Demian Berjman <db...@despegar.com>wrote:
>> 
>>> Hi.
>>> 
>>> Anyone has build a retry system (durable) for the producer in case the
>>> kakfa cluster is down? We have certain messages that must be sent because
>>> there are after a transaction that cannot be undo.
>>> 
>>> We could set the property "message.send.max.retries", but if the producer
>>> goes down, we lost the messages.
>>> 
>>> Thanks,
>>> 

Re: kafka producer - retry messages

Posted by Otis Gospodnetic <ot...@gmail.com>.
Hi,

Right, broker on localhost and localhost connection don't help if the
broker is actually down.  It's not only about network reachability and
such.  We write to FS (yes, file system) as "well, what is the simplest
thing that we can do and where we are least likely to hit some other
issues? Write to FS.".  Yes, much like Kafka brokers themselves, but this
helps when Kafka is down for some reason.

Otis
--
Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/


On Thu, Nov 28, 2013 at 8:04 PM, Philip O'Toole <ph...@loggly.com> wrote:

> Sure, and the disk could go bad, the machine itself could fail.
>
> My point is that my experience of Kafka 0.72 has been that it is very
> reliable. The only time I have seen it go down is when the disk underneath
> fills up. So if one is going to write all the code to stream to disk
> *efficiently* from in-process, one should trade off that cost versus
> connecting to a process over localhost, which has been shown to do a very
> good job of just that.
>
> It's fair to ask what if the broker process goes down. But it's fair to
> ask what if there is a bug in the stream-to-disk code you write? What if
> your process goes down?
>
> I am not saying I am right. Just that engineering is about trade-offs, and
> a Kafka instance running right there on the same machine, might provide the
> reliability required. But it might not. As always, YMMV.
>
> Philip
>
> > On Nov 28, 2013, at 4:39 PM, Diego Parra <di...@gmail.com> wrote:
> >
> > Philip, what about if the broker goes down?
> > I may be missing something.
> >
> > Diego.
> > El 28/11/2013 21:09, "Philip O'Toole" <ph...@loggly.com> escribió:
> >
> >> By FS I guess you mean file system.
> >>
> >> In that case, if one is that concerned, why not run a single Kafka
> broker
> >> on the same machine, and connect to it over localhost? And disable ZK
> mode
> >> too, perhaps.
> >>
> >> I may be missing something, but I never fully understand why people try
> >> really hard to build a stream-to-disk backup approach, when they might
> be
> >> able to couple tightly to Kafka, which, well, just streams to disk.
> >>
> >> Philip
> >>
> >> On Nov 28, 2013, at 3:58 PM, Otis Gospodnetic <
> otis.gospodnetic@gmail.com>
> >> wrote:
> >>
> >>> Hi,
> >>>
> >>> We've done this at Sematext, where we use Kafka in all 3
> >> products/services
> >>> you see in my signature.  When we fail to push a message into Kafka we
> >>> store it in the FS and from there we can process it later.
> >>>
> >>> Otis
> >>> --
> >>> Performance Monitoring * Log Analytics * Search Analytics
> >>> Solr & Elasticsearch Support * http://sematext.com/
> >>>
> >>>
> >>> On Thu, Nov 28, 2013 at 9:37 AM, Demian Berjman <dberjman@despegar.com
> >>> wrote:
> >>>
> >>>> Hi.
> >>>>
> >>>> Anyone has build a retry system (durable) for the producer in case the
> >>>> kakfa cluster is down? We have certain messages that must be sent
> >> because
> >>>> there are after a transaction that cannot be undo.
> >>>>
> >>>> We could set the property "message.send.max.retries", but if the
> >> producer
> >>>> goes down, we lost the messages.
> >>>>
> >>>> Thanks,
> >>
>

Re: kafka producer - retry messages

Posted by Philip O'Toole <ph...@loggly.com>.
Sure, and the disk could go bad, the machine itself could fail. 

My point is that my experience of Kafka 0.72 has been that it is very reliable. The only time I have seen it go down is when the disk underneath fills up. So if one is going to write all the code to stream to disk *efficiently* from in-process, one should trade off that cost versus connecting to a process over localhost, which has been shown to do a very good job of just that. 

It's fair to ask what if the broker process goes down. But it's fair to ask what if there is a bug in the stream-to-disk code you write? What if your process goes down?

I am not saying I am right. Just that engineering is about trade-offs, and a Kafka instance running right there on the same machine, might provide the reliability required. But it might not. As always, YMMV. 

Philip

> On Nov 28, 2013, at 4:39 PM, Diego Parra <di...@gmail.com> wrote:
> 
> Philip, what about if the broker goes down?
> I may be missing something.
> 
> Diego.
> El 28/11/2013 21:09, "Philip O'Toole" <ph...@loggly.com> escribió:
> 
>> By FS I guess you mean file system.
>> 
>> In that case, if one is that concerned, why not run a single Kafka broker
>> on the same machine, and connect to it over localhost? And disable ZK mode
>> too, perhaps.
>> 
>> I may be missing something, but I never fully understand why people try
>> really hard to build a stream-to-disk backup approach, when they might be
>> able to couple tightly to Kafka, which, well, just streams to disk.
>> 
>> Philip
>> 
>> On Nov 28, 2013, at 3:58 PM, Otis Gospodnetic <ot...@gmail.com>
>> wrote:
>> 
>>> Hi,
>>> 
>>> We've done this at Sematext, where we use Kafka in all 3
>> products/services
>>> you see in my signature.  When we fail to push a message into Kafka we
>>> store it in the FS and from there we can process it later.
>>> 
>>> Otis
>>> --
>>> Performance Monitoring * Log Analytics * Search Analytics
>>> Solr & Elasticsearch Support * http://sematext.com/
>>> 
>>> 
>>> On Thu, Nov 28, 2013 at 9:37 AM, Demian Berjman <dberjman@despegar.com
>>> wrote:
>>> 
>>>> Hi.
>>>> 
>>>> Anyone has build a retry system (durable) for the producer in case the
>>>> kakfa cluster is down? We have certain messages that must be sent
>> because
>>>> there are after a transaction that cannot be undo.
>>>> 
>>>> We could set the property "message.send.max.retries", but if the
>> producer
>>>> goes down, we lost the messages.
>>>> 
>>>> Thanks,
>> 

Re: kafka producer - retry messages

Posted by Diego Parra <di...@gmail.com>.
Philip, what about if the broker goes down?
I may be missing something.

Diego.
 El 28/11/2013 21:09, "Philip O'Toole" <ph...@loggly.com> escribió:

> By FS I guess you mean file system.
>
> In that case, if one is that concerned, why not run a single Kafka broker
> on the same machine, and connect to it over localhost? And disable ZK mode
> too, perhaps.
>
> I may be missing something, but I never fully understand why people try
> really hard to build a stream-to-disk backup approach, when they might be
> able to couple tightly to Kafka, which, well, just streams to disk.
>
> Philip
>
> On Nov 28, 2013, at 3:58 PM, Otis Gospodnetic <ot...@gmail.com>
> wrote:
>
> > Hi,
> >
> > We've done this at Sematext, where we use Kafka in all 3
> products/services
> > you see in my signature.  When we fail to push a message into Kafka we
> > store it in the FS and from there we can process it later.
> >
> > Otis
> > --
> > Performance Monitoring * Log Analytics * Search Analytics
> > Solr & Elasticsearch Support * http://sematext.com/
> >
> >
> > On Thu, Nov 28, 2013 at 9:37 AM, Demian Berjman <dberjman@despegar.com
> >wrote:
> >
> >> Hi.
> >>
> >> Anyone has build a retry system (durable) for the producer in case the
> >> kakfa cluster is down? We have certain messages that must be sent
> because
> >> there are after a transaction that cannot be undo.
> >>
> >> We could set the property "message.send.max.retries", but if the
> producer
> >> goes down, we lost the messages.
> >>
> >> Thanks,
> >>
>

Re: kafka producer - retry messages

Posted by Philip O'Toole <ph...@loggly.com>.
By FS I guess you mean file system. 

In that case, if one is that concerned, why not run a single Kafka broker on the same machine, and connect to it over localhost? And disable ZK mode too, perhaps. 

I may be missing something, but I never fully understand why people try really hard to build a stream-to-disk backup approach, when they might be able to couple tightly to Kafka, which, well, just streams to disk. 

Philip

On Nov 28, 2013, at 3:58 PM, Otis Gospodnetic <ot...@gmail.com> wrote:

> Hi,
> 
> We've done this at Sematext, where we use Kafka in all 3 products/services
> you see in my signature.  When we fail to push a message into Kafka we
> store it in the FS and from there we can process it later.
> 
> Otis
> --
> Performance Monitoring * Log Analytics * Search Analytics
> Solr & Elasticsearch Support * http://sematext.com/
> 
> 
> On Thu, Nov 28, 2013 at 9:37 AM, Demian Berjman <db...@despegar.com>wrote:
> 
>> Hi.
>> 
>> Anyone has build a retry system (durable) for the producer in case the
>> kakfa cluster is down? We have certain messages that must be sent because
>> there are after a transaction that cannot be undo.
>> 
>> We could set the property "message.send.max.retries", but if the producer
>> goes down, we lost the messages.
>> 
>> Thanks,
>> 

Re: kafka producer - retry messages

Posted by Otis Gospodnetic <ot...@gmail.com>.
Hi,

We've done this at Sematext, where we use Kafka in all 3 products/services
you see in my signature.  When we fail to push a message into Kafka we
store it in the FS and from there we can process it later.

Otis
--
Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/


On Thu, Nov 28, 2013 at 9:37 AM, Demian Berjman <db...@despegar.com>wrote:

> Hi.
>
> Anyone has build a retry system (durable) for the producer in case the
> kakfa cluster is down? We have certain messages that must be sent because
> there are after a transaction that cannot be undo.
>
> We could set the property "message.send.max.retries", but if the producer
> goes down, we lost the messages.
>
> Thanks,
>