You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Bhavesh Mistry <mi...@gmail.com> on 2014/05/06 03:30:34 UTC
QOS on Producer Side
We are using Kafka for operation metrics and we do not want to loose any
data at all if there is issue with Network or all brokers needs to be
rebooted or operation reason down time while all producers are still
producing data on front end side. We use async to publish messages and we
are using Kafka version 0.8.0.
Has any one implemented buffering on local disk (on producer side) and
transmit messages when network connection is restored? How do I get
handle to list of messages async thread could not transfer after x reties
? I know new producer API has callback interface, but is per message not
per producer instance ?
Is this final new Producer API ?
http://empathybox.com/kafka-javadoc/index.html?kafka/clients/producer/KafkaProducer.html
is there a plan to add method like. onFailure(List<Messages> messages,
Exception exception ) ?
Basically, I have to address the QOS on producer side, and be able to
buffer on disk and retransmit all message to partitions that are reserved
for messages that happened in past.
How does Linked-in handle QOS on producer side ?
Is there any plan to add this QOS feature on Producer Side with Strategies to
store and retransmit the message ? If we do get the list of messages is
call back, will it be compressed data ?
I would appreciate Kafka Developers and others feedback on how to implement
QOS.
Thanks,
Bhavesh
Re: QOS on Producer Side
Posted by Jun Rao <ju...@gmail.com>.
The new producer should work with the 0.8.0 broker.
Thanks,
Jun
On Tue, May 6, 2014 at 11:30 AM, Bhavesh Mistry
<mi...@gmail.com>wrote:
> So to retrieve the message that was not transmitted due to error, I will
> have to store the message in each call back the message and store this in a
> file. Is there an alternative solution ?
>
> What is best way to achieve QOS on producer side ?
>
> Also, will new Producer API will work with Broker version 0.8.0 ? Please
> let me know.
>
> Thanks,
>
> Bhavesh
>
>
> On Tue, May 6, 2014 at 8:46 AM, Jun Rao <ju...@gmail.com> wrote:
>
> > The callback will be invoked on completion. The exception tells you
> whether
> > the send completed or not. This api will be available in the next
> release.
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Mon, May 5, 2014 at 10:03 PM, Bhavesh Mistry
> > <mi...@gmail.com>wrote:
> >
> > > Thanks for answers.
> > >
> > > Does the callback get call on failure only or for success as well ?
> > Also,
> > > how do I do this on Kafka 0.8.0 ? Is there any plan for adding
> > buffering
> > > on disk for next version ? Also, when application restart, kafka
> > producer
> > > will have to transmit the messages (buffered messages on disk) ?
> > >
> > > I can not upgrade to Kafka 0.8.1.1 right now. how can I achieve this
> on
> > > Kafka version 0.8.0. Also, when retransmitting the messages I wanted
> to
> > > transfer dedicated Kafka back-up or reserved partition for topic
> > (near-real
> > > time topic).
> > >
> > > How would you handle retransmission of messages ?
> > >
> > > Any idea about implementation will be very helpful.
> > >
> > > Thanks,
> > >
> > > Bhavesh
> > >
> > > On Mon, May 5, 2014 at 9:29 PM, Jun Rao <ju...@gmail.com> wrote:
> > >
> > > > At LinkedIn, we only buffer messages in memory in the producer. We
> try
> > as
> > > > much as we can to make the Kafka cluster always available (with
> > > > replication). The "on failure" feature you mentioned can be
> implemented
> > > by
> > > > providing a callback. You can see the example in the javadoc.
> > > >
> > > > *send
> > > > <
> > > >
> > >
> >
> http://empathybox.com/kafka-javadoc/kafka/clients/producer/KafkaProducer.html#send(kafka.clients.producer.ProducerRecord
> > > > ,
> > > > kafka.clients.producer.Callback)>*(ProducerRecord<
> > > >
> > >
> >
> http://empathybox.com/kafka-javadoc/kafka/clients/producer/ProducerRecord.html
> > > > >
> > > > record, Callback<
> > > >
> > http://empathybox.com/kafka-javadoc/kafka/clients/producer/Callback.html
> > > >
> > > > callback)
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > > On Mon, May 5, 2014 at 6:30 PM, Bhavesh Mistry
> > > > <mi...@gmail.com>wrote:
> > > >
> > > > > We are using Kafka for operation metrics and we do not want to
> loose
> > > any
> > > > > data at all if there is issue with Network or all brokers needs to
> be
> > > > > rebooted or operation reason down time while all producers are
> still
> > > > > producing data on front end side. We use async to publish messages
> > and
> > > > we
> > > > > are using Kafka version 0.8.0.
> > > > >
> > > > >
> > > > >
> > > > > Has any one implemented buffering on local disk (on producer side)
> > and
> > > > > transmit messages when network connection is restored? How do I
> get
> > > > > handle to list of messages async thread could not transfer after x
> > > reties
> > > > > ? I know new producer API has callback interface, but is per
> message
> > > not
> > > > > per producer instance ?
> > > > >
> > > > >
> > > > >
> > > > > Is this final new Producer API ?
> > > > >
> > > > >
> > > >
> > >
> >
> http://empathybox.com/kafka-javadoc/index.html?kafka/clients/producer/KafkaProducer.html
> > > > >
> > > > >
> > > > > is there a plan to add method like. onFailure(List<Messages>
> > messages,
> > > > > Exception exception ) ?
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > Basically, I have to address the QOS on producer side, and be able
> to
> > > > > buffer on disk and retransmit all message to partitions that are
> > > reserved
> > > > > for messages that happened in past.
> > > > >
> > > > >
> > > > >
> > > > > How does Linked-in handle QOS on producer side ?
> > > > >
> > > > >
> > > > > Is there any plan to add this QOS feature on Producer Side with
> > > > Strategies
> > > > > to
> > > > > store and retransmit the message ? If we do get the list of
> messages
> > > is
> > > > > call back, will it be compressed data ?
> > > > >
> > > > >
> > > > >
> > > > > I would appreciate Kafka Developers and others feedback on how to
> > > > implement
> > > > > QOS.
> > > > >
> > > > >
> > > > > Thanks,
> > > > >
> > > > >
> > > > > Bhavesh
> > > > >
> > > >
> > >
> >
>
Re: QOS on Producer Side
Posted by Bhavesh Mistry <mi...@gmail.com>.
So to retrieve the message that was not transmitted due to error, I will
have to store the message in each call back the message and store this in a
file. Is there an alternative solution ?
What is best way to achieve QOS on producer side ?
Also, will new Producer API will work with Broker version 0.8.0 ? Please
let me know.
Thanks,
Bhavesh
On Tue, May 6, 2014 at 8:46 AM, Jun Rao <ju...@gmail.com> wrote:
> The callback will be invoked on completion. The exception tells you whether
> the send completed or not. This api will be available in the next release.
>
> Thanks,
>
> Jun
>
>
> On Mon, May 5, 2014 at 10:03 PM, Bhavesh Mistry
> <mi...@gmail.com>wrote:
>
> > Thanks for answers.
> >
> > Does the callback get call on failure only or for success as well ?
> Also,
> > how do I do this on Kafka 0.8.0 ? Is there any plan for adding
> buffering
> > on disk for next version ? Also, when application restart, kafka
> producer
> > will have to transmit the messages (buffered messages on disk) ?
> >
> > I can not upgrade to Kafka 0.8.1.1 right now. how can I achieve this on
> > Kafka version 0.8.0. Also, when retransmitting the messages I wanted to
> > transfer dedicated Kafka back-up or reserved partition for topic
> (near-real
> > time topic).
> >
> > How would you handle retransmission of messages ?
> >
> > Any idea about implementation will be very helpful.
> >
> > Thanks,
> >
> > Bhavesh
> >
> > On Mon, May 5, 2014 at 9:29 PM, Jun Rao <ju...@gmail.com> wrote:
> >
> > > At LinkedIn, we only buffer messages in memory in the producer. We try
> as
> > > much as we can to make the Kafka cluster always available (with
> > > replication). The "on failure" feature you mentioned can be implemented
> > by
> > > providing a callback. You can see the example in the javadoc.
> > >
> > > *send
> > > <
> > >
> >
> http://empathybox.com/kafka-javadoc/kafka/clients/producer/KafkaProducer.html#send(kafka.clients.producer.ProducerRecord
> > > ,
> > > kafka.clients.producer.Callback)>*(ProducerRecord<
> > >
> >
> http://empathybox.com/kafka-javadoc/kafka/clients/producer/ProducerRecord.html
> > > >
> > > record, Callback<
> > >
> http://empathybox.com/kafka-javadoc/kafka/clients/producer/Callback.html
> > >
> > > callback)
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Mon, May 5, 2014 at 6:30 PM, Bhavesh Mistry
> > > <mi...@gmail.com>wrote:
> > >
> > > > We are using Kafka for operation metrics and we do not want to loose
> > any
> > > > data at all if there is issue with Network or all brokers needs to be
> > > > rebooted or operation reason down time while all producers are still
> > > > producing data on front end side. We use async to publish messages
> and
> > > we
> > > > are using Kafka version 0.8.0.
> > > >
> > > >
> > > >
> > > > Has any one implemented buffering on local disk (on producer side)
> and
> > > > transmit messages when network connection is restored? How do I get
> > > > handle to list of messages async thread could not transfer after x
> > reties
> > > > ? I know new producer API has callback interface, but is per message
> > not
> > > > per producer instance ?
> > > >
> > > >
> > > >
> > > > Is this final new Producer API ?
> > > >
> > > >
> > >
> >
> http://empathybox.com/kafka-javadoc/index.html?kafka/clients/producer/KafkaProducer.html
> > > >
> > > >
> > > > is there a plan to add method like. onFailure(List<Messages>
> messages,
> > > > Exception exception ) ?
> > > >
> > > >
> > > >
> > > >
> > > > Basically, I have to address the QOS on producer side, and be able to
> > > > buffer on disk and retransmit all message to partitions that are
> > reserved
> > > > for messages that happened in past.
> > > >
> > > >
> > > >
> > > > How does Linked-in handle QOS on producer side ?
> > > >
> > > >
> > > > Is there any plan to add this QOS feature on Producer Side with
> > > Strategies
> > > > to
> > > > store and retransmit the message ? If we do get the list of messages
> > is
> > > > call back, will it be compressed data ?
> > > >
> > > >
> > > >
> > > > I would appreciate Kafka Developers and others feedback on how to
> > > implement
> > > > QOS.
> > > >
> > > >
> > > > Thanks,
> > > >
> > > >
> > > > Bhavesh
> > > >
> > >
> >
>
Re: QOS on Producer Side
Posted by Bhavesh Mistry <mi...@gmail.com>.
So to retrieve the message that was not transmitted due to error, I will
have to store the message in each call back the message and store this in a
file. Is there an alternative solution ?
What is best way to achieve QOS on producer side ?
Also, will new Producer API will work with Broker version 0.8.0 ? Please
let me know.
Thanks,
Bhavesh
On Tue, May 6, 2014 at 8:46 AM, Jun Rao <ju...@gmail.com> wrote:
> The callback will be invoked on completion. The exception tells you whether
> the send completed or not. This api will be available in the next release.
>
> Thanks,
>
> Jun
>
>
> On Mon, May 5, 2014 at 10:03 PM, Bhavesh Mistry
> <mi...@gmail.com>wrote:
>
> > Thanks for answers.
> >
> > Does the callback get call on failure only or for success as well ?
> Also,
> > how do I do this on Kafka 0.8.0 ? Is there any plan for adding
> buffering
> > on disk for next version ? Also, when application restart, kafka
> producer
> > will have to transmit the messages (buffered messages on disk) ?
> >
> > I can not upgrade to Kafka 0.8.1.1 right now. how can I achieve this on
> > Kafka version 0.8.0. Also, when retransmitting the messages I wanted to
> > transfer dedicated Kafka back-up or reserved partition for topic
> (near-real
> > time topic).
> >
> > How would you handle retransmission of messages ?
> >
> > Any idea about implementation will be very helpful.
> >
> > Thanks,
> >
> > Bhavesh
> >
> > On Mon, May 5, 2014 at 9:29 PM, Jun Rao <ju...@gmail.com> wrote:
> >
> > > At LinkedIn, we only buffer messages in memory in the producer. We try
> as
> > > much as we can to make the Kafka cluster always available (with
> > > replication). The "on failure" feature you mentioned can be implemented
> > by
> > > providing a callback. You can see the example in the javadoc.
> > >
> > > *send
> > > <
> > >
> >
> http://empathybox.com/kafka-javadoc/kafka/clients/producer/KafkaProducer.html#send(kafka.clients.producer.ProducerRecord
> > > ,
> > > kafka.clients.producer.Callback)>*(ProducerRecord<
> > >
> >
> http://empathybox.com/kafka-javadoc/kafka/clients/producer/ProducerRecord.html
> > > >
> > > record, Callback<
> > >
> http://empathybox.com/kafka-javadoc/kafka/clients/producer/Callback.html
> > >
> > > callback)
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Mon, May 5, 2014 at 6:30 PM, Bhavesh Mistry
> > > <mi...@gmail.com>wrote:
> > >
> > > > We are using Kafka for operation metrics and we do not want to loose
> > any
> > > > data at all if there is issue with Network or all brokers needs to be
> > > > rebooted or operation reason down time while all producers are still
> > > > producing data on front end side. We use async to publish messages
> and
> > > we
> > > > are using Kafka version 0.8.0.
> > > >
> > > >
> > > >
> > > > Has any one implemented buffering on local disk (on producer side)
> and
> > > > transmit messages when network connection is restored? How do I get
> > > > handle to list of messages async thread could not transfer after x
> > reties
> > > > ? I know new producer API has callback interface, but is per message
> > not
> > > > per producer instance ?
> > > >
> > > >
> > > >
> > > > Is this final new Producer API ?
> > > >
> > > >
> > >
> >
> http://empathybox.com/kafka-javadoc/index.html?kafka/clients/producer/KafkaProducer.html
> > > >
> > > >
> > > > is there a plan to add method like. onFailure(List<Messages>
> messages,
> > > > Exception exception ) ?
> > > >
> > > >
> > > >
> > > >
> > > > Basically, I have to address the QOS on producer side, and be able to
> > > > buffer on disk and retransmit all message to partitions that are
> > reserved
> > > > for messages that happened in past.
> > > >
> > > >
> > > >
> > > > How does Linked-in handle QOS on producer side ?
> > > >
> > > >
> > > > Is there any plan to add this QOS feature on Producer Side with
> > > Strategies
> > > > to
> > > > store and retransmit the message ? If we do get the list of messages
> > is
> > > > call back, will it be compressed data ?
> > > >
> > > >
> > > >
> > > > I would appreciate Kafka Developers and others feedback on how to
> > > implement
> > > > QOS.
> > > >
> > > >
> > > > Thanks,
> > > >
> > > >
> > > > Bhavesh
> > > >
> > >
> >
>
Re: QOS on Producer Side
Posted by Jun Rao <ju...@gmail.com>.
The callback will be invoked on completion. The exception tells you whether
the send completed or not. This api will be available in the next release.
Thanks,
Jun
On Mon, May 5, 2014 at 10:03 PM, Bhavesh Mistry
<mi...@gmail.com>wrote:
> Thanks for answers.
>
> Does the callback get call on failure only or for success as well ? Also,
> how do I do this on Kafka 0.8.0 ? Is there any plan for adding buffering
> on disk for next version ? Also, when application restart, kafka producer
> will have to transmit the messages (buffered messages on disk) ?
>
> I can not upgrade to Kafka 0.8.1.1 right now. how can I achieve this on
> Kafka version 0.8.0. Also, when retransmitting the messages I wanted to
> transfer dedicated Kafka back-up or reserved partition for topic (near-real
> time topic).
>
> How would you handle retransmission of messages ?
>
> Any idea about implementation will be very helpful.
>
> Thanks,
>
> Bhavesh
>
> On Mon, May 5, 2014 at 9:29 PM, Jun Rao <ju...@gmail.com> wrote:
>
> > At LinkedIn, we only buffer messages in memory in the producer. We try as
> > much as we can to make the Kafka cluster always available (with
> > replication). The "on failure" feature you mentioned can be implemented
> by
> > providing a callback. You can see the example in the javadoc.
> >
> > *send
> > <
> >
> http://empathybox.com/kafka-javadoc/kafka/clients/producer/KafkaProducer.html#send(kafka.clients.producer.ProducerRecord
> > ,
> > kafka.clients.producer.Callback)>*(ProducerRecord<
> >
> http://empathybox.com/kafka-javadoc/kafka/clients/producer/ProducerRecord.html
> > >
> > record, Callback<
> > http://empathybox.com/kafka-javadoc/kafka/clients/producer/Callback.html
> >
> > callback)
> >
> > Thanks,
> >
> > Jun
> >
> > On Mon, May 5, 2014 at 6:30 PM, Bhavesh Mistry
> > <mi...@gmail.com>wrote:
> >
> > > We are using Kafka for operation metrics and we do not want to loose
> any
> > > data at all if there is issue with Network or all brokers needs to be
> > > rebooted or operation reason down time while all producers are still
> > > producing data on front end side. We use async to publish messages and
> > we
> > > are using Kafka version 0.8.0.
> > >
> > >
> > >
> > > Has any one implemented buffering on local disk (on producer side) and
> > > transmit messages when network connection is restored? How do I get
> > > handle to list of messages async thread could not transfer after x
> reties
> > > ? I know new producer API has callback interface, but is per message
> not
> > > per producer instance ?
> > >
> > >
> > >
> > > Is this final new Producer API ?
> > >
> > >
> >
> http://empathybox.com/kafka-javadoc/index.html?kafka/clients/producer/KafkaProducer.html
> > >
> > >
> > > is there a plan to add method like. onFailure(List<Messages> messages,
> > > Exception exception ) ?
> > >
> > >
> > >
> > >
> > > Basically, I have to address the QOS on producer side, and be able to
> > > buffer on disk and retransmit all message to partitions that are
> reserved
> > > for messages that happened in past.
> > >
> > >
> > >
> > > How does Linked-in handle QOS on producer side ?
> > >
> > >
> > > Is there any plan to add this QOS feature on Producer Side with
> > Strategies
> > > to
> > > store and retransmit the message ? If we do get the list of messages
> is
> > > call back, will it be compressed data ?
> > >
> > >
> > >
> > > I would appreciate Kafka Developers and others feedback on how to
> > implement
> > > QOS.
> > >
> > >
> > > Thanks,
> > >
> > >
> > > Bhavesh
> > >
> >
>
Re: QOS on Producer Side
Posted by Bhavesh Mistry <mi...@gmail.com>.
Thanks for answers.
Does the callback get call on failure only or for success as well ? Also,
how do I do this on Kafka 0.8.0 ? Is there any plan for adding buffering
on disk for next version ? Also, when application restart, kafka producer
will have to transmit the messages (buffered messages on disk) ?
I can not upgrade to Kafka 0.8.1.1 right now. how can I achieve this on
Kafka version 0.8.0. Also, when retransmitting the messages I wanted to
transfer dedicated Kafka back-up or reserved partition for topic (near-real
time topic).
How would you handle retransmission of messages ?
Any idea about implementation will be very helpful.
Thanks,
Bhavesh
On Mon, May 5, 2014 at 9:29 PM, Jun Rao <ju...@gmail.com> wrote:
> At LinkedIn, we only buffer messages in memory in the producer. We try as
> much as we can to make the Kafka cluster always available (with
> replication). The "on failure" feature you mentioned can be implemented by
> providing a callback. You can see the example in the javadoc.
>
> *send
> <
> http://empathybox.com/kafka-javadoc/kafka/clients/producer/KafkaProducer.html#send(kafka.clients.producer.ProducerRecord
> ,
> kafka.clients.producer.Callback)>*(ProducerRecord<
> http://empathybox.com/kafka-javadoc/kafka/clients/producer/ProducerRecord.html
> >
> record, Callback<
> http://empathybox.com/kafka-javadoc/kafka/clients/producer/Callback.html>
> callback)
>
> Thanks,
>
> Jun
>
> On Mon, May 5, 2014 at 6:30 PM, Bhavesh Mistry
> <mi...@gmail.com>wrote:
>
> > We are using Kafka for operation metrics and we do not want to loose any
> > data at all if there is issue with Network or all brokers needs to be
> > rebooted or operation reason down time while all producers are still
> > producing data on front end side. We use async to publish messages and
> we
> > are using Kafka version 0.8.0.
> >
> >
> >
> > Has any one implemented buffering on local disk (on producer side) and
> > transmit messages when network connection is restored? How do I get
> > handle to list of messages async thread could not transfer after x reties
> > ? I know new producer API has callback interface, but is per message not
> > per producer instance ?
> >
> >
> >
> > Is this final new Producer API ?
> >
> >
> http://empathybox.com/kafka-javadoc/index.html?kafka/clients/producer/KafkaProducer.html
> >
> >
> > is there a plan to add method like. onFailure(List<Messages> messages,
> > Exception exception ) ?
> >
> >
> >
> >
> > Basically, I have to address the QOS on producer side, and be able to
> > buffer on disk and retransmit all message to partitions that are reserved
> > for messages that happened in past.
> >
> >
> >
> > How does Linked-in handle QOS on producer side ?
> >
> >
> > Is there any plan to add this QOS feature on Producer Side with
> Strategies
> > to
> > store and retransmit the message ? If we do get the list of messages is
> > call back, will it be compressed data ?
> >
> >
> >
> > I would appreciate Kafka Developers and others feedback on how to
> implement
> > QOS.
> >
> >
> > Thanks,
> >
> >
> > Bhavesh
> >
>
Re: QOS on Producer Side
Posted by Jun Rao <ju...@gmail.com>.
At LinkedIn, we only buffer messages in memory in the producer. We try as
much as we can to make the Kafka cluster always available (with
replication). The "on failure" feature you mentioned can be implemented by
providing a callback. You can see the example in the javadoc.
*send
<http://empathybox.com/kafka-javadoc/kafka/clients/producer/KafkaProducer.html#send(kafka.clients.producer.ProducerRecord,
kafka.clients.producer.Callback)>*(ProducerRecord<http://empathybox.com/kafka-javadoc/kafka/clients/producer/ProducerRecord.html>
record, Callback<http://empathybox.com/kafka-javadoc/kafka/clients/producer/Callback.html>
callback)
Thanks,
Jun
On Mon, May 5, 2014 at 6:30 PM, Bhavesh Mistry
<mi...@gmail.com>wrote:
> We are using Kafka for operation metrics and we do not want to loose any
> data at all if there is issue with Network or all brokers needs to be
> rebooted or operation reason down time while all producers are still
> producing data on front end side. We use async to publish messages and we
> are using Kafka version 0.8.0.
>
>
>
> Has any one implemented buffering on local disk (on producer side) and
> transmit messages when network connection is restored? How do I get
> handle to list of messages async thread could not transfer after x reties
> ? I know new producer API has callback interface, but is per message not
> per producer instance ?
>
>
>
> Is this final new Producer API ?
>
> http://empathybox.com/kafka-javadoc/index.html?kafka/clients/producer/KafkaProducer.html
>
>
> is there a plan to add method like. onFailure(List<Messages> messages,
> Exception exception ) ?
>
>
>
>
> Basically, I have to address the QOS on producer side, and be able to
> buffer on disk and retransmit all message to partitions that are reserved
> for messages that happened in past.
>
>
>
> How does Linked-in handle QOS on producer side ?
>
>
> Is there any plan to add this QOS feature on Producer Side with Strategies
> to
> store and retransmit the message ? If we do get the list of messages is
> call back, will it be compressed data ?
>
>
>
> I would appreciate Kafka Developers and others feedback on how to implement
> QOS.
>
>
> Thanks,
>
>
> Bhavesh
>