You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Christian Csar <ca...@gmail.com> on 2014/05/02 04:22:59 UTC
0.8.1 Java Producer API Callbacks
I'm looking at using the java producer api for 0.8.1 and I'm slightly
confused by this passage from section 4.4 of
https://kafka.apache.org/documentation.html#theproducer
"Note that as of Kafka 0.8.1 the async producer does not have a
callback, which could be used to register handlers to catch send errors.
Adding such callback functionality is proposed for Kafka 0.9, see
[Proposed Producer
API](https://cwiki.apache.org/confluence/display/KAFKA/Client+Rewrite#ClientRewrite-ProposedProducerAPI)."
org.apache.kafka.clients.producer.KafkaProducer in 0.8.1 appears to have
public Future<RecordMetadata> send(ProducerRecord record, Callback
callback) which looks like the mentioned callback.
How do the callbacks with the async producer? Is it as described in the
comment on the send method (see
https://github.com/apache/kafka/blob/0.8.1/clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java#L151
for reference)?
Looking around it seems plausible the language in the documentation
might refer to a separate sort of callback that existed in 0.7 but not
0.8. In our use case we have something useful to do if we can detect
messages failing to be sent.
Christian
Re: 0.8.1 Java Producer API Callbacks
Posted by Jay Kreps <ja...@gmail.com>.
This summary is correct. If you are just starting development now it is
probably reasonable to start with the new producer. We would certainly
appreciate any feedback on it.
My recommendation would be to build the producer off trunk as there were a
few bug fixes since 0.8.1.x that are worth getting.
-Jay
On Fri, May 2, 2014 at 9:49 AM, Neha Narkhede <ne...@gmail.com>wrote:
> Hi Christian,
>
> As Jun mentioned, the new producer is marked beta since it is new. By 0.9
> or even sooner, we'd expect it to be deployed at LinkedIn and a few other
> companies in a stable state. The APIs, guarantees and features will not
> change by 0.9, just the stability.
>
> Thanks,
> Neha
>
>
> On Fri, May 2, 2014 at 7:50 AM, Jun Rao <ju...@gmail.com> wrote:
>
> > It's beta mostly because it's new.
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Thu, May 1, 2014 at 10:25 PM, cacsar@gmail.com <ca...@gmail.com>
> > wrote:
> >
> > > Thanks, that's quite helpful. According to this post,
> > > http://blog.empathybox.com/ , it looks like it will be beta then which
> > > seems good enough. Assuming that the beta designation is correct, is
> that
> > > because it won't have as many features/the same flexibility as it's
> > > expected to by 0.9?
> > > Christian
> > >
> > >
> > > On Thu, May 1, 2014 at 9:19 PM, Neha Narkhede <neha.narkhede@gmail.com
> > > >wrote:
> > >
> > > > The javadoc of the new producer is here for now -
> > > >
> > > >
> > >
> >
> http://empathybox.com/kafka-javadoc/index.html?kafka/clients/producer/KafkaProducer.html
> > > >
> > > > Thanks,
> > > > Neha
> > > >
> > > >
> > > > On Thu, May 1, 2014 at 9:16 PM, Jun Rao <ju...@gmail.com> wrote:
> > > >
> > > > > The new producer (that supports callbacks) is in trunk. It will be
> > > > released
> > > > > in 0.8.2. You can look at the java doc of KafkaProducer for the
> api.
> > > > >
> > > > > Thanks
> > > > >
> > > > > Jun
> > > > >
> > > > >
> > > > > On Thu, May 1, 2014 at 8:43 PM, Christian Csar <ca...@gmail.com>
> > > wrote:
> > > > >
> > > > > > On 05/01/2014 07:22 PM, Christian Csar wrote:
> > > > > > > I'm looking at using the java producer api for 0.8.1 and I'm
> > > slightly
> > > > > > > confused by this passage from section 4.4 of
> > > > > > > https://kafka.apache.org/documentation.html#theproducer
> > > > > > > "Note that as of Kafka 0.8.1 the async producer does not have a
> > > > > > > callback, which could be used to register handlers to catch
> send
> > > > > errors.
> > > > > > > Adding such callback functionality is proposed for Kafka 0.9,
> see
> > > > > > > [Proposed Producer
> > > > > > > API](
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Client+Rewrite#ClientRewrite-ProposedProducerAPI
> > > > > )
> > > > > > ."
> > > > > > >
> > > > > > > org.apache.kafka.clients.producer.KafkaProducer in 0.8.1
> appears
> > to
> > > > > have
> > > > > > > public Future<RecordMetadata> send(ProducerRecord record,
> > Callback
> > > > > > > callback) which looks like the mentioned callback.
> > > > > > >
> > > > > > > How do the callbacks with the async producer? Is it as
> described
> > in
> > > > the
> > > > > > > comment on the send method (see
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/kafka/blob/0.8.1/clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java#L151
> > > > > > > for reference)?
> > > > > > >
> > > > > > > Looking around it seems plausible the language in the
> > documentation
> > > > > > > might refer to a separate sort of callback that existed in 0.7
> > but
> > > > not
> > > > > > > 0.8. In our use case we have something useful to do if we can
> > > detect
> > > > > > > messages failing to be sent.
> > > > > > >
> > > > > > > Christian
> > > > > > >
> > > > > >
> > > > > > It appears that I was looking at the Java client rather than the
> > > Scala
> > > > > > java api referenced by the documentation
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/kafka/blob/0.8.1/core/src/main/scala/kafka/javaapi/producer/Producer.scala
> > > > > >
> > > > > > Are both of these currently suited for use from java and still
> > > > > > supported? Given the support for callbacks in the event of
> failure
> > I
> > > am
> > > > > > inclined to use the Java one despite the currently limited
> support
> > > for
> > > > > > specifying partitioners (though it supports specifying the
> > partition)
> > > > or
> > > > > > encoders.
> > > > > >
> > > > > > Any guidance on this would be appreciated.
> > > > > >
> > > > > > Christian
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: 0.8.1 Java Producer API Callbacks
Posted by Neha Narkhede <ne...@gmail.com>.
Hi Christian,
As Jun mentioned, the new producer is marked beta since it is new. By 0.9
or even sooner, we'd expect it to be deployed at LinkedIn and a few other
companies in a stable state. The APIs, guarantees and features will not
change by 0.9, just the stability.
Thanks,
Neha
On Fri, May 2, 2014 at 7:50 AM, Jun Rao <ju...@gmail.com> wrote:
> It's beta mostly because it's new.
>
> Thanks,
>
> Jun
>
>
> On Thu, May 1, 2014 at 10:25 PM, cacsar@gmail.com <ca...@gmail.com>
> wrote:
>
> > Thanks, that's quite helpful. According to this post,
> > http://blog.empathybox.com/ , it looks like it will be beta then which
> > seems good enough. Assuming that the beta designation is correct, is that
> > because it won't have as many features/the same flexibility as it's
> > expected to by 0.9?
> > Christian
> >
> >
> > On Thu, May 1, 2014 at 9:19 PM, Neha Narkhede <neha.narkhede@gmail.com
> > >wrote:
> >
> > > The javadoc of the new producer is here for now -
> > >
> > >
> >
> http://empathybox.com/kafka-javadoc/index.html?kafka/clients/producer/KafkaProducer.html
> > >
> > > Thanks,
> > > Neha
> > >
> > >
> > > On Thu, May 1, 2014 at 9:16 PM, Jun Rao <ju...@gmail.com> wrote:
> > >
> > > > The new producer (that supports callbacks) is in trunk. It will be
> > > released
> > > > in 0.8.2. You can look at the java doc of KafkaProducer for the api.
> > > >
> > > > Thanks
> > > >
> > > > Jun
> > > >
> > > >
> > > > On Thu, May 1, 2014 at 8:43 PM, Christian Csar <ca...@gmail.com>
> > wrote:
> > > >
> > > > > On 05/01/2014 07:22 PM, Christian Csar wrote:
> > > > > > I'm looking at using the java producer api for 0.8.1 and I'm
> > slightly
> > > > > > confused by this passage from section 4.4 of
> > > > > > https://kafka.apache.org/documentation.html#theproducer
> > > > > > "Note that as of Kafka 0.8.1 the async producer does not have a
> > > > > > callback, which could be used to register handlers to catch send
> > > > errors.
> > > > > > Adding such callback functionality is proposed for Kafka 0.9, see
> > > > > > [Proposed Producer
> > > > > > API](
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Client+Rewrite#ClientRewrite-ProposedProducerAPI
> > > > )
> > > > > ."
> > > > > >
> > > > > > org.apache.kafka.clients.producer.KafkaProducer in 0.8.1 appears
> to
> > > > have
> > > > > > public Future<RecordMetadata> send(ProducerRecord record,
> Callback
> > > > > > callback) which looks like the mentioned callback.
> > > > > >
> > > > > > How do the callbacks with the async producer? Is it as described
> in
> > > the
> > > > > > comment on the send method (see
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/kafka/blob/0.8.1/clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java#L151
> > > > > > for reference)?
> > > > > >
> > > > > > Looking around it seems plausible the language in the
> documentation
> > > > > > might refer to a separate sort of callback that existed in 0.7
> but
> > > not
> > > > > > 0.8. In our use case we have something useful to do if we can
> > detect
> > > > > > messages failing to be sent.
> > > > > >
> > > > > > Christian
> > > > > >
> > > > >
> > > > > It appears that I was looking at the Java client rather than the
> > Scala
> > > > > java api referenced by the documentation
> > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/kafka/blob/0.8.1/core/src/main/scala/kafka/javaapi/producer/Producer.scala
> > > > >
> > > > > Are both of these currently suited for use from java and still
> > > > > supported? Given the support for callbacks in the event of failure
> I
> > am
> > > > > inclined to use the Java one despite the currently limited support
> > for
> > > > > specifying partitioners (though it supports specifying the
> partition)
> > > or
> > > > > encoders.
> > > > >
> > > > > Any guidance on this would be appreciated.
> > > > >
> > > > > Christian
> > > > >
> > > > >
> > > >
> > >
> >
>
Re: 0.8.1 Java Producer API Callbacks
Posted by Jun Rao <ju...@gmail.com>.
It's beta mostly because it's new.
Thanks,
Jun
On Thu, May 1, 2014 at 10:25 PM, cacsar@gmail.com <ca...@gmail.com> wrote:
> Thanks, that's quite helpful. According to this post,
> http://blog.empathybox.com/ , it looks like it will be beta then which
> seems good enough. Assuming that the beta designation is correct, is that
> because it won't have as many features/the same flexibility as it's
> expected to by 0.9?
> Christian
>
>
> On Thu, May 1, 2014 at 9:19 PM, Neha Narkhede <neha.narkhede@gmail.com
> >wrote:
>
> > The javadoc of the new producer is here for now -
> >
> >
> http://empathybox.com/kafka-javadoc/index.html?kafka/clients/producer/KafkaProducer.html
> >
> > Thanks,
> > Neha
> >
> >
> > On Thu, May 1, 2014 at 9:16 PM, Jun Rao <ju...@gmail.com> wrote:
> >
> > > The new producer (that supports callbacks) is in trunk. It will be
> > released
> > > in 0.8.2. You can look at the java doc of KafkaProducer for the api.
> > >
> > > Thanks
> > >
> > > Jun
> > >
> > >
> > > On Thu, May 1, 2014 at 8:43 PM, Christian Csar <ca...@gmail.com>
> wrote:
> > >
> > > > On 05/01/2014 07:22 PM, Christian Csar wrote:
> > > > > I'm looking at using the java producer api for 0.8.1 and I'm
> slightly
> > > > > confused by this passage from section 4.4 of
> > > > > https://kafka.apache.org/documentation.html#theproducer
> > > > > "Note that as of Kafka 0.8.1 the async producer does not have a
> > > > > callback, which could be used to register handlers to catch send
> > > errors.
> > > > > Adding such callback functionality is proposed for Kafka 0.9, see
> > > > > [Proposed Producer
> > > > > API](
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Client+Rewrite#ClientRewrite-ProposedProducerAPI
> > > )
> > > > ."
> > > > >
> > > > > org.apache.kafka.clients.producer.KafkaProducer in 0.8.1 appears to
> > > have
> > > > > public Future<RecordMetadata> send(ProducerRecord record, Callback
> > > > > callback) which looks like the mentioned callback.
> > > > >
> > > > > How do the callbacks with the async producer? Is it as described in
> > the
> > > > > comment on the send method (see
> > > > >
> > > >
> > >
> >
> https://github.com/apache/kafka/blob/0.8.1/clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java#L151
> > > > > for reference)?
> > > > >
> > > > > Looking around it seems plausible the language in the documentation
> > > > > might refer to a separate sort of callback that existed in 0.7 but
> > not
> > > > > 0.8. In our use case we have something useful to do if we can
> detect
> > > > > messages failing to be sent.
> > > > >
> > > > > Christian
> > > > >
> > > >
> > > > It appears that I was looking at the Java client rather than the
> Scala
> > > > java api referenced by the documentation
> > > >
> > > >
> > >
> >
> https://github.com/apache/kafka/blob/0.8.1/core/src/main/scala/kafka/javaapi/producer/Producer.scala
> > > >
> > > > Are both of these currently suited for use from java and still
> > > > supported? Given the support for callbacks in the event of failure I
> am
> > > > inclined to use the Java one despite the currently limited support
> for
> > > > specifying partitioners (though it supports specifying the partition)
> > or
> > > > encoders.
> > > >
> > > > Any guidance on this would be appreciated.
> > > >
> > > > Christian
> > > >
> > > >
> > >
> >
>
Re: 0.8.1 Java Producer API Callbacks
Posted by "cacsar@gmail.com" <ca...@gmail.com>.
Thanks, that's quite helpful. According to this post,
http://blog.empathybox.com/ , it looks like it will be beta then which
seems good enough. Assuming that the beta designation is correct, is that
because it won't have as many features/the same flexibility as it's
expected to by 0.9?
Christian
On Thu, May 1, 2014 at 9:19 PM, Neha Narkhede <ne...@gmail.com>wrote:
> The javadoc of the new producer is here for now -
>
> http://empathybox.com/kafka-javadoc/index.html?kafka/clients/producer/KafkaProducer.html
>
> Thanks,
> Neha
>
>
> On Thu, May 1, 2014 at 9:16 PM, Jun Rao <ju...@gmail.com> wrote:
>
> > The new producer (that supports callbacks) is in trunk. It will be
> released
> > in 0.8.2. You can look at the java doc of KafkaProducer for the api.
> >
> > Thanks
> >
> > Jun
> >
> >
> > On Thu, May 1, 2014 at 8:43 PM, Christian Csar <ca...@gmail.com> wrote:
> >
> > > On 05/01/2014 07:22 PM, Christian Csar wrote:
> > > > I'm looking at using the java producer api for 0.8.1 and I'm slightly
> > > > confused by this passage from section 4.4 of
> > > > https://kafka.apache.org/documentation.html#theproducer
> > > > "Note that as of Kafka 0.8.1 the async producer does not have a
> > > > callback, which could be used to register handlers to catch send
> > errors.
> > > > Adding such callback functionality is proposed for Kafka 0.9, see
> > > > [Proposed Producer
> > > > API](
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Client+Rewrite#ClientRewrite-ProposedProducerAPI
> > )
> > > ."
> > > >
> > > > org.apache.kafka.clients.producer.KafkaProducer in 0.8.1 appears to
> > have
> > > > public Future<RecordMetadata> send(ProducerRecord record, Callback
> > > > callback) which looks like the mentioned callback.
> > > >
> > > > How do the callbacks with the async producer? Is it as described in
> the
> > > > comment on the send method (see
> > > >
> > >
> >
> https://github.com/apache/kafka/blob/0.8.1/clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java#L151
> > > > for reference)?
> > > >
> > > > Looking around it seems plausible the language in the documentation
> > > > might refer to a separate sort of callback that existed in 0.7 but
> not
> > > > 0.8. In our use case we have something useful to do if we can detect
> > > > messages failing to be sent.
> > > >
> > > > Christian
> > > >
> > >
> > > It appears that I was looking at the Java client rather than the Scala
> > > java api referenced by the documentation
> > >
> > >
> >
> https://github.com/apache/kafka/blob/0.8.1/core/src/main/scala/kafka/javaapi/producer/Producer.scala
> > >
> > > Are both of these currently suited for use from java and still
> > > supported? Given the support for callbacks in the event of failure I am
> > > inclined to use the Java one despite the currently limited support for
> > > specifying partitioners (though it supports specifying the partition)
> or
> > > encoders.
> > >
> > > Any guidance on this would be appreciated.
> > >
> > > Christian
> > >
> > >
> >
>
Re: 0.8.1 Java Producer API Callbacks
Posted by Neha Narkhede <ne...@gmail.com>.
The javadoc of the new producer is here for now -
http://empathybox.com/kafka-javadoc/index.html?kafka/clients/producer/KafkaProducer.html
Thanks,
Neha
On Thu, May 1, 2014 at 9:16 PM, Jun Rao <ju...@gmail.com> wrote:
> The new producer (that supports callbacks) is in trunk. It will be released
> in 0.8.2. You can look at the java doc of KafkaProducer for the api.
>
> Thanks
>
> Jun
>
>
> On Thu, May 1, 2014 at 8:43 PM, Christian Csar <ca...@gmail.com> wrote:
>
> > On 05/01/2014 07:22 PM, Christian Csar wrote:
> > > I'm looking at using the java producer api for 0.8.1 and I'm slightly
> > > confused by this passage from section 4.4 of
> > > https://kafka.apache.org/documentation.html#theproducer
> > > "Note that as of Kafka 0.8.1 the async producer does not have a
> > > callback, which could be used to register handlers to catch send
> errors.
> > > Adding such callback functionality is proposed for Kafka 0.9, see
> > > [Proposed Producer
> > > API](
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Client+Rewrite#ClientRewrite-ProposedProducerAPI
> )
> > ."
> > >
> > > org.apache.kafka.clients.producer.KafkaProducer in 0.8.1 appears to
> have
> > > public Future<RecordMetadata> send(ProducerRecord record, Callback
> > > callback) which looks like the mentioned callback.
> > >
> > > How do the callbacks with the async producer? Is it as described in the
> > > comment on the send method (see
> > >
> >
> https://github.com/apache/kafka/blob/0.8.1/clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java#L151
> > > for reference)?
> > >
> > > Looking around it seems plausible the language in the documentation
> > > might refer to a separate sort of callback that existed in 0.7 but not
> > > 0.8. In our use case we have something useful to do if we can detect
> > > messages failing to be sent.
> > >
> > > Christian
> > >
> >
> > It appears that I was looking at the Java client rather than the Scala
> > java api referenced by the documentation
> >
> >
> https://github.com/apache/kafka/blob/0.8.1/core/src/main/scala/kafka/javaapi/producer/Producer.scala
> >
> > Are both of these currently suited for use from java and still
> > supported? Given the support for callbacks in the event of failure I am
> > inclined to use the Java one despite the currently limited support for
> > specifying partitioners (though it supports specifying the partition) or
> > encoders.
> >
> > Any guidance on this would be appreciated.
> >
> > Christian
> >
> >
>
Re: 0.8.1 Java Producer API Callbacks
Posted by Jun Rao <ju...@gmail.com>.
The new producer (that supports callbacks) is in trunk. It will be released
in 0.8.2. You can look at the java doc of KafkaProducer for the api.
Thanks
Jun
On Thu, May 1, 2014 at 8:43 PM, Christian Csar <ca...@gmail.com> wrote:
> On 05/01/2014 07:22 PM, Christian Csar wrote:
> > I'm looking at using the java producer api for 0.8.1 and I'm slightly
> > confused by this passage from section 4.4 of
> > https://kafka.apache.org/documentation.html#theproducer
> > "Note that as of Kafka 0.8.1 the async producer does not have a
> > callback, which could be used to register handlers to catch send errors.
> > Adding such callback functionality is proposed for Kafka 0.9, see
> > [Proposed Producer
> > API](
> https://cwiki.apache.org/confluence/display/KAFKA/Client+Rewrite#ClientRewrite-ProposedProducerAPI)
> ."
> >
> > org.apache.kafka.clients.producer.KafkaProducer in 0.8.1 appears to have
> > public Future<RecordMetadata> send(ProducerRecord record, Callback
> > callback) which looks like the mentioned callback.
> >
> > How do the callbacks with the async producer? Is it as described in the
> > comment on the send method (see
> >
> https://github.com/apache/kafka/blob/0.8.1/clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java#L151
> > for reference)?
> >
> > Looking around it seems plausible the language in the documentation
> > might refer to a separate sort of callback that existed in 0.7 but not
> > 0.8. In our use case we have something useful to do if we can detect
> > messages failing to be sent.
> >
> > Christian
> >
>
> It appears that I was looking at the Java client rather than the Scala
> java api referenced by the documentation
>
> https://github.com/apache/kafka/blob/0.8.1/core/src/main/scala/kafka/javaapi/producer/Producer.scala
>
> Are both of these currently suited for use from java and still
> supported? Given the support for callbacks in the event of failure I am
> inclined to use the Java one despite the currently limited support for
> specifying partitioners (though it supports specifying the partition) or
> encoders.
>
> Any guidance on this would be appreciated.
>
> Christian
>
>
Re: 0.8.1 Java Producer API Callbacks
Posted by Christian Csar <ca...@gmail.com>.
On 05/01/2014 07:22 PM, Christian Csar wrote:
> I'm looking at using the java producer api for 0.8.1 and I'm slightly
> confused by this passage from section 4.4 of
> https://kafka.apache.org/documentation.html#theproducer
> "Note that as of Kafka 0.8.1 the async producer does not have a
> callback, which could be used to register handlers to catch send errors.
> Adding such callback functionality is proposed for Kafka 0.9, see
> [Proposed Producer
> API](https://cwiki.apache.org/confluence/display/KAFKA/Client+Rewrite#ClientRewrite-ProposedProducerAPI)."
>
> org.apache.kafka.clients.producer.KafkaProducer in 0.8.1 appears to have
> public Future<RecordMetadata> send(ProducerRecord record, Callback
> callback) which looks like the mentioned callback.
>
> How do the callbacks with the async producer? Is it as described in the
> comment on the send method (see
> https://github.com/apache/kafka/blob/0.8.1/clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java#L151
> for reference)?
>
> Looking around it seems plausible the language in the documentation
> might refer to a separate sort of callback that existed in 0.7 but not
> 0.8. In our use case we have something useful to do if we can detect
> messages failing to be sent.
>
> Christian
>
It appears that I was looking at the Java client rather than the Scala
java api referenced by the documentation
https://github.com/apache/kafka/blob/0.8.1/core/src/main/scala/kafka/javaapi/producer/Producer.scala
Are both of these currently suited for use from java and still
supported? Given the support for callbacks in the event of failure I am
inclined to use the Java one despite the currently limited support for
specifying partitioners (though it supports specifying the partition) or
encoders.
Any guidance on this would be appreciated.
Christian