You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Jonathan Weeks <jo...@gmail.com> on 2014/08/29 19:09:46 UTC
Trunk backwards compatibility (producer / consumer client questions)
Thanks, Jay. Follow-up questions:
Some of our services will produce and consume. Is there consumer code on trunk that is backwards compatible with an existing 0.8.1.1 broker cluster? If not 0.8.1.1, will the consumer code on trunk work with a 0.8.2 broker cluster when 0.8.2 is released?
(Our code is scala, BTW)
Best Regards,
-Jonathan
On Aug 26, 2014, at 5:55 PM, Jay Kreps <ja...@gmail.com> wrote:
> Also, Jonathan, to answer your question, the new producer on trunk is
> running in prod for some use cases at LinkedIn and can be used with
> any 0.8.x. version.
>
> -Jay
>
> On Tue, Aug 26, 2014 at 12:38 PM, Jonathan Weeks
> <jo...@gmail.com> wrote:
>> I am interested in this very topic as well. Also, can the trunk version of the producer be used with an existing 0.8.1.1 broker installation, or does one need to wait for 0.8.2 (at least)?
>>
>> Thanks,
>>
>> -Jonathan
>>
>> On Aug 26, 2014, at 12:35 PM, Ryan Persaud <Ry...@symantec.com> wrote:
>>
>>> Hello,
>>>
>>> I'm looking to insert log lines from log files into kafka, but I'm concerned with handling asynchronous send() failures. Specifically, if some of the log lines fail to send, I want to be notified of the failure so that I can attempt to resend them.
>>>
>>> Based on previous threads on the mailing list (http://comments.gmane.org/gmane.comp.apache.kafka.user/1322), I know that the trunk version of kafka supports callbacks for dealing with failures. However, the callback function is not passed any metadata that can be used by the producer end to reference the original message. Including the key of the message in the RecordMetadata seems like it would be really useful for recovery purposes. Is anyone using the callback functionality to trigger resends of failed messages? If so, how are they tying the callbacks to messages? Is anyone using other methods for handling async errors/resending today? I can’t imagine that I am the only one trying to do this. I asked this question on the IRC channel today, and it sparked some discussion, but I wanted to hear from a wider audience.
>>>
>>> Thanks for the information,
>>> -Ryan
>>>
>>
Re: Trunk backwards compatibility (producer / consumer client questions)
Posted by Jay Kreps <ja...@gmail.com>.
Yes, the protocol for metadata, fetch, and produce is the same across
both clients and all 0.8.x versions.
-Jay
On Fri, Aug 29, 2014 at 10:09 AM, Jonathan Weeks
<jo...@gmail.com> wrote:
> Thanks, Jay. Follow-up questions:
>
> Some of our services will produce and consume. Is there consumer code on trunk that is backwards compatible with an existing 0.8.1.1 broker cluster? If not 0.8.1.1, will the consumer code on trunk work with a 0.8.2 broker cluster when 0.8.2 is released?
>
> (Our code is scala, BTW)
>
> Best Regards,
>
> -Jonathan
>
>
> On Aug 26, 2014, at 5:55 PM, Jay Kreps <ja...@gmail.com> wrote:
>
>> Also, Jonathan, to answer your question, the new producer on trunk is
>> running in prod for some use cases at LinkedIn and can be used with
>> any 0.8.x. version.
>>
>> -Jay
>>
>> On Tue, Aug 26, 2014 at 12:38 PM, Jonathan Weeks
>> <jo...@gmail.com> wrote:
>>> I am interested in this very topic as well. Also, can the trunk version of the producer be used with an existing 0.8.1.1 broker installation, or does one need to wait for 0.8.2 (at least)?
>>>
>>> Thanks,
>>>
>>> -Jonathan
>>>
>>> On Aug 26, 2014, at 12:35 PM, Ryan Persaud <Ry...@symantec.com> wrote:
>>>
>>>> Hello,
>>>>
>>>> I'm looking to insert log lines from log files into kafka, but I'm concerned with handling asynchronous send() failures. Specifically, if some of the log lines fail to send, I want to be notified of the failure so that I can attempt to resend them.
>>>>
>>>> Based on previous threads on the mailing list (http://comments.gmane.org/gmane.comp.apache.kafka.user/1322), I know that the trunk version of kafka supports callbacks for dealing with failures. However, the callback function is not passed any metadata that can be used by the producer end to reference the original message. Including the key of the message in the RecordMetadata seems like it would be really useful for recovery purposes. Is anyone using the callback functionality to trigger resends of failed messages? If so, how are they tying the callbacks to messages? Is anyone using other methods for handling async errors/resending today? I can’t imagine that I am the only one trying to do this. I asked this question on the IRC channel today, and it sparked some discussion, but I wanted to hear from a wider audience.
>>>>
>>>> Thanks for the information,
>>>> -Ryan
>>>>
>>>
>
Re: Trunk backwards compatibility (producer / consumer client
questions)
Posted by Joel Koshy <jj...@gmail.com>.
I think Jun was referring to "consumer" clients. The new producer is
compatible with existing brokers. However the new consumer requires a
server-side consumer coordinator.
On Fri, Aug 29, 2014 at 10:37:12AM -0700, Jonathan Weeks wrote:
> Hi Jun,
>
> Jay indicated that the new producer client on trunk is backwards compatible with 0.8.1.1 (see thread below) — can you elaborate?
>
> Given the consumer re-write for 0.9, I can definitely see how that would break backwards compatibility, but Jay indicates that the producer on the trunk will work with older existing brokers...
>
> Thanks,
>
> -Jonathan
>
> On Aug 29, 2014, at 10:32 AM, Jun Rao <ju...@gmail.com> wrote:
>
> > The old clients with be compatible with the new broker. However, in order
> > to use the new clients, you will need to upgrade to the new broker first.
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Fri, Aug 29, 2014 at 10:09 AM, Jonathan Weeks <jo...@gmail.com>
> > wrote:
> >
> >> Thanks, Jay. Follow-up questions:
> >>
> >> Some of our services will produce and consume. Is there consumer code on
> >> trunk that is backwards compatible with an existing 0.8.1.1 broker cluster?
> >> If not 0.8.1.1, will the consumer code on trunk work with a 0.8.2 broker
> >> cluster when 0.8.2 is released?
> >>
> >> (Our code is scala, BTW)
> >>
> >> Best Regards,
> >>
> >> -Jonathan
> >>
> >>
> >> On Aug 26, 2014, at 5:55 PM, Jay Kreps <ja...@gmail.com> wrote:
> >>
> >>> Also, Jonathan, to answer your question, the new producer on trunk is
> >>> running in prod for some use cases at LinkedIn and can be used with
> >>> any 0.8.x. version.
> >>>
> >>> -Jay
> >>>
> >>> On Tue, Aug 26, 2014 at 12:38 PM, Jonathan Weeks
> >>> <jo...@gmail.com> wrote:
> >>>> I am interested in this very topic as well. Also, can the trunk version
> >> of the producer be used with an existing 0.8.1.1 broker installation, or
> >> does one need to wait for 0.8.2 (at least)?
> >>>>
> >>>> Thanks,
> >>>>
> >>>> -Jonathan
> >>>>
> >>>> On Aug 26, 2014, at 12:35 PM, Ryan Persaud <Ry...@symantec.com>
> >> wrote:
> >>>>
> >>>>> Hello,
> >>>>>
> >>>>> I'm looking to insert log lines from log files into kafka, but I'm
> >> concerned with handling asynchronous send() failures. Specifically, if
> >> some of the log lines fail to send, I want to be notified of the failure so
> >> that I can attempt to resend them.
> >>>>>
> >>>>> Based on previous threads on the mailing list (
> >> http://comments.gmane.org/gmane.comp.apache.kafka.user/1322), I know that
> >> the trunk version of kafka supports callbacks for dealing with failures.
> >> However, the callback function is not passed any metadata that can be used
> >> by the producer end to reference the original message. Including the key
> >> of the message in the RecordMetadata seems like it would be really useful
> >> for recovery purposes. Is anyone using the callback functionality to
> >> trigger resends of failed messages? If so, how are they tying the
> >> callbacks to messages? Is anyone using other methods for handling async
> >> errors/resending today? I can’t imagine that I am the only one trying to
> >> do this. I asked this question on the IRC channel today, and it sparked
> >> some discussion, but I wanted to hear from a wider audience.
> >>>>>
> >>>>> Thanks for the information,
> >>>>> -Ryan
> >>>>>
> >>>>
> >>
> >>
>
Re: Trunk backwards compatibility (producer / consumer client questions)
Posted by Jun Rao <ju...@gmail.com>.
Yes, the new producer in trunk is compatible with 0.8.1.1 broker. In trunk,
we have evolved the format of OffsetCommitRequest, which is used by the old
consumer (and will be used by the new consumer). So, to use the consumer in
trunk, you will need to upgrade the broker first.
Thanks,
Jun
On Fri, Aug 29, 2014 at 10:37 AM, Jonathan Weeks <jo...@gmail.com>
wrote:
> Hi Jun,
>
> Jay indicated that the new producer client on trunk is backwards
> compatible with 0.8.1.1 (see thread below) — can you elaborate?
>
> Given the consumer re-write for 0.9, I can definitely see how that would
> break backwards compatibility, but Jay indicates that the producer on the
> trunk will work with older existing brokers...
>
> Thanks,
>
> -Jonathan
>
> On Aug 29, 2014, at 10:32 AM, Jun Rao <ju...@gmail.com> wrote:
>
> > The old clients with be compatible with the new broker. However, in order
> > to use the new clients, you will need to upgrade to the new broker first.
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Fri, Aug 29, 2014 at 10:09 AM, Jonathan Weeks <
> jonathanbweeks@gmail.com>
> > wrote:
> >
> >> Thanks, Jay. Follow-up questions:
> >>
> >> Some of our services will produce and consume. Is there consumer code on
> >> trunk that is backwards compatible with an existing 0.8.1.1 broker
> cluster?
> >> If not 0.8.1.1, will the consumer code on trunk work with a 0.8.2 broker
> >> cluster when 0.8.2 is released?
> >>
> >> (Our code is scala, BTW)
> >>
> >> Best Regards,
> >>
> >> -Jonathan
> >>
> >>
> >> On Aug 26, 2014, at 5:55 PM, Jay Kreps <ja...@gmail.com> wrote:
> >>
> >>> Also, Jonathan, to answer your question, the new producer on trunk is
> >>> running in prod for some use cases at LinkedIn and can be used with
> >>> any 0.8.x. version.
> >>>
> >>> -Jay
> >>>
> >>> On Tue, Aug 26, 2014 at 12:38 PM, Jonathan Weeks
> >>> <jo...@gmail.com> wrote:
> >>>> I am interested in this very topic as well. Also, can the trunk
> version
> >> of the producer be used with an existing 0.8.1.1 broker installation, or
> >> does one need to wait for 0.8.2 (at least)?
> >>>>
> >>>> Thanks,
> >>>>
> >>>> -Jonathan
> >>>>
> >>>> On Aug 26, 2014, at 12:35 PM, Ryan Persaud <Ryan_Persaud@symantec.com
> >
> >> wrote:
> >>>>
> >>>>> Hello,
> >>>>>
> >>>>> I'm looking to insert log lines from log files into kafka, but I'm
> >> concerned with handling asynchronous send() failures. Specifically, if
> >> some of the log lines fail to send, I want to be notified of the
> failure so
> >> that I can attempt to resend them.
> >>>>>
> >>>>> Based on previous threads on the mailing list (
> >> http://comments.gmane.org/gmane.comp.apache.kafka.user/1322), I know
> that
> >> the trunk version of kafka supports callbacks for dealing with failures.
> >> However, the callback function is not passed any metadata that can be
> used
> >> by the producer end to reference the original message. Including the
> key
> >> of the message in the RecordMetadata seems like it would be really
> useful
> >> for recovery purposes. Is anyone using the callback functionality to
> >> trigger resends of failed messages? If so, how are they tying the
> >> callbacks to messages? Is anyone using other methods for handling async
> >> errors/resending today? I can’t imagine that I am the only one trying
> to
> >> do this. I asked this question on the IRC channel today, and it sparked
> >> some discussion, but I wanted to hear from a wider audience.
> >>>>>
> >>>>> Thanks for the information,
> >>>>> -Ryan
> >>>>>
> >>>>
> >>
> >>
>
>
Re: Trunk backwards compatibility (producer / consumer client questions)
Posted by Jonathan Weeks <jo...@gmail.com>.
Hi Jun,
Jay indicated that the new producer client on trunk is backwards compatible with 0.8.1.1 (see thread below) — can you elaborate?
Given the consumer re-write for 0.9, I can definitely see how that would break backwards compatibility, but Jay indicates that the producer on the trunk will work with older existing brokers...
Thanks,
-Jonathan
On Aug 29, 2014, at 10:32 AM, Jun Rao <ju...@gmail.com> wrote:
> The old clients with be compatible with the new broker. However, in order
> to use the new clients, you will need to upgrade to the new broker first.
>
> Thanks,
>
> Jun
>
>
> On Fri, Aug 29, 2014 at 10:09 AM, Jonathan Weeks <jo...@gmail.com>
> wrote:
>
>> Thanks, Jay. Follow-up questions:
>>
>> Some of our services will produce and consume. Is there consumer code on
>> trunk that is backwards compatible with an existing 0.8.1.1 broker cluster?
>> If not 0.8.1.1, will the consumer code on trunk work with a 0.8.2 broker
>> cluster when 0.8.2 is released?
>>
>> (Our code is scala, BTW)
>>
>> Best Regards,
>>
>> -Jonathan
>>
>>
>> On Aug 26, 2014, at 5:55 PM, Jay Kreps <ja...@gmail.com> wrote:
>>
>>> Also, Jonathan, to answer your question, the new producer on trunk is
>>> running in prod for some use cases at LinkedIn and can be used with
>>> any 0.8.x. version.
>>>
>>> -Jay
>>>
>>> On Tue, Aug 26, 2014 at 12:38 PM, Jonathan Weeks
>>> <jo...@gmail.com> wrote:
>>>> I am interested in this very topic as well. Also, can the trunk version
>> of the producer be used with an existing 0.8.1.1 broker installation, or
>> does one need to wait for 0.8.2 (at least)?
>>>>
>>>> Thanks,
>>>>
>>>> -Jonathan
>>>>
>>>> On Aug 26, 2014, at 12:35 PM, Ryan Persaud <Ry...@symantec.com>
>> wrote:
>>>>
>>>>> Hello,
>>>>>
>>>>> I'm looking to insert log lines from log files into kafka, but I'm
>> concerned with handling asynchronous send() failures. Specifically, if
>> some of the log lines fail to send, I want to be notified of the failure so
>> that I can attempt to resend them.
>>>>>
>>>>> Based on previous threads on the mailing list (
>> http://comments.gmane.org/gmane.comp.apache.kafka.user/1322), I know that
>> the trunk version of kafka supports callbacks for dealing with failures.
>> However, the callback function is not passed any metadata that can be used
>> by the producer end to reference the original message. Including the key
>> of the message in the RecordMetadata seems like it would be really useful
>> for recovery purposes. Is anyone using the callback functionality to
>> trigger resends of failed messages? If so, how are they tying the
>> callbacks to messages? Is anyone using other methods for handling async
>> errors/resending today? I can’t imagine that I am the only one trying to
>> do this. I asked this question on the IRC channel today, and it sparked
>> some discussion, but I wanted to hear from a wider audience.
>>>>>
>>>>> Thanks for the information,
>>>>> -Ryan
>>>>>
>>>>
>>
>>
Re: Trunk backwards compatibility (producer / consumer client questions)
Posted by Jun Rao <ju...@gmail.com>.
The old clients with be compatible with the new broker. However, in order
to use the new clients, you will need to upgrade to the new broker first.
Thanks,
Jun
On Fri, Aug 29, 2014 at 10:09 AM, Jonathan Weeks <jo...@gmail.com>
wrote:
> Thanks, Jay. Follow-up questions:
>
> Some of our services will produce and consume. Is there consumer code on
> trunk that is backwards compatible with an existing 0.8.1.1 broker cluster?
> If not 0.8.1.1, will the consumer code on trunk work with a 0.8.2 broker
> cluster when 0.8.2 is released?
>
> (Our code is scala, BTW)
>
> Best Regards,
>
> -Jonathan
>
>
> On Aug 26, 2014, at 5:55 PM, Jay Kreps <ja...@gmail.com> wrote:
>
> > Also, Jonathan, to answer your question, the new producer on trunk is
> > running in prod for some use cases at LinkedIn and can be used with
> > any 0.8.x. version.
> >
> > -Jay
> >
> > On Tue, Aug 26, 2014 at 12:38 PM, Jonathan Weeks
> > <jo...@gmail.com> wrote:
> >> I am interested in this very topic as well. Also, can the trunk version
> of the producer be used with an existing 0.8.1.1 broker installation, or
> does one need to wait for 0.8.2 (at least)?
> >>
> >> Thanks,
> >>
> >> -Jonathan
> >>
> >> On Aug 26, 2014, at 12:35 PM, Ryan Persaud <Ry...@symantec.com>
> wrote:
> >>
> >>> Hello,
> >>>
> >>> I'm looking to insert log lines from log files into kafka, but I'm
> concerned with handling asynchronous send() failures. Specifically, if
> some of the log lines fail to send, I want to be notified of the failure so
> that I can attempt to resend them.
> >>>
> >>> Based on previous threads on the mailing list (
> http://comments.gmane.org/gmane.comp.apache.kafka.user/1322), I know that
> the trunk version of kafka supports callbacks for dealing with failures.
> However, the callback function is not passed any metadata that can be used
> by the producer end to reference the original message. Including the key
> of the message in the RecordMetadata seems like it would be really useful
> for recovery purposes. Is anyone using the callback functionality to
> trigger resends of failed messages? If so, how are they tying the
> callbacks to messages? Is anyone using other methods for handling async
> errors/resending today? I can’t imagine that I am the only one trying to
> do this. I asked this question on the IRC channel today, and it sparked
> some discussion, but I wanted to hear from a wider audience.
> >>>
> >>> Thanks for the information,
> >>> -Ryan
> >>>
> >>
>
>