You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Sachin Mittal <sj...@gmail.com> on 2017/07/18 07:17:00 UTC
Kafka compatibility matrix needed
Hi,
I would like some help/information on what client versions are compatible
with what broker versions in kafka.
Some table like this would be good
server
client 0.8 0.9 0.10 0.11
0.8 yes ? ? ?
0.9 ? yes ? ?
0.10 ? ? yes ?
0.11 ? ? ? yes
So if question marks are filled it would be of great help.
Reason I am asking is many times we need to use other libraries/frameworks
to pull/push data from/into kafka and sometimes these support only a
particular version of clients.
Like right now I am trying to pull data from kafka via druid/tranquility
and they have clients of version 0.8.x implemented but my broker is running
0.10.x.
Also if such a table can be posted on kafka documentation page or github
page that would be great.
Thanks
Sachin
Re: Kafka compatibility matrix needed
Posted by Ismael Juma <is...@juma.me.uk>.
Also, make
On Wed, Jul 19, 2017 at 5:10 AM, Ismael Juma <is...@juma.me.uk> wrote:
> Hi Anish,
>
> What should happen in that case is that the broker down converts the
> messages to a format supported by the older clients. We have tests for this
> scenario so it would be good to understand what is different in your
> environment. Please file a JIRA and provide as much information as possible.
>
> Ismael
>
> On Wed, Jul 19, 2017 at 12:39 AM, Anish Mashankar <
> anish@systeminsights.com> wrote:
>
>> After upgrading Kafka 0.10.0 to 0.11.0, and changing the Message protocol
>> to 0.11 on brokers, the consumers with version 0.8.2.1 started reporting
>> Invalid message error logs.
>>
>> On Tue, Jul 18, 2017 at 6:37 PM Ismael Juma <is...@juma.me.uk> wrote:
>>
>> > Hi all,
>> >
>> > 0.8.x clients should work with 0.9.x, 0.10.x and 0.11.x brokers. We have
>> > system tests for all the relevant combinations. One thing to be careful
>> > about is that Scala consumers and Java consumers store offsets and group
>> > management information differently and the Java consumer was only
>> > introduced in 0.9.0.0.
>> >
>> > Ismael
>> >
>> > On Tue, Jul 18, 2017 at 5:18 AM, John Yost <ho...@gmail.com>
>> wrote:
>> >
>> > > Hi Everyone,
>> > >
>> > > I personally found that the 0.8.x clients do not work with 0.10.0. We
>> > > upgraded our clients (KafkaSpout and custom consumers) to 0.9.0.1 and
>> > then
>> > > Kafka produce/consume worked fine.
>> > >
>> > > --John
>> > >
>> > > On Tue, Jul 18, 2017 at 6:36 AM, Sachin Mittal <sj...@gmail.com>
>> > wrote:
>> > >
>> > > > OK.
>> > > >
>> > > > Just a doubt I have is that my broker is 0.10.2 and producer is
>> also of
>> > > the
>> > > > same version and writes to a topic.
>> > > > Then I have a client from version 0.8.2 trying to fetch these
>> messages
>> > > and
>> > > > what I see is that all messages are getting dropped.
>> > > >
>> > > > However since this is an older client I tried to fetch its offsets
>> > using:
>> > > > kafka-run-class kafka.tools.ConsumerOffsetChecker --zookeeper
>> > > > localhost:2181 --group test
>> > > >
>> > > > I saw that lag is 0.
>> > > >
>> > > > So probably better question would be can 0.8.2 read a message
>> written
>> > by
>> > > > 0.10.2
>> > > >
>> > > > Thanks
>> > > > Sachin
>> > > >
>> > > >
>> > > >
>> > > > On Tue, Jul 18, 2017 at 3:26 PM, Tom Crayford <tcrayford@heroku.com
>> >
>> > > > wrote:
>> > > >
>> > > > > All broker versions support all older client versions
>> > > > >
>> > > > > On Tue, Jul 18, 2017 at 10:15 AM, Sachin Mittal <
>> sjmittal@gmail.com>
>> > > > > wrote:
>> > > > >
>> > > > > > Hi,
>> > > > > > This gives me some information but still not the complete
>> picture.
>> > > > > >
>> > > > > > It says:
>> > > > > > 0.10.2, Java clients have acquired the ability to communicate
>> with
>> > > > older
>> > > > > > brokers.
>> > > > > >
>> > > > > > It also says
>> > > > > > Version 0.11.0 brokers support 0.8.x and newer clients
>> > > > > >
>> > > > > > Question is does 0.10.2 broker support 0.8.x clients?
>> > > > > >
>> > > > > > This may solve answer my temporary doubt, but a compatibility
>> > matrix
>> > > > will
>> > > > > > be much helpful to have a yes/no answer.
>> > > > > >
>> > > > > > Thanks
>> > > > > > Sachin
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > > On Tue, Jul 18, 2017 at 2:38 PM, Michal Borowiecki <
>> > > > > > michal.borowiecki@openbet.com> wrote:
>> > > > > >
>> > > > > >> Have you seen this: http://kafka.apache.org/
>> > > > documentation.html#upgrade
>> > > > > >>
>> > > > > >> Starting with version 0.10.2, Java clients (producer and
>> consumer)
>> > > > have
>> > > > > >> acquired the ability to communicate with older brokers. Version
>> > > 0.11.0
>> > > > > >> clients can talk to version 0.10.0 or newer brokers. However,
>> if
>> > > your
>> > > > > >> brokers are older than 0.10.0, you must upgrade all the
>> brokers in
>> > > the
>> > > > > >> Kafka cluster before upgrading your clients. Version 0.11.0
>> > brokers
>> > > > > support
>> > > > > >> 0.8.x and newer clients.
>> > > > > >>
>> > > > > >> Hope that helps.
>> > > > > >>
>> > > > > >> Cheers,
>> > > > > >>
>> > > > > >> Michał
>> > > > > >>
>> > > > > >> On 18/07/17 08:17, Sachin Mittal wrote:
>> > > > > >>
>> > > > > >> Hi,
>> > > > > >> I would like some help/information on what client versions are
>> > > > > compatible
>> > > > > >> with what broker versions in kafka.
>> > > > > >>
>> > > > > >> Some table like this would be good
>> > > > > >>
>> > > > > >> server
>> > > > > >> client 0.8 0.9 0.10 0.11
>> > > > > >> 0.8 yes ? ? ?
>> > > > > >> 0.9 ? yes ? ?
>> > > > > >> 0.10 ? ? yes ?
>> > > > > >> 0.11 ? ? ? yes
>> > > > > >>
>> > > > > >> So if question marks are filled it would be of great help.
>> > > > > >>
>> > > > > >> Reason I am asking is many times we need to use other
>> > > > > libraries/frameworks
>> > > > > >> to pull/push data from/into kafka and sometimes these support
>> > only a
>> > > > > >> particular version of clients.
>> > > > > >>
>> > > > > >> Like right now I am trying to pull data from kafka via
>> > > > druid/tranquility
>> > > > > >> and they have clients of version 0.8.x implemented but my
>> broker
>> > is
>> > > > > running
>> > > > > >> 0.10.x.
>> > > > > >>
>> > > > > >> Also if such a table can be posted on kafka documentation page
>> or
>> > > > github
>> > > > > >> page that would be great.
>> > > > > >>
>> > > > > >> Thanks
>> > > > > >> Sachin
>> > > > > >>
>> > > > > >>
>> > > > > >>
>> > > > > >> --
>> > > > > >> <http://www.openbet.com/> Michal Borowiecki
>> > > > > >> Senior Software Engineer L4
>> > > > > >> T: +44 208 742 1600 <+44%2020%208742%201600>
>> > <+44%2020%208742%201600>
>> > > > > >>
>> > > > > >>
>> > > > > >> +44 203 249 8448 <+44%2020%203249%208448>
>> > <+44%2020%203249%208448>
>> > > > > >>
>> > > > > >>
>> > > > > >>
>> > > > > >> E: michal.borowiecki@openbet.com
>> > > > > >> W: www.openbet.com
>> > > > > >> OpenBet Ltd
>> > > > > >>
>> > > > > >> Chiswick Park Building 9
>> > > > > >>
>> > > > > >> 566 Chiswick High Rd
>> > > > > >>
>> > > > > >> London
>> > > > > >>
>> > > > > >> W4 5XT
>> > > > > >>
>> > > > > >> UK
>> > > > > >> <https://www.openbet.com/email_promo>
>> > > > > >> This message is confidential and intended only for the
>> addressee.
>> > If
>> > > > you
>> > > > > >> have received this message in error, please immediately notify
>> the
>> > > > > >> postmaster@openbet.com and delete it from your system as well
>> as
>> > > any
>> > > > > >> copies. The content of e-mails as well as traffic data may be
>> > > > monitored
>> > > > > by
>> > > > > >> OpenBet for employment and security purposes. To protect the
>> > > > environment
>> > > > > >> please do not print this e-mail unless necessary. OpenBet Ltd.
>> > > > > Registered
>> > > > > >> Office: Chiswick Park Building 9, 566 Chiswick High Road,
>> London,
>> > W4
>> > > > > 5XT,
>> > > > > >> United Kingdom. A company registered in England and Wales.
>> > > Registered
>> > > > > no.
>> > > > > >> 3134634. VAT no. GB927523612
>> > > > > >>
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> --
>>
>> Regards,
>> Anish Samir Mashankar
>> R&D Engineer
>> System Insights
>> +91-9789870733
>>
>
>
Re: Kafka compatibility matrix needed
Posted by Ismael Juma <is...@juma.me.uk>.
Hi Anish,
What should happen in that case is that the broker down converts the
messages to a format supported by the older clients. We have tests for this
scenario so it would be good to understand what is different in your
environment. Please file a JIRA and provide as much information as possible.
Ismael
On Wed, Jul 19, 2017 at 12:39 AM, Anish Mashankar <an...@systeminsights.com>
wrote:
> After upgrading Kafka 0.10.0 to 0.11.0, and changing the Message protocol
> to 0.11 on brokers, the consumers with version 0.8.2.1 started reporting
> Invalid message error logs.
>
> On Tue, Jul 18, 2017 at 6:37 PM Ismael Juma <is...@juma.me.uk> wrote:
>
> > Hi all,
> >
> > 0.8.x clients should work with 0.9.x, 0.10.x and 0.11.x brokers. We have
> > system tests for all the relevant combinations. One thing to be careful
> > about is that Scala consumers and Java consumers store offsets and group
> > management information differently and the Java consumer was only
> > introduced in 0.9.0.0.
> >
> > Ismael
> >
> > On Tue, Jul 18, 2017 at 5:18 AM, John Yost <ho...@gmail.com> wrote:
> >
> > > Hi Everyone,
> > >
> > > I personally found that the 0.8.x clients do not work with 0.10.0. We
> > > upgraded our clients (KafkaSpout and custom consumers) to 0.9.0.1 and
> > then
> > > Kafka produce/consume worked fine.
> > >
> > > --John
> > >
> > > On Tue, Jul 18, 2017 at 6:36 AM, Sachin Mittal <sj...@gmail.com>
> > wrote:
> > >
> > > > OK.
> > > >
> > > > Just a doubt I have is that my broker is 0.10.2 and producer is also
> of
> > > the
> > > > same version and writes to a topic.
> > > > Then I have a client from version 0.8.2 trying to fetch these
> messages
> > > and
> > > > what I see is that all messages are getting dropped.
> > > >
> > > > However since this is an older client I tried to fetch its offsets
> > using:
> > > > kafka-run-class kafka.tools.ConsumerOffsetChecker --zookeeper
> > > > localhost:2181 --group test
> > > >
> > > > I saw that lag is 0.
> > > >
> > > > So probably better question would be can 0.8.2 read a message written
> > by
> > > > 0.10.2
> > > >
> > > > Thanks
> > > > Sachin
> > > >
> > > >
> > > >
> > > > On Tue, Jul 18, 2017 at 3:26 PM, Tom Crayford <tc...@heroku.com>
> > > > wrote:
> > > >
> > > > > All broker versions support all older client versions
> > > > >
> > > > > On Tue, Jul 18, 2017 at 10:15 AM, Sachin Mittal <
> sjmittal@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi,
> > > > > > This gives me some information but still not the complete
> picture.
> > > > > >
> > > > > > It says:
> > > > > > 0.10.2, Java clients have acquired the ability to communicate
> with
> > > > older
> > > > > > brokers.
> > > > > >
> > > > > > It also says
> > > > > > Version 0.11.0 brokers support 0.8.x and newer clients
> > > > > >
> > > > > > Question is does 0.10.2 broker support 0.8.x clients?
> > > > > >
> > > > > > This may solve answer my temporary doubt, but a compatibility
> > matrix
> > > > will
> > > > > > be much helpful to have a yes/no answer.
> > > > > >
> > > > > > Thanks
> > > > > > Sachin
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Tue, Jul 18, 2017 at 2:38 PM, Michal Borowiecki <
> > > > > > michal.borowiecki@openbet.com> wrote:
> > > > > >
> > > > > >> Have you seen this: http://kafka.apache.org/
> > > > documentation.html#upgrade
> > > > > >>
> > > > > >> Starting with version 0.10.2, Java clients (producer and
> consumer)
> > > > have
> > > > > >> acquired the ability to communicate with older brokers. Version
> > > 0.11.0
> > > > > >> clients can talk to version 0.10.0 or newer brokers. However, if
> > > your
> > > > > >> brokers are older than 0.10.0, you must upgrade all the brokers
> in
> > > the
> > > > > >> Kafka cluster before upgrading your clients. Version 0.11.0
> > brokers
> > > > > support
> > > > > >> 0.8.x and newer clients.
> > > > > >>
> > > > > >> Hope that helps.
> > > > > >>
> > > > > >> Cheers,
> > > > > >>
> > > > > >> Michał
> > > > > >>
> > > > > >> On 18/07/17 08:17, Sachin Mittal wrote:
> > > > > >>
> > > > > >> Hi,
> > > > > >> I would like some help/information on what client versions are
> > > > > compatible
> > > > > >> with what broker versions in kafka.
> > > > > >>
> > > > > >> Some table like this would be good
> > > > > >>
> > > > > >> server
> > > > > >> client 0.8 0.9 0.10 0.11
> > > > > >> 0.8 yes ? ? ?
> > > > > >> 0.9 ? yes ? ?
> > > > > >> 0.10 ? ? yes ?
> > > > > >> 0.11 ? ? ? yes
> > > > > >>
> > > > > >> So if question marks are filled it would be of great help.
> > > > > >>
> > > > > >> Reason I am asking is many times we need to use other
> > > > > libraries/frameworks
> > > > > >> to pull/push data from/into kafka and sometimes these support
> > only a
> > > > > >> particular version of clients.
> > > > > >>
> > > > > >> Like right now I am trying to pull data from kafka via
> > > > druid/tranquility
> > > > > >> and they have clients of version 0.8.x implemented but my broker
> > is
> > > > > running
> > > > > >> 0.10.x.
> > > > > >>
> > > > > >> Also if such a table can be posted on kafka documentation page
> or
> > > > github
> > > > > >> page that would be great.
> > > > > >>
> > > > > >> Thanks
> > > > > >> Sachin
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> --
> > > > > >> <http://www.openbet.com/> Michal Borowiecki
> > > > > >> Senior Software Engineer L4
> > > > > >> T: +44 208 742 1600 <+44%2020%208742%201600>
> > <+44%2020%208742%201600>
> > > > > >>
> > > > > >>
> > > > > >> +44 203 249 8448 <+44%2020%203249%208448>
> > <+44%2020%203249%208448>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> E: michal.borowiecki@openbet.com
> > > > > >> W: www.openbet.com
> > > > > >> OpenBet Ltd
> > > > > >>
> > > > > >> Chiswick Park Building 9
> > > > > >>
> > > > > >> 566 Chiswick High Rd
> > > > > >>
> > > > > >> London
> > > > > >>
> > > > > >> W4 5XT
> > > > > >>
> > > > > >> UK
> > > > > >> <https://www.openbet.com/email_promo>
> > > > > >> This message is confidential and intended only for the
> addressee.
> > If
> > > > you
> > > > > >> have received this message in error, please immediately notify
> the
> > > > > >> postmaster@openbet.com and delete it from your system as well
> as
> > > any
> > > > > >> copies. The content of e-mails as well as traffic data may be
> > > > monitored
> > > > > by
> > > > > >> OpenBet for employment and security purposes. To protect the
> > > > environment
> > > > > >> please do not print this e-mail unless necessary. OpenBet Ltd.
> > > > > Registered
> > > > > >> Office: Chiswick Park Building 9, 566 Chiswick High Road,
> London,
> > W4
> > > > > 5XT,
> > > > > >> United Kingdom. A company registered in England and Wales.
> > > Registered
> > > > > no.
> > > > > >> 3134634. VAT no. GB927523612
> > > > > >>
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> --
>
> Regards,
> Anish Samir Mashankar
> R&D Engineer
> System Insights
> +91-9789870733
>
Re: Kafka compatibility matrix needed
Posted by Anish Mashankar <an...@systeminsights.com>.
After upgrading Kafka 0.10.0 to 0.11.0, and changing the Message protocol
to 0.11 on brokers, the consumers with version 0.8.2.1 started reporting
Invalid message error logs.
On Tue, Jul 18, 2017 at 6:37 PM Ismael Juma <is...@juma.me.uk> wrote:
> Hi all,
>
> 0.8.x clients should work with 0.9.x, 0.10.x and 0.11.x brokers. We have
> system tests for all the relevant combinations. One thing to be careful
> about is that Scala consumers and Java consumers store offsets and group
> management information differently and the Java consumer was only
> introduced in 0.9.0.0.
>
> Ismael
>
> On Tue, Jul 18, 2017 at 5:18 AM, John Yost <ho...@gmail.com> wrote:
>
> > Hi Everyone,
> >
> > I personally found that the 0.8.x clients do not work with 0.10.0. We
> > upgraded our clients (KafkaSpout and custom consumers) to 0.9.0.1 and
> then
> > Kafka produce/consume worked fine.
> >
> > --John
> >
> > On Tue, Jul 18, 2017 at 6:36 AM, Sachin Mittal <sj...@gmail.com>
> wrote:
> >
> > > OK.
> > >
> > > Just a doubt I have is that my broker is 0.10.2 and producer is also of
> > the
> > > same version and writes to a topic.
> > > Then I have a client from version 0.8.2 trying to fetch these messages
> > and
> > > what I see is that all messages are getting dropped.
> > >
> > > However since this is an older client I tried to fetch its offsets
> using:
> > > kafka-run-class kafka.tools.ConsumerOffsetChecker --zookeeper
> > > localhost:2181 --group test
> > >
> > > I saw that lag is 0.
> > >
> > > So probably better question would be can 0.8.2 read a message written
> by
> > > 0.10.2
> > >
> > > Thanks
> > > Sachin
> > >
> > >
> > >
> > > On Tue, Jul 18, 2017 at 3:26 PM, Tom Crayford <tc...@heroku.com>
> > > wrote:
> > >
> > > > All broker versions support all older client versions
> > > >
> > > > On Tue, Jul 18, 2017 at 10:15 AM, Sachin Mittal <sj...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi,
> > > > > This gives me some information but still not the complete picture.
> > > > >
> > > > > It says:
> > > > > 0.10.2, Java clients have acquired the ability to communicate with
> > > older
> > > > > brokers.
> > > > >
> > > > > It also says
> > > > > Version 0.11.0 brokers support 0.8.x and newer clients
> > > > >
> > > > > Question is does 0.10.2 broker support 0.8.x clients?
> > > > >
> > > > > This may solve answer my temporary doubt, but a compatibility
> matrix
> > > will
> > > > > be much helpful to have a yes/no answer.
> > > > >
> > > > > Thanks
> > > > > Sachin
> > > > >
> > > > >
> > > > >
> > > > > On Tue, Jul 18, 2017 at 2:38 PM, Michal Borowiecki <
> > > > > michal.borowiecki@openbet.com> wrote:
> > > > >
> > > > >> Have you seen this: http://kafka.apache.org/
> > > documentation.html#upgrade
> > > > >>
> > > > >> Starting with version 0.10.2, Java clients (producer and consumer)
> > > have
> > > > >> acquired the ability to communicate with older brokers. Version
> > 0.11.0
> > > > >> clients can talk to version 0.10.0 or newer brokers. However, if
> > your
> > > > >> brokers are older than 0.10.0, you must upgrade all the brokers in
> > the
> > > > >> Kafka cluster before upgrading your clients. Version 0.11.0
> brokers
> > > > support
> > > > >> 0.8.x and newer clients.
> > > > >>
> > > > >> Hope that helps.
> > > > >>
> > > > >> Cheers,
> > > > >>
> > > > >> Michał
> > > > >>
> > > > >> On 18/07/17 08:17, Sachin Mittal wrote:
> > > > >>
> > > > >> Hi,
> > > > >> I would like some help/information on what client versions are
> > > > compatible
> > > > >> with what broker versions in kafka.
> > > > >>
> > > > >> Some table like this would be good
> > > > >>
> > > > >> server
> > > > >> client 0.8 0.9 0.10 0.11
> > > > >> 0.8 yes ? ? ?
> > > > >> 0.9 ? yes ? ?
> > > > >> 0.10 ? ? yes ?
> > > > >> 0.11 ? ? ? yes
> > > > >>
> > > > >> So if question marks are filled it would be of great help.
> > > > >>
> > > > >> Reason I am asking is many times we need to use other
> > > > libraries/frameworks
> > > > >> to pull/push data from/into kafka and sometimes these support
> only a
> > > > >> particular version of clients.
> > > > >>
> > > > >> Like right now I am trying to pull data from kafka via
> > > druid/tranquility
> > > > >> and they have clients of version 0.8.x implemented but my broker
> is
> > > > running
> > > > >> 0.10.x.
> > > > >>
> > > > >> Also if such a table can be posted on kafka documentation page or
> > > github
> > > > >> page that would be great.
> > > > >>
> > > > >> Thanks
> > > > >> Sachin
> > > > >>
> > > > >>
> > > > >>
> > > > >> --
> > > > >> <http://www.openbet.com/> Michal Borowiecki
> > > > >> Senior Software Engineer L4
> > > > >> T: +44 208 742 1600 <+44%2020%208742%201600>
> <+44%2020%208742%201600>
> > > > >>
> > > > >>
> > > > >> +44 203 249 8448 <+44%2020%203249%208448>
> <+44%2020%203249%208448>
> > > > >>
> > > > >>
> > > > >>
> > > > >> E: michal.borowiecki@openbet.com
> > > > >> W: www.openbet.com
> > > > >> OpenBet Ltd
> > > > >>
> > > > >> Chiswick Park Building 9
> > > > >>
> > > > >> 566 Chiswick High Rd
> > > > >>
> > > > >> London
> > > > >>
> > > > >> W4 5XT
> > > > >>
> > > > >> UK
> > > > >> <https://www.openbet.com/email_promo>
> > > > >> This message is confidential and intended only for the addressee.
> If
> > > you
> > > > >> have received this message in error, please immediately notify the
> > > > >> postmaster@openbet.com and delete it from your system as well as
> > any
> > > > >> copies. The content of e-mails as well as traffic data may be
> > > monitored
> > > > by
> > > > >> OpenBet for employment and security purposes. To protect the
> > > environment
> > > > >> please do not print this e-mail unless necessary. OpenBet Ltd.
> > > > Registered
> > > > >> Office: Chiswick Park Building 9, 566 Chiswick High Road, London,
> W4
> > > > 5XT,
> > > > >> United Kingdom. A company registered in England and Wales.
> > Registered
> > > > no.
> > > > >> 3134634. VAT no. GB927523612
> > > > >>
> > > > >
> > > > >
> > > >
> > >
> >
>
--
Regards,
Anish Samir Mashankar
R&D Engineer
System Insights
+91-9789870733
Re: Kafka compatibility matrix needed
Posted by Sachin Mittal <sj...@gmail.com>.
Hi,
Are you referring to this topic level config?
message.format.version
Specify the message format version the broker will use to append messages
to the logs. The value should be a valid ApiVersion. Some examples are:
0.8.2, 0.9.0.0, 0.10.0, check ApiVersion for more details. By setting a
particular message format version, the user is certifying that all the
existing messages on disk are smaller or equal than the specified version.
Setting this value incorrectly will cause consumers with older versions to
break as they will receive messages with a format that they don't
understand.
So in my case since broker is of version 0.10.2 by default every topic is
created with this format? A default producer of that version will publish
message of this format?
And as a result a client of older version is not able to consume message of
this format from that topic?
Am I right in understanding the case?
Also in my case for this to work I need to set message.format.version =
0.8.2 for that topic and produce message from my default 0.10.2 client.
Then the consumer client using version 0.8.2 will correct consume message
produced by that producer.
Is this correct way of handling this case?
Thanks
Sachin
On Wed, Jul 19, 2017 at 5:43 PM, Ismael Juma <is...@juma.me.uk> wrote:
> Hi Sachin,
>
> Yes, that should work as the broker will down convert the messages for the
> older consumer. The one thing to avoid is decreasing the message format
> version for an existing topic. That is not supported.
>
> As usual, you should test this in a non production environment first. Also,
> one additional comment inline.
>
> On Wed, Jul 19, 2017 at 1:19 AM, Sachin Mittal <sj...@gmail.com> wrote:
> >
> > I btw have doubt that different versions work on same topic as message
> > format has changed from 0.8.x through 0.11.x
> >
>
> This should work fine, we have tests that update the message format version
> while clients are producing/consuming to verify this scenario.
>
> Ismael
>
Re: Kafka compatibility matrix needed
Posted by Ismael Juma <is...@juma.me.uk>.
Hi Sachin,
Yes, that should work as the broker will down convert the messages for the
older consumer. The one thing to avoid is decreasing the message format
version for an existing topic. That is not supported.
As usual, you should test this in a non production environment first. Also,
one additional comment inline.
On Wed, Jul 19, 2017 at 1:19 AM, Sachin Mittal <sj...@gmail.com> wrote:
>
> I btw have doubt that different versions work on same topic as message
> format has changed from 0.8.x through 0.11.x
>
This should work fine, we have tests that update the message format version
while clients are producing/consuming to verify this scenario.
Ismael
Re: Kafka compatibility matrix needed
Posted by Sachin Mittal <sj...@gmail.com>.
Hi,
So can anyone confirm if messages to topic produced by java client of
0.10.2 can be consumed by java client 0.8.2.
Perhaps brokers support older versions of client, but would need some
information on this case as it is not very clear.
In our case we have control over the producer but not consumer (as it is
some third party api).
So if while producing can we specify protocol version for 0.8.2, that can
also work, provided older clients work.
I btw have doubt that different versions work on same topic as message
format has changed from 0.8.x through 0.11.x
Please let us know your thoughts.
Thanks
Sachin
On Tue, Jul 18, 2017 at 6:37 PM, Ismael Juma <is...@juma.me.uk> wrote:
> Hi all,
>
> 0.8.x clients should work with 0.9.x, 0.10.x and 0.11.x brokers. We have
> system tests for all the relevant combinations. One thing to be careful
> about is that Scala consumers and Java consumers store offsets and group
> management information differently and the Java consumer was only
> introduced in 0.9.0.0.
>
> Ismael
>
> On Tue, Jul 18, 2017 at 5:18 AM, John Yost <ho...@gmail.com> wrote:
>
> > Hi Everyone,
> >
> > I personally found that the 0.8.x clients do not work with 0.10.0. We
> > upgraded our clients (KafkaSpout and custom consumers) to 0.9.0.1 and
> then
> > Kafka produce/consume worked fine.
> >
> > --John
> >
> > On Tue, Jul 18, 2017 at 6:36 AM, Sachin Mittal <sj...@gmail.com>
> wrote:
> >
> > > OK.
> > >
> > > Just a doubt I have is that my broker is 0.10.2 and producer is also of
> > the
> > > same version and writes to a topic.
> > > Then I have a client from version 0.8.2 trying to fetch these messages
> > and
> > > what I see is that all messages are getting dropped.
> > >
> > > However since this is an older client I tried to fetch its offsets
> using:
> > > kafka-run-class kafka.tools.ConsumerOffsetChecker --zookeeper
> > > localhost:2181 --group test
> > >
> > > I saw that lag is 0.
> > >
> > > So probably better question would be can 0.8.2 read a message written
> by
> > > 0.10.2
> > >
> > > Thanks
> > > Sachin
> > >
> > >
> > >
> > > On Tue, Jul 18, 2017 at 3:26 PM, Tom Crayford <tc...@heroku.com>
> > > wrote:
> > >
> > > > All broker versions support all older client versions
> > > >
> > > > On Tue, Jul 18, 2017 at 10:15 AM, Sachin Mittal <sj...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi,
> > > > > This gives me some information but still not the complete picture.
> > > > >
> > > > > It says:
> > > > > 0.10.2, Java clients have acquired the ability to communicate with
> > > older
> > > > > brokers.
> > > > >
> > > > > It also says
> > > > > Version 0.11.0 brokers support 0.8.x and newer clients
> > > > >
> > > > > Question is does 0.10.2 broker support 0.8.x clients?
> > > > >
> > > > > This may solve answer my temporary doubt, but a compatibility
> matrix
> > > will
> > > > > be much helpful to have a yes/no answer.
> > > > >
> > > > > Thanks
> > > > > Sachin
> > > > >
> > > > >
> > > > >
> > > > > On Tue, Jul 18, 2017 at 2:38 PM, Michal Borowiecki <
> > > > > michal.borowiecki@openbet.com> wrote:
> > > > >
> > > > >> Have you seen this: http://kafka.apache.org/
> > > documentation.html#upgrade
> > > > >>
> > > > >> Starting with version 0.10.2, Java clients (producer and consumer)
> > > have
> > > > >> acquired the ability to communicate with older brokers. Version
> > 0.11.0
> > > > >> clients can talk to version 0.10.0 or newer brokers. However, if
> > your
> > > > >> brokers are older than 0.10.0, you must upgrade all the brokers in
> > the
> > > > >> Kafka cluster before upgrading your clients. Version 0.11.0
> brokers
> > > > support
> > > > >> 0.8.x and newer clients.
> > > > >>
> > > > >> Hope that helps.
> > > > >>
> > > > >> Cheers,
> > > > >>
> > > > >> Michał
> > > > >>
> > > > >> On 18/07/17 08:17, Sachin Mittal wrote:
> > > > >>
> > > > >> Hi,
> > > > >> I would like some help/information on what client versions are
> > > > compatible
> > > > >> with what broker versions in kafka.
> > > > >>
> > > > >> Some table like this would be good
> > > > >>
> > > > >> server
> > > > >> client 0.8 0.9 0.10 0.11
> > > > >> 0.8 yes ? ? ?
> > > > >> 0.9 ? yes ? ?
> > > > >> 0.10 ? ? yes ?
> > > > >> 0.11 ? ? ? yes
> > > > >>
> > > > >> So if question marks are filled it would be of great help.
> > > > >>
> > > > >> Reason I am asking is many times we need to use other
> > > > libraries/frameworks
> > > > >> to pull/push data from/into kafka and sometimes these support
> only a
> > > > >> particular version of clients.
> > > > >>
> > > > >> Like right now I am trying to pull data from kafka via
> > > druid/tranquility
> > > > >> and they have clients of version 0.8.x implemented but my broker
> is
> > > > running
> > > > >> 0.10.x.
> > > > >>
> > > > >> Also if such a table can be posted on kafka documentation page or
> > > github
> > > > >> page that would be great.
> > > > >>
> > > > >> Thanks
> > > > >> Sachin
> > > > >>
> > > > >>
> > > > >>
> > > > >> --
> > > > >> <http://www.openbet.com/> Michal Borowiecki
> > > > >> Senior Software Engineer L4
> > > > >> T: +44 208 742 1600 <+44%2020%208742%201600>
> > > > >>
> > > > >>
> > > > >> +44 203 249 8448 <+44%2020%203249%208448>
> > > > >>
> > > > >>
> > > > >>
> > > > >> E: michal.borowiecki@openbet.com
> > > > >> W: www.openbet.com
> > > > >> OpenBet Ltd
> > > > >>
> > > > >> Chiswick Park Building 9
> > > > >>
> > > > >> 566 Chiswick High Rd
> > > > >>
> > > > >> London
> > > > >>
> > > > >> W4 5XT
> > > > >>
> > > > >> UK
> > > > >> <https://www.openbet.com/email_promo>
> > > > >> This message is confidential and intended only for the addressee.
> If
> > > you
> > > > >> have received this message in error, please immediately notify the
> > > > >> postmaster@openbet.com and delete it from your system as well as
> > any
> > > > >> copies. The content of e-mails as well as traffic data may be
> > > monitored
> > > > by
> > > > >> OpenBet for employment and security purposes. To protect the
> > > environment
> > > > >> please do not print this e-mail unless necessary. OpenBet Ltd.
> > > > Registered
> > > > >> Office: Chiswick Park Building 9, 566 Chiswick High Road, London,
> W4
> > > > 5XT,
> > > > >> United Kingdom. A company registered in England and Wales.
> > Registered
> > > > no.
> > > > >> 3134634. VAT no. GB927523612
> > > > >>
> > > > >
> > > > >
> > > >
> > >
> >
>
Re: Kafka compatibility matrix needed
Posted by Ismael Juma <is...@juma.me.uk>.
Hi all,
0.8.x clients should work with 0.9.x, 0.10.x and 0.11.x brokers. We have
system tests for all the relevant combinations. One thing to be careful
about is that Scala consumers and Java consumers store offsets and group
management information differently and the Java consumer was only
introduced in 0.9.0.0.
Ismael
On Tue, Jul 18, 2017 at 5:18 AM, John Yost <ho...@gmail.com> wrote:
> Hi Everyone,
>
> I personally found that the 0.8.x clients do not work with 0.10.0. We
> upgraded our clients (KafkaSpout and custom consumers) to 0.9.0.1 and then
> Kafka produce/consume worked fine.
>
> --John
>
> On Tue, Jul 18, 2017 at 6:36 AM, Sachin Mittal <sj...@gmail.com> wrote:
>
> > OK.
> >
> > Just a doubt I have is that my broker is 0.10.2 and producer is also of
> the
> > same version and writes to a topic.
> > Then I have a client from version 0.8.2 trying to fetch these messages
> and
> > what I see is that all messages are getting dropped.
> >
> > However since this is an older client I tried to fetch its offsets using:
> > kafka-run-class kafka.tools.ConsumerOffsetChecker --zookeeper
> > localhost:2181 --group test
> >
> > I saw that lag is 0.
> >
> > So probably better question would be can 0.8.2 read a message written by
> > 0.10.2
> >
> > Thanks
> > Sachin
> >
> >
> >
> > On Tue, Jul 18, 2017 at 3:26 PM, Tom Crayford <tc...@heroku.com>
> > wrote:
> >
> > > All broker versions support all older client versions
> > >
> > > On Tue, Jul 18, 2017 at 10:15 AM, Sachin Mittal <sj...@gmail.com>
> > > wrote:
> > >
> > > > Hi,
> > > > This gives me some information but still not the complete picture.
> > > >
> > > > It says:
> > > > 0.10.2, Java clients have acquired the ability to communicate with
> > older
> > > > brokers.
> > > >
> > > > It also says
> > > > Version 0.11.0 brokers support 0.8.x and newer clients
> > > >
> > > > Question is does 0.10.2 broker support 0.8.x clients?
> > > >
> > > > This may solve answer my temporary doubt, but a compatibility matrix
> > will
> > > > be much helpful to have a yes/no answer.
> > > >
> > > > Thanks
> > > > Sachin
> > > >
> > > >
> > > >
> > > > On Tue, Jul 18, 2017 at 2:38 PM, Michal Borowiecki <
> > > > michal.borowiecki@openbet.com> wrote:
> > > >
> > > >> Have you seen this: http://kafka.apache.org/
> > documentation.html#upgrade
> > > >>
> > > >> Starting with version 0.10.2, Java clients (producer and consumer)
> > have
> > > >> acquired the ability to communicate with older brokers. Version
> 0.11.0
> > > >> clients can talk to version 0.10.0 or newer brokers. However, if
> your
> > > >> brokers are older than 0.10.0, you must upgrade all the brokers in
> the
> > > >> Kafka cluster before upgrading your clients. Version 0.11.0 brokers
> > > support
> > > >> 0.8.x and newer clients.
> > > >>
> > > >> Hope that helps.
> > > >>
> > > >> Cheers,
> > > >>
> > > >> Michał
> > > >>
> > > >> On 18/07/17 08:17, Sachin Mittal wrote:
> > > >>
> > > >> Hi,
> > > >> I would like some help/information on what client versions are
> > > compatible
> > > >> with what broker versions in kafka.
> > > >>
> > > >> Some table like this would be good
> > > >>
> > > >> server
> > > >> client 0.8 0.9 0.10 0.11
> > > >> 0.8 yes ? ? ?
> > > >> 0.9 ? yes ? ?
> > > >> 0.10 ? ? yes ?
> > > >> 0.11 ? ? ? yes
> > > >>
> > > >> So if question marks are filled it would be of great help.
> > > >>
> > > >> Reason I am asking is many times we need to use other
> > > libraries/frameworks
> > > >> to pull/push data from/into kafka and sometimes these support only a
> > > >> particular version of clients.
> > > >>
> > > >> Like right now I am trying to pull data from kafka via
> > druid/tranquility
> > > >> and they have clients of version 0.8.x implemented but my broker is
> > > running
> > > >> 0.10.x.
> > > >>
> > > >> Also if such a table can be posted on kafka documentation page or
> > github
> > > >> page that would be great.
> > > >>
> > > >> Thanks
> > > >> Sachin
> > > >>
> > > >>
> > > >>
> > > >> --
> > > >> <http://www.openbet.com/> Michal Borowiecki
> > > >> Senior Software Engineer L4
> > > >> T: +44 208 742 1600 <+44%2020%208742%201600>
> > > >>
> > > >>
> > > >> +44 203 249 8448 <+44%2020%203249%208448>
> > > >>
> > > >>
> > > >>
> > > >> E: michal.borowiecki@openbet.com
> > > >> W: www.openbet.com
> > > >> OpenBet Ltd
> > > >>
> > > >> Chiswick Park Building 9
> > > >>
> > > >> 566 Chiswick High Rd
> > > >>
> > > >> London
> > > >>
> > > >> W4 5XT
> > > >>
> > > >> UK
> > > >> <https://www.openbet.com/email_promo>
> > > >> This message is confidential and intended only for the addressee. If
> > you
> > > >> have received this message in error, please immediately notify the
> > > >> postmaster@openbet.com and delete it from your system as well as
> any
> > > >> copies. The content of e-mails as well as traffic data may be
> > monitored
> > > by
> > > >> OpenBet for employment and security purposes. To protect the
> > environment
> > > >> please do not print this e-mail unless necessary. OpenBet Ltd.
> > > Registered
> > > >> Office: Chiswick Park Building 9, 566 Chiswick High Road, London, W4
> > > 5XT,
> > > >> United Kingdom. A company registered in England and Wales.
> Registered
> > > no.
> > > >> 3134634. VAT no. GB927523612
> > > >>
> > > >
> > > >
> > >
> >
>
Re: Kafka compatibility matrix needed
Posted by John Yost <ho...@gmail.com>.
Hi Everyone,
I personally found that the 0.8.x clients do not work with 0.10.0. We
upgraded our clients (KafkaSpout and custom consumers) to 0.9.0.1 and then
Kafka produce/consume worked fine.
--John
On Tue, Jul 18, 2017 at 6:36 AM, Sachin Mittal <sj...@gmail.com> wrote:
> OK.
>
> Just a doubt I have is that my broker is 0.10.2 and producer is also of the
> same version and writes to a topic.
> Then I have a client from version 0.8.2 trying to fetch these messages and
> what I see is that all messages are getting dropped.
>
> However since this is an older client I tried to fetch its offsets using:
> kafka-run-class kafka.tools.ConsumerOffsetChecker --zookeeper
> localhost:2181 --group test
>
> I saw that lag is 0.
>
> So probably better question would be can 0.8.2 read a message written by
> 0.10.2
>
> Thanks
> Sachin
>
>
>
> On Tue, Jul 18, 2017 at 3:26 PM, Tom Crayford <tc...@heroku.com>
> wrote:
>
> > All broker versions support all older client versions
> >
> > On Tue, Jul 18, 2017 at 10:15 AM, Sachin Mittal <sj...@gmail.com>
> > wrote:
> >
> > > Hi,
> > > This gives me some information but still not the complete picture.
> > >
> > > It says:
> > > 0.10.2, Java clients have acquired the ability to communicate with
> older
> > > brokers.
> > >
> > > It also says
> > > Version 0.11.0 brokers support 0.8.x and newer clients
> > >
> > > Question is does 0.10.2 broker support 0.8.x clients?
> > >
> > > This may solve answer my temporary doubt, but a compatibility matrix
> will
> > > be much helpful to have a yes/no answer.
> > >
> > > Thanks
> > > Sachin
> > >
> > >
> > >
> > > On Tue, Jul 18, 2017 at 2:38 PM, Michal Borowiecki <
> > > michal.borowiecki@openbet.com> wrote:
> > >
> > >> Have you seen this: http://kafka.apache.org/
> documentation.html#upgrade
> > >>
> > >> Starting with version 0.10.2, Java clients (producer and consumer)
> have
> > >> acquired the ability to communicate with older brokers. Version 0.11.0
> > >> clients can talk to version 0.10.0 or newer brokers. However, if your
> > >> brokers are older than 0.10.0, you must upgrade all the brokers in the
> > >> Kafka cluster before upgrading your clients. Version 0.11.0 brokers
> > support
> > >> 0.8.x and newer clients.
> > >>
> > >> Hope that helps.
> > >>
> > >> Cheers,
> > >>
> > >> Michał
> > >>
> > >> On 18/07/17 08:17, Sachin Mittal wrote:
> > >>
> > >> Hi,
> > >> I would like some help/information on what client versions are
> > compatible
> > >> with what broker versions in kafka.
> > >>
> > >> Some table like this would be good
> > >>
> > >> server
> > >> client 0.8 0.9 0.10 0.11
> > >> 0.8 yes ? ? ?
> > >> 0.9 ? yes ? ?
> > >> 0.10 ? ? yes ?
> > >> 0.11 ? ? ? yes
> > >>
> > >> So if question marks are filled it would be of great help.
> > >>
> > >> Reason I am asking is many times we need to use other
> > libraries/frameworks
> > >> to pull/push data from/into kafka and sometimes these support only a
> > >> particular version of clients.
> > >>
> > >> Like right now I am trying to pull data from kafka via
> druid/tranquility
> > >> and they have clients of version 0.8.x implemented but my broker is
> > running
> > >> 0.10.x.
> > >>
> > >> Also if such a table can be posted on kafka documentation page or
> github
> > >> page that would be great.
> > >>
> > >> Thanks
> > >> Sachin
> > >>
> > >>
> > >>
> > >> --
> > >> <http://www.openbet.com/> Michal Borowiecki
> > >> Senior Software Engineer L4
> > >> T: +44 208 742 1600 <+44%2020%208742%201600>
> > >>
> > >>
> > >> +44 203 249 8448 <+44%2020%203249%208448>
> > >>
> > >>
> > >>
> > >> E: michal.borowiecki@openbet.com
> > >> W: www.openbet.com
> > >> OpenBet Ltd
> > >>
> > >> Chiswick Park Building 9
> > >>
> > >> 566 Chiswick High Rd
> > >>
> > >> London
> > >>
> > >> W4 5XT
> > >>
> > >> UK
> > >> <https://www.openbet.com/email_promo>
> > >> This message is confidential and intended only for the addressee. If
> you
> > >> have received this message in error, please immediately notify the
> > >> postmaster@openbet.com and delete it from your system as well as any
> > >> copies. The content of e-mails as well as traffic data may be
> monitored
> > by
> > >> OpenBet for employment and security purposes. To protect the
> environment
> > >> please do not print this e-mail unless necessary. OpenBet Ltd.
> > Registered
> > >> Office: Chiswick Park Building 9, 566 Chiswick High Road, London, W4
> > 5XT,
> > >> United Kingdom. A company registered in England and Wales. Registered
> > no.
> > >> 3134634. VAT no. GB927523612
> > >>
> > >
> > >
> >
>
Re: Kafka compatibility matrix needed
Posted by Sachin Mittal <sj...@gmail.com>.
OK.
Just a doubt I have is that my broker is 0.10.2 and producer is also of the
same version and writes to a topic.
Then I have a client from version 0.8.2 trying to fetch these messages and
what I see is that all messages are getting dropped.
However since this is an older client I tried to fetch its offsets using:
kafka-run-class kafka.tools.ConsumerOffsetChecker --zookeeper
localhost:2181 --group test
I saw that lag is 0.
So probably better question would be can 0.8.2 read a message written by
0.10.2
Thanks
Sachin
On Tue, Jul 18, 2017 at 3:26 PM, Tom Crayford <tc...@heroku.com> wrote:
> All broker versions support all older client versions
>
> On Tue, Jul 18, 2017 at 10:15 AM, Sachin Mittal <sj...@gmail.com>
> wrote:
>
> > Hi,
> > This gives me some information but still not the complete picture.
> >
> > It says:
> > 0.10.2, Java clients have acquired the ability to communicate with older
> > brokers.
> >
> > It also says
> > Version 0.11.0 brokers support 0.8.x and newer clients
> >
> > Question is does 0.10.2 broker support 0.8.x clients?
> >
> > This may solve answer my temporary doubt, but a compatibility matrix will
> > be much helpful to have a yes/no answer.
> >
> > Thanks
> > Sachin
> >
> >
> >
> > On Tue, Jul 18, 2017 at 2:38 PM, Michal Borowiecki <
> > michal.borowiecki@openbet.com> wrote:
> >
> >> Have you seen this: http://kafka.apache.org/documentation.html#upgrade
> >>
> >> Starting with version 0.10.2, Java clients (producer and consumer) have
> >> acquired the ability to communicate with older brokers. Version 0.11.0
> >> clients can talk to version 0.10.0 or newer brokers. However, if your
> >> brokers are older than 0.10.0, you must upgrade all the brokers in the
> >> Kafka cluster before upgrading your clients. Version 0.11.0 brokers
> support
> >> 0.8.x and newer clients.
> >>
> >> Hope that helps.
> >>
> >> Cheers,
> >>
> >> Michał
> >>
> >> On 18/07/17 08:17, Sachin Mittal wrote:
> >>
> >> Hi,
> >> I would like some help/information on what client versions are
> compatible
> >> with what broker versions in kafka.
> >>
> >> Some table like this would be good
> >>
> >> server
> >> client 0.8 0.9 0.10 0.11
> >> 0.8 yes ? ? ?
> >> 0.9 ? yes ? ?
> >> 0.10 ? ? yes ?
> >> 0.11 ? ? ? yes
> >>
> >> So if question marks are filled it would be of great help.
> >>
> >> Reason I am asking is many times we need to use other
> libraries/frameworks
> >> to pull/push data from/into kafka and sometimes these support only a
> >> particular version of clients.
> >>
> >> Like right now I am trying to pull data from kafka via druid/tranquility
> >> and they have clients of version 0.8.x implemented but my broker is
> running
> >> 0.10.x.
> >>
> >> Also if such a table can be posted on kafka documentation page or github
> >> page that would be great.
> >>
> >> Thanks
> >> Sachin
> >>
> >>
> >>
> >> --
> >> <http://www.openbet.com/> Michal Borowiecki
> >> Senior Software Engineer L4
> >> T: +44 208 742 1600 <+44%2020%208742%201600>
> >>
> >>
> >> +44 203 249 8448 <+44%2020%203249%208448>
> >>
> >>
> >>
> >> E: michal.borowiecki@openbet.com
> >> W: www.openbet.com
> >> OpenBet Ltd
> >>
> >> Chiswick Park Building 9
> >>
> >> 566 Chiswick High Rd
> >>
> >> London
> >>
> >> W4 5XT
> >>
> >> UK
> >> <https://www.openbet.com/email_promo>
> >> This message is confidential and intended only for the addressee. If you
> >> have received this message in error, please immediately notify the
> >> postmaster@openbet.com and delete it from your system as well as any
> >> copies. The content of e-mails as well as traffic data may be monitored
> by
> >> OpenBet for employment and security purposes. To protect the environment
> >> please do not print this e-mail unless necessary. OpenBet Ltd.
> Registered
> >> Office: Chiswick Park Building 9, 566 Chiswick High Road, London, W4
> 5XT,
> >> United Kingdom. A company registered in England and Wales. Registered
> no.
> >> 3134634. VAT no. GB927523612
> >>
> >
> >
>
Re: Kafka compatibility matrix needed
Posted by Tom Crayford <tc...@heroku.com>.
All broker versions support all older client versions
On Tue, Jul 18, 2017 at 10:15 AM, Sachin Mittal <sj...@gmail.com> wrote:
> Hi,
> This gives me some information but still not the complete picture.
>
> It says:
> 0.10.2, Java clients have acquired the ability to communicate with older
> brokers.
>
> It also says
> Version 0.11.0 brokers support 0.8.x and newer clients
>
> Question is does 0.10.2 broker support 0.8.x clients?
>
> This may solve answer my temporary doubt, but a compatibility matrix will
> be much helpful to have a yes/no answer.
>
> Thanks
> Sachin
>
>
>
> On Tue, Jul 18, 2017 at 2:38 PM, Michal Borowiecki <
> michal.borowiecki@openbet.com> wrote:
>
>> Have you seen this: http://kafka.apache.org/documentation.html#upgrade
>>
>> Starting with version 0.10.2, Java clients (producer and consumer) have
>> acquired the ability to communicate with older brokers. Version 0.11.0
>> clients can talk to version 0.10.0 or newer brokers. However, if your
>> brokers are older than 0.10.0, you must upgrade all the brokers in the
>> Kafka cluster before upgrading your clients. Version 0.11.0 brokers support
>> 0.8.x and newer clients.
>>
>> Hope that helps.
>>
>> Cheers,
>>
>> Michał
>>
>> On 18/07/17 08:17, Sachin Mittal wrote:
>>
>> Hi,
>> I would like some help/information on what client versions are compatible
>> with what broker versions in kafka.
>>
>> Some table like this would be good
>>
>> server
>> client 0.8 0.9 0.10 0.11
>> 0.8 yes ? ? ?
>> 0.9 ? yes ? ?
>> 0.10 ? ? yes ?
>> 0.11 ? ? ? yes
>>
>> So if question marks are filled it would be of great help.
>>
>> Reason I am asking is many times we need to use other libraries/frameworks
>> to pull/push data from/into kafka and sometimes these support only a
>> particular version of clients.
>>
>> Like right now I am trying to pull data from kafka via druid/tranquility
>> and they have clients of version 0.8.x implemented but my broker is running
>> 0.10.x.
>>
>> Also if such a table can be posted on kafka documentation page or github
>> page that would be great.
>>
>> Thanks
>> Sachin
>>
>>
>>
>> --
>> <http://www.openbet.com/> Michal Borowiecki
>> Senior Software Engineer L4
>> T: +44 208 742 1600 <+44%2020%208742%201600>
>>
>>
>> +44 203 249 8448 <+44%2020%203249%208448>
>>
>>
>>
>> E: michal.borowiecki@openbet.com
>> W: www.openbet.com
>> OpenBet Ltd
>>
>> Chiswick Park Building 9
>>
>> 566 Chiswick High Rd
>>
>> London
>>
>> W4 5XT
>>
>> UK
>> <https://www.openbet.com/email_promo>
>> This message is confidential and intended only for the addressee. If you
>> have received this message in error, please immediately notify the
>> postmaster@openbet.com and delete it from your system as well as any
>> copies. The content of e-mails as well as traffic data may be monitored by
>> OpenBet for employment and security purposes. To protect the environment
>> please do not print this e-mail unless necessary. OpenBet Ltd. Registered
>> Office: Chiswick Park Building 9, 566 Chiswick High Road, London, W4 5XT,
>> United Kingdom. A company registered in England and Wales. Registered no.
>> 3134634. VAT no. GB927523612
>>
>
>
Re: Kafka compatibility matrix needed
Posted by Sachin Mittal <sj...@gmail.com>.
Hi,
This gives me some information but still not the complete picture.
It says:
0.10.2, Java clients have acquired the ability to communicate with older
brokers.
It also says
Version 0.11.0 brokers support 0.8.x and newer clients
Question is does 0.10.2 broker support 0.8.x clients?
This may solve answer my temporary doubt, but a compatibility matrix will
be much helpful to have a yes/no answer.
Thanks
Sachin
On Tue, Jul 18, 2017 at 2:38 PM, Michal Borowiecki <
michal.borowiecki@openbet.com> wrote:
> Have you seen this: http://kafka.apache.org/documentation.html#upgrade
>
> Starting with version 0.10.2, Java clients (producer and consumer) have
> acquired the ability to communicate with older brokers. Version 0.11.0
> clients can talk to version 0.10.0 or newer brokers. However, if your
> brokers are older than 0.10.0, you must upgrade all the brokers in the
> Kafka cluster before upgrading your clients. Version 0.11.0 brokers support
> 0.8.x and newer clients.
>
> Hope that helps.
>
> Cheers,
>
> Michał
>
> On 18/07/17 08:17, Sachin Mittal wrote:
>
> Hi,
> I would like some help/information on what client versions are compatible
> with what broker versions in kafka.
>
> Some table like this would be good
>
> server
> client 0.8 0.9 0.10 0.11
> 0.8 yes ? ? ?
> 0.9 ? yes ? ?
> 0.10 ? ? yes ?
> 0.11 ? ? ? yes
>
> So if question marks are filled it would be of great help.
>
> Reason I am asking is many times we need to use other libraries/frameworks
> to pull/push data from/into kafka and sometimes these support only a
> particular version of clients.
>
> Like right now I am trying to pull data from kafka via druid/tranquility
> and they have clients of version 0.8.x implemented but my broker is running
> 0.10.x.
>
> Also if such a table can be posted on kafka documentation page or github
> page that would be great.
>
> Thanks
> Sachin
>
>
>
> --
> <http://www.openbet.com/> Michal Borowiecki
> Senior Software Engineer L4
> T: +44 208 742 1600
>
>
> +44 203 249 8448
>
>
>
> E: michal.borowiecki@openbet.com
> W: www.openbet.com
> OpenBet Ltd
>
> Chiswick Park Building 9
>
> 566 Chiswick High Rd
>
> London
>
> W4 5XT
>
> UK
> <https://www.openbet.com/email_promo>
> This message is confidential and intended only for the addressee. If you
> have received this message in error, please immediately notify the
> postmaster@openbet.com and delete it from your system as well as any
> copies. The content of e-mails as well as traffic data may be monitored by
> OpenBet for employment and security purposes. To protect the environment
> please do not print this e-mail unless necessary. OpenBet Ltd. Registered
> Office: Chiswick Park Building 9, 566 Chiswick High Road, London, W4 5XT,
> United Kingdom. A company registered in England and Wales. Registered no.
> 3134634. VAT no. GB927523612
>
Re: Kafka compatibility matrix needed
Posted by Michal Borowiecki <mi...@openbet.com>.
Have you seen this: http://kafka.apache.org/documentation.html#upgrade
> Starting with version 0.10.2, Java clients (producer and consumer)
> have acquired the ability to communicate with older brokers. Version
> 0.11.0 clients can talk to version 0.10.0 or newer brokers. However,
> if your brokers are older than 0.10.0, you must upgrade all the
> brokers in the Kafka cluster before upgrading your clients. Version
> 0.11.0 brokers support 0.8.x and newer clients.
Hope that helps.
Cheers,
Michał
On 18/07/17 08:17, Sachin Mittal wrote:
> Hi,
> I would like some help/information on what client versions are compatible
> with what broker versions in kafka.
>
> Some table like this would be good
>
> server
> client 0.8 0.9 0.10 0.11
> 0.8 yes ? ? ?
> 0.9 ? yes ? ?
> 0.10 ? ? yes ?
> 0.11 ? ? ? yes
>
> So if question marks are filled it would be of great help.
>
> Reason I am asking is many times we need to use other libraries/frameworks
> to pull/push data from/into kafka and sometimes these support only a
> particular version of clients.
>
> Like right now I am trying to pull data from kafka via druid/tranquility
> and they have clients of version 0.8.x implemented but my broker is running
> 0.10.x.
>
> Also if such a table can be posted on kafka documentation page or github
> page that would be great.
>
> Thanks
> Sachin
>
--
Signature
<http://www.openbet.com/> Michal Borowiecki
Senior Software Engineer L4
T: +44 208 742 1600
+44 203 249 8448
E: michal.borowiecki@openbet.com
W: www.openbet.com <http://www.openbet.com/>
OpenBet Ltd
Chiswick Park Building 9
566 Chiswick High Rd
London
W4 5XT
UK
<https://www.openbet.com/email_promo>
This message is confidential and intended only for the addressee. If you
have received this message in error, please immediately notify the
postmaster@openbet.com <ma...@openbet.com> and delete it
from your system as well as any copies. The content of e-mails as well
as traffic data may be monitored by OpenBet for employment and security
purposes. To protect the environment please do not print this e-mail
unless necessary. OpenBet Ltd. Registered Office: Chiswick Park Building
9, 566 Chiswick High Road, London, W4 5XT, United Kingdom. A company
registered in England and Wales. Registered no. 3134634. VAT no.
GB927523612