Fwd: Question on KafkaInputOperator - with kafka 0.8.2.1

[Thomas Weise <th...@gmail.com>]

We have had this question often.

I think it would be good to bump the version in malhar-contrib as most
users are on 0.8.2?

---------- Forwarded message ----------
From: Thomas Weise <th...@gmail.com>
Date: Tue, Jul 5, 2016 at 2:20 PM
Subject: Re: Question on KafkaInputOperator - with kafka 0.8.2.1
To: users@apex.apache.org


Hi Eric,

You need to use the 0.8.1 version of the client library. It can talk to the
0.8.2 cluster.

Thanks,
Thomas


On Tue, Jul 5, 2016 at 2:16 PM, Martin, Eric <Er...@capitalone.com>
wrote:

> Hi all,
>
>
>
> I am using the 0.8.x version of the Kafka Input Operators. When using
> these operators, I am able to run the application fine while using kafka
> version 0.8.1.1. However, if I use version 0.8.2.x, I am not able to run
> the application even though the kafka cluster that I am connecting is
> running on 0.8.2.1.
>
>
>
> The exception that I am getting is:
>
> *java.lang.NoSuchMethodError:
> kafka.cluster.Broker.getConnectionString()Ljava/lang/String;*
>
>
>
> *at
> com.datatorrent.contrib.kafka.KafkaMetadataUtil.getBrokers(KafkaMetadataUtil.java:114)*
>
> *at
> com.datatorrent.contrib.kafka.KafkaConsumer.initBrokers(KafkaConsumer.java:131)*
>
> *at
> com.datatorrent.contrib.kafka.AbstractKafkaInputOperator.definePartitions(AbstractKafkaInputOperator.java:488)*
>
>
>
> It would appear that the getConnectionString() method is not supported in
> kafka 0.8.2.x. Using 0.8.1.x is not a huge issue for most things, but I am
> looking to keep track of offsets in topics and this only appears to be
> available with kafka 0.8.2.x.
>
>
>
> Is there any known work around for this issue, so that I can use 0.8.2.x
> in the application?
>
>
>
> Thanks,
>
> Eric
>
> ------------------------------
>
> The information contained in this e-mail is confidential and/or
> proprietary to Capital One and/or its affiliates and may only be used
> solely in performance of work or services for Capital One. The information
> transmitted herewith is intended only for use by the individual or entity
> to which it is addressed. If the reader of this message is not the intended
> recipient, you are hereby notified that any review, retransmission,
> dissemination, distribution, copying or other use of, or taking of any
> action in reliance upon this information is strictly prohibited. If you
> have received this communication in error, please contact the sender and
> delete the material from your computer.
>

Re: Question on KafkaInputOperator - with kafka 0.8.2.1

[Thomas Weise <th...@datatorrent.com>]

I don't see the need to support 0.8.1 client. We should depend on what
users need.



On Wed, Jul 6, 2016 at 8:30 PM, Siyuan Hua <si...@datatorrent.com> wrote:

> Thomas,
>
> I think a if condition check should be able to make this util class support
> both 0.8.1.* and 0.8.2.*
>
> Regards,
> Siyuan
>
> On Tue, Jul 5, 2016 at 2:22 PM, Thomas Weise <th...@gmail.com>
> wrote:
>
> > We have had this question often.
> >
> > I think it would be good to bump the version in malhar-contrib as most
> > users are on 0.8.2?
> >
> > ---------- Forwarded message ----------
> > From: Thomas Weise <th...@gmail.com>
> > Date: Tue, Jul 5, 2016 at 2:20 PM
> > Subject: Re: Question on KafkaInputOperator - with kafka 0.8.2.1
> > To: users@apex.apache.org
> >
> >
> > Hi Eric,
> >
> > You need to use the 0.8.1 version of the client library. It can talk to
> the
> > 0.8.2 cluster.
> >
> > Thanks,
> > Thomas
> >
> >
> > On Tue, Jul 5, 2016 at 2:16 PM, Martin, Eric <
> Eric.Martin2@capitalone.com>
> > wrote:
> >
> > > Hi all,
> > >
> > >
> > >
> > > I am using the 0.8.x version of the Kafka Input Operators. When using
> > > these operators, I am able to run the application fine while using
> kafka
> > > version 0.8.1.1. However, if I use version 0.8.2.x, I am not able to
> run
> > > the application even though the kafka cluster that I am connecting is
> > > running on 0.8.2.1.
> > >
> > >
> > >
> > > The exception that I am getting is:
> > >
> > > *java.lang.NoSuchMethodError:
> > > kafka.cluster.Broker.getConnectionString()Ljava/lang/String;*
> > >
> > >
> > >
> > > *at
> > >
> >
> com.datatorrent.contrib.kafka.KafkaMetadataUtil.getBrokers(KafkaMetadataUtil.java:114)*
> > >
> > > *at
> > >
> >
> com.datatorrent.contrib.kafka.KafkaConsumer.initBrokers(KafkaConsumer.java:131)*
> > >
> > > *at
> > >
> >
> com.datatorrent.contrib.kafka.AbstractKafkaInputOperator.definePartitions(AbstractKafkaInputOperator.java:488)*
> > >
> > >
> > >
> > > It would appear that the getConnectionString() method is not supported
> in
> > > kafka 0.8.2.x. Using 0.8.1.x is not a huge issue for most things, but I
> > am
> > > looking to keep track of offsets in topics and this only appears to be
> > > available with kafka 0.8.2.x.
> > >
> > >
> > >
> > > Is there any known work around for this issue, so that I can use
> 0.8.2.x
> > > in the application?
> > >
> > >
> > >
> > > Thanks,
> > >
> > > Eric
> > >
> > > ------------------------------
> > >
> > > The information contained in this e-mail is confidential and/or
> > > proprietary to Capital One and/or its affiliates and may only be used
> > > solely in performance of work or services for Capital One. The
> > information
> > > transmitted herewith is intended only for use by the individual or
> entity
> > > to which it is addressed. If the reader of this message is not the
> > intended
> > > recipient, you are hereby notified that any review, retransmission,
> > > dissemination, distribution, copying or other use of, or taking of any
> > > action in reliance upon this information is strictly prohibited. If you
> > > have received this communication in error, please contact the sender
> and
> > > delete the material from your computer.
> > >
> >
>

Re: Question on KafkaInputOperator - with kafka 0.8.2.1

[Siyuan Hua <si...@datatorrent.com>]

Thomas,

I think a if condition check should be able to make this util class support
both 0.8.1.* and 0.8.2.*

Regards,
Siyuan

On Tue, Jul 5, 2016 at 2:22 PM, Thomas Weise <th...@gmail.com> wrote:

> We have had this question often.
>
> I think it would be good to bump the version in malhar-contrib as most
> users are on 0.8.2?
>
> ---------- Forwarded message ----------
> From: Thomas Weise <th...@gmail.com>
> Date: Tue, Jul 5, 2016 at 2:20 PM
> Subject: Re: Question on KafkaInputOperator - with kafka 0.8.2.1
> To: users@apex.apache.org
>
>
> Hi Eric,
>
> You need to use the 0.8.1 version of the client library. It can talk to the
> 0.8.2 cluster.
>
> Thanks,
> Thomas
>
>
> On Tue, Jul 5, 2016 at 2:16 PM, Martin, Eric <Er...@capitalone.com>
> wrote:
>
> > Hi all,
> >
> >
> >
> > I am using the 0.8.x version of the Kafka Input Operators. When using
> > these operators, I am able to run the application fine while using kafka
> > version 0.8.1.1. However, if I use version 0.8.2.x, I am not able to run
> > the application even though the kafka cluster that I am connecting is
> > running on 0.8.2.1.
> >
> >
> >
> > The exception that I am getting is:
> >
> > *java.lang.NoSuchMethodError:
> > kafka.cluster.Broker.getConnectionString()Ljava/lang/String;*
> >
> >
> >
> > *at
> >
> com.datatorrent.contrib.kafka.KafkaMetadataUtil.getBrokers(KafkaMetadataUtil.java:114)*
> >
> > *at
> >
> com.datatorrent.contrib.kafka.KafkaConsumer.initBrokers(KafkaConsumer.java:131)*
> >
> > *at
> >
> com.datatorrent.contrib.kafka.AbstractKafkaInputOperator.definePartitions(AbstractKafkaInputOperator.java:488)*
> >
> >
> >
> > It would appear that the getConnectionString() method is not supported in
> > kafka 0.8.2.x. Using 0.8.1.x is not a huge issue for most things, but I
> am
> > looking to keep track of offsets in topics and this only appears to be
> > available with kafka 0.8.2.x.
> >
> >
> >
> > Is there any known work around for this issue, so that I can use 0.8.2.x
> > in the application?
> >
> >
> >
> > Thanks,
> >
> > Eric
> >
> > ------------------------------
> >
> > The information contained in this e-mail is confidential and/or
> > proprietary to Capital One and/or its affiliates and may only be used
> > solely in performance of work or services for Capital One. The
> information
> > transmitted herewith is intended only for use by the individual or entity
> > to which it is addressed. If the reader of this message is not the
> intended
> > recipient, you are hereby notified that any review, retransmission,
> > dissemination, distribution, copying or other use of, or taking of any
> > action in reliance upon this information is strictly prohibited. If you
> > have received this communication in error, please contact the sender and
> > delete the material from your computer.
> >
>