Apache Kafka 0.8.2 Consumer Example

[Ricardo Ferreira <jr...@gmail.com>]

Hi,

I had a client running on Kafka 0.8.1 using the High-Level consumer API
working fine.

Today I updated my Kafka installation for the 0.8.2 version (tried all
versions of Scala) but the consumer doesn't get any messages. I tested
using the kafka-console-consumer.sh utility tool and works fine, only my
Java program that not.

Did I miss something? I heard that the API changed, so I'd like to know if
someone can share a simple client with me.

Please, respond directly to me or just reply all because I am not currently
subscribed to the group.

Thanks,

Ricardo Ferreira

Re: Apache Kafka 0.8.2 Consumer Example

[Achanta Vamsi Subhash <ac...@flipkart.com>]

High level consumer of 0.8.1 works fine with 0.8.2. In extra, you can
change the config to use kafka for offsets storage instead of zookeeper.
There are some extra config parameters added as well as explained in the
wiki.
http://kafka.apache.org/documentation.html#consumerconfigs

For low-level consumer, you can look here:
https://cwiki.apache.org/confluence/display/KAFKA/Committing+and+fetching+consumer+offsets+in+Kafka


On Mon, Feb 9, 2015 at 10:23 AM, Jaikiran Pai <ja...@gmail.com>
wrote:

> Ricardo,
>
> What does your consumer code look like? If it's too big to have that code
> inline in this mail, you can perhaps put it in a github repo or even a gist.
>
> -Jaikiran
>
>
> On Monday 09 February 2015 12:54 AM, Ricardo Ferreira wrote:
>
>> Hi Gwen,
>>
>> Sorry, both the consumer and the broker are 0.8.2?
>> A = Yes
>>
>> So what's on 0.8.1?
>> A = It works fine using 0.8.1 for server AND client.
>>
>> You probably know the consumer group of your application. Can you use the
>> offset checker tool on that?
>> A = Yes, I know from the consumer, and the offset checker gave me nothing
>> about that group.
>>
>> Thanks,
>>
>> Ricardo
>>
>> On Sun, Feb 8, 2015 at 1:19 PM, Gwen Shapira <gs...@cloudera.com>
>> wrote:
>>
>>  Sorry, both the consumer and the broker are 0.8.2?
>>>
>>> So what's on 0.8.1?
>>>
>>> I seriously doubt downgrading is the solution.
>>>
>>> You probably know the consumer group of your application. Can you use the
>>> offset checker tool on that?
>>>
>>> Gwen
>>>
>>> On Sun, Feb 8, 2015 at 9:01 AM, Ricardo Ferreira <
>>> jricardoferreira@gmail.com> wrote:
>>>
>>>  Hi Gwen,
>>>>
>>>> Thanks for the response.
>>>>
>>>> In my case, I have both consumer application and the server versions in
>>>> 0.8.2, Scala 2.10.
>>>>
>>>> No errors are thrown, and my *zookeeper.session.timeout.ms
>>>> <http://zookeeper.session.timeout.ms>* property is set to 500, although
>>>> I tried 5000 and also didn't worked.
>>>>
>>>> I checked the offset checker tool, but it asks for a group in which I
>>>> don't know which group the kafka-console-producer is using. I tried the
>>>> consumer group but it gave the following message:
>>>>
>>>> Exiting due to: org.apache.zookeeper.KeeperException$NoNodeException:
>>>> KeeperErrorCode = NoNode for /consumers/test-consumer-
>>>> group/offsets/test/0.
>>>>
>>>> Perhaps the solution is downgrade the consumer libs to 0.8.1?
>>>>
>>>> Thanks,
>>>>
>>>> Ricardo
>>>>
>>>> On Sun, Feb 8, 2015 at 11:27 AM, Gwen Shapira <gs...@cloudera.com>
>>>> wrote:
>>>>
>>>>  I have a 0.8.1 high level consumer working fine with 0.8.2 server. Few
>>>>> of them actually :)
>>>>> AFAIK the API did not change.
>>>>>
>>>>> Do you see any error messages? Do you have timeout configured on the
>>>>> consumer? What does the offset checker tool say?
>>>>>
>>>>> On Fri, Feb 6, 2015 at 4:49 PM, Ricardo Ferreira <
>>>>> jricardoferreira@gmail.com> wrote:
>>>>>
>>>>>  Hi,
>>>>>>
>>>>>> I had a client running on Kafka 0.8.1 using the High-Level consumer
>>>>>> API
>>>>>> working fine.
>>>>>>
>>>>>> Today I updated my Kafka installation for the 0.8.2 version (tried all
>>>>>> versions of Scala) but the consumer doesn't get any messages. I tested
>>>>>> using the kafka-console-consumer.sh utility tool and works fine, only
>>>>>> my
>>>>>> Java program that not.
>>>>>>
>>>>>> Did I miss something? I heard that the API changed, so I'd like to
>>>>>> know
>>>>>> if
>>>>>> someone can share a simple client with me.
>>>>>>
>>>>>> Please, respond directly to me or just reply all because I am not
>>>>>> currently
>>>>>> subscribed to the group.
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Ricardo Ferreira
>>>>>>
>>>>>>
>>>>>
>


-- 
Regards
Vamsi Subhash

Re: Apache Kafka 0.8.2 Consumer Example

[Jaikiran Pai <ja...@gmail.com>]

Ricardo,

What does your consumer code look like? If it's too big to have that 
code inline in this mail, you can perhaps put it in a github repo or 
even a gist.

-Jaikiran

On Monday 09 February 2015 12:54 AM, Ricardo Ferreira wrote:
> Hi Gwen,
>
> Sorry, both the consumer and the broker are 0.8.2?
> A = Yes
>
> So what's on 0.8.1?
> A = It works fine using 0.8.1 for server AND client.
>
> You probably know the consumer group of your application. Can you use the
> offset checker tool on that?
> A = Yes, I know from the consumer, and the offset checker gave me nothing
> about that group.
>
> Thanks,
>
> Ricardo
>
> On Sun, Feb 8, 2015 at 1:19 PM, Gwen Shapira <gs...@cloudera.com> wrote:
>
>> Sorry, both the consumer and the broker are 0.8.2?
>>
>> So what's on 0.8.1?
>>
>> I seriously doubt downgrading is the solution.
>>
>> You probably know the consumer group of your application. Can you use the
>> offset checker tool on that?
>>
>> Gwen
>>
>> On Sun, Feb 8, 2015 at 9:01 AM, Ricardo Ferreira <
>> jricardoferreira@gmail.com> wrote:
>>
>>> Hi Gwen,
>>>
>>> Thanks for the response.
>>>
>>> In my case, I have both consumer application and the server versions in
>>> 0.8.2, Scala 2.10.
>>>
>>> No errors are thrown, and my *zookeeper.session.timeout.ms
>>> <http://zookeeper.session.timeout.ms>* property is set to 500, although
>>> I tried 5000 and also didn't worked.
>>>
>>> I checked the offset checker tool, but it asks for a group in which I
>>> don't know which group the kafka-console-producer is using. I tried the
>>> consumer group but it gave the following message:
>>>
>>> Exiting due to: org.apache.zookeeper.KeeperException$NoNodeException:
>>> KeeperErrorCode = NoNode for /consumers/test-consumer-group/offsets/test/0.
>>>
>>> Perhaps the solution is downgrade the consumer libs to 0.8.1?
>>>
>>> Thanks,
>>>
>>> Ricardo
>>>
>>> On Sun, Feb 8, 2015 at 11:27 AM, Gwen Shapira <gs...@cloudera.com>
>>> wrote:
>>>
>>>> I have a 0.8.1 high level consumer working fine with 0.8.2 server. Few
>>>> of them actually :)
>>>> AFAIK the API did not change.
>>>>
>>>> Do you see any error messages? Do you have timeout configured on the
>>>> consumer? What does the offset checker tool say?
>>>>
>>>> On Fri, Feb 6, 2015 at 4:49 PM, Ricardo Ferreira <
>>>> jricardoferreira@gmail.com> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I had a client running on Kafka 0.8.1 using the High-Level consumer API
>>>>> working fine.
>>>>>
>>>>> Today I updated my Kafka installation for the 0.8.2 version (tried all
>>>>> versions of Scala) but the consumer doesn't get any messages. I tested
>>>>> using the kafka-console-consumer.sh utility tool and works fine, only my
>>>>> Java program that not.
>>>>>
>>>>> Did I miss something? I heard that the API changed, so I'd like to know
>>>>> if
>>>>> someone can share a simple client with me.
>>>>>
>>>>> Please, respond directly to me or just reply all because I am not
>>>>> currently
>>>>> subscribed to the group.
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Ricardo Ferreira
>>>>>
>>>>

Re: Apache Kafka 0.8.2 Consumer Example

[Gwen Shapira <gs...@cloudera.com>]

I'm guessing the upgrade changed your broker configuration file
(server.properties).

Perhaps take a look and see if things like max.message.bytes are still
where you want them?

Gwen

On Sun, Feb 8, 2015 at 11:24 AM, Ricardo Ferreira <
jricardoferreira@gmail.com> wrote:

> Hi Gwen,
>
> Sorry, both the consumer and the broker are 0.8.2?
> A = Yes
>
> So what's on 0.8.1?
> A = It works fine using 0.8.1 for server AND client.
>
> You probably know the consumer group of your application. Can you use the
> offset checker tool on that?
> A = Yes, I know from the consumer, and the offset checker gave me nothing
> about that group.
>
> Thanks,
>
> Ricardo
>
> On Sun, Feb 8, 2015 at 1:19 PM, Gwen Shapira <gs...@cloudera.com>
> wrote:
>
>> Sorry, both the consumer and the broker are 0.8.2?
>>
>> So what's on 0.8.1?
>>
>> I seriously doubt downgrading is the solution.
>>
>> You probably know the consumer group of your application. Can you use the
>> offset checker tool on that?
>>
>> Gwen
>>
>> On Sun, Feb 8, 2015 at 9:01 AM, Ricardo Ferreira <
>> jricardoferreira@gmail.com> wrote:
>>
>>> Hi Gwen,
>>>
>>> Thanks for the response.
>>>
>>> In my case, I have both consumer application and the server versions in
>>> 0.8.2, Scala 2.10.
>>>
>>> No errors are thrown, and my *zookeeper.session.timeout.ms
>>> <http://zookeeper.session.timeout.ms>* property is set to 500, although
>>> I tried 5000 and also didn't worked.
>>>
>>> I checked the offset checker tool, but it asks for a group in which I
>>> don't know which group the kafka-console-producer is using. I tried the
>>> consumer group but it gave the following message:
>>>
>>> Exiting due to: org.apache.zookeeper.KeeperException$NoNodeException:
>>> KeeperErrorCode = NoNode for /consumers/test-consumer-group/offsets/test/0.
>>>
>>> Perhaps the solution is downgrade the consumer libs to 0.8.1?
>>>
>>> Thanks,
>>>
>>> Ricardo
>>>
>>> On Sun, Feb 8, 2015 at 11:27 AM, Gwen Shapira <gs...@cloudera.com>
>>> wrote:
>>>
>>>> I have a 0.8.1 high level consumer working fine with 0.8.2 server. Few
>>>> of them actually :)
>>>> AFAIK the API did not change.
>>>>
>>>> Do you see any error messages? Do you have timeout configured on the
>>>> consumer? What does the offset checker tool say?
>>>>
>>>> On Fri, Feb 6, 2015 at 4:49 PM, Ricardo Ferreira <
>>>> jricardoferreira@gmail.com> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I had a client running on Kafka 0.8.1 using the High-Level consumer API
>>>>> working fine.
>>>>>
>>>>> Today I updated my Kafka installation for the 0.8.2 version (tried all
>>>>> versions of Scala) but the consumer doesn't get any messages. I tested
>>>>> using the kafka-console-consumer.sh utility tool and works fine, only
>>>>> my
>>>>> Java program that not.
>>>>>
>>>>> Did I miss something? I heard that the API changed, so I'd like to
>>>>> know if
>>>>> someone can share a simple client with me.
>>>>>
>>>>> Please, respond directly to me or just reply all because I am not
>>>>> currently
>>>>> subscribed to the group.
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Ricardo Ferreira
>>>>>
>>>>
>>>>
>>>
>>
>

Re: Apache Kafka 0.8.2 Consumer Example

[Ricardo Ferreira <jr...@gmail.com>]

Hi Gwen,

Sorry, both the consumer and the broker are 0.8.2?
A = Yes

So what's on 0.8.1?
A = It works fine using 0.8.1 for server AND client.

You probably know the consumer group of your application. Can you use the
offset checker tool on that?
A = Yes, I know from the consumer, and the offset checker gave me nothing
about that group.

Thanks,

Ricardo

On Sun, Feb 8, 2015 at 1:19 PM, Gwen Shapira <gs...@cloudera.com> wrote:

> Sorry, both the consumer and the broker are 0.8.2?
>
> So what's on 0.8.1?
>
> I seriously doubt downgrading is the solution.
>
> You probably know the consumer group of your application. Can you use the
> offset checker tool on that?
>
> Gwen
>
> On Sun, Feb 8, 2015 at 9:01 AM, Ricardo Ferreira <
> jricardoferreira@gmail.com> wrote:
>
>> Hi Gwen,
>>
>> Thanks for the response.
>>
>> In my case, I have both consumer application and the server versions in
>> 0.8.2, Scala 2.10.
>>
>> No errors are thrown, and my *zookeeper.session.timeout.ms
>> <http://zookeeper.session.timeout.ms>* property is set to 500, although
>> I tried 5000 and also didn't worked.
>>
>> I checked the offset checker tool, but it asks for a group in which I
>> don't know which group the kafka-console-producer is using. I tried the
>> consumer group but it gave the following message:
>>
>> Exiting due to: org.apache.zookeeper.KeeperException$NoNodeException:
>> KeeperErrorCode = NoNode for /consumers/test-consumer-group/offsets/test/0.
>>
>> Perhaps the solution is downgrade the consumer libs to 0.8.1?
>>
>> Thanks,
>>
>> Ricardo
>>
>> On Sun, Feb 8, 2015 at 11:27 AM, Gwen Shapira <gs...@cloudera.com>
>> wrote:
>>
>>> I have a 0.8.1 high level consumer working fine with 0.8.2 server. Few
>>> of them actually :)
>>> AFAIK the API did not change.
>>>
>>> Do you see any error messages? Do you have timeout configured on the
>>> consumer? What does the offset checker tool say?
>>>
>>> On Fri, Feb 6, 2015 at 4:49 PM, Ricardo Ferreira <
>>> jricardoferreira@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> I had a client running on Kafka 0.8.1 using the High-Level consumer API
>>>> working fine.
>>>>
>>>> Today I updated my Kafka installation for the 0.8.2 version (tried all
>>>> versions of Scala) but the consumer doesn't get any messages. I tested
>>>> using the kafka-console-consumer.sh utility tool and works fine, only my
>>>> Java program that not.
>>>>
>>>> Did I miss something? I heard that the API changed, so I'd like to know
>>>> if
>>>> someone can share a simple client with me.
>>>>
>>>> Please, respond directly to me or just reply all because I am not
>>>> currently
>>>> subscribed to the group.
>>>>
>>>> Thanks,
>>>>
>>>> Ricardo Ferreira
>>>>
>>>
>>>
>>
>

Re: Apache Kafka 0.8.2 Consumer Example

[Gwen Shapira <gs...@cloudera.com>]

Sorry, both the consumer and the broker are 0.8.2?

So what's on 0.8.1?

I seriously doubt downgrading is the solution.

You probably know the consumer group of your application. Can you use the
offset checker tool on that?

Gwen

On Sun, Feb 8, 2015 at 9:01 AM, Ricardo Ferreira <jricardoferreira@gmail.com
> wrote:

> Hi Gwen,
>
> Thanks for the response.
>
> In my case, I have both consumer application and the server versions in
> 0.8.2, Scala 2.10.
>
> No errors are thrown, and my *zookeeper.session.timeout.ms
> <http://zookeeper.session.timeout.ms>* property is set to 500, although I
> tried 5000 and also didn't worked.
>
> I checked the offset checker tool, but it asks for a group in which I
> don't know which group the kafka-console-producer is using. I tried the
> consumer group but it gave the following message:
>
> Exiting due to: org.apache.zookeeper.KeeperException$NoNodeException:
> KeeperErrorCode = NoNode for /consumers/test-consumer-group/offsets/test/0.
>
> Perhaps the solution is downgrade the consumer libs to 0.8.1?
>
> Thanks,
>
> Ricardo
>
> On Sun, Feb 8, 2015 at 11:27 AM, Gwen Shapira <gs...@cloudera.com>
> wrote:
>
>> I have a 0.8.1 high level consumer working fine with 0.8.2 server. Few of
>> them actually :)
>> AFAIK the API did not change.
>>
>> Do you see any error messages? Do you have timeout configured on the
>> consumer? What does the offset checker tool say?
>>
>> On Fri, Feb 6, 2015 at 4:49 PM, Ricardo Ferreira <
>> jricardoferreira@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> I had a client running on Kafka 0.8.1 using the High-Level consumer API
>>> working fine.
>>>
>>> Today I updated my Kafka installation for the 0.8.2 version (tried all
>>> versions of Scala) but the consumer doesn't get any messages. I tested
>>> using the kafka-console-consumer.sh utility tool and works fine, only my
>>> Java program that not.
>>>
>>> Did I miss something? I heard that the API changed, so I'd like to know
>>> if
>>> someone can share a simple client with me.
>>>
>>> Please, respond directly to me or just reply all because I am not
>>> currently
>>> subscribed to the group.
>>>
>>> Thanks,
>>>
>>> Ricardo Ferreira
>>>
>>
>>
>

Re: Apache Kafka 0.8.2 Consumer Example

[Ricardo Ferreira <jr...@gmail.com>]

Hi Gwen,

Thanks for the response.

In my case, I have both consumer application and the server versions in
0.8.2, Scala 2.10.

No errors are thrown, and my *zookeeper.session.timeout.ms
<http://zookeeper.session.timeout.ms>* property is set to 500, although I
tried 5000 and also didn't worked.

I checked the offset checker tool, but it asks for a group in which I don't
know which group the kafka-console-producer is using. I tried the consumer
group but it gave the following message:

Exiting due to: org.apache.zookeeper.KeeperException$NoNodeException:
KeeperErrorCode = NoNode for /consumers/test-consumer-group/offsets/test/0.

Perhaps the solution is downgrade the consumer libs to 0.8.1?

Thanks,

Ricardo

On Sun, Feb 8, 2015 at 11:27 AM, Gwen Shapira <gs...@cloudera.com> wrote:

> I have a 0.8.1 high level consumer working fine with 0.8.2 server. Few of
> them actually :)
> AFAIK the API did not change.
>
> Do you see any error messages? Do you have timeout configured on the
> consumer? What does the offset checker tool say?
>
> On Fri, Feb 6, 2015 at 4:49 PM, Ricardo Ferreira <
> jricardoferreira@gmail.com> wrote:
>
>> Hi,
>>
>> I had a client running on Kafka 0.8.1 using the High-Level consumer API
>> working fine.
>>
>> Today I updated my Kafka installation for the 0.8.2 version (tried all
>> versions of Scala) but the consumer doesn't get any messages. I tested
>> using the kafka-console-consumer.sh utility tool and works fine, only my
>> Java program that not.
>>
>> Did I miss something? I heard that the API changed, so I'd like to know if
>> someone can share a simple client with me.
>>
>> Please, respond directly to me or just reply all because I am not
>> currently
>> subscribed to the group.
>>
>> Thanks,
>>
>> Ricardo Ferreira
>>
>
>

Re: Apache Kafka 0.8.2 Consumer Example

[Gwen Shapira <gs...@cloudera.com>]

I have a 0.8.1 high level consumer working fine with 0.8.2 server. Few of
them actually :)
AFAIK the API did not change.

Do you see any error messages? Do you have timeout configured on the
consumer? What does the offset checker tool say?

On Fri, Feb 6, 2015 at 4:49 PM, Ricardo Ferreira <jricardoferreira@gmail.com
> wrote:

> Hi,
>
> I had a client running on Kafka 0.8.1 using the High-Level consumer API
> working fine.
>
> Today I updated my Kafka installation for the 0.8.2 version (tried all
> versions of Scala) but the consumer doesn't get any messages. I tested
> using the kafka-console-consumer.sh utility tool and works fine, only my
> Java program that not.
>
> Did I miss something? I heard that the API changed, so I'd like to know if
> someone can share a simple client with me.
>
> Please, respond directly to me or just reply all because I am not currently
> subscribed to the group.
>
> Thanks,
>
> Ricardo Ferreira
>