Re: Trident Kafka Spout - Ack count increasing even though no messages are processed

[Nathan Marz <na...@nathanmarz.com>]

Nimbus has a Thrift API, so you can query it with any programming language.
Here's the Thrift definition:
https://github.com/apache/storm/blob/master/storm-core/src/storm.thrift

On Tue, Sep 30, 2014 at 5:32 PM, Ratay, Steve <St...@teradata.com>
wrote:

>  What is the Nimbus API?  Is it this class?
> https://storm.incubator.apache.org/apidocs/backtype/storm/generated/Nimbus.Client.html
> Or something else?
>
>  Thanks, Steve
>
>   From: Deepak Subhramanian <de...@gmail.com>
> Reply-To: "user@storm.apache.org" <us...@storm.apache.org>
> Date: Tuesday, September 30, 2014 at 4:28 PM
> To: "user@storm.apache.org" <us...@storm.apache.org>, Nishantha Pelendage <
> pnpsampath@gmail.com>
> Subject: Re: Trident Kafka Spout - Ack count increasing even though no
> messages are processed
>
>   Thank you very much for the inputs. I will checkout the Nimbus api.
>
> On Tue, Sep 30, 2014 at 9:02 PM, Nathan Marz <na...@nathanmarz.com>
> wrote:
>
>> You would have to manually count up the tuples going onto
>> non-coordination streams. So ignore "$coord-*" streams, ack/fail streams,
>> system streams, and metrics streams. You can get this all from the Nimbus
>> API.
>>
>> On Mon, Sep 29, 2014 at 3:59 PM, Deepak Subhramanian <
>> deepak.subhramanian@gmail.com> wrote:
>>
>>> Thanks Nathan for the explanation. Is there a way to get the actual no
>>> of messages processed in trident from the storm gui or using the storm api ?
>>>
>>> On Sat, Sep 27, 2014 at 9:20 PM, Nathan Marz <na...@nathanmarz.com>
>>> wrote:
>>>
>>>> Trident executes a batch every 500ms (by default). A batch involves a
>>>> bunch of coordination messages going out to all the bolts to coordinate the
>>>> batch (even if the batch is empty). So that's what you're seeing.
>>>>
>>>> On Fri, Sep 26, 2014 at 12:32 PM, Deepak Subhramanian <
>>>> deepak.subhramanian@gmail.com> wrote:
>>>>
>>>>> Hi,
>>>>> I deployed a basic trident topology in HDP storm cluster. I am using
>>>>>
>>>>> TransactionalTridentKafkaSpout to process messages from Kafka Queue .
>>>>> For some reason the ack count in Storm GUI in spout and bolt increases by
>>>>> 20 every second even if there is no messages in Kafka queue .
>>>>>
>>>>> Is there any reason ack is called by Trident even though no messages
>>>>> are getting processed
>>>>> Thanks, D
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>>   --
>>>>  Twitter: @nathanmarz
>>>>  http://nathanmarz.com
>>>>
>>>
>>>
>>>
>>>   --
>>> Deepak Subhramanian
>>>
>>
>>
>>
>>  --
>>  Twitter: @nathanmarz
>>  http://nathanmarz.com
>>
>
>
>
>  --
> Deepak Subhramanian
>



-- 
Twitter: @nathanmarz
http://nathanmarz.com

Re: Trident Kafka Spout - Ack count increasing even though no messages are processed

[Deepak Subhramanian <de...@gmail.com>]

Thanks Nathan.

Steve
Here is a python and ruby example I found.
http://tutorials.github.io/pages/retrieving-storm-data-from-nimbus.html#.VCsYYSldWEd


On Tue, Sep 30, 2014 at 11:51 PM, Nathan Marz <na...@nathanmarz.com> wrote:

> Nimbus has a Thrift API, so you can query it with any programming
> language. Here's the Thrift definition:
> https://github.com/apache/storm/blob/master/storm-core/src/storm.thrift
>
> On Tue, Sep 30, 2014 at 5:32 PM, Ratay, Steve <St...@teradata.com>
> wrote:
>
>>  What is the Nimbus API?  Is it this class?
>> https://storm.incubator.apache.org/apidocs/backtype/storm/generated/Nimbus.Client.html
>> Or something else?
>>
>>  Thanks, Steve
>>
>>   From: Deepak Subhramanian <de...@gmail.com>
>> Reply-To: "user@storm.apache.org" <us...@storm.apache.org>
>> Date: Tuesday, September 30, 2014 at 4:28 PM
>> To: "user@storm.apache.org" <us...@storm.apache.org>, Nishantha Pelendage
>> <pn...@gmail.com>
>> Subject: Re: Trident Kafka Spout - Ack count increasing even though no
>> messages are processed
>>
>>   Thank you very much for the inputs. I will checkout the Nimbus api.
>>
>> On Tue, Sep 30, 2014 at 9:02 PM, Nathan Marz <na...@nathanmarz.com>
>> wrote:
>>
>>> You would have to manually count up the tuples going onto
>>> non-coordination streams. So ignore "$coord-*" streams, ack/fail streams,
>>> system streams, and metrics streams. You can get this all from the Nimbus
>>> API.
>>>
>>> On Mon, Sep 29, 2014 at 3:59 PM, Deepak Subhramanian <
>>> deepak.subhramanian@gmail.com> wrote:
>>>
>>>> Thanks Nathan for the explanation. Is there a way to get the actual no
>>>> of messages processed in trident from the storm gui or using the storm api ?
>>>>
>>>> On Sat, Sep 27, 2014 at 9:20 PM, Nathan Marz <na...@nathanmarz.com>
>>>> wrote:
>>>>
>>>>> Trident executes a batch every 500ms (by default). A batch involves a
>>>>> bunch of coordination messages going out to all the bolts to coordinate the
>>>>> batch (even if the batch is empty). So that's what you're seeing.
>>>>>
>>>>> On Fri, Sep 26, 2014 at 12:32 PM, Deepak Subhramanian <
>>>>> deepak.subhramanian@gmail.com> wrote:
>>>>>
>>>>>> Hi,
>>>>>> I deployed a basic trident topology in HDP storm cluster. I am using
>>>>>>
>>>>>> TransactionalTridentKafkaSpout to process messages from Kafka Queue .
>>>>>> For some reason the ack count in Storm GUI in spout and bolt increases by
>>>>>> 20 every second even if there is no messages in Kafka queue .
>>>>>>
>>>>>> Is there any reason ack is called by Trident even though no messages
>>>>>> are getting processed
>>>>>> Thanks, D
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>>   --
>>>>>  Twitter: @nathanmarz
>>>>>  http://nathanmarz.com
>>>>>
>>>>
>>>>
>>>>
>>>>   --
>>>> Deepak Subhramanian
>>>>
>>>
>>>
>>>
>>>  --
>>>  Twitter: @nathanmarz
>>>  http://nathanmarz.com
>>>
>>
>>
>>
>>  --
>> Deepak Subhramanian
>>
>
>
>
> --
> Twitter: @nathanmarz
> http://nathanmarz.com
>



-- 
Deepak Subhramanian