topologys down slow over time ("emitted" count double of "transferred")

[Subrat Basnet <su...@myktm.com>]

Hi everyone,

Please see attached Storm-ui screenshot.

I have nothing in the logs, but somehow my topology slows down over time.

It starts out by processing 50,000+ records in the first 10 minutes.. now its gone down to 2000 only.

Any ideas where the bottle neck is in storm? I have checked my spout data sources, and there are no bottlenecks there.

The thing I noticed is, the number of data “emitted” is almost double the “transferred”. Could that be it? How can I fix this situation?

Thanks
Subrat

Re: topologys down slow over time ("emitted" count double of "transferred")

[Jeffery Maass <ma...@gmail.com>]

Subrat:

The numbers themselves don't match a 1:1 to a single tuple.  They are
counters of activity.  So, if  spout emits a tuple once, and that tuple
travels 2 streams, and each instance of that tuple is transferred, then the
transferred counter is incremented twice.  On any page, go to the bottom
and click - "show system stats".  It will show you system bolts which
interact with system streams.  Click "Show Visualization", play around with
the options and, things become even clearer.

Metrics!:
Transmit metrics from storm: https://github.com/verisign/storm-graphite
Record and view metrics from storm:
http://graphite.readthedocs.org/en/latest/

I notice that there are no Ackd and no Failed tuples in your spout.  Are
you intentionally using an ureliable spout?

Some questions to consider:

Do you have proof that your spout has something to crawl?  If there isn't
anything to emit, the number emitted will "look" slower.

Was your bolt operating at capacity 17 hours ago?  ( Was it's Capacity #
closer to 1 ? )



Thank you for your time!

+++++++++++++++++++++
Jeff Maass <ma...@gmail.com>
linkedin.com/in/jeffmaass
stackoverflow.com/users/373418/maassql
+++++++++++++++++++++


On Thu, May 21, 2015 at 10:25 PM, Subrat Basnet <su...@myktm.com> wrote:

> Hi everyone,
>
> Please see attached Storm-ui screenshot.
>
> I have nothing in the logs, but somehow my topology slows down over time.
>
> It starts out by processing 50,000+ records in the first 10 minutes.. now
> its gone down to 2000 only.
>
> Any ideas where the bottle neck is in storm? I have checked my spout data
> sources, and there are no bottlenecks there.
>
> The thing I noticed is, the number of data “emitted” is almost double the
> “transferred”. Could that be it? How can I fix this situation?
>
> Thanks
> Subrat
>
>