tuple loss

[李家宏 <jh...@gmail.com>]

I am running storm-0.9.0.1 with zmq as transport layer, and I set zmq.hwm =
2000 to avoid significant memory leaks, Also I disableed ackers.

I found from UI that some components executed much less tuples than being
transferred to them.
Let's say, a topology looks like this : componentA -> componentB.
ComponentA transferred 100M tuples while componentB only executed 1M
tuples. I'am wondering where do the left tuples gone. If they were queueing
in the transport layer, then the queue should be keep increasing and thus
the momory usage, But I didn't see any noticeable memeory leaks.

Does storm have the guarantee that all tuples transferred from taskA to
taskB should be received by taskB, moreover, the tulpes are in order.

Regards

-- 

======================================================

Gvain

Email: jh.li.em@gmail.com

Re: tuple loss

[Bobby Evans <ev...@yahoo-inc.com>]

My understanding is that zeromq will throw away tuples after it has buffered a certain amount.  If you are sending them faster then it is able to send downstream.

- Bobby

From: 李家宏 <jh...@gmail.com>>
Reply-To: "user@storm.incubator.apache.org<ma...@storm.incubator.apache.org>" <us...@storm.incubator.apache.org>>
Date: Wednesday, May 14, 2014 at 4:06 AM
To: dev <de...@storm.incubator.apache.org>>, user <us...@storm.incubator.apache.org>>
Subject: tuple loss

I am running storm-0.9.0.1 with zmq as transport layer, and I set zmq.hwm = 2000 to avoid significant memory leaks, Also I disableed ackers.

I found from UI that some components executed much less tuples than being transferred to them.
Let's say, a topology looks like this : componentA -> componentB. ComponentA transferred 100M tuples while componentB only executed 1M tuples. I'am wondering where do the left tuples gone. If they were queueing in the transport layer, then the queue should be keep increasing and thus the momory usage, But I didn't see any noticeable memeory leaks.

Does storm have the guarantee that all tuples transferred from taskA to taskB should be received by taskB, moreover, the tulpes are in order.

Regards

--

======================================================

Gvain

Email: jh.li.em@gmail.com<ma...@gmail.com>