You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by "Tao, Jing" <jt...@webmd.net> on 2015/01/28 22:13:03 UTC
Timed-out tuples replayed over and over
Hi,
I have a topology where one of the bolts is taking 2 seconds to complete due to a webservice call which is timing out (I set request timeout to 2 sec).
I have the following configs for the topology:
config.put(Config.TOPOLOGY_MESSAGE_TIMEOUT_SECS, 60)
config.setMaxSpoutPending(2000)
config.setNumAckers(4)
config.setNumWorkers(3)
What I am seeing is, many message will timeout, and gets replayed every few minutes. I'm assuming they are repeatedly timing out? But if each tuple takes 2 seconds, and TOPOLOGY_MESSAGE_TIMEOUT_SECS is 60, why are they timing out so much? I had a total of 2250 tuples, and the queue still has not cleared up after almost 2 hours.
When the webservice call does not timeout, 2250 tuples only takes a few minutes to complete.
Any ideas?
Thanks,
Jing
Re: Timed-out tuples replayed over and over
Posted by Nathan Leung <nc...@gmail.com>.
sorry I thought by queue you meant the spout output queue. tuples that are
timed out, as you read, are still processed by storm.
On Thu, Jan 29, 2015 at 4:01 PM, Tao, Jing <jt...@webmd.net> wrote:
> Based on this thread
> http://stackoverflow.com/questions/26536525/what-is-the-point-of-timing-out-tuples
> it sounds like timed-out tuples still flow through the topology. So that
> would explain why my queue never cleared.
>
>
>
> *From:* Nathan Leung [mailto:ncleung@gmail.com]
> *Sent:* Wednesday, January 28, 2015 5:20 PM
> *To:* user
> *Subject:* RE: Timed-out tuples replayed over and over
>
>
>
> I'm not sure, I would have to run a test to see if this is possible.
>
> Maybe someone else knows off hand. If it's possible to re-emit while the
> tuple is still enqueued, I can see how you might run into a situation where
> you can never work the queue off.
>
> On Jan 28, 2015 4:56 PM, "Tao, Jing" <jt...@webmd.net> wrote:
>
> Can multiple copies of the same tuple be in the queue at the same time?
> Say 1999 tuples timeout, the queue is still reduced to size 1999, correct?
> What I’m seeing is that the queue size seems to never shrink once timeouts
> start happening.
>
>
>
> I’ve seen timeouts previously due to large volume of data, but they did
> clear up after a few minutes.
>
> This time, it did not clear after even 2 hours. The Spout’s Acked #
> stayed the same for 2 hours.
>
>
>
> *From:* Nathan Leung [mailto:ncleung@gmail.com]
> *Sent:* Wednesday, January 28, 2015 4:20 PM
> *To:* user
> *Subject:* Re: Timed-out tuples replayed over and over
>
>
>
> Your max spout pending is 2000. In a worst case scenario, where you have
> 1 spout task, 1 bolt task, and every request is timing out, if your spout
> output queue is full it will take 4000 seconds to drain. This is not
> accounting for any tuples that need to be replayed, which will cause the
> full queue drain to take even longer.
>
>
>
> On Wed, Jan 28, 2015 at 4:13 PM, Tao, Jing <jt...@webmd.net> wrote:
>
> Hi,
>
>
>
> I have a topology where one of the bolts is taking 2 seconds to complete
> due to a webservice call which is timing out (I set request timeout to 2
> sec).
>
>
>
> I have the following configs for the topology:
>
> config.put(Config.TOPOLOGY_MESSAGE_TIMEOUT_SECS, 60)
>
> config.setMaxSpoutPending(2000)
>
> config.setNumAckers(4)
>
> config.setNumWorkers(3)
>
>
>
> What I am seeing is, many message will timeout, and gets replayed every
> few minutes. I’m assuming they are repeatedly timing out? But if each
> tuple takes 2 seconds, and TOPOLOGY_MESSAGE_TIMEOUT_SECS is 60, why are
> they timing out so much? I had a total of 2250 tuples, and the queue still
> has not cleared up after almost 2 hours.
>
>
>
> When the webservice call does not timeout, 2250 tuples only takes a few
> minutes to complete.
>
>
>
> Any ideas?
>
>
>
> Thanks,
>
> Jing
>
>
>
RE: Timed-out tuples replayed over and over
Posted by "Tao, Jing" <jt...@webmd.net>.
Based on this thread http://stackoverflow.com/questions/26536525/what-is-the-point-of-timing-out-tuples it sounds like timed-out tuples still flow through the topology. So that would explain why my queue never cleared.
From: Nathan Leung [mailto:ncleung@gmail.com]
Sent: Wednesday, January 28, 2015 5:20 PM
To: user
Subject: RE: Timed-out tuples replayed over and over
I'm not sure, I would have to run a test to see if this is possible.
Maybe someone else knows off hand. If it's possible to re-emit while the tuple is still enqueued, I can see how you might run into a situation where you can never work the queue off.
On Jan 28, 2015 4:56 PM, "Tao, Jing" <jt...@webmd.net>> wrote:
Can multiple copies of the same tuple be in the queue at the same time? Say 1999 tuples timeout, the queue is still reduced to size 1999, correct? What I’m seeing is that the queue size seems to never shrink once timeouts start happening.
I’ve seen timeouts previously due to large volume of data, but they did clear up after a few minutes.
This time, it did not clear after even 2 hours. The Spout’s Acked # stayed the same for 2 hours.
From: Nathan Leung [mailto:ncleung@gmail.com<ma...@gmail.com>]
Sent: Wednesday, January 28, 2015 4:20 PM
To: user
Subject: Re: Timed-out tuples replayed over and over
Your max spout pending is 2000. In a worst case scenario, where you have 1 spout task, 1 bolt task, and every request is timing out, if your spout output queue is full it will take 4000 seconds to drain. This is not accounting for any tuples that need to be replayed, which will cause the full queue drain to take even longer.
On Wed, Jan 28, 2015 at 4:13 PM, Tao, Jing <jt...@webmd.net>> wrote:
Hi,
I have a topology where one of the bolts is taking 2 seconds to complete due to a webservice call which is timing out (I set request timeout to 2 sec).
I have the following configs for the topology:
config.put(Config.TOPOLOGY_MESSAGE_TIMEOUT_SECS, 60)
config.setMaxSpoutPending(2000)
config.setNumAckers(4)
config.setNumWorkers(3)
What I am seeing is, many message will timeout, and gets replayed every few minutes. I’m assuming they are repeatedly timing out? But if each tuple takes 2 seconds, and TOPOLOGY_MESSAGE_TIMEOUT_SECS is 60, why are they timing out so much? I had a total of 2250 tuples, and the queue still has not cleared up after almost 2 hours.
When the webservice call does not timeout, 2250 tuples only takes a few minutes to complete.
Any ideas?
Thanks,
Jing
RE: Timed-out tuples replayed over and over
Posted by Nathan Leung <nc...@gmail.com>.
I'm not sure, I would have to run a test to see if this is possible.
Maybe someone else knows off hand. If it's possible to re-emit while the
tuple is still enqueued, I can see how you might run into a situation where
you can never work the queue off.
On Jan 28, 2015 4:56 PM, "Tao, Jing" <jt...@webmd.net> wrote:
> Can multiple copies of the same tuple be in the queue at the same time?
> Say 1999 tuples timeout, the queue is still reduced to size 1999, correct?
> What I’m seeing is that the queue size seems to never shrink once timeouts
> start happening.
>
>
>
> I’ve seen timeouts previously due to large volume of data, but they did
> clear up after a few minutes.
>
> This time, it did not clear after even 2 hours. The Spout’s Acked #
> stayed the same for 2 hours.
>
>
>
> *From:* Nathan Leung [mailto:ncleung@gmail.com]
> *Sent:* Wednesday, January 28, 2015 4:20 PM
> *To:* user
> *Subject:* Re: Timed-out tuples replayed over and over
>
>
>
> Your max spout pending is 2000. In a worst case scenario, where you have
> 1 spout task, 1 bolt task, and every request is timing out, if your spout
> output queue is full it will take 4000 seconds to drain. This is not
> accounting for any tuples that need to be replayed, which will cause the
> full queue drain to take even longer.
>
>
>
> On Wed, Jan 28, 2015 at 4:13 PM, Tao, Jing <jt...@webmd.net> wrote:
>
> Hi,
>
>
>
> I have a topology where one of the bolts is taking 2 seconds to complete
> due to a webservice call which is timing out (I set request timeout to 2
> sec).
>
>
>
> I have the following configs for the topology:
>
> config.put(Config.TOPOLOGY_MESSAGE_TIMEOUT_SECS, 60)
>
> config.setMaxSpoutPending(2000)
>
> config.setNumAckers(4)
>
> config.setNumWorkers(3)
>
>
>
> What I am seeing is, many message will timeout, and gets replayed every
> few minutes. I’m assuming they are repeatedly timing out? But if each
> tuple takes 2 seconds, and TOPOLOGY_MESSAGE_TIMEOUT_SECS is 60, why are
> they timing out so much? I had a total of 2250 tuples, and the queue still
> has not cleared up after almost 2 hours.
>
>
>
> When the webservice call does not timeout, 2250 tuples only takes a few
> minutes to complete.
>
>
>
> Any ideas?
>
>
>
> Thanks,
>
> Jing
>
>
>
RE: Timed-out tuples replayed over and over
Posted by "Tao, Jing" <jt...@webmd.net>.
Can multiple copies of the same tuple be in the queue at the same time? Say 1999 tuples timeout, the queue is still reduced to size 1999, correct? What I’m seeing is that the queue size seems to never shrink once timeouts start happening.
I’ve seen timeouts previously due to large volume of data, but they did clear up after a few minutes.
This time, it did not clear after even 2 hours. The Spout’s Acked # stayed the same for 2 hours.
From: Nathan Leung [mailto:ncleung@gmail.com]
Sent: Wednesday, January 28, 2015 4:20 PM
To: user
Subject: Re: Timed-out tuples replayed over and over
Your max spout pending is 2000. In a worst case scenario, where you have 1 spout task, 1 bolt task, and every request is timing out, if your spout output queue is full it will take 4000 seconds to drain. This is not accounting for any tuples that need to be replayed, which will cause the full queue drain to take even longer.
On Wed, Jan 28, 2015 at 4:13 PM, Tao, Jing <jt...@webmd.net>> wrote:
Hi,
I have a topology where one of the bolts is taking 2 seconds to complete due to a webservice call which is timing out (I set request timeout to 2 sec).
I have the following configs for the topology:
config.put(Config.TOPOLOGY_MESSAGE_TIMEOUT_SECS, 60)
config.setMaxSpoutPending(2000)
config.setNumAckers(4)
config.setNumWorkers(3)
What I am seeing is, many message will timeout, and gets replayed every few minutes. I’m assuming they are repeatedly timing out? But if each tuple takes 2 seconds, and TOPOLOGY_MESSAGE_TIMEOUT_SECS is 60, why are they timing out so much? I had a total of 2250 tuples, and the queue still has not cleared up after almost 2 hours.
When the webservice call does not timeout, 2250 tuples only takes a few minutes to complete.
Any ideas?
Thanks,
Jing
Re: Timed-out tuples replayed over and over
Posted by Nathan Leung <nc...@gmail.com>.
Your max spout pending is 2000. In a worst case scenario, where you have 1
spout task, 1 bolt task, and every request is timing out, if your spout
output queue is full it will take 4000 seconds to drain. This is not
accounting for any tuples that need to be replayed, which will cause the
full queue drain to take even longer.
On Wed, Jan 28, 2015 at 4:13 PM, Tao, Jing <jt...@webmd.net> wrote:
> Hi,
>
>
>
> I have a topology where one of the bolts is taking 2 seconds to complete
> due to a webservice call which is timing out (I set request timeout to 2
> sec).
>
>
>
> I have the following configs for the topology:
>
> config.put(Config.TOPOLOGY_MESSAGE_TIMEOUT_SECS, 60)
>
> config.setMaxSpoutPending(2000)
>
> config.setNumAckers(4)
>
> config.setNumWorkers(3)
>
>
>
> What I am seeing is, many message will timeout, and gets replayed every
> few minutes. I’m assuming they are repeatedly timing out? But if each
> tuple takes 2 seconds, and TOPOLOGY_MESSAGE_TIMEOUT_SECS is 60, why are
> they timing out so much? I had a total of 2250 tuples, and the queue still
> has not cleared up after almost 2 hours.
>
>
>
> When the webservice call does not timeout, 2250 tuples only takes a few
> minutes to complete.
>
>
>
> Any ideas?
>
>
>
> Thanks,
>
> Jing
>