You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Brian O'Neill <bo...@alumni.brown.edu> on 2013/11/07 18:35:23 UTC

Worker stops heartbeating.

We are running on 0.9.0-rc2, and have workers that stop heart beating.
(tailing the worker log, we see the WorkerHeartbeat message stop)

At the time the worker stops heart beating, memory seems fine, CPU seems
fine. (jconsole)
The VM seems to continue processing data and is healthy enough for us to
take a thread dump. (jstack)
Nothing immediately jumped out in the thread dump.

The worker is eventually killed, and re-spawned.

What is the best way to diagnose the issue?

Any ideas?

-brian

---
Brian O'Neill
Chief Architect
Health Market Science
The Science of Better Results
2700 Horizon Drive € King of Prussia, PA € 19406
M: 215.588.6024 € @boneill42 <http://www.twitter.com/boneill42>   €
healthmarketscience.com


This information transmitted in this email message is for the intended
recipient only and may contain confidential and/or privileged material. If
you received this email in error and are not the intended recipient, or the
person responsible to deliver it to the intended recipient, please contact
the sender at the email above and delete this email and any attachments and
destroy any copies thereof. Any review, retransmission, dissemination,
copying or other use of, or taking any action in reliance upon, this
information by persons or entities other than the intended recipient is
strictly prohibited.
 



Re: Worker stops heartbeating.

Posted by Brian O'Neill <bo...@alumni.brown.edu>.
Thanks for the quick response.
We actually tried lowering that to 4, but it didn¹t have any effect.

---
Brian O'Neill
Chief Architect
Health Market Science
The Science of Better Results
2700 Horizon Drive € King of Prussia, PA € 19406
M: 215.588.6024 € @boneill42 <http://www.twitter.com/boneill42>   €
healthmarketscience.com


This information transmitted in this email message is for the intended
recipient only and may contain confidential and/or privileged material. If
you received this email in error and are not the intended recipient, or the
person responsible to deliver it to the intended recipient, please contact
the sender at the email above and delete this email and any attachments and
destroy any copies thereof. Any review, retransmission, dissemination,
copying or other use of, or taking any action in reliance upon, this
information by persons or entities other than the intended recipient is
strictly prohibited.
 


From:  SKG <sk...@gmail.com>
Date:  Thursday, November 7, 2013 at 12:45 PM
To:  <st...@googlegroups.com>
Cc:  <us...@storm.incubator.apache.org>, Brian O'Neill
<bo...@alumni.brown.edu>
Subject:  Re: Worker stops heartbeating.

May be the hearbeat thread is not processed by worker java process.

Check this parameter in defaults.yaml of supervisor server.

topology.receiver.buffer.size: 8 # setting it too high causes a lot of
problems (heartbeat thread gets starved, throughput plummets)

On Thursday, November 7, 2013 11:05:23 PM UTC+5:30, Brian O'Neill wrote:
> 
> We are running on 0.9.0-rc2, and have workers that stop heart beating.
> (tailing the worker log, we see the WorkerHeartbeat message stop)
> 
> At the time the worker stops heart beating, memory seems fine, CPU seems fine.
> (jconsole)
> The VM seems to continue processing data and is healthy enough for us to take
> a thread dump. (jstack)
> Nothing immediately jumped out in the thread dump.
> 
> The worker is eventually killed, and re-spawned.
> 
> What is the best way to diagnose the issue?
> 
> Any ideas?
> 
> -brian
> 
> ---
> Brian O'Neill
> Chief Architect
> Health Market Science
> The Science of Better Results
> 2700 Horizon Drive € King of Prussia, PA € 19406
> M: 215.588.6024 € @boneill42 <http://www.twitter.com/boneill42>   €
> healthmarketscience.com
> 
> 
> This information transmitted in this email message is for the intended
> recipient only and may contain confidential and/or privileged material. If you
> received this email in error and are not the intended recipient, or the person
> responsible to deliver it to the intended recipient, please contact the sender
> at the email above and delete this email and any attachments and destroy any
> copies thereof. Any review, retransmission, dissemination, copying or other
> use of, or taking any action in reliance upon, this information by persons or
> entities other than the intended recipient is strictly prohibited.
>  



Re: Worker stops heartbeating.

Posted by SKG <sk...@gmail.com>.
May be the hearbeat thread is not processed by worker java process.

Check this parameter in defaults.yaml of supervisor server.

topology.receiver.buffer.size: 8 # setting it too high causes a lot of 
problems (heartbeat thread gets starved, throughput plummets)

On Thursday, November 7, 2013 11:05:23 PM UTC+5:30, Brian O'Neill wrote:
>
>
> We are running on 0.9.0-rc2, and have workers that stop heart beating.
> (tailing the worker log, we see the WorkerHeartbeat message stop)
>
> At the time the worker stops heart beating, memory seems fine, CPU seems 
> fine. (jconsole)
> The VM seems to continue processing data and is healthy enough for us to 
> take a thread dump. (jstack)
> Nothing immediately jumped out in the thread dump.
>
> The worker is eventually killed, and re-spawned.
>
> What is the best way to diagnose the issue?
>
> Any ideas?
>
> -brian
>
> ---
>
> Brian O'Neill
>
> Chief Architect
>
> *Health Market Science*
>
> *The Science of Better Results*
>
> 2700 Horizon Drive • King of Prussia, PA • 19406
>
> M: 215.588.6024 • @boneill42 <http://www.twitter.com/boneill42>  •  
>
> healthmarketscience.com
>
>
> This information transmitted in this email message is for the intended 
> recipient only and may contain confidential and/or privileged material. If 
> you received this email in error and are not the intended recipient, or the 
> person responsible to deliver it to the intended recipient, please contact 
> the sender at the email above and delete this email and any attachments and 
> destroy any copies thereof. Any review, retransmission, dissemination, 
> copying or other use of, or taking any action in reliance upon, this 
> information by persons or entities other than the intended recipient is 
> strictly prohibited.
>
>  
>
>