You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Grega Kešpret <gr...@celtra.com> on 2014/01/06 17:44:58 UTC
Worker hangs with 100% CPU in Standalone cluster
Hi,
we are seeing several times a day one worker in a Standalone cluster hang
up with 100% CPU at the last task and doesn't proceed. After we restart the
job, it completes successfully.
We are using Spark v0.8.1-incubating.
Attached please find jstack logs of Worker and CoarseGrainedExecutorBackend
JVM processes.
Grega
--
[image: Inline image 1]
*Grega Kešpret*
Analytics engineer
Celtra — Rich Media Mobile Advertising
celtra.com <http://www.celtra.com/> |
@celtramobile<http://www.twitter.com/celtramobile>
Re: Worker hangs with 100% CPU in Standalone cluster
Posted by Grega Kešpret <gr...@celtra.com>.
Andrew yes, that's right. Don't forget to implement hashCode() and equals()
when dealing with standard classes (you may not always want to/be able to
use case classes in Scala), as it can lead to some nasty bugs :-).
Grega
On Thu, Jan 16, 2014 at 9:42 PM, Andrew Ash <an...@andrewash.com> wrote:
> It sounds like the takeaway is that if you're using custom classes, you
> need to make sure that their hashCode() and equals() methods are
> value-based?
>
>
> On Thu, Jan 16, 2014 at 12:08 PM, Patrick Wendell <pw...@gmail.com>wrote:
>
>> Thanks for following up and explaining this one! Definitely something
>> other users might run into...
>>
>>
>> On Thu, Jan 16, 2014 at 5:58 AM, Grega Kešpret <gr...@celtra.com> wrote:
>>
>>> Just to follow up, we have since pinpointed the problem to be in
>>> application code (not Spark). In some cases, there was an infinite loop in
>>> Scala HashTable linear probing algorithm, where an element's next() pointed
>>> at itself. It was probably caused by wrong hashCode() and equals() methods
>>> on the object we were storing.
>>>
>>> Milos, we also have Master node separate from Worker nodes. Could
>>> someone from Spark team comment about that?
>>>
>>> Grega
>>> --
>>> [image: Inline image 1]
>>> *Grega Kešpret*
>>> Analytics engineer
>>>
>>> Celtra — Rich Media Mobile Advertising
>>> celtra.com <http://www.celtra.com/> | @celtramobile<http://www.twitter.com/celtramobile>
>>>
>>>
>>> On Thu, Jan 16, 2014 at 2:46 PM, Milos Nikolic <
>>> milos.nikolic83@gmail.com> wrote:
>>>
>>>> Hello,
>>>>
>>>> I’m facing the same (or similar) problem. In my case, the last two
>>>> tasks hang in a map function following sc.sequenceFile(…). It happens from
>>>> time to time (more often with TorrentBroadcast than HttpBroadcast) and
>>>> after restarting it works fine.
>>>>
>>>> The problem always happens on the same node — on the node that plays
>>>> the roles of the master and one worker. Once this node becomes master-only
>>>> (i.e., I removed this nodes from conf/slaves), the problem is gone.
>>>>
>>>> Does that mean that the master and workers have to be on separate
>>>> nodes?
>>>>
>>>> Best,
>>>> Milos
>>>>
>>>>
>>>> On Jan 6, 2014, at 5:44 PM, Grega Kešpret <gr...@celtra.com> wrote:
>>>>
>>>> Hi,
>>>>
>>>> we are seeing several times a day one worker in a Standalone cluster
>>>> hang up with 100% CPU at the last task and doesn't proceed. After we
>>>> restart the job, it completes successfully.
>>>>
>>>> We are using Spark v0.8.1-incubating.
>>>>
>>>> Attached please find jstack logs of Worker
>>>> and CoarseGrainedExecutorBackend JVM processes.
>>>>
>>>> Grega
>>>> --
>>>> <celtra_logo.png>
>>>> *Grega Kešpret*
>>>> Analytics engineer
>>>>
>>>> Celtra — Rich Media Mobile Advertising
>>>> celtra.com <http://www.celtra.com/> | @celtramobile<http://www.twitter.com/celtramobile>
>>>> <logs.zip>
>>>>
>>>>
>>>>
>>>
>>
>
Re: Worker hangs with 100% CPU in Standalone cluster
Posted by Andrew Ash <an...@andrewash.com>.
It sounds like the takeaway is that if you're using custom classes, you
need to make sure that their hashCode() and equals() methods are
value-based?
On Thu, Jan 16, 2014 at 12:08 PM, Patrick Wendell <pw...@gmail.com>wrote:
> Thanks for following up and explaining this one! Definitely something
> other users might run into...
>
>
> On Thu, Jan 16, 2014 at 5:58 AM, Grega Kešpret <gr...@celtra.com> wrote:
>
>> Just to follow up, we have since pinpointed the problem to be in
>> application code (not Spark). In some cases, there was an infinite loop in
>> Scala HashTable linear probing algorithm, where an element's next() pointed
>> at itself. It was probably caused by wrong hashCode() and equals() methods
>> on the object we were storing.
>>
>> Milos, we also have Master node separate from Worker nodes. Could someone
>> from Spark team comment about that?
>>
>> Grega
>> --
>> [image: Inline image 1]
>> *Grega Kešpret*
>> Analytics engineer
>>
>> Celtra — Rich Media Mobile Advertising
>> celtra.com <http://www.celtra.com/> | @celtramobile<http://www.twitter.com/celtramobile>
>>
>>
>> On Thu, Jan 16, 2014 at 2:46 PM, Milos Nikolic <milos.nikolic83@gmail.com
>> > wrote:
>>
>>> Hello,
>>>
>>> I’m facing the same (or similar) problem. In my case, the last two tasks
>>> hang in a map function following sc.sequenceFile(…). It happens from time
>>> to time (more often with TorrentBroadcast than HttpBroadcast) and after
>>> restarting it works fine.
>>>
>>> The problem always happens on the same node — on the node that plays the
>>> roles of the master and one worker. Once this node becomes master-only
>>> (i.e., I removed this nodes from conf/slaves), the problem is gone.
>>>
>>> Does that mean that the master and workers have to be on separate nodes?
>>>
>>> Best,
>>> Milos
>>>
>>>
>>> On Jan 6, 2014, at 5:44 PM, Grega Kešpret <gr...@celtra.com> wrote:
>>>
>>> Hi,
>>>
>>> we are seeing several times a day one worker in a Standalone cluster
>>> hang up with 100% CPU at the last task and doesn't proceed. After we
>>> restart the job, it completes successfully.
>>>
>>> We are using Spark v0.8.1-incubating.
>>>
>>> Attached please find jstack logs of Worker
>>> and CoarseGrainedExecutorBackend JVM processes.
>>>
>>> Grega
>>> --
>>> <celtra_logo.png>
>>> *Grega Kešpret*
>>> Analytics engineer
>>>
>>> Celtra — Rich Media Mobile Advertising
>>> celtra.com <http://www.celtra.com/> | @celtramobile<http://www.twitter.com/celtramobile>
>>> <logs.zip>
>>>
>>>
>>>
>>
>
Re: Worker hangs with 100% CPU in Standalone cluster
Posted by Patrick Wendell <pw...@gmail.com>.
Thanks for following up and explaining this one! Definitely something other
users might run into...
On Thu, Jan 16, 2014 at 5:58 AM, Grega Kešpret <gr...@celtra.com> wrote:
> Just to follow up, we have since pinpointed the problem to be in
> application code (not Spark). In some cases, there was an infinite loop in
> Scala HashTable linear probing algorithm, where an element's next() pointed
> at itself. It was probably caused by wrong hashCode() and equals() methods
> on the object we were storing.
>
> Milos, we also have Master node separate from Worker nodes. Could someone
> from Spark team comment about that?
>
> Grega
> --
> [image: Inline image 1]
> *Grega Kešpret*
> Analytics engineer
>
> Celtra — Rich Media Mobile Advertising
> celtra.com <http://www.celtra.com/> | @celtramobile<http://www.twitter.com/celtramobile>
>
>
> On Thu, Jan 16, 2014 at 2:46 PM, Milos Nikolic <mi...@gmail.com>wrote:
>
>> Hello,
>>
>> I’m facing the same (or similar) problem. In my case, the last two tasks
>> hang in a map function following sc.sequenceFile(…). It happens from time
>> to time (more often with TorrentBroadcast than HttpBroadcast) and after
>> restarting it works fine.
>>
>> The problem always happens on the same node — on the node that plays the
>> roles of the master and one worker. Once this node becomes master-only
>> (i.e., I removed this nodes from conf/slaves), the problem is gone.
>>
>> Does that mean that the master and workers have to be on separate nodes?
>>
>> Best,
>> Milos
>>
>>
>> On Jan 6, 2014, at 5:44 PM, Grega Kešpret <gr...@celtra.com> wrote:
>>
>> Hi,
>>
>> we are seeing several times a day one worker in a Standalone cluster hang
>> up with 100% CPU at the last task and doesn't proceed. After we restart the
>> job, it completes successfully.
>>
>> We are using Spark v0.8.1-incubating.
>>
>> Attached please find jstack logs of Worker
>> and CoarseGrainedExecutorBackend JVM processes.
>>
>> Grega
>> --
>> <celtra_logo.png>
>> *Grega Kešpret*
>> Analytics engineer
>>
>> Celtra — Rich Media Mobile Advertising
>> celtra.com <http://www.celtra.com/> | @celtramobile<http://www.twitter.com/celtramobile>
>> <logs.zip>
>>
>>
>>
>
Re: Worker hangs with 100% CPU in Standalone cluster
Posted by Grega Kešpret <gr...@celtra.com>.
Just to follow up, we have since pinpointed the problem to be in
application code (not Spark). In some cases, there was an infinite loop in
Scala HashTable linear probing algorithm, where an element's next() pointed
at itself. It was probably caused by wrong hashCode() and equals() methods
on the object we were storing.
Milos, we also have Master node separate from Worker nodes. Could someone
from Spark team comment about that?
Grega
--
[image: Inline image 1]
*Grega Kešpret*
Analytics engineer
Celtra — Rich Media Mobile Advertising
celtra.com <http://www.celtra.com/> |
@celtramobile<http://www.twitter.com/celtramobile>
On Thu, Jan 16, 2014 at 2:46 PM, Milos Nikolic <mi...@gmail.com>wrote:
> Hello,
>
> I’m facing the same (or similar) problem. In my case, the last two tasks
> hang in a map function following sc.sequenceFile(…). It happens from time
> to time (more often with TorrentBroadcast than HttpBroadcast) and after
> restarting it works fine.
>
> The problem always happens on the same node — on the node that plays the
> roles of the master and one worker. Once this node becomes master-only
> (i.e., I removed this nodes from conf/slaves), the problem is gone.
>
> Does that mean that the master and workers have to be on separate nodes?
>
> Best,
> Milos
>
>
> On Jan 6, 2014, at 5:44 PM, Grega Kešpret <gr...@celtra.com> wrote:
>
> Hi,
>
> we are seeing several times a day one worker in a Standalone cluster hang
> up with 100% CPU at the last task and doesn't proceed. After we restart the
> job, it completes successfully.
>
> We are using Spark v0.8.1-incubating.
>
> Attached please find jstack logs of Worker
> and CoarseGrainedExecutorBackend JVM processes.
>
> Grega
> --
> <celtra_logo.png>
> *Grega Kešpret*
> Analytics engineer
>
> Celtra — Rich Media Mobile Advertising
> celtra.com <http://www.celtra.com/> | @celtramobile<http://www.twitter.com/celtramobile>
> <logs.zip>
>
>
>
Re: Worker hangs with 100% CPU in Standalone cluster
Posted by Milos Nikolic <mi...@gmail.com>.
Hello,
I’m facing the same (or similar) problem. In my case, the last two tasks hang in a map function following sc.sequenceFile(…). It happens from time to time (more often with TorrentBroadcast than HttpBroadcast) and after restarting it works fine.
The problem always happens on the same node — on the node that plays the roles of the master and one worker. Once this node becomes master-only (i.e., I removed this nodes from conf/slaves), the problem is gone.
Does that mean that the master and workers have to be on separate nodes?
Best,
Milos
On Jan 6, 2014, at 5:44 PM, Grega Kešpret <gr...@celtra.com> wrote:
> Hi,
>
> we are seeing several times a day one worker in a Standalone cluster hang up with 100% CPU at the last task and doesn't proceed. After we restart the job, it completes successfully.
>
> We are using Spark v0.8.1-incubating.
>
> Attached please find jstack logs of Worker and CoarseGrainedExecutorBackend JVM processes.
>
> Grega
> --
> <celtra_logo.png>
> Grega Kešpret
> Analytics engineer
>
> Celtra — Rich Media Mobile Advertising
> celtra.com | @celtramobile
> <logs.zip>