You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Michael Albert <m_...@yahoo.com.INVALID> on 2015/02/06 03:04:37 UTC
Spark stalls or hangs: is this a clue? remote fetches seem to never
return?
Greetings!
Again, thanks to all who have given suggestions.I am still trying to diagnose a problem where I have processes than run for one or several hours but intermittently stall or hang.By "stall" I mean that there is no CPU usage on the workers or the driver, nor network activity, nor do I see disk activity.It just hangs.
Using the Application Master to find which workers still had active tasks, I then went to that machine and looked in the user logs.In one of the users log's "stderr" files, it ends with "Started 50 remote fetches...."Should there be a message saying that the fetch was completed?Any suggestions as to how I might diagnose why the fetch was not completed?
Thanks!-Mike
Here is the last part of the log:15/02/06 01:33:46 INFO storage.MemoryStore: ensureFreeSpace(5368) called with curMem=875861, maxMem=231564902415/02/06 01:33:46 INFO storage.MemoryStore: Block broadcast_10 stored as values in memory (estimated size 5.2 KB, free 2.2 GB)15/02/06 01:33:46 INFO spark.MapOutputTrackerWorker: Don't have map outputs for shuffle 5, fetching them15/02/06 01:33:46 INFO spark.MapOutputTrackerWorker: Doing the fetch; tracker actor = Actor[akka.tcp://sparkDriver@ip-10-171-0-208.ec2.internal:44124/user/MapOutputTracker#-878402310]15/02/06 01:33:46 INFO spark.MapOutputTrackerWorker: Don't have map outputs for shuffle 5, fetching them15/02/06 01:33:46 INFO spark.MapOutputTrackerWorker: Got the output locations15/02/06 01:33:46 INFO storage.ShuffleBlockFetcherIterator: Getting 300 non-empty blocks out of 300 blocks15/02/06 01:33:46 INFO storage.ShuffleBlockFetcherIterator: Getting 300 non-empty blocks out of 300 blocks15/02/06 01:33:46 INFO storage.ShuffleBlockFetcherIterator: Started 50 remote fetches in 47 ms15/02/06 01:33:46 INFO storage.ShuffleBlockFetcherIterator: Started 50 remote fetches in 48 msIt's been like that for half and hour.
Thanks!-Mike
Re: Spark stalls or hangs: is this a clue? remote fetches seem to never return?
Posted by Xuefeng Wu <be...@gmail.com>.
what's the dump info by jstack?
Yours, Xuefeng Wu 吴雪峰 敬上
> On 2015年2月6日, at 上午10:20, Michael Albert <m_...@yahoo.com.INVALID> wrote:
>
> My apologies for following up my own post, but I thought this might be of interest.
>
> I terminated the java process corresponding to executor which had opened the stderr file mentioned below (kill <pid>).
> Then my spark job completed without error (it was actually almost finished).
>
> Now I am completely confused :-).
>
> Thanks!
> -Mike
>
>
> From: Michael Albert <m_...@yahoo.com.INVALID>
> To: "user@spark.apache.org" <us...@spark.apache.org>
> Sent: Thursday, February 5, 2015 9:04 PM
> Subject: Spark stalls or hangs: is this a clue? remote fetches seem to never return?
>
> Greetings!
>
> Again, thanks to all who have given suggestions.
> I am still trying to diagnose a problem where I have processes than run for one or several hours but intermittently stall or hang.
> By "stall" I mean that there is no CPU usage on the workers or the driver, nor network activity, nor do I see disk activity.
> It just hangs.
>
> Using the Application Master to find which workers still had active tasks, I then went to that machine and looked in the user logs.
> In one of the users log's "stderr" files, it ends with "Started 50 remote fetches...."
> Should there be a message saying that the fetch was completed?
> Any suggestions as to how I might diagnose why the fetch was not completed?
>
> Thanks!
> -Mike
>
> Here is the last part of the log:
> 15/02/06 01:33:46 INFO storage.MemoryStore: ensureFreeSpace(5368) called with curMem=875861, maxMem=2315649024
> 15/02/06 01:33:46 INFO storage.MemoryStore: Block broadcast_10 stored as values in memory (estimated size 5.2 KB, free 2.2 GB)
> 15/02/06 01:33:46 INFO spark.MapOutputTrackerWorker: Don't have map outputs for shuffle 5, fetching them
> 15/02/06 01:33:46 INFO spark.MapOutputTrackerWorker: Doing the fetch; tracker actor = Actor[akka.tcp://sparkDriver@ip-10-171-0-208.ec2.internal:44124/user/MapOutputTracker#-878402310]
> 15/02/06 01:33:46 INFO spark.MapOutputTrackerWorker: Don't have map outputs for shuffle 5, fetching them
> 15/02/06 01:33:46 INFO spark.MapOutputTrackerWorker: Got the output locations
> 15/02/06 01:33:46 INFO storage.ShuffleBlockFetcherIterator: Getting 300 non-empty blocks out of 300 blocks
> 15/02/06 01:33:46 INFO storage.ShuffleBlockFetcherIterator: Getting 300 non-empty blocks out of 300 blocks
> 15/02/06 01:33:46 INFO storage.ShuffleBlockFetcherIterator: Started 50 remote fetches in 47 ms
> 15/02/06 01:33:46 INFO storage.ShuffleBlockFetcherIterator: Started 50 remote fetches in 48 ms
> It's been like that for half and hour.
>
> Thanks!
> -Mike
>
>
>
>
Re: Spark stalls or hangs: is this a clue? remote fetches seem to
never return?
Posted by Michael Albert <m_...@yahoo.com.INVALID>.
My apologies for following up my own post, but I thought this might be of interest.
I terminated the java process corresponding to executor which had opened the stderr file mentioned below (kill <pid>).Then my spark job completed without error (it was actually almost finished).
Now I am completely confused :-).
Thanks!-Mike
From: Michael Albert <m_...@yahoo.com.INVALID>
To: "user@spark.apache.org" <us...@spark.apache.org>
Sent: Thursday, February 5, 2015 9:04 PM
Subject: Spark stalls or hangs: is this a clue? remote fetches seem to never return?
Greetings!
Again, thanks to all who have given suggestions.I am still trying to diagnose a problem where I have processes than run for one or several hours but intermittently stall or hang.By "stall" I mean that there is no CPU usage on the workers or the driver, nor network activity, nor do I see disk activity.It just hangs.
Using the Application Master to find which workers still had active tasks, I then went to that machine and looked in the user logs.In one of the users log's "stderr" files, it ends with "Started 50 remote fetches...."Should there be a message saying that the fetch was completed?Any suggestions as to how I might diagnose why the fetch was not completed?
Thanks!-Mike
Here is the last part of the log:15/02/06 01:33:46 INFO storage.MemoryStore: ensureFreeSpace(5368) called with curMem=875861, maxMem=231564902415/02/06 01:33:46 INFO storage.MemoryStore: Block broadcast_10 stored as values in memory (estimated size 5.2 KB, free 2.2 GB)15/02/06 01:33:46 INFO spark.MapOutputTrackerWorker: Don't have map outputs for shuffle 5, fetching them15/02/06 01:33:46 INFO spark.MapOutputTrackerWorker: Doing the fetch; tracker actor = Actor[akka.tcp://sparkDriver@ip-10-171-0-208.ec2.internal:44124/user/MapOutputTracker#-878402310]15/02/06 01:33:46 INFO spark.MapOutputTrackerWorker: Don't have map outputs for shuffle 5, fetching them15/02/06 01:33:46 INFO spark.MapOutputTrackerWorker: Got the output locations15/02/06 01:33:46 INFO storage.ShuffleBlockFetcherIterator: Getting 300 non-empty blocks out of 300 blocks15/02/06 01:33:46 INFO storage.ShuffleBlockFetcherIterator: Getting 300 non-empty blocks out of 300 blocks15/02/06 01:33:46 INFO storage.ShuffleBlockFetcherIterator: Started 50 remote fetches in 47 ms15/02/06 01:33:46 INFO storage.ShuffleBlockFetcherIterator: Started 50 remote fetches in 48 msIt's been like that for half and hour.
Thanks!-Mike