You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Edward Capriolo <ed...@gmail.com> on 2010/03/02 21:49:17 UTC
"lost task" suspect Distributed Cache to blame
We are happily launching jobs on our 18.3 cluster at a good clip now.
We have one job in which EVERY map attempt fails with the same
message. Rerunning the job, and it usually runs to completion.
stderr logs
Exception in thread "main" java.lang.NullPointerException
at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2186)
The second launch all runs well, but it takes 40 minutes for the first
job to fail with retries and all.
...
2010-03-02 13:17:32,090 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201002111021_0407_m_000058_0: Task
attempt_201002111021_0407_m_000058_0 failed to report status for 600
seconds. Killing!
2010-03-02 13:17:32,098 INFO org.apache.hadoop.mapred.TaskTracker:
Process Thread Dump: lost task
....
Many dumped threads
I noticed some threads are blocked:
Thread 249 (Thread-131):
State: BLOCKED
Blocked count: 2
Waited count: 1
Blocked on java.util.TreeMap@295acf3
Blocked by 129 (Thread-65)
Stack:
org.apache.hadoop.filecache.DistributedCache.getLocalCache(DistributedCache.java:161)
org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:140)
This job is somewhat special (for us) in that in involves shipping
large files over distributed cache. My working theory is that
something goes wrong with the distributed cache/job tracker and the
Job/Task/Tips never have a chance.
Has anyone ever experienced something like this?
Thank you,
Edward
Re: "lost task" suspect Distributed Cache to blame
Posted by Edward Capriolo <ed...@gmail.com>.
On Tue, Mar 2, 2010 at 6:55 PM, Allen Wittenauer
<aw...@linkedin.com> wrote:
>
>
>
> On 3/2/10 12:49 PM, "Edward Capriolo" <ed...@gmail.com> wrote:
>
>> This job is somewhat special (for us) in that in involves shipping
>> large files over distributed cache. My working theory is that
>> something goes wrong with the distributed cache/job tracker and the
>> Job/Task/Tips never have a chance.
>>
>> Has anyone ever experienced something like this?
>
> How big is big and what archiving format are you using? Jar has issues with
> files >2gb.
>
> I've also noticed that dcache seems to get written to -every- tmp dir on a
> node, which seems a bit ridiculous.
>
>
They are data files not jars. They are a few GB each. Today the
process ran but a couple tasks got this...
java.lang.NullPointerException
at org.apache.hadoop.fs.FileUtil.getDU(FileUtil.java:406)
...
at org.apache.hadoop.filecache.DistributedCache.getLocalCache(DistributedCache.java:176)
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:140)
I was thinking this could be my problem:
http://mail-archives.apache.org/mod_mbox/hadoop-core-commits/200903.mbox/%3C20090326222615.2486023889D0@eris.apache.org%3E
Not sure though. As I said job replays but some days our pipeline
loses 45 minutes to this :( Would like to get to the bottom of it.
Re: STDERR and location of new files
Posted by prasenjit mukherjee <pr...@gmail.com>.
I sympathize with you. I actually automated what you just mentioned.
Remotely issue a find/grep command in the userlog directory on all
the nodes. Painful, but that seems to be the only option.
-Prasen
On Wed, Mar 3, 2010 at 8:33 AM, deqiang sun <de...@bcm.edu> wrote:
> I found the stderr log on datanodes.
> However, I think that's not convenient at all.
> I do see on the jobtracker page there's link to the datanodes. But the link is broken, it is not working.
> I have to login, tar, scp to get the log files on each individual datanode.
>
> Is there a better way to do that? Or is there a better way to do the debugging and tunning?
>
> Thanks, ds
>
>
> On Mar 2, 2010, at 6:04 PM, deqiang sun wrote:
>
>> Hi everyone,
>>
>> I am running a perl script through streaming at the beginning of which I print a test identifier to STDERR:
>> print STDERR "whereisthis\n";
>>
>>
>> However, I can NOT find such "whereisthis" after I checked the log files and the web interface.
>>
>> Could anyone tell me how to fix this?
>>
>> Should I turn on the STDERR somewhere?
>>
>> I also tried to create and write this identifier to a file. I can NOT find the file either.
>>
>> Thanks, ds
>
> deqiang sun
> Baylor College of Medicine
> deqiangs@bcm.edu
>
>
>
>
>
Re: STDERR and location of new files
Posted by deqiang sun <de...@bcm.edu>.
I found the stderr log on datanodes.
However, I think that's not convenient at all.
I do see on the jobtracker page there's link to the datanodes. But the link is broken, it is not working.
I have to login, tar, scp to get the log files on each individual datanode.
Is there a better way to do that? Or is there a better way to do the debugging and tunning?
Thanks, ds
On Mar 2, 2010, at 6:04 PM, deqiang sun wrote:
> Hi everyone,
>
> I am running a perl script through streaming at the beginning of which I print a test identifier to STDERR:
> print STDERR "whereisthis\n";
>
>
> However, I can NOT find such "whereisthis" after I checked the log files and the web interface.
>
> Could anyone tell me how to fix this?
>
> Should I turn on the STDERR somewhere?
>
> I also tried to create and write this identifier to a file. I can NOT find the file either.
>
> Thanks, ds
deqiang sun
Baylor College of Medicine
deqiangs@bcm.edu
STDERR and location of new files
Posted by deqiang sun <de...@bcm.edu>.
Hi everyone,
I am running a perl script through streaming at the beginning of which I print a test identifier to STDERR:
print STDERR "whereisthis\n";
However, I can NOT find such "whereisthis" after I checked the log files and the web interface.
Could anyone tell me how to fix this?
Should I turn on the STDERR somewhere?
I also tried to create and write this identifier to a file. I can NOT find the file either.
Thanks, ds
Re: "lost task" suspect Distributed Cache to blame
Posted by Allen Wittenauer <aw...@linkedin.com>.
On 3/2/10 12:49 PM, "Edward Capriolo" <ed...@gmail.com> wrote:
> This job is somewhat special (for us) in that in involves shipping
> large files over distributed cache. My working theory is that
> something goes wrong with the distributed cache/job tracker and the
> Job/Task/Tips never have a chance.
>
> Has anyone ever experienced something like this?
How big is big and what archiving format are you using? Jar has issues with
files >2gb.
I've also noticed that dcache seems to get written to -every- tmp dir on a
node, which seems a bit ridiculous.