You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Mayuran Yogarajah <ma...@casalemedia.com> on 2009/06/07 02:55:35 UTC

Every time the mapping phase finishes I see this

There are always a few 'Failed/Killed Task Attempts' and when I view the 
logs for
these I see:

- some that are empty, ie stdout/stderr/syslog logs are all blank
- several that say:

2009-06-06 20:47:15,309 WARN org.apache.hadoop.mapred.TaskTracker: Error running child
java.io.IOException: Filesystem closed
	at org.apache.hadoop.dfs.DFSClient.checkOpen(DFSClient.java:195)
	at org.apache.hadoop.dfs.DFSClient.access$600(DFSClient.java:59)
	at org.apache.hadoop.dfs.DFSClient$DFSInputStream.close(DFSClient.java:1359)
	at java.io.FilterInputStream.close(FilterInputStream.java:159)
	at org.apache.hadoop.mapred.LineRecordReader$LineReader.close(LineRecordReader.java:103)
	at org.apache.hadoop.mapred.LineRecordReader.close(LineRecordReader.java:301)
	at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.close(MapTask.java:173)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:231)
	at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2198)



Any idea why this happens? I don't understand why I'd be seeing these 
only as the mappers get to
100%.

thanks

Re: Every time the mapping phase finishes I see this

Posted by Mayuran Yogarajah <ma...@casalemedia.com>.
I should mention..these are Hadoop streaming jobs, Hadoop version 
hadoop-0.18.3.

Any idea about the empty stdout/stderr/syslog logs? I have no way to 
really track down
whats causing them. 

thanks


Steve Loughran wrote:
> Mayuran Yogarajah wrote:
>   
>> There are always a few 'Failed/Killed Task Attempts' and when I view the
>> logs for
>> these I see:
>>
>> - some that are empty, ie stdout/stderr/syslog logs are all blank
>> - several that say:
>>
>> 2009-06-06 20:47:15,309 WARN org.apache.hadoop.mapred.TaskTracker: Error
>> running child
>> java.io.IOException: Filesystem closed
>>     at org.apache.hadoop.dfs.DFSClient.checkOpen(DFSClient.java:195)
>>     at org.apache.hadoop.dfs.DFSClient.access$600(DFSClient.java:59)
>>     at
>> org.apache.hadoop.dfs.DFSClient$DFSInputStream.close(DFSClient.java:1359)
>>     at java.io.FilterInputStream.close(FilterInputStream.java:159)
>>     at
>> org.apache.hadoop.mapred.LineRecordReader$LineReader.close(LineRecordReader.java:103)
>>
>>     at
>> org.apache.hadoop.mapred.LineRecordReader.close(LineRecordReader.java:301)
>>     at
>> org.apache.hadoop.mapred.MapTask$TrackedRecordReader.close(MapTask.java:173)
>>
>>     at org.apache.hadoop.mapred.MapTask.run(MapTask.java:231)
>>     at
>> org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2198)
>>
>>
>>
>> Any idea why this happens? I don't understand why I'd be seeing these
>> only as the mappers get to
>> 100%.
>>     
>
> Seen this when something in the same process got a FileSystem reference
> by FileSystem.get() and then called close() on it -it closes the client
> for every thread/class that has a reference to the same object.
>
>
> We're planning on adding more diagnostics, by tracking who closed the
> filesystem
> https://issues.apache.org/jira/browse/HADOOP-5933
>

Re: Every time the mapping phase finishes I see this

Posted by Steve Loughran <st...@apache.org>.
Mayuran Yogarajah wrote:
> There are always a few 'Failed/Killed Task Attempts' and when I view the 
> logs for
> these I see:
> 
> - some that are empty, ie stdout/stderr/syslog logs are all blank
> - several that say:
> 
> 2009-06-06 20:47:15,309 WARN org.apache.hadoop.mapred.TaskTracker: Error 
> running child
> java.io.IOException: Filesystem closed
>     at org.apache.hadoop.dfs.DFSClient.checkOpen(DFSClient.java:195)
>     at org.apache.hadoop.dfs.DFSClient.access$600(DFSClient.java:59)
>     at 
> org.apache.hadoop.dfs.DFSClient$DFSInputStream.close(DFSClient.java:1359)
>     at java.io.FilterInputStream.close(FilterInputStream.java:159)
>     at 
> org.apache.hadoop.mapred.LineRecordReader$LineReader.close(LineRecordReader.java:103) 
> 
>     at 
> org.apache.hadoop.mapred.LineRecordReader.close(LineRecordReader.java:301)
>     at 
> org.apache.hadoop.mapred.MapTask$TrackedRecordReader.close(MapTask.java:173) 
> 
>     at org.apache.hadoop.mapred.MapTask.run(MapTask.java:231)
>     at 
> org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2198)
> 
> 
> 
> Any idea why this happens? I don't understand why I'd be seeing these 
> only as the mappers get to
> 100%.

Seen this when something in the same process got a FileSystem reference 
by FileSystem.get() and then called close() on it -it closes the client 
for every thread/class that has a reference to the same object.


We're planning on adding more diagnostics, by tracking who closed the 
filesystem
https://issues.apache.org/jira/browse/HADOOP-5933