You are viewing a plain text version of this content. The canonical link for it is here.

Posted to common-user@hadoop.apache.org by Rakhi Khatwani <ra...@gmail.com> on 2009/04/05 08:48:05 UTC

My Map Tasks are getting killed

Hi,
      I am executing a job on ec2 (set up on cluster with 18 nodes... my job
has 7 Map tasks). however my tasks gets killed without reporting an error.

i even tried going through the logs, which happens to be fine.

On the UI the tasks fail and the status shows as KILLED (error column being
empty).
and eventually the job fails.

what could go wrong?? I am finding it weird since this is the 1st time i am
comming across this kinda failure.

Thanks,
Raakhi

Re: My Map Tasks are getting killed

Posted by Rakhi Khatwani <ra...@gmail.com>.

One thing i noticed is 3 of my regions servers shut down:
giving the following errors:

2009-04-05 00:08:45,727 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner
7674788444062751362 lease expired
2009-04-05 00:08:45,731 ERROR
org.apache.hadoop.hbase.regionserver.StoreFileScanner: [B@132a2bc closing
scanner
java.io.IOException: Filesystem closed
       at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:198)
       at org.apache.hadoop.hdfs.DFSClient.access$600(DFSClient.java:65)
       at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.close(DFSClient.java:1573)
       at java.io.FilterInputStream.close(FilterInputStream.java:155)
       at
org.apache.hadoop.hbase.io.SequenceFile$Reader.close(SequenceFile.java:1598)
       at org.apache.hadoop.hbase.io.MapFile$Reader.close(MapFile.java:586)
       at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.close(StoreFileScanner.java:356)
       at
org.apache.hadoop.hbase.regionserver.HStoreScanner.closeScanner(HStoreScanner.java:289)
       at
org.apache.hadoop.hbase.regionserver.HStoreScanner.doClose(HStoreScanner.java:309)
       at
org.apache.hadoop.hbase.regionserver.HStoreScanner.close(HStoreScanner.java:303)
       at
org.apache.hadoop.hbase.regionserver.HRegion$HScanner.closeScanner(HRegion.java:2119)
       at
org.apache.hadoop.hbase.regionserver.HRegion$HScanner.close(HRegion.java:2139)
       at
org.apache.hadoop.hbase.regionserver.HRegionServer$ScannerListener.leaseExpired(HRegionServer.java:1759)
       at org.apache.hadoop.hbase.Leases.run(Leases.java:95)
 2009-04-05 00:08:45,732 ERROR
org.apache.hadoop.hbase.regionserver.StoreFileScanner: [B@159f8be closing
scanner
java.io.IOException: Filesystem closed
       at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:198)
       at org.apache.hadoop.hdfs.DFSClient.access$600(DFSClient.java:65)
       at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.close(DFSClient.java:1573)
       at java.io.FilterInputStream.close(FilterInputStream.java:155)
       at
org.apache.hadoop.hbase.io.SequenceFile$Reader.close(SequenceFile.java:1598)
       at org.apache.hadoop.hbase.io.MapFile$Reader.close(MapFile.java:586)
       at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.close(StoreFileScanner.java:356)
       at
org.apache.hadoop.hbase.regionserver.HStoreScanner.closeScanner(HStoreScanner.java:289)
       at
org.apache.hadoop.hbase.regionserver.HStoreScanner.doClose(HStoreScanner.java:309)
       at
org.apache.hadoop.hbase.regionserver.HStoreScanner.close(HStoreScanner.java:303)
       at
org.apache.hadoop.hbase.regionserver.HRegion$HScanner.closeScanner(HRegion.java:2119)
       at
org.apache.hadoop.hbase.regionserver.HRegion$HScanner.close(HRegion.java:2139)
       at
org.apache.hadoop.hbase.regionserver.HRegionServer$ScannerListener.leaseExpired(HRegionServer.java:1759)
       at org.apache.hadoop.hbase.Leases.run(Leases.java:95)
2009-04-05 00:08:45,732 ERROR
org.apache.hadoop.hbase.regionserver.StoreFileScanner: [B@1d9ba08 closing
scanner
java.io.IOException: Filesystem closed
       at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:198)
       at org.apache.hadoop.hdfs.DFSClient.access$600(DFSClient.java:65)
       at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.close(DFSClient.java:1573)
       at java.io.FilterInputStream.close(FilterInputStream.java:155)
       at
org.apache.hadoop.hbase.io.SequenceFile$Reader.close(SequenceFile.java:1598)
       at org.apache.hadoop.hbase.io.MapFile$Reader.close(MapFile.java:586)
       ......

 what could be the reason why this is happening?

is it a problem because i am performing a split on the table inside my map??

Thanks
Raakhi.


>
>
> On Sun, Apr 5, 2009 at 12:18 PM, Rakhi Khatwani <ra...@gmail.com>wrote:
>
>> Hi,
>>       I am executing a job on ec2 (set up on cluster with 18 nodes... my
>> job has 7 Map tasks). however my tasks gets killed without reporting an
>> error.
>>
>> i even tried going through the logs, which happens to be fine.
>>
>> On the UI the tasks fail and the status shows as KILLED (error column
>> being empty).
>> and eventually the job fails.
>>
>> what could go wrong?? I am finding it weird since this is the 1st time i
>> am comming across this kinda failure.
>>
>> Thanks,
>> Raakhi
>>
>
>