[jira] Commented: (HADOOP-18) Crash with multiple temp directories

["Rod Taylor (JIRA)" <ji...@apache.org>]

    [ http://issues.apache.org/jira/browse/HADOOP-18?page=comments#action_12368835 ] 

Rod Taylor commented on HADOOP-18:
----------------------------------

Finally figured it out. One temp directory was filling up (different sizes) and the fetch was aborting BUT I didn't see this in the logs for the longest time.

The patch in NUTCH-143 helped track down the issue because it caused the error to be noticed be the scripts driving the code in a location close to where the error took place rather than several steps (possibly many hours) later..

Please close this bug.

> Crash with multiple temp directories
> ------------------------------------
>
>          Key: HADOOP-18
>          URL: http://issues.apache.org/jira/browse/HADOOP-18
>      Project: Hadoop
>         Type: Bug
>   Components: mapred
>     Reporter: Rod Taylor
>     Priority: Critical

>
> A brief read of the code indicated it may be possible to use multiple local directories using something like the below:
>   <property>
>     <name>mapred.local.dir</name>
>     <value>/local,/local1,/local2</value>
>     <description>The local directory where MapReduce stores intermediate
>     data files.
>     </description>
>   </property>
> This failed with the below exception during either the generate or update phase (not entirely sure which).
> java.lang.ArrayIndexOutOfBoundsException
>         at java.util.zip.CRC32.update(CRC32.java:51)
>         at org.apache.nutch.fs.NFSDataInputStream$Checker.read(NFSDataInputStream.java:92)
>         at org.apache.nutch.fs.NFSDataInputStream$PositionCache.read(NFSDataInputStream.java:156)
>         at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
>         at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
>         at java.io.BufferedInputStream.read(BufferedInputStream.java:313)
>         at java.io.DataInputStream.readFully(DataInputStream.java:176)
>         at org.apache.nutch.io.DataOutputBuffer$Buffer.write(DataOutputBuffer.java:55)
>         at org.apache.nutch.io.DataOutputBuffer.write(DataOutputBuffer.java:89)
>         at org.apache.nutch.io.SequenceFile$Reader.next(SequenceFile.java:378)
>         at org.apache.nutch.io.SequenceFile$Reader.next(SequenceFile.java:301)
>         at org.apache.nutch.io.SequenceFile$Reader.next(SequenceFile.java:323)
>         at org.apache.nutch.mapred.SequenceFileRecordReader.next(SequenceFileRecordReader.java:60)
>         at org.apache.nutch.segment.SegmentReader$InputFormat$1.next(SegmentReader.java:80)
>         at org.apache.nutch.mapred.MapTask$2.next(MapTask.java:106)
>         at org.apache.nutch.mapred.MapRunner.run(MapRunner.java:48)
>         at org.apache.nutch.mapred.MapTask.run(MapTask.java:116)
>         at org.apache.nutch.mapred.TaskTracker$Child.main(TaskTracker.java:604)

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira