[jira] [Commented] (TEZ-1387) Add proper diagonstics message for disk issues

["Hitesh Shah (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/TEZ-1387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14568233#comment-14568233 ] 

Hitesh Shah commented on TEZ-1387:
----------------------------------

[~zjffdu] Can you confirm that we are already reporting back the yarn exit code if an attempt fails via the termination cause? If yes, we can close this out as  a dup. 

> Add proper diagonstics message for disk issues
> ----------------------------------------------
>
>                 Key: TEZ-1387
>                 URL: https://issues.apache.org/jira/browse/TEZ-1387
>             Project: Apache Tez
>          Issue Type: Sub-task
>            Reporter: Yesha Vora
>
> Tez prints 'java.io.IOException: Spill failed' message where disks are full. It should print better diagnostic message such as  "disk is full" .
> {noformat}
> 2014-06-13 12:09:37,202 INFO [main] org.apache.tez.runtime.library.common.sort.impl.dflt.DefaultSorter: (EQUATOR) 146679246 kvi 36669804(146679216)
> 2014-06-13 12:09:37,383 WARN [SpillThread [finalreduce_] org.apache.tez.runtime.library.common.sort.impl.dflt.DefaultSorter: Got an exception in sortAndSpill
> org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for attempt_1402677732456_0109_1_00_000003_1_10003_spill_0.out
>         at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:402)
>         at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150)
>         at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131)
>         at org.apache.tez.runtime.library.common.task.local.output.TezTaskOutputFiles.getSpillFileForWrite(TezTaskOutputFiles.java:183)
>         at org.apache.tez.runtime.library.common.sort.impl.dflt.DefaultSorter.spill(DefaultSorter.java:739)
>         at org.apache.tez.runtime.library.common.sort.impl.dflt.DefaultSorter.sortAndSpill(DefaultSorter.java:723)
>         at org.apache.tez.runtime.library.common.sort.impl.dflt.DefaultSorter$SpillThread.run(DefaultSorter.java:655)
> 2014-06-13 12:09:37,389 INFO [main] org.apache.tez.runtime.LogicalIOProcessorRuntimeTask: Final Counters : Counters: 10 [[org.apache.tez.common.counters.TaskCounter SPLIT_RAW_BYTES=221, SPILLED_RECORDS=0, INPUT_RECORDS_PROCESSED=16398, OUTPUT_RECORDS=16397, OUTPUT_BYTES=173778252, OUTPUT_BYTES_WITH_OVERHEAD=0, OUTPUT_BYTES_PHYSICAL=0, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0]]
> 2014-06-13 12:09:37,390 INFO [Tez Container Heartbeat Thread [container_1402677732456_0109_01_000021]] org.apache.hadoop.mapred.YarnTezDagChild: Heartbeat thread interrupted.  stopped: true error: false
> 2014-06-13 12:09:37,390 INFO [Tez Container Heartbeat Thread [container_1402677732456_0109_01_000021]] org.apache.hadoop.mapred.YarnTezDagChild: Current task marked as complete. Stopping heartbeat thread and allowing normal container shutdown
> 2014-06-13 12:09:37,390 FATAL [main] org.apache.hadoop.mapred.YarnTezDagChild: Error running child : java.io.IOException: Spill failed
>         at org.apache.tez.runtime.library.common.sort.impl.dflt.DefaultSorter.checkSpillException(DefaultSorter.java:686)
>         at org.apache.tez.runtime.library.common.sort.impl.dflt.DefaultSorter.collect(DefaultSorter.java:211)
>         at org.apache.tez.runtime.library.common.sort.impl.dflt.DefaultSorter.write(DefaultSorter.java:185)
>         at org.apache.tez.runtime.library.output.OnFileSortedOutput$1.write(OnFileSortedOutput.java:116)
>         at org.apache.tez.mapreduce.processor.map.MapProcessor$NewOutputCollector.write(MapProcessor.java:373)
>         at org.apache.tez.mapreduce.hadoop.mapreduce.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:90)
>         at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:112)
>         at org.apache.hadoop.mapreduce.Mapper.map(Mapper.java:124)
>         at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
>         at org.apache.tez.mapreduce.processor.map.MapProcessor.runNewMapper(MapProcessor.java:247)
>         at org.apache.tez.mapreduce.processor.map.MapProcessor.run(MapProcessor.java:134)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)