[jira] [Created] (TEZ-2378) In case Fetcher (unordered) fails to do local fetch, log in debug mode to reduce log size

["Rajesh Balamohan (JIRA)" <ji...@apache.org>]

Rajesh Balamohan created TEZ-2378:
-------------------------------------

             Summary: In case Fetcher (unordered) fails to do local fetch, log in debug mode to reduce log size
                 Key: TEZ-2378
                 URL: https://issues.apache.org/jira/browse/TEZ-2378
             Project: Apache Tez
          Issue Type: Bug
            Reporter: Rajesh Balamohan


Following can be logged as debug mode as opposed to WARN level. May be counters can be added later to track the number of times it failed to do local-fetch.

{noformat}
2015-04-28 05:41:45,487 WARN [Fetcher [Map_5] #15] shuffle.Fetcher: Failed to shuffle output of InputAttemptIdentifier [inputIdentifier=InputIdentifier [inputIndex=81], attemptNumber=0, pathComponent=attempt_1429683757595_0485_1_03_000081_0_10003, fetchTypeInfo=FINAL_MERGE_ENABLED, spillEventId=-1] from cn047-10.l42scl.hortonworks.com(local fetch)
org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find output/attempt_1429683757595_0485_1_03_000081_0_10003/file.out.index in any of the configured local directories
        at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:449)
        at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:164)
        at org.apache.tez.runtime.library.common.shuffle.Fetcher.getShuffleInputFileName(Fetcher.java:612)
        at org.apache.tez.runtime.library.common.shuffle.Fetcher.getTezIndexRecord(Fetcher.java:592)
        at org.apache.tez.runtime.library.common.shuffle.Fetcher.doLocalDiskFetch(Fetcher.java:537)
        at org.apache.tez.runtime.library.common.shuffle.Fetcher.doSharedFetch(Fetcher.java:353)
        at org.apache.tez.runtime.library.common.shuffle.Fetcher.callInternal(Fetcher.java:192)
        at org.apache.tez.runtime.library.common.shuffle.Fetcher.callInternal(Fetcher.java:72)
        at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)