You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@tez.apache.org by "Rajesh Balamohan (Jira)" <ji...@apache.org> on 2020/06/23 09:56:00 UTC
[jira] [Created] (TEZ-4194) NPE in FetcherOrderedGrouped
Rajesh Balamohan created TEZ-4194:
-------------------------------------
Summary: NPE in FetcherOrderedGrouped
Key: TEZ-4194
URL: https://issues.apache.org/jira/browse/TEZ-4194
Project: Apache Tez
Issue Type: Bug
Affects Versions: 0.9.1
Reporter: Rajesh Balamohan
When running store_sales data generation (@ 1 TB scale) on a cloud environment, this was observed.
This causes task to fail and re-execute. One in 2 or 4 runs gets into this. Weirdly, not sure why its trying to call fetcher when all inputs are downloaded as per log.
{noformat}
2020-06-23 08:49:34,546 [INFO] [Fetcher_O {Map_1} #1] |orderedgrouped.ShuffleScheduler|: All inputs fetched for input vertex : Map 1
2020-06-23 08:49:34,546 [INFO] [Fetcher_O {Map_1} #1] |orderedgrouped.ShuffleScheduler|: copy(2385 (spillsFetched=2385) of 2385. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.01 MB/s)
2020-06-23 08:49:34,546 [INFO] [ShuffleAndMergeRunner {Map_1}] |orderedgrouped.ShuffleScheduler|: Shutting down FetchScheduler for input: Map_1, wasInterrupted=false
2020-06-23 08:49:34,547 [INFO] [ShuffleAndMergeRunner {Map_1}] |orderedgrouped.ShuffleScheduler|: copy(2385 (spillsFetched=2385) of 2385. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.01 MB/s)
2020-06-23 08:49:34,548 [INFO] [ShuffleAndMergeRunner {Map_1}] |orderedgrouped.ShuffleScheduler|: Shutting down fetchers for input: Map_1, shutdown timetaken: 0 ms, hasFetcherExecutorStopped: true
2020-06-23 08:49:34,549 [INFO] [ShuffleAndMergeRunner {Map_1}] |orderedgrouped.MergeManager|: TotalInMemFetchStats: count=2115, totalSize=28250798, min=2011, max=17268, avg=1.0
2020-06-23 08:49:34,549 [INFO] [ShuffleAndMergeRunner {Map_1}] |orderedgrouped.MergeManager|: finalMerge with #inMemoryOutputs=2115, size=28250798 and #onDiskOutputs=144, size=3487553
2020-06-23 08:49:34,686 [INFO] [Fetcher_O {Map_1} #3] |orderedgrouped.Shuffle|: Map_1: Setting throwable in reportException with message [null] from thread [Fetcher_O {Map_1} #3
2020-06-23 08:49:34,687 [INFO] [Fetcher_O {Map_1} #3] |orderedgrouped.ShuffleScheduler|: Map_1: Already shutdown. Ignoring fetch complete
2020-06-23 08:49:35,044 [ERROR] [ShuffleAndMergeRunner {Map_1}] |orderedgrouped.Shuffle|: Map_1: ShuffleRunner failed with error
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.Shuffle$ShuffleError: error in shuffle in Fetcher_O {Map_1} #3
at org.apache.tez.runtime.library.common.shuffle.orderedgrouped.Shuffle$RunShuffleCallable.callInternal(Shuffle.java:332)
at org.apache.tez.runtime.library.common.shuffle.orderedgrouped.Shuffle$RunShuffleCallable.callInternal(Shuffle.java:288)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69)
at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NullPointerException
at org.apache.tez.runtime.library.common.shuffle.orderedgrouped.FetcherOrderedGrouped.setupConnection(FetcherOrderedGrouped.java:354)
at org.apache.tez.runtime.library.common.shuffle.orderedgrouped.FetcherOrderedGrouped.copyFromHost(FetcherOrderedGrouped.java:263)
at org.apache.tez.runtime.library.common.shuffle.orderedgrouped.FetcherOrderedGrouped.fetchNext(FetcherOrderedGrouped.java:182)
at org.apache.tez.runtime.library.common.shuffle.orderedgrouped.FetcherOrderedGrouped.callInternal(FetcherOrderedGrouped.java:194)
at org.apache.tez.runtime.library.common.shuffle.orderedgrouped.FetcherOrderedGrouped.callInternal(FetcherOrderedGrouped.java:57)
... 7 more
2020-06-23 08:49:35,046 [INFO] [ShuffleAndMergeRunner {Map_1}] |task.TezTaskRunner2|: Received notification of a failure which will cause the task to die
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.Shuffle$ShuffleError: error in shuffle in Fetcher_O {Map_1} #3
at org.apache.tez.runtime.library.common.shuffle.orderedgrouped.Shuffle$RunShuffleCallable.callInternal(Shuffle.java:332)
at org.apache.tez.runtime.library.common.shuffle.orderedgrouped.Shuffle$RunShuffleCallable.callInternal(Shuffle.java:288)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69)
at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NullPointerException
at org.apache.tez.runtime.library.common.shuffle.orderedgrouped.FetcherOrderedGrouped.setupConnection(FetcherOrderedGrouped.java:354)
at org.apache.tez.runtime.library.common.shuffle.orderedgrouped.FetcherOrderedGrouped.copyFromHost(FetcherOrderedGrouped.java:263)
at org.apache.tez.runtime.library.common.shuffle.orderedgrouped.FetcherOrderedGrouped.fetchNext(FetcherOrderedGrouped.java:182)
at org.apache.tez.runtime.library.common.shuffle.orderedgrouped.FetcherOrderedGrouped.callInternal(FetcherOrderedGrouped.java:194)
at org.apache.tez.runtime.library.common.shuffle.orderedgrouped.FetcherOrderedGrouped.callInternal(FetcherOrderedGrouped.java:57)
... 7 more
{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)