You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@tez.apache.org by "Kuhu Shukla (JIRA)" <ji...@apache.org> on 2017/10/27 20:56:00 UTC
[jira] [Reopened] (TEZ-3738) TestUnorderedPartitionedKVWriter fails
due to RejectedExecutionException
[ https://issues.apache.org/jira/browse/TEZ-3738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kuhu Shukla reopened TEZ-3738:
------------------------------
Reopening this since the test fails intermittently with the same error.
Eg. https://builds.apache.org/job/PreCommit-TEZ-Build/2671/testReport/junit/org.apache.tez.runtime.library.common.writers/TestUnorderedPartitionedKVWriter/testMergeBuffersAndSpill_test_false__NONE__/
Stack trace:
{code}
java.util.concurrent.RejectedExecutionException: Task com.google.common.util.concurrent.ListenableFutureTask@5d1a2c96 rejected from java.util.concurrent.ThreadPoolExecutor@230e3749[Running, pool size = 2, active threads = 2, queued tasks = 0, completed tasks = 1]
at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379)
at com.google.common.util.concurrent.MoreExecutors$ListeningDecorator.execute(MoreExecutors.java:440)
at com.google.common.util.concurrent.AbstractListeningExecutorService.submit(AbstractListeningExecutorService.java:56)
at org.apache.tez.runtime.library.common.writers.UnorderedPartitionedKVWriter.scheduleSpill(UnorderedPartitionedKVWriter.java:503)
at org.apache.tez.runtime.library.common.writers.UnorderedPartitionedKVWriter.mayBeSpill(UnorderedPartitionedKVWriter.java:477)
at org.apache.tez.runtime.library.common.writers.UnorderedPartitionedKVWriter.getNextAvailableBuffer(UnorderedPartitionedKVWriter.java:541)
at org.apache.tez.runtime.library.common.writers.UnorderedPartitionedKVWriter.setupNextBuffer(UnorderedPartitionedKVWriter.java:467)
at org.apache.tez.runtime.library.common.writers.UnorderedPartitionedKVWriter.write(UnorderedPartitionedKVWriter.java:413)
at org.apache.tez.runtime.library.common.writers.UnorderedPartitionedKVWriter.write(UnorderedPartitionedKVWriter.java:365)
at org.apache.tez.runtime.library.common.writers.TestUnorderedPartitionedKVWriter.baseTest(TestUnorderedPartitionedKVWriter.java:1133)
at org.apache.tez.runtime.library.common.writers.TestUnorderedPartitionedKVWriter.baseTest(TestUnorderedPartitionedKVWriter.java:1077)
at org.apache.tez.runtime.library.common.writers.TestUnorderedPartitionedKVWriter.testMergeBuffersAndSpill(TestUnorderedPartitionedKVWriter.java:269)
{code}
Standard output :
{code}
2017-10-21 01:38:20,402 INFO [main] writers.TestUnorderedPartitionedKVWriter (TestUnorderedPartitionedKVWriter.java:setup(149)) - Setup. Using test dir: /tmp/TestUnorderedPartitionedKVWriter
2017-10-21 01:38:20,424 INFO [Thread-83] writers.BaseUnorderedPartitionedKVWriter (BaseUnorderedPartitionedKVWriter.java:<init>(148)) - Instantiating Partitioner: [org.apache.tez.runtime.library.common.writers.TestUnorderedPartitionedKVWriter$PartitionerForTest]
2017-10-21 01:38:20,425 INFO [Thread-83] writers.UnorderedPartitionedKVWriter (UnorderedPartitionedKVWriter.java:<init>(285)) - destinationVertexName: numBuffers=2, sizePerBuffer=1024, skipBuffers=false, numPartitions=10, availableMemory=2048, maxSingleBufferSizeBytes=2048, pipelinedShuffle=false, isFinalMergeEnabled=true, numPartitions=10, reportPartitionStats=NONE
2017-10-21 01:38:20,425 INFO [Thread-83] writers.UnorderedPartitionedKVWriter (UnorderedPartitionedKVWriter.java:setupNextBuffer(460)) - destinationVertexName: Moving to next buffer. Total filled buffers: 0
2017-10-21 01:38:20,425 INFO [Thread-83] writers.UnorderedPartitionedKVWriter (UnorderedPartitionedKVWriter.java:setupNextBuffer(460)) - destinationVertexName: Moving to next buffer. Total filled buffers: 0
2017-10-21 01:38:20,428 INFO [UnorderedOutSpiller {destinationVertexName} #0] writers.UnorderedPartitionedKVWriter (UnorderedPartitionedKVWriter.java:callInternal(638)) - destinationVertexName: Finished spill 0
2017-10-21 01:38:20,428 INFO [Thread-83] writers.UnorderedPartitionedKVWriter (UnorderedPartitionedKVWriter.java:setupNextBuffer(460)) - destinationVertexName: Moving to next buffer. Total filled buffers: 0
2017-10-21 01:38:20,435 INFO [UnorderedOutSpiller {destinationVertexName} #1] writers.UnorderedPartitionedKVWriter (UnorderedPartitionedKVWriter.java:callInternal(638)) - destinationVertexName: Finished spill 1
2017-10-21 01:38:20,437 INFO [Thread-83] writers.UnorderedPartitionedKVWriter (UnorderedPartitionedKVWriter.java:setupNextBuffer(460)) - destinationVertexName: Moving to next buffer. Total filled buffers: 0
2017-10-21 01:38:20,438 INFO [UnorderedOutSpiller {destinationVertexName} #0] writers.UnorderedPartitionedKVWriter (UnorderedPartitionedKVWriter.java:callInternal(638)) - destinationVertexName: Finished spill 2
2017-10-21 01:38:20,438 INFO [main] writers.TestUnorderedPartitionedKVWriter (TestUnorderedPartitionedKVWriter.java:cleanup(157)) - CleanUp
{code}
I cannot tell if this is broken recently(since most commits don't seem pertinent to this) or has just been flaky enough to have not failed since June 2017(approx.).
Even though the test fails irrespective of the value of isFinalMergeEnabled, the following code to me seemed suspect since we don't know if the spilling phase has finished as the numSpills could change after mergeAll?
{code}
if (isFinalMergeEnabled) {
if (numSpills.get() > 0) {
mergeAll();
} else {
finalSpill();
}
updateTezCountersAndNotify();
eventList.add(generateVMEvent());
eventList.add(generateDMEvent());
}
{code}
Appreciate any thoughts, comments from [~rajesh.balamohan], [~harishjp] to help understand the issue and if I have missed something. Thanks a lot!
> TestUnorderedPartitionedKVWriter fails due to RejectedExecutionException
> ------------------------------------------------------------------------
>
> Key: TEZ-3738
> URL: https://issues.apache.org/jira/browse/TEZ-3738
> Project: Apache Tez
> Issue Type: Bug
> Reporter: Jason Lowe
>
> TestUnorderedPartitionedKVWriter is failing in recent precommit builds. Stacktrace to follow.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)