You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@tez.apache.org by "Tassapol Athiapinya (JIRA)" <ji...@apache.org> on 2014/05/06 02:48:14 UTC
[jira] [Commented] (TEZ-1088) Flaky Test:
TestFaultTolerance.testInputFailureCausesRerunAttemptWithinMaxAttemptSuccess
[ https://issues.apache.org/jira/browse/TEZ-1088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13990149#comment-13990149 ]
Tassapol Athiapinya commented on TEZ-1088:
------------------------------------------
The reason of flaky test is race condition of events seen by a task. In this test, v2-task0 has input failure. v2-task1 has *no* input failure. However, v2-task1 can see events of rerun upstream attempt caused by v2-task0. Once that happens, TestInput has code path to set input version with the largest value.
{code:title=TestInput line 301}
synchronized (inputReady) {
inputReady.set(maxInputVersionSeen);
LOG.info("Notifying done with " + maxInputVersionSeen);
inputReady.notifyAll();
}
{code}
Since v2-task1 attempt gets input version 1 number higher than expected, test will fail.
One workaround fix is to let v2-task1 to accept a value one version higher. In this test case, we don't exercise limit on max failed attempt. Even if v2-task1 sees only attempt with no rerun, test will still pass once v2-task1 itself does rerun (which bumps actual output by 1 to reach expected output).
> Flaky Test: TestFaultTolerance.testInputFailureCausesRerunAttemptWithinMaxAttemptSuccess
> ----------------------------------------------------------------------------------------
>
> Key: TEZ-1088
> URL: https://issues.apache.org/jira/browse/TEZ-1088
> Project: Apache Tez
> Issue Type: Bug
> Reporter: Hitesh Shah
> Assignee: Tassapol Athiapinya
> Attachments: org.apache.tez.test.TestFaultTolerance.zip, syslog_dag_1398715972246_0001_9
>
>
> 2014-04-28 20:14:19,100 INFO [AsyncDispatcher event handler] org.apache.tez.dag.history.recovery.RecoveryService: DAG completed, dagId=dag_1398715972246_0001_9, queueSize=0
> 2014-04-28 20:14:19,147 INFO [AsyncDispatcher event handler] org.apache.tez.dag.history.HistoryEventHandler: [HISTORY][DAG:dag_1398715972246_0001_9][Event:DAG_FINISHED]: dagId=dag_1398715972246_0001_9, startTime=1398716046982, finishTime=1398716059090, timeTaken=12108, status=FAILED, diagnostics=Vertex re-running, vertexName=v1, vertexId=vertex_1398715972246_0001_9_00
> Vertex failed, vertexName=v2, vertexId=vertex_1398715972246_0001_9_01, diagnostics=[Task failed, taskId=task_1398715972246_0001_9_01_000001, diagnostics=[AttemptID:attempt_1398715972246_0001_9_01_000001_0 Info:Error: exceptionThrown=java.lang.RuntimeException: Expected output mismatch of current FailingProcessor: attempt_1398715972246_0001_9_01_000001_0_10001 dag: testInputFailureCausesRerunAttemptWithinMaxAttemptSuccess vertex: v2 taskIndex: 1 taskAttempt: 0
> Expected output: 4 got: 5
> at org.apache.tez.test.TestProcessor.throwException(TestProcessor.java:98)
> at org.apache.tez.test.TestProcessor.run(TestProcessor.java:250)
> at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:307)
> at org.apache.hadoop.mapred.YarnTezDagChild$5.run(YarnTezDagChild.java:581)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
> at org.apache.hadoop.mapred.YarnTezDagChild.main(YarnTezDagChild.java:570)
> , errorMessage=Expected output mismatch of current FailingProcessor: attempt_1398715972246_0001_9_01_000001_0_10001 dag: testInputFailureCausesRerunAttemptWithinMaxAttemptSuccess vertex: v2 taskIndex: 1 taskAttempt: 0
> Expected output: 4 got: 5
> Container killed by the ApplicationMaster.
> Container killed on request. Exit code is 143
> , AttemptID:attempt_1398715972246_0001_9_01_000001_1 Info:Error: exceptionThrown=java.lang.RuntimeException: Expected output mismatch of current FailingProcessor: attempt_1398715972246_0001_9_01_000001_1_10011 dag: testInputFailureCausesRerunAttemptWithinMaxAttemptSuccess vertex: v2 taskIndex: 1 taskAttempt: 1
> Expected output: 4 got: 5
> at org.apache.tez.test.TestProcessor.throwException(TestProcessor.java:98)
> at org.apache.tez.test.TestProcessor.run(TestProcessor.java:250)
> at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:307)
> at org.apache.hadoop.mapred.YarnTezDagChild$5.run(YarnTezDagChild.java:581)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
> at org.apache.hadoop.mapred.YarnTezDagChild.main(YarnTezDagChild.java:570)
> , errorMessage=Expected output mismatch of current FailingProcessor: attempt_1398715972246_0001_9_01_000001_1_10011 dag: testInputFailureCausesRerunAttemptWithinMaxAttemptSuccess vertex: v2 taskIndex: 1 taskAttempt: 1
> Expected output: 4 got: 5
> Container released by application, AttemptID:attempt_1398715972246_0001_9_01_000001_2 Info:Error: exceptionThrown=java.lang.RuntimeException: Expected output mismatch of current FailingProcessor: attempt_1398715972246_0001_9_01_000001_2_10003 dag: testInputFailureCausesRerunAttemptWithinMaxAttemptSuccess vertex: v2 taskIndex: 1 taskAttempt: 2
> Expected output: 4 got: 6
> at org.apache.tez.test.TestProcessor.throwException(TestProcessor.java:98)
> at org.apache.tez.test.TestProcessor.run(TestProcessor.java:250)
> at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:307)
> at org.apache.hadoop.mapred.YarnTezDagChild$5.run(YarnTezDagChild.java:581)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
> at org.apache.hadoop.mapred.YarnTezDagChild.main(YarnTezDagChild.java:570)
> , errorMessage=Expected output mismatch of current FailingProcessor: attempt_1398715972246_0001_9_01_000001_2_10003 dag: testInputFailureCausesRerunAttemptWithinMaxAttemptSuccess vertex: v2 taskIndex: 1 taskAttempt: 2
> Expected output: 4 got: 6
> Container released by application, AttemptID:attempt_1398715972246_0001_9_01_000001_3 Info:Error: exceptionThrown=java.lang.RuntimeException: Expected output mismatch of current FailingProcessor: attempt_1398715972246_0001_9_01_000001_3_10001 dag: testInputFailureCausesRerunAttemptWithinMaxAttemptSuccess vertex: v2 taskIndex: 1 taskAttempt: 3
> Expected output: 4 got: 8
> at org.apache.tez.test.TestProcessor.throwException(TestProcessor.java:98)
> at org.apache.tez.test.TestProcessor.run(TestProcessor.java:250)
> at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:307)
> at org.apache.hadoop.mapred.YarnTezDagChild$5.run(YarnTezDagChild.java:581)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
> at org.apache.hadoop.mapred.YarnTezDagChild.main(YarnTezDagChild.java:570)
> , errorMessage=Expected output mismatch of current FailingProcessor: attempt_1398715972246_0001_9_01_000001_3_10001 dag: testInputFailureCausesRerunAttemptWithinMaxAttemptSuccess vertex: v2 taskIndex: 1 taskAttempt: 3
> Expected output: 4 got: 8], Vertex failed as one or more tasks failed. failedTasks:1]
> DAG failed due to vertex failure. failedVertices:1 killedVertices:0, counters=Counters: 12, org.apache.tez.common.counters.DAGCounter, NUM_FAILED_TASKS=6, TOTAL_LAUNCHED_TASKS=9, File System Counters, HDFS: BYTES_READ=0, HDFS: BYTES_WRITTEN=0, HDFS: READ_OPS=0, HDFS: LARGE_READ_OPS=0, HDFS: WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, GC_TIME_MILLIS=21, CPU_MILLISECONDS=-12330, PHYSICAL_MEMORY_BYTES=548294656, VIRTUAL_MEMORY_BYTES=4034506752, COMMITTED_HEAP_BYTES=310968320
> 2014-04-28 20:14:19,147 INFO [AsyncDispatcher event handler] org.apache.tez.dag.app.dag.impl.DAGImpl: DAG: dag_1398715972246_0001_9 finished with state: FAILED
> 2014-04-28 20:14:19,148 INFO [AsyncDispatcher event handler] org.apache.tez.dag.app.dag.impl.DAGImpl: dag_1398715972246_0001_9 transitioned from RUNNING to FAILED
> 2014-04-28 20:14:19,148 INFO [AsyncDispatcher event handler] org.apache.tez.dag.app.DAGAppMaster: DAG completed, dagId=dag_1398715972246_0001_9, dagState=FAILED
> 2014-04-28 20:14:19,148 INFO [AsyncDispatcher event handler] org.apache.tez.common.TezUtils: Redirecting log files based on addend: dag_1398715972246_0001_9_post
> 2014-04-28 20:14:19,148 INFO [ContainerLauncher #4] org.apache.tez.dag.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_STOP_REQUEST
--
This message was sent by Atlassian JIRA
(v6.2#6252)