[jira] [Commented] (HIVE-28962) Prevent committing outputs if an exception was thrown in the Tez processor

["Denys Kuzmenko (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HIVE-28962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17954394#comment-17954394 ] 

Denys Kuzmenko commented on HIVE-28962:
---------------------------------------

Merged to master
Thanks [~simhadri-g] and [~abstractdog] for the review!

> Prevent committing outputs if an exception was thrown in the Tez processor
> --------------------------------------------------------------------------
>
>                 Key: HIVE-28962
>                 URL: https://issues.apache.org/jira/browse/HIVE-28962
>             Project: Hive
>          Issue Type: Bug
>          Components: Iceberg integration
>            Reporter: Denys Kuzmenko
>            Assignee: Denys Kuzmenko
>            Priority: Major
>              Labels: pull-request-available
>
> Communication failure between coordinator and executor led to a task kill (06:04:18.328Z)
> {code:java}
> app=query-coordinator-0/2025-05-16-06-00_query-coordinator_query-coordinator-0-7_638e93f4-c948-4fc0-ad05-1451fc4b7c53_0.log:<14>1 2025-05-16T06:04:18.328Z query-coordinator-0-7 query-coordinator 1 638e93f4-c948-4fc0-ad05-1451fc4b7c53 [mdc@38374 class="HistoryEventHandler.criticalEvents" level="INFO" thread="Dispatcher thread {Central}"] [HISTORY][DAG:dag_1746256596566_0014_1634][Event:TASK_ATTEMPT_FINISHED]: vertexName=Reducer 3, taskAttemptId=attempt_1746256596566_0014_1634_07_000070_13, creationTime=1747375458134, allocationTime=1747375458277, startTime=1747375458294, finishTime=1747375458328, timeTaken=34, status=KILLED, errorEnum=NODE_FAILED, diagnostics=Node with same host and port but with new unique ID pinged, nodeHttpAddress=<>, counters=Counters: 1, org.apache.tez.common.counters.TaskCounter, TASK_DURATION_MILLIS=34
> {code:java}
>  {code}
> then, an InterruptedException was thrown by `ReduceRecordProcessor.init()` because of task attempt kill (06:04:27.442Z)
> {code:java}
> <11>1 2025-05-16T06:04:27.442Z query-executor-0-16 query-executor 1 71c82503-aeaa-4ca8-ad1c-c01db6df4f9d [mdc@38374 class="tez.TezProcessor" dagId="dag_1746256596566_0014_1634" fragmentId="1746256596566_0014_1634_07_000070_13" level="ERROR" queryId="hive_20250516055906_b7f7cff1-fc84-44d2-a7c5-30bd9aa428c5" thread="TezTR-596566_14_1634_7_70_13"] java.lang.InterruptedException\r at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056)\r at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2090)\r at org.apache.tez.runtime.InputReadyTracker$InputReadyMonitor.awaitCondition(InputReadyTracker.java:147)\r at org.apache.tez.runtime.InputReadyTracker.waitForAllInputsReady(InputReadyTracker.java:107)\r at org.apache.tez.runtime.api.impl.TezProcessorContextImpl.waitForAllInputsReady(TezProcessorContextImpl.java:138)\r at org.apache.tez.runtime.api.impl.TezProcessorContextImpl.waitForAllInputsReady(TezProcessorContextImpl.java:133)\r at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.init(ReduceRecordProcessor.java:122)\r at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:296)\r at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:280)\r at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)\r at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:86)\r at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:72)\r at java.base/java.security.AccessController.doPrivileged(Native Method)\r at java.base/javax.security.auth.Subject.doAs(Subject.java:423)\r at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899)\r at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:72)\r at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:42)\r at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)\r at org.apache.hadoop.hive.llap.daemon.impl.StatsRecordingThreadPool$WrappedCallable.call(StatsRecordingThreadPool.java:118)\r at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)\r at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\r at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\r at java.base/java.lang.Thread.run(Thread.java:829)\r
> {code}
> however, it still allowed task to commit and created a commit manifest (06:04:27.442Z)
> {code:java}
>  
> <14>1 2025-05-16T06:04:27.442Z query-executor-0-16 query-executor 1 71c82503-aeaa-4ca8-ad1c-c01db6df4f9d [mdc@38374 class="hive.HiveIcebergOutputCommitter" dagId="dag_1746256596566_0014_1634" fragmentId="1746256596566_0014_1634_07_000070_13" level="INFO" queryId="hive_20250516055906_b7f7cff1-fc84-44d2-a7c5-30bd9aa428c5" thread="TezTR-596566_14_1634_7_70_13"] Created Iceberg commitTask manifest file: [s3a://temp/hive_20250516055906_b7f7cff1-fc84-44d2-a7c5-30bd9aa428c5-job_17462565965667_0014/task-70.forCommit]\rFilesForCommit\{dataFiles=[], deleteFiles=[], replacedDataFiles=[], referencedDataFiles=[]}
> {code}
> that is a regression introduced by HIVE-24857



--
This message was sent by Atlassian Jira
(v8.20.10#820010)