You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Sergey Shelukhin (JIRA)" <ji...@apache.org> on 2016/11/22 00:54:58 UTC
[jira] [Created] (HIVE-15254) LLAP: send finishable state with the
task, not (just) as a message
Sergey Shelukhin created HIVE-15254:
---------------------------------------
Summary: LLAP: send finishable state with the task, not (just) as a message
Key: HIVE-15254
URL: https://issues.apache.org/jira/browse/HIVE-15254
Project: Hive
Issue Type: Bug
Reporter: Sergey Shelukhin
{noformat}
2016-11-18 20:20:01,472 . [HISTORY][DAG:dag_1478967587833_2622_1][Event:VERTEX_FINISHED]: vertexName=Map 7,
2016-11-18 20:20:01,500 . [HISTORY][DAG:dag_1478967587833_2622_1][Event:VERTEX_FINISHED]: vertexName=Map 4,
2016-11-18 20:20:01,503 . [HISTORY][DAG:dag_1478967587833_2622_1][Event:VERTEX_FINISHED]: vertexName=Map 5,
2016-11-18 20:20:01,505 . [HISTORY][DAG:dag_1478967587833_2622_1][Event:VERTEX_FINISHED]: vertexName=Map 8,
2016-11-18 20:20:01,530 . [HISTORY][DAG:dag_1478967587833_2622_1][Event:VERTEX_FINISHED]: vertexName=Map 6,
2016-11-18 20:20:01,870 . [HISTORY][DAG:dag_1478967587833_2622_1][Event:VERTEX_FINISHED]: vertexName=Map 3,
...
2016-11-18T20:26:59,410 INFO [Wait-Queue-Scheduler-0 ()] org.apache.hadoop.hive.llap.daemon.impl.TaskExecutorService: Attempting to execute TaskWrapper{task=attempt_1478967587833_2622_1_06_000024_1, inWaitQueue=true, inPreemptionQueue=false, registeredForNotifications=true, canFinish=false, firstAttemptStartTime=1479500401952, dagStartTime=1479500397795, withinDagPriority=64, vertexParallelism= 107, selfAndUpstreamParallelism= 131, selfAndUpstreamComplete= 87}
2016-11-18T20:26:59,411 INFO [Wait-Queue-Scheduler-0 ()] org.apache.hadoop.hive.llap.daemon.impl.TaskExecutorService: attempt_1478967587833_2622_1_06_000024_1 is not finishable. Adding it to pre-emption queue
2016-11-18T20:26:59,415 INFO [Task-Executor-14 (1478967587833_2622_1_06_000024_1)] org.apache.tez.runtime.LogicalIOProcessorRuntimeTask: Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : ... inputSpecList=[
{{ sourceVertexName=a11, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInputLegacy }},
{{ sourceVertexName=Map 5, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.UnorderedKVInput }},
{{ sourceVertexName=Map 6, physicalEdgeCount=3, inputClassName=org.apache.tez.runtime.library.input.UnorderedKVInput }},
{{ sourceVertexName=Map 8, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.UnorderedKVInput }},
{{ sourceVertexName=Map 4, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.UnorderedKVInput }},
{{ sourceVertexName=Map 7, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.UnorderedKVInput }},
{{ sourceVertexName=Map 3, physicalEdgeCount=17, inputClassName=org.apache.tez.runtime.library.input.UnorderedKVInput }}, ],
{noformat}
The task should be finishable but isn't.
That can be a problem when adding to wait queue and then rejecting the least important task (the one being submitted, or an old evicted one) - wait queue comparator takes finishable state into account, so while this task should be able to kick any non-finishable task out of the queue, it might actually get rejected because it "looks" non-finishable at submission time.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)