You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@tez.apache.org by "Rajesh Balamohan (JIRA)" <ji...@apache.org> on 2015/12/14 04:01:46 UTC
[jira] [Comment Edited] (TEZ-2684)
ShuffleVertexManager.parsePartitionStats throws IllegalStateException:
Stats should be initialized
[ https://issues.apache.org/jira/browse/TEZ-2684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15055342#comment-15055342 ]
Rajesh Balamohan edited comment on TEZ-2684 at 12/14/15 3:01 AM:
-----------------------------------------------------------------
Rebasing the patch for branch-0.7. This would fail TestVertexImp.testTez2684 which should get fixed once TEZ-2987 is fixed. (\cc [~bikassaha])
was (Author: rajesh.balamohan):
Rebasing the patch for branch-0.7. This would fail TestVertexImp.testTez2684 which should get fixed once TEZ-2987 is fixed.
> ShuffleVertexManager.parsePartitionStats throws IllegalStateException: Stats should be initialized
> --------------------------------------------------------------------------------------------------
>
> Key: TEZ-2684
> URL: https://issues.apache.org/jira/browse/TEZ-2684
> Project: Apache Tez
> Issue Type: Bug
> Affects Versions: 0.8.0-alpha
> Environment: Hive on Tez
> Reporter: Wei Zheng
> Assignee: Rajesh Balamohan
> Fix For: 0.8.0-alpha, 0.7.1
>
> Attachments: 0803_02.patch, TEZ-2684.1.patch, TEZ-2684.2.patch, TEZ-2684.2.patch.branch-0.7, dynamic_partition_pruning.q, hivelog.tar.gz
>
>
> When I run hive qfile test (attached) using TestMiniTezCliDriver. My WIP patch is also attached for problem reproduction purpose, as well as hive.log.
> Here's the explain and backtrace I got from qfile output:
> {code}
> EXPLAIN select count(*) from srcpart join srcpart_date on (srcpart.ds = srcpart_date.ds) where srcpart_date.`date` = '2008-04-08'
> POSTHOOK: type: QUERY
> STAGE DEPENDENCIES:
> Stage-1 is a root stage
> Stage-0 depends on stages: Stage-1
> STAGE PLANS:
> Stage: Stage-1
> Tez
> Edges:
> Reducer 2 <- Map 1 (SIMPLE_EDGE), Map 4 (SIMPLE_EDGE)
> Reducer 3 <- Reducer 2 (SIMPLE_EDGE)
> DagName: wzheng_20150803161620_55c139de-c26c-467f-b592-7d4333053ac6:38
> Vertices:
> Map 1
> Map Operator Tree:
> TableScan
> alias: srcpart
> filterExpr: ds is not null (type: boolean)
> Statistics: Num rows: 2000 Data size: 21248 Basic stats: COMPLETE Column stats: NONE
> Reduce Output Operator
> key expressions: ds (type: string)
> sort order: +
> Map-reduce partition columns: ds (type: string)
> Statistics: Num rows: 2000 Data size: 21248 Basic stats: COMPLETE Column stats: NONE
> Map 4
> Map Operator Tree:
> TableScan
> alias: srcpart_date
> filterExpr: (ds is not null and (date = '2008-04-08')) (type: boolean)
> Statistics: Num rows: 2 Data size: 42 Basic stats: COMPLETE Column stats: NONE
> Filter Operator
> predicate: (ds is not null and (date = '2008-04-08')) (type: boolean)
> Statistics: Num rows: 1 Data size: 21 Basic stats: COMPLETE Column stats: NONE
> Reduce Output Operator
> key expressions: ds (type: string)
> sort order: +
> Map-reduce partition columns: ds (type: string)
> Statistics: Num rows: 1 Data size: 21 Basic stats: COMPLETE Column stats: NONE
> Select Operator
> expressions: ds (type: string)
> outputColumnNames: _col0
> Statistics: Num rows: 1 Data size: 21 Basic stats: COMPLETE Column stats: NONE
> Group By Operator
> keys: _col0 (type: string)
> mode: hash
> outputColumnNames: _col0
> Statistics: Num rows: 1 Data size: 21 Basic stats: COMPLETE Column stats: NONE
> Dynamic Partitioning Event Operator
> Target Input: srcpart
> Partition key expr: ds
> Statistics: Num rows: 1 Data size: 21 Basic stats: COMPLETE Column stats: NONE
> Target column: ds
> Target Vertex: Map 1
> Reducer 2
> Reduce Operator Tree:
> Merge Join Operator
> condition map:
> Inner Join 0 to 1
> keys:
> 0 ds (type: string)
> 1 ds (type: string)
> Statistics: Num rows: 2200 Data size: 23372 Basic stats: COMPLETE Column stats: NONE
> Group By Operator
> aggregations: count()
> mode: hash
> outputColumnNames: _col0
> Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
> Reduce Output Operator
> sort order:
> Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
> value expressions: _col0 (type: bigint)
> Reducer 3
> Reduce Operator Tree:
> Group By Operator
> aggregations: count(VALUE._col0)
> mode: mergepartial
> outputColumnNames: _col0
> Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
> File Output Operator
> compressed: false
> Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
> table:
> input format: org.apache.hadoop.mapred.TextInputFormat
> output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
> serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
> Stage: Stage-0
> Fetch Operator
> limit: -1
> Processor Tree:
> ListSink
> PREHOOK: query: select count(*) from srcpart join srcpart_date on (srcpart.ds = srcpart_date.ds) where srcpart_date.`date` = '2008-04-08'
> PREHOOK: type: QUERY
> PREHOOK: Input: default@srcpart
> PREHOOK: Input: default@srcpart@ds=2008-04-08/hr=11
> PREHOOK: Input: default@srcpart@ds=2008-04-08/hr=12
> PREHOOK: Input: default@srcpart@ds=2008-04-09/hr=11
> PREHOOK: Input: default@srcpart@ds=2008-04-09/hr=12
> PREHOOK: Input: default@srcpart_date
> PREHOOK: Output: file:/Users/wzheng/bf/hive/itests/qtest/target/tmp/localscratchdir/93b335b5-3ced-4f4d-abdd-2fd5defd11e4/hive_2015-08-03_16-16-21_046_5066458626645110592-1/-mr-10001
> Status: Failed
> Vertex failed, vertexName=Reducer 2, vertexId=vertex_1438643776809_0001_8_02, diagnostics=[Vertex vertex_1438643776809_0001_8_02 [Reducer 2] killed/failed due to:AM_USERCODE_FAILURE, Exception in VertexManager, vertex:vertex_1438643776809_0001_8_02 [Reducer 2], java.lang.IllegalStateException: Stats should be initialized
> at com.google.common.base.Preconditions.checkState(Preconditions.java:149)
> at org.apache.tez.dag.library.vertexmanager.ShuffleVertexManager.parsePartitionStats(ShuffleVertexManager.java:535)
> at org.apache.tez.dag.library.vertexmanager.ShuffleVertexManager.onVertexManagerEventReceived(ShuffleVertexManager.java:575)
> at org.apache.tez.dag.app.dag.impl.VertexManager$VertexManagerEventReceived.invoke(VertexManager.java:602)
> at org.apache.tez.dag.app.dag.impl.VertexManager$VertexManagerEvent$1.run(VertexManager.java:643)
> at org.apache.tez.dag.app.dag.impl.VertexManager$VertexManagerEvent$1.run(VertexManager.java:638)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
> at org.apache.tez.dag.app.dag.impl.VertexManager$VertexManagerEvent.call(VertexManager.java:638)
> at org.apache.tez.dag.app.dag.impl.VertexManager$VertexManagerEvent.call(VertexManager.java:627)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> ]
> Vertex killed, vertexName=Reducer 3, vertexId=vertex_1438643776809_0001_8_03, diagnostics=[Vertex received Kill in INITED state., Vertex vertex_1438643776809_0001_8_03 [Reducer 3] killed/failed due to:null]
> Vertex killed, vertexName=Map 1, vertexId=vertex_1438643776809_0001_8_01, diagnostics=[Vertex received Kill in INITED state., Vertex vertex_1438643776809_0001_8_01 [Map 1] killed/failed due to:null]
> DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:2
> FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed, vertexName=Reducer 2, vertexId=vertex_1438643776809_0001_8_02, diagnostics=[Vertex vertex_1438643776809_0001_8_02 [Reducer 2] killed/failed due to:AM_USERCODE_FAILURE, Exception in VertexManager, vertex:vertex_1438643776809_0001_8_02 [Reducer 2], java.lang.IllegalStateException: Stats should be initialized
> at com.google.common.base.Preconditions.checkState(Preconditions.java:149)
> at org.apache.tez.dag.library.vertexmanager.ShuffleVertexManager.parsePartitionStats(ShuffleVertexManager.java:535)
> at org.apache.tez.dag.library.vertexmanager.ShuffleVertexManager.onVertexManagerEventReceived(ShuffleVertexManager.java:575)
> at org.apache.tez.dag.app.dag.impl.VertexManager$VertexManagerEventReceived.invoke(VertexManager.java:602)
> at org.apache.tez.dag.app.dag.impl.VertexManager$VertexManagerEvent$1.run(VertexManager.java:643)
> at org.apache.tez.dag.app.dag.impl.VertexManager$VertexManagerEvent$1.run(VertexManager.java:638)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
> at org.apache.tez.dag.app.dag.impl.VertexManager$VertexManagerEvent.call(VertexManager.java:638)
> at org.apache.tez.dag.app.dag.impl.VertexManager$VertexManagerEvent.call(VertexManager.java:627)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> ]Vertex killed, vertexName=Reducer 3, vertexId=vertex_1438643776809_0001_8_03, diagnostics=[Vertex received Kill in INITED state., Vertex vertex_1438643776809_0001_8_03 [Reducer 3] killed/failed due to:null]Vertex killed, vertexName=Map 1, vertexId=vertex_1438643776809_0001_8_01, diagnostics=[Vertex received Kill in INITED state., Vertex vertex_1438643776809_0001_8_01 [Map 1] killed/failed due to:null]DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:2
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)