[jira] [Commented] (HIVE-20839) "Cannot find field" error during dynamically partitioned hash join

["Jason Dere (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HIVE-20839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16669100#comment-16669100 ] 

Jason Dere commented on HIVE-20839:
-----------------------------------

Issue seems to be that there are several column expressions that end up being converted to constant value 'SSS' during constant folding. Some of these constant columns end up being pruned from the query, and others might be join keys which are required in the query. The problem was that ExprNodeDescUtils.resolveJoinKeysAsRSColumns() (which is used to setup the join key expressions in the case of dynamically partitioned hashjoin) was trying to resolve the join key expression for the 'SSS' column and picked the wrong column - it was also constant 'SSS', but it was actually a different column that ended up getting pruned.

The ExprNodeConstDesc actually has a getFoldedFromCol() field that shows what column this constant expression has been folded from, I think this can be used to help resolve to the correct expression.

> "Cannot find field" error during dynamically partitioned hash join
> ------------------------------------------------------------------
>
>                 Key: HIVE-20839
>                 URL: https://issues.apache.org/jira/browse/HIVE-20839
>             Project: Hive
>          Issue Type: Bug
>          Components: Query Processor
>            Reporter: Jason Dere
>            Assignee: Jason Dere
>            Priority: Major
>
> Occurs in some cases in the non-CBO optimized queries, either if CBO is disabled or has failed due to error.
> {noformat}
> 2018-10-11T04:40:22,724 ERROR [TezTR-85144_8944_1085_28_996_2 (1539092085144_8944_1085_28_000996_2)] tez.ReduceRecordProcessor: Hit error while closing operators - failing tree
> 2018-10-11T04:40:22,724 ERROR [TezTR-85144_8944_1085_28_996_2 (1539092085144_8944_1085_28_000996_2)] tez.TezProcessor: java.lang.RuntimeException: cannot find field _col304 from [0:_col0, 1:_col1, 2:_col2, 3:_col3, 4:_col4, 5:_col5, 6:_col6, 7:_col7, 8:_col8, 9:_col9, 10:_col10, 11:_col11, 12:_col12, 13:_col13, 14:_col15, 15:_col16, 16:_col17, 17:_col18, 18:_col19, 19:_col20, 20:_col21, 21:_col22, 22:_col23, 23:_col24, 24:_col25, 25:_col26, 26:_col27, 27:_col28, 28:_col29, 29:_col30, 30:_col31, 31:_col32, 32:_col33, 33:_col34, 34:_col35, 35:_col36, 36:_col37, 37:_col38, 38:_col39, 39:_col40, 40:_col41, 41:_col42, 42:_col43, 43:_col44, 44:_col45, 45:_col46, 46:_col47, 47:_col48, 48:_col49, 49:_col50, 50:_col51, 51:_col52, 52:_col53, 53:_col54, 54:_col55, 55:_col56, 56:_col57, 57:_col58, 58:_col59, 59:_col60, 60:_col61, 61:_col62, 62:_col63, 63:_col64, 64:_col65, 65:_col66, 66:_col67, 67:_col68, 68:_col70, 69:_col72, 70:_col73, 71:_col74, 72:_col75, 73:_col76, 74:_col77, 75:_col78, 76:_col79, 77:_col80, 78:_col81, 79:_col82, 80:_col83, 81:_col84, 82:_col85, 83:_col86, 84:_col87, 85:_col88, 86:_col89, 87:_col90, 88:_col91, 89:_col92, 90:_col93, 91:_col94, 92:_col95, 93:_col96, 94:_col97, 95:_col98, 96:_col99, 97:_col100, 98:_col101, 99:_col102, 100:_col103, 101:_col104, 102:_col105, 103:_col106, 104:_col107, 105:_col108, 106:_col109, 107:_col110, 108:_col111, 109:_col112, 110:_col113, 111:_col114, 112:_col115, 113:_col116, 114:_col117, 115:_col118, 116:_col119, 117:_col120, 118:_col121, 119:_col122, 120:_col123, 121:_col124, 122:_col125, 123:_col126, 124:_col127, 125:_col128, 126:_col129, 127:_col130, 128:_col131, 129:_col132, 130:_col133, 131:_col134, 132:_col135, 133:_col136, 134:_col137, 135:_col138, 136:_col139, 137:_col140, 138:_col141, 139:_col142, 140:_col143, 141:_col144, 142:_col145, 143:_col146, 144:_col147, 145:_col148, 146:_col149, 147:_col150, 148:_col151, 149:_col152, 150:_col153, 151:_col154, 152:_col155, 153:_col156, 154:_col157, 155:_col158, 156:_col159, 157:_col160, 158:_col161, 159:_col162, 160:_col163, 161:_col164, 162:_col165, 163:_col166, 164:_col167, 165:_col168, 166:_col169, 167:_col170, 168:_col171, 169:_col318]
> at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.getStandardStructFieldRef(ObjectInspectorUtils.java:485)
> at org.apache.hadoop.hive.serde2.objectinspector.StandardStructObjectInspector.getStructFieldRef(StandardStructObjectInspector.java:153)
> at org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator.initialize(ExprNodeColumnEvaluator.java:80)
> at org.apache.hadoop.hive.ql.exec.JoinUtil.getObjectInspectorsFromEvaluators(JoinUtil.java:91)
> at org.apache.hadoop.hive.ql.exec.AbstractMapJoinOperator.initializeOp(AbstractMapJoinOperator.java:74)
> at org.apache.hadoop.hive.ql.exec.MapJoinOperator.initializeOp(MapJoinOperator.java:144)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:374)
> at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.init(ReduceRecordProcessor.java:195)
> at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:188)
> at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:172)
> at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:370)
> at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869)
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)