You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by Mohit Sabharwal <mo...@cloudera.com> on 2014/12/29 21:34:59 UTC
Review Request 29465: HIVE-9222: Fix ordering differences due to Java
8 (Part 4)
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29465/
-----------------------------------------------------------
Review request for hive.
Bugs: HIVE-9222
https://issues.apache.org/jira/browse/HIVE-9222
Repository: hive-git
Description
-------
This patch fixes the following tests:
(1) TestNegativeCliDriver.testNegativeCliDriver: unset_view_property.q and unset_table_property.q
DDLSemanticAnalyzer.analyzeAlterTableProps() gets table properties via getProps() which must be an insert order map.
(2) TestCliDriver.testCliDriver_overridden_confs
VerifyOverriddenConfigsHook emits overridden configs. Changed SessionState.overriddenConfigurations to insert order map.
(3) TestNegativeCliDriver.testNegativeCliDriver_columnstats_partlvl_invalid_values
ColumnStatsSemanticAnalyzer.getPartKeyValuePairsFromAST() gets ((ASTNode) tree.getChild(0) in different order between Java 7 and Java 8. The order is different in HiveParser.statement() itself in ParseDriver.parse() so this difference comes from antlr library.
Generated java version specific output.
(4) TestMinimrCliDriver.testCliDriver_list_bucket_dml_10, TestCliDriver tests: stats_list_bucket.q, list_bucket_dml_12.q and list_bucket_dml_13.q
Looks like these need rebase after HIVE-9206? Not sure what happened here...
(5) TestCliDriver.testCliDriver: mapjoin_hook.q, auto_join_without_localtask.q, auto_join25.q, multiMapJoin2.q
PrintCompletedTasksHook prints completed task list, which depends on the list of tasks added to runnable task list in DriverContext. Some on these tasks may get filtered. We see that different tasks are getting filtered out by the condition resolver in ConditionTask in Java 8 compared to Java 7.
ConditionalTask.execute() calls ConditionalResolverCommonJoin.resolveDriverAlias() via getTasks(), which returns a single task based on task to alias map. The next mapred task in the task list gets filtered out by the resolver in ConditionalTask.resolveTask(). In other words, the the mapred task that shows up first will be kept and the next one will be filtered. Converted task to alias map to an insert order map so order is same with Java 8 and Java 7.
Diffs
-----
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/CommonJoinTaskDispatcher.java 3df1c2652f13dba8f1b0aa720f2649b738c576a1
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/MapJoinResolver.java c37b6fd57335f284587d0ca6ff5f8d479fafa0c5
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/SortMergeJoinTaskDispatcher.java 3fcccb0fdf918627c6e8b046120442f1a3a1869b
ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 537708271cae1deeb2ac3de2dd876559528949d3
ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 1b5864e220fbaa1ac7d8f338844c3fae41d9ecec
ql/src/test/org/apache/hadoop/hive/ql/plan/TestConditionalResolverCommonJoin.java 3af0257477e17b217e01c0285532cf2a3e961caa
ql/src/test/queries/clientnegative/columnstats_partlvl_invalid_values.q 34f91fc8d1de846af0c0b6d1961139f08a34cf20
ql/src/test/results/clientnegative/columnstats_partlvl_invalid_values.q.java1.7.out PRE-CREATION
ql/src/test/results/clientnegative/columnstats_partlvl_invalid_values.q.java1.8.out PRE-CREATION
ql/src/test/results/clientnegative/columnstats_partlvl_invalid_values.q.out 3261f78fab2341208c61d3cd5728f9a9d710ec80
ql/src/test/results/clientnegative/unset_table_property.q.out 8f94c9d28dc7cbf03688e56de21e9f0ba84afa73
ql/src/test/results/clientpositive/auto_join_without_localtask.q.out 8bb92e48a4e26eb72e80460cf9e13b5cbba8755c
ql/src/test/results/clientpositive/list_bucket_dml_10.q.java1.7.out PRE-CREATION
ql/src/test/results/clientpositive/list_bucket_dml_10.q.java1.8.out PRE-CREATION
ql/src/test/results/clientpositive/list_bucket_dml_10.q.out 92c4d4455d3d6b6ef0fc4a5db686c3ec7597454b
ql/src/test/results/clientpositive/list_bucket_dml_12.q.java1.8.out 68856e0fe4450e49726cd01448423f2edbceb47b
ql/src/test/results/clientpositive/multiMapJoin2.q.out eb3bc9c9ff9464a52eb63adc5edaffe18c188abb
ql/src/test/results/clientpositive/stats_list_bucket.q.java1.8.out 80a0f9c8683eb527c05854a38b1f6f03d48dd467
Diff: https://reviews.apache.org/r/29465/diff/
Testing
-------
Tested with Java 8 and Java 7.
Thanks,
Mohit Sabharwal