You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hive.apache.org by za...@apache.org on 2025/04/30 11:05:56 UTC
(hive) branch master updated: HIVE-27102: Upgrade Calcite version from 1.25.0 to 1.33.0 (#5196)
This is an automated email from the ASF dual-hosted git repository.
zabetak pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hive.git
The following commit(s) were added to refs/heads/master by this push:
new 208488089df HIVE-27102: Upgrade Calcite version from 1.25.0 to 1.33.0 (#5196)
208488089df is described below
commit 208488089df71e8a0dafd95b2812e5e4864baed9
Author: Soumyakanti Das <so...@gmail.com>
AuthorDate: Wed Apr 30 04:05:48 2025 -0700
HIVE-27102: Upgrade Calcite version from 1.25.0 to 1.33.0 (#5196)
1. Upgrade Calcite version from 1.25.0 to 1.33.0 and update code to address breaking changes and avoid compilation failures:
* Add HiveJdbcImplementor#visit(JdbcTableScan) to address breaking change from CALCITE-4640.
* Add HiveRuleConfig to address the removal of RelRule.Config.EMPTY by CALCITE-4839. This allows to instantiate RelRule subclasses without relying on the immutables annotation processor.
* Adapt RelNode constructors and pass empty hints to address breaking change from in CALCITE-4640.
* Simplify HiveDruidRules by using the config from respective rules in Calcite.
* Adapt HiveRelFactories due to changes in ProjectFactory API (CALCITE-4199, CALCITE-5127).
2. Upgrade Avatica version from 1.12.0 to 1.23.0 (required by Calcite).
3. Add explicit dependency to org.immutables:value-annotations to avoid compilation failures due to missing Value annotation.
Since we don't want to propagate the dependency to dependent projects we declared it at provided scope. The problem appears cause annotation pre-processing is enabled.
4. Add explicit dependency to org.locationtech.jts:jts-core and declare it at runtime scope.
The jts-core is a transitive dependency used by calcite-core. In order for CBO to function properly the jar must be present in the classpath. If the jar is missing, Hiveserver2 will fail during startup with ClassNotFoundException.
Normally we wouldn't need to explicitly declare transitive dependencies, but we are forced to do so cause calcite is partially shaded in hive-exec module. We are already doing the same for various other calcite deps such as janino, avatica, etc.
5. Register HiveIn as SqlKind.OTHER_FUNCTION and distinguish it from SqlKind.IN which is now reserved for usage inside RexSubQuery
6. Replace checks for SqlKind.IN in RexCall, which can no longer appear, with checks on HiveIn operator directly.
7. Implement new rules and converters to transform/expand the internal SEARCH operator to traditional conjunctions/disjunctions of standard comparisons operators, IN, and BETWEEN.
8. Copy and use Lopt classes from Calcite with the fix for (CALCITE-6737) to avoid changes in join order when queries contain self-joins.
9. Move rollup aggregation logic from HiveRelBuilder to the appropriate SqlAggFunction subclass and remove redundant overrides in HiveMaterializedViewRule.
CALCITE-4342 made rollup strategy a first class citizen of the SqlAggFunction interface so each function can return its own rollup if available.
It's no longer necessary to pass from the RelBuilder and basically whenever we need to obtain a rollup we have to ask directly the SqlAggFunction.
As part of CALCITE-4342, the MaterializedViewRule#getRollup method was deprecated so this refactoring is necessary for the Hive MV rules to function properly.
10. Make HivePreFilteringRule a reduction rule to address fixpoint bug (StackOverflowError) triggered by changes in RelMetadataQuery#getPulledUpPredicates (CALCITE-5036).
11. Add HiveTypeFactory to avoid redundant nullability CAST to ARRAY that in some cases (lateral_view_outer.q) leads to wrong results. The redundant CAST is a side-effect from fixing CALCITE-4603.
12. Handle ROW type literals in RexNodeConverter#transformInToOrOperands
After the upgrade a ROW RexCall with constants is reduced/folded to a RexLiteral of ROW types so we must adapt the IN transformation logic otherwise the ROW specific code will not kick in.
Although this is merely an optimization it has impact on correctness (due to existing Hive bugs) so it's necessary to keep it working.
13. Enhance FilterSelectivityEstimator to provide better estimates for NOT
14. Use RelBuilder instead of RexBuilder in TestHivePushdownSnapshotFilterRule.
Code is more readable and avoids creating invalid expressions with wrong types as it was the case before.
15. Disable RelBuilder simplifications in TestHivePointLookupOptimizerRule to avoid the introduction of SEARCH in the input plans. The rule is meant to handle disjunctions (OR) so we want to keep the input plans intact in order to properly test the rule. If the input plan is simplified to SEARCH or something else then test coverage will change that is not desired.
The plan and behavior changes that occur as part of the upgrade can be summarized into the following categories. An additional (R*) classifier, is used to denote subtle regressions that will be fixed by follow-up patches. Each change is associated with some represantive .q.out files that illustrate the overall impact on query plans.
Category I: Changes that are mostly cosmetic and do not affect the performance or behavior of the queries.
1. Changes in the output of EXPLAIN EXTENDED (OPTIMIZED SQL)
* Less parentheses [auto_join_reordering_values.q.out]
* Less spaces when formatting IN, ARRAY, etc. [alter_partition_coltype.q.out, autoColumnStats_5a.q.out]
* VALUES clause with one entry becomes SELECT with literals [acid_nullscan.q.out]
* New OPTIMIZED SQL entries since various bugs in RelToSqlConverter were fixed [pcs.q.out]
* Explicit CROSS JOIN clause instead of comma separated sub-queries [tez_fixed_bucket_pruning.q.out]
2. Different formatting (toString) for DOUBLE literals [rule_exclusion_config.q.out]
3. Extra "fields" attributes in the output of EXPLAIN FORMATTED (R*) [concat_op.q.out]
4. IN clause literals sorted alphabetically [dynamic_partition_pruning_2.q.out]
5. Different aliases for expressions inside projects (e.g., lots of EXPR$ in CBO plans) [cardinality_preserving_join_opt.q.out]
6. Result changes due to missing ORDER BY in queries [partition_wise_fileformat2.q.out]
Category II: Changes that can slightly affect the performance of queries increasing/decreasing CPU cycles.
1. Predicate order changes in filters [auto_join5.q.out]
2. Redundant IS NOT NULL predicates (R*) [explainuser_4.q.out, pointlookup5.q.out]
3. Redundant CAST around COALESCE (R*) [cbo_aggregate_reduce_functions_rule.q.out]
3. Pull-up constants from nullsafe predicates [is_distinct_from.q.out, join_nullsafe.q.out, orc_predicate_pushdown.q.out]
4. Residual filter predicates in joins (R*) [ppd_join.q.out]
Category III: Changes that can lead to significant changes in query performance.
1. Expression simplifications in project/filter/join
* IN, BETWEEN, and comparison operators (<,>,>=,<=,==,<>) are simplified together by exploiting the SEARCH operator [bucketsortoptimize_insert_7.q.out]
* x NOT BETWEEN 10 AND 20 is transformed to x < 10 OR x > 20 [udf_between.q.out]
* x <> 10 is transformed to x > 10 OR x < 10 (R*) [ppd_join3.q.out]
* Simplifications with arithmetic operators [materialized_view_rewrite_8.q.out, tez_vector_dynpart_hashjoin_2.q.out]
2. Disjunctive predicate pushdown for joins (R*) [cbo_join_transitive_pred_loop_1.q.out, correlationoptimizer8.q.out]
3. LEFT OUTER to INNER joins due to CALCITE-5247 [subquery_ANY.q.out]
4. Better orders for LEFT/RIGHT/ANTI joins due to CALCITE-4208 [auto_join30.q.out, subquery_join_rewrite.q.out]
5. Unconditional OR to IN tranformations [in_typecheck_char.q.out, pointlookup2.q.out]
6. Extra partition pruning due to simplifications [select_unquote_or.q.out]
Co-authored-by: Stamatis Zampetakis <za...@gmail.com>
Co-authored-by: Krisztian Kasa <ka...@gmail.com>
---
.../org/apache/hadoop/hive/ql/log/PerfLogger.java | 1 +
.../src/test/results/positive/hbase_queries.q.out | 2 +-
...elete_iceberg_copy_on_write_unpartitioned.q.out | 8 +-
.../results/positive/delete_iceberg_mixed.q.out | 4 +-
.../positive/llap/iceberg_bucket_map_join_7.q.out | 80 +-
.../positive/llap/llap_iceberg_read_orc.q.out | 4 +-
...merge_iceberg_copy_on_write_unpartitioned.q.out | 4 +-
.../test/results/positive/mv_iceberg_orc4.q.out | 2 +-
.../update_iceberg_copy_on_write_partitioned.q.out | 14 +-
...pdate_iceberg_copy_on_write_unpartitioned.q.out | 12 +-
.../insert_into_dynamic_partitions.q.out | 2 +-
.../results/clientpositive/insert_into_table.q.out | 2 +-
.../insert_overwrite_dynamic_partitions.q.out | 2 +-
.../clientpositive/insert_overwrite_table.q.out | 2 +-
.../txn/compactor/TestMaterializedViewRebuild.java | 2 +-
packaging/pom.xml | 4 +
pom.xml | 18 +-
ql/pom.xml | 8 +
.../AlterMaterializedViewRebuildAnalyzer.java | 7 +-
.../hadoop/hive/ql/optimizer/calcite/Bug.java | 10 +
.../hive/ql/optimizer/calcite/HiveCalciteUtil.java | 4 +-
.../hive/ql/optimizer/calcite/HiveRelBuilder.java | 24 -
.../ql/optimizer/calcite/HiveRelFactories.java | 7 +-
.../ql/optimizer/calcite/HiveRexExecutorImpl.java | 2 +-
.../hive/ql/optimizer/calcite/HiveTypeFactory.java | 54 +
.../hive/ql/optimizer/calcite/RangeConverter.java | 129 ++
.../ql/optimizer/calcite/SearchTransformer.java | 139 ++
.../calcite/functions/HiveMergeableAggregate.java | 3 +-
.../calcite/functions/HiveSqlCountAggFunction.java | 8 +-
.../functions/HiveSqlMinMaxAggFunction.java | 5 +
.../calcite/functions/HiveSqlSumAggFunction.java | 5 +
.../HiveSqlSumEmptyIsZeroAggFunction.java | 5 +
.../optimizer/calcite/reloperators/HiveFilter.java | 2 -
.../ql/optimizer/calcite/reloperators/HiveIn.java | 10 +-
.../reloperators/jdbc/JdbcHiveTableScan.java | 5 +-
.../rules/CommonRelSubExprRegisterRule.java | 11 +-
.../ql/optimizer/calcite/rules/CteRuleConfig.java | 36 +-
.../rules/HiveAggregateJoinTransposeRule.java | 3 +-
.../calcite/rules/HiveAggregateSplitRule.java | 4 +-
.../ql/optimizer/calcite/rules/HiveDruidRules.java | 125 +-
.../calcite/rules/HiveFilterJoinRule.java | 25 +-
.../calcite/rules/HiveFilterSortPredicates.java | 7 +-
.../calcite/rules/HiveInBetweenExpandRule.java | 30 +-
.../optimizer/calcite/rules/HiveLoptJoinTree.java | 223 +++
.../optimizer/calcite/rules/HiveLoptMultiJoin.java | 871 ++++++++
.../calcite/rules/HiveLoptOptimizeJoinRule.java | 2082 ++++++++++++++++++++
.../calcite/rules/HiveLoptSemiJoinOptimizer.java | 861 ++++++++
.../rules/HivePointLookupOptimizerRule.java | 4 +-
.../calcite/rules/HivePreFilteringRule.java | 27 +-
.../calcite/rules/HiveReduceExpressionsRule.java | 12 +-
.../rules/HiveReduceExpressionsWithStatsRule.java | 2 +-
.../calcite/rules/HiveRemoveEmptySingleRules.java | 86 +-
.../calcite/rules/HiveRexShuttleTransformRule.java | 57 +
.../ql/optimizer/calcite/rules/HiveRuleConfig.java | 66 +
.../optimizer/calcite/rules/HiveSearchRules.java | 50 +
.../calcite/rules/jdbc/HiveJdbcImplementor.java | 5 +
.../rules/jdbc/JDBCExpandExpressionsRule.java | 10 +-
.../rules/jdbc/JDBCExtractJoinFilterRule.java | 13 +-
.../calcite/rules/jdbc/JDBCFilterJoinRule.java | 7 +-
.../calcite/rules/jdbc/JDBCRexCallValidator.java | 4 +
.../HiveAugmentSnapshotMaterializationRule.java | 32 +-
.../rules/views/HiveMaterializedViewRule.java | 11 -
.../views/HivePushdownSnapshotFilterRule.java | 18 +-
.../rules/views/HiveRowIsDeletedPropagator.java | 15 +-
.../calcite/stats/FilterSelectivityEstimator.java | 69 +-
.../calcite/translator/ExprNodeConverter.java | 12 +-
.../calcite/translator/RexNodeConverter.java | 22 +-
.../calcite/translator/SqlFunctionConverter.java | 1 -
.../hadoop/hive/ql/parse/CalcitePlanner.java | 29 +-
.../hive/ql/parse/type/HiveFunctionHelper.java | 3 +-
.../hive/ql/parse/type/RexNodeExprFactory.java | 6 +-
.../hive/ql/io/TestSymlinkTextInputFormat.java | 2 +-
.../rules/TestHivePointLookupOptimizerRule.java | 23 +-
.../views/TestHivePushdownSnapshotFilterRule.java | 9 +-
.../stats/TestFilterSelectivityEstimator.java | 29 +-
.../hive/ql/plan/mapping/TestCounterMapping.java | 2 +-
.../encryption_join_unencrypted_tbl.q.out | 6 +-
.../clientpositive/llap/acid_no_buckets.q.out | 12 +-
.../clientpositive/llap/acid_nullscan.q.out | 2 +-
.../llap/alter_partition_coltype.q.out | 2 +-
.../clientpositive/llap/ambiguitycheck.q.out | 2 +-
.../llap/annotate_stats_filter.q.out | 8 +-
.../clientpositive/llap/autoColumnStats_5a.q.out | 4 +-
.../results/clientpositive/llap/auto_join21.q.out | 4 +-
.../results/clientpositive/llap/auto_join28.q.out | 4 +-
.../results/clientpositive/llap/auto_join29.q.out | 12 +-
.../results/clientpositive/llap/auto_join30.q.out | 42 +-
.../results/clientpositive/llap/auto_join5.q.out | 4 +-
.../llap/auto_join_reordering_values.q.out | 2 +-
.../clientpositive/llap/bucket_map_join_tez2.q.out | 2 +-
.../clientpositive/llap/bucket_map_join_tez3.q.out | 32 +-
.../clientpositive/llap/bucketpruning1.q.out | 44 +-
.../llap/bucketsortoptimize_insert_7.q.out | 24 +-
.../llap/cardinality_preserving_join_opt.q.out | 14 +-
.../llap/cbo_aggregate_reduce_functions_rule.q.out | 2 +-
.../llap/cbo_join_transitive_pred_loop_1.q.out | 2 +-
.../llap/cbo_join_transitive_pred_loop_3.q.out | 2 +-
.../llap/cbo_join_transitive_pred_loop_4.q.out | 2 +-
.../llap/cbo_ppd_non_deterministic.q.out | 4 +-
.../llap/cbo_proj_join_transpose.q.out | 2 +-
.../llap/cbo_rp_outer_join_ppr.q.out | 8 +-
.../clientpositive/llap/cbo_rp_simple_select.q.out | 4 +-
.../clientpositive/llap/cbo_simple_select.q.out | 4 +-
.../results/clientpositive/llap/concat_op.q.out | 6 +-
.../clientpositive/llap/constant_prop_3.q.out | 4 +-
.../llap/constraints_explain_ddl.q.out | 26 +-
.../llap/constraints_optimization.q.out | 32 +-
.../llap/correlationoptimizer10.q.out | 32 +-
.../llap/correlationoptimizer8.q.out | 48 +-
.../llap/correlationoptimizer9.q.out | 8 +-
ql/src/test/results/clientpositive/llap/ctas.q.out | 2 +-
.../clientpositive/llap/cte_cbo_plan_json.q.out | 2 +-
.../llap/dynamic_partition_pruning_2.q.out | 12 +-
.../clientpositive/llap/estimate_pkfk_push.q.out | 4 +-
.../llap/excluded_rule_explain.q.out | 8 +-
.../clientpositive/llap/explainuser_1.q.out | 4 +-
.../clientpositive/llap/explainuser_4.q.out | 56 +-
.../clientpositive/llap/external_jdbc_table2.q.out | 8 +-
.../clientpositive/llap/external_jdbc_table4.q.out | 6 +-
.../clientpositive/llap/filter_cond_pushdown.q.out | 30 +-
.../llap/filter_join_breaktask.q.out | 2 +-
.../clientpositive/llap/filter_numeric.q.out | 4 +-
.../clientpositive/llap/folder_predicate.q.out | 4 +-
.../clientpositive/llap/fouter_join_ppr.q.out | 32 +-
.../clientpositive/llap/groupby_position.q.out | 4 +-
.../llap/groupby_rollup_empty2.q.out | 4 +-
.../clientpositive/llap/in_typecheck_char.q.out | 6 +-
.../test/results/clientpositive/llap/input4.q.out | 2 +-
.../results/clientpositive/llap/input_part1.q.out | 2 +-
.../clientpositive/llap/intersect_all_rj.q.out | 4 +-
.../clientpositive/llap/is_distinct_from.q.out | 4 +-
.../test/results/clientpositive/llap/join21.q.out | 4 +-
.../test/results/clientpositive/llap/join26.q.out | 2 +-
.../test/results/clientpositive/llap/join32.q.out | 2 +-
.../clientpositive/llap/join32_lessSize.q.out | 6 +-
.../test/results/clientpositive/llap/join33.q.out | 2 +-
.../test/results/clientpositive/llap/join34.q.out | 40 +-
.../test/results/clientpositive/llap/join35.q.out | 38 +-
.../test/results/clientpositive/llap/join5.q.out | 4 +-
.../test/results/clientpositive/llap/join9.q.out | 2 +-
.../clientpositive/llap/join_filters_overlap.q.out | 36 +-
.../llap/join_is_not_distinct_from.q.out | 14 +-
.../clientpositive/llap/join_nullsafe.q.out | 14 +-
.../clientpositive/llap/list_bucket_dml_1.q.out | 2 +-
.../clientpositive/llap/list_bucket_dml_11.q.out | 2 +-
.../clientpositive/llap/list_bucket_dml_12.q.out | 4 +-
.../clientpositive/llap/list_bucket_dml_13.q.out | 2 +-
.../clientpositive/llap/list_bucket_dml_2.q.out | 2 +-
.../clientpositive/llap/list_bucket_dml_3.q.out | 2 +-
.../clientpositive/llap/list_bucket_dml_4.q.out | 2 +-
.../clientpositive/llap/list_bucket_dml_5.q.out | 2 +-
.../clientpositive/llap/list_bucket_dml_9.q.out | 2 +-
.../llap/list_bucket_query_multiskew_1.q.out | 8 +-
.../llap/list_bucket_query_multiskew_2.q.out | 6 +-
.../llap/list_bucket_query_multiskew_3.q.out | 6 +-
.../clientpositive/llap/louter_join_ppr.q.out | 32 +-
.../results/clientpositive/llap/masking_1.q.out | 20 +-
.../results/clientpositive/llap/masking_12.q.out | 8 +-
.../results/clientpositive/llap/masking_13.q.out | 4 +-
.../clientpositive/llap/masking_1_newdb.q.out | 4 +-
.../results/clientpositive/llap/masking_2.q.out | 16 +-
.../results/clientpositive/llap/masking_3.q.out | 4 +-
.../results/clientpositive/llap/masking_6.q.out | 8 +-
.../results/clientpositive/llap/masking_7.q.out | 8 +-
.../llap/materialized_view_create.q.out | 2 +-
.../llap/materialized_view_partitioned_2.q.out | 2 +-
.../llap/materialized_view_rewrite_7.q.out | 14 +-
.../llap/materialized_view_rewrite_8.q.out | 2 +-
.../llap/materialized_view_rewrite_by_text_9.q.out | 2 +-
.../llap/materialized_view_rewrite_ssb.q.out | 4 +-
.../llap/materialized_view_rewrite_ssb_2.q.out | 4 +-
...terialized_view_rewrite_ssb_grouping_sets.q.out | 16 +-
.../clientpositive/llap/multi_in_clause.q.out | 4 +-
.../llap/murmur_hash_migration2.q.out | 2 +-
.../results/clientpositive/llap/notInTest.q.out | 2 +-
.../llap/orc_predicate_pushdown.q.out | 16 +-
.../clientpositive/llap/outer_join_ppr.q.out | 16 +-
.../llap/parquet_predicate_pushdown.q.out | 16 +-
.../llap/parquet_vectorization_0.q.out | 8 +-
.../llap/partition_wise_fileformat2.q.out | 100 +-
ql/src/test/results/clientpositive/llap/pcr.q.out | 29 +-
ql/src/test/results/clientpositive/llap/pcs.q.out | 31 +-
.../results/clientpositive/llap/plan_json.q.out | 2 +-
.../results/clientpositive/llap/pointlookup2.q.out | 76 +-
.../results/clientpositive/llap/pointlookup3.q.out | 69 +-
.../results/clientpositive/llap/pointlookup4.q.out | 4 +-
.../results/clientpositive/llap/pointlookup5.q.out | 8 +-
.../results/clientpositive/llap/pointlookup6.q.out | 4 +-
.../results/clientpositive/llap/ppd_gby_join.q.out | 122 +-
.../results/clientpositive/llap/ppd_join.q.out | 58 +-
.../results/clientpositive/llap/ppd_join2.q.out | 60 +-
.../results/clientpositive/llap/ppd_join3.q.out | 76 +-
.../clientpositive/llap/ppd_outer_join1.q.out | 8 +-
.../clientpositive/llap/ppd_outer_join2.q.out | 8 +-
.../clientpositive/llap/ppd_outer_join3.q.out | 8 +-
.../clientpositive/llap/ppd_outer_join4.q.out | 16 +-
.../llap/rand_partitionpruner3.q.out | 4 +-
.../clientpositive/llap/rcfile_null_value.q.out | 4 +-
.../clientpositive/llap/remove_exprs_stats.q.out | 4 +-
.../llap/retry_failure_stat_changes.q.out | 6 +-
.../clientpositive/llap/router_join_ppr.q.out | 32 +-
.../llap/rule_exclusion_config.q.out | 12 +-
.../clientpositive/llap/runtime_stats_hs2.q.out | 4 +-
.../clientpositive/llap/scratch_col_issue.q.out | 4 +-
.../clientpositive/llap/select_unquote_or.q.out | 12 +-
.../results/clientpositive/llap/sharedwork.q.out | 20 +-
.../clientpositive/llap/stats_histogram.q.out | 10 +-
.../clientpositive/llap/stats_histogram_null.q.out | 4 +-
.../results/clientpositive/llap/subquery_ALL.q.out | 36 +-
.../results/clientpositive/llap/subquery_ANY.q.out | 185 +-
.../clientpositive/llap/subquery_corr_join.q.out | 2 +-
.../results/clientpositive/llap/subquery_in.q.out | 4 +-
.../clientpositive/llap/subquery_in_having.q.out | 4 +-
.../llap/subquery_join_rewrite.q.out | 23 +-
.../clientpositive/llap/subquery_multi.q.out | 10 +-
.../clientpositive/llap/subquery_notexists.q.out | 2 +-
.../clientpositive/llap/subquery_notin.q.out | 200 +-
.../llap/subquery_notin_having.q.out | 4 +-
.../clientpositive/llap/subquery_select.q.out | 28 +-
.../llap/subquery_unqualcolumnrefs.q.out | 2 +-
.../clientpositive/llap/subquery_views.q.out | 4 +-
.../llap/subquery_with_corr_and_mv.q.out | 2 +-
.../llap/temp_table_alter_partition_coltype.q.out | 2 +-
.../llap/tez_dynpart_hashjoin_1.q.out | 47 +-
.../llap/tez_dynpart_hashjoin_2.q.out | 12 -
.../llap/tez_fixed_bucket_pruning.q.out | 25 +-
.../clientpositive/llap/tez_smb_reduce_side.q.out | 4 +-
.../clientpositive/llap/tez_union_group_by.q.out | 2 +-
.../llap/tez_vector_dynpart_hashjoin_1.q.out | 42 +-
.../llap/tez_vector_dynpart_hashjoin_2.q.out | 12 -
.../clientpositive/llap/topnkey_inner_join2.q.out | 4 +-
.../results/clientpositive/llap/udf_between.q.out | 8 +-
.../results/clientpositive/llap/udtf_explode.q.out | 9 +-
.../test/results/clientpositive/llap/union22.q.out | 4 +-
.../llap/vector_between_columns.q.out | 6 +-
.../clientpositive/llap/vector_between_in.q.out | 66 +-
.../clientpositive/llap/vector_case_when_2.q.out | 16 +-
.../clientpositive/llap/vector_coalesce_3.q.out | 4 +-
.../llap/vector_decimal_expressions.q.out | 12 +-
.../clientpositive/llap/vector_interval_2.q.out | 12 +-
.../clientpositive/llap/vector_join30.q.out | 85 +-
.../llap/vector_mr_diff_schema_alias.q.out | 4 +-
.../llap/vector_windowing_multipartitioning.q.out | 30 +-
.../llap/vector_windowing_navfn.q.out | 6 +-
.../clientpositive/llap/vectorization_0.q.out | 8 +-
.../clientpositive/llap/vectorized_case.q.out | 12 +-
.../llap/vectorized_string_funcs.q.out | 10 +-
.../perf/tpcds30tb/tez/cbo_ext_query1.q.out | 6 +-
.../perf/tpcds30tb/tez/cbo_query10.q.out | 2 +-
.../perf/tpcds30tb/tez/cbo_query11.q.out | 16 +-
.../perf/tpcds30tb/tez/cbo_query12.q.out | 2 +-
.../perf/tpcds30tb/tez/cbo_query13.q.out | 16 +-
.../perf/tpcds30tb/tez/cbo_query15.q.out | 8 +-
.../perf/tpcds30tb/tez/cbo_query16.q.out | 2 +-
.../perf/tpcds30tb/tez/cbo_query18.q.out | 12 +-
.../perf/tpcds30tb/tez/cbo_query19.q.out | 4 +-
.../perf/tpcds30tb/tez/cbo_query2.q.out | 6 +-
.../perf/tpcds30tb/tez/cbo_query20.q.out | 2 +-
.../perf/tpcds30tb/tez/cbo_query21.q.out | 4 +-
.../perf/tpcds30tb/tez/cbo_query23.q.out | 18 +-
.../perf/tpcds30tb/tez/cbo_query24.q.out | 12 +-
.../perf/tpcds30tb/tez/cbo_query27.q.out | 2 +-
.../perf/tpcds30tb/tez/cbo_query31.q.out | 4 +-
.../perf/tpcds30tb/tez/cbo_query34.q.out | 6 +-
.../perf/tpcds30tb/tez/cbo_query36.q.out | 2 +-
.../perf/tpcds30tb/tez/cbo_query37.q.out | 2 +-
.../perf/tpcds30tb/tez/cbo_query4.q.out | 24 +-
.../perf/tpcds30tb/tez/cbo_query40.q.out | 4 +-
.../perf/tpcds30tb/tez/cbo_query41.q.out | 2 +-
.../perf/tpcds30tb/tez/cbo_query43.q.out | 2 +-
.../perf/tpcds30tb/tez/cbo_query46.q.out | 4 +-
.../perf/tpcds30tb/tez/cbo_query47.q.out | 14 +-
.../perf/tpcds30tb/tez/cbo_query48.q.out | 8 +-
.../perf/tpcds30tb/tez/cbo_query53.q.out | 2 +-
.../perf/tpcds30tb/tez/cbo_query56.q.out | 8 +-
.../perf/tpcds30tb/tez/cbo_query57.q.out | 14 +-
.../perf/tpcds30tb/tez/cbo_query58.q.out | 2 +-
.../perf/tpcds30tb/tez/cbo_query59.q.out | 6 +-
.../perf/tpcds30tb/tez/cbo_query6.q.out | 6 +-
.../perf/tpcds30tb/tez/cbo_query62.q.out | 4 +-
.../perf/tpcds30tb/tez/cbo_query63.q.out | 2 +-
.../perf/tpcds30tb/tez/cbo_query64.q.out | 12 +-
.../perf/tpcds30tb/tez/cbo_query65.q.out | 2 +-
.../perf/tpcds30tb/tez/cbo_query66.q.out | 16 +-
.../perf/tpcds30tb/tez/cbo_query67.q.out | 2 +-
.../perf/tpcds30tb/tez/cbo_query71.q.out | 2 +-
.../perf/tpcds30tb/tez/cbo_query72.q.out | 6 +-
.../perf/tpcds30tb/tez/cbo_query73.q.out | 4 +-
.../perf/tpcds30tb/tez/cbo_query74.q.out | 4 +-
.../perf/tpcds30tb/tez/cbo_query78.q.out | 68 +-
.../perf/tpcds30tb/tez/cbo_query79.q.out | 2 +-
.../perf/tpcds30tb/tez/cbo_query8.q.out | 6 +-
.../perf/tpcds30tb/tez/cbo_query82.q.out | 2 +-
.../perf/tpcds30tb/tez/cbo_query83.q.out | 2 +-
.../perf/tpcds30tb/tez/cbo_query84.q.out | 4 +-
.../perf/tpcds30tb/tez/cbo_query85.q.out | 28 +-
.../perf/tpcds30tb/tez/cbo_query88.q.out | 18 +-
.../perf/tpcds30tb/tez/cbo_query89.q.out | 2 +-
.../perf/tpcds30tb/tez/cbo_query9.q.out | 10 +-
.../perf/tpcds30tb/tez/cbo_query91.q.out | 2 +-
.../perf/tpcds30tb/tez/cbo_query98.q.out | 2 +-
.../perf/tpcds30tb/tez/cbo_query99.q.out | 4 +-
.../perf/tpcds30tb/tez/query10.q.out | 4 +-
.../perf/tpcds30tb/tez/query12.q.out | 4 +-
.../perf/tpcds30tb/tez/query13.q.out | 36 +-
.../perf/tpcds30tb/tez/query15.q.out | 4 +-
.../perf/tpcds30tb/tez/query16.q.out | 4 +-
.../perf/tpcds30tb/tez/query18.q.out | 8 +-
.../perf/tpcds30tb/tez/query20.q.out | 4 +-
.../perf/tpcds30tb/tez/query27.q.out | 4 +-
.../perf/tpcds30tb/tez/query34.q.out | 8 +-
.../perf/tpcds30tb/tez/query36.q.out | 4 +-
.../perf/tpcds30tb/tez/query37.q.out | 4 +-
.../perf/tpcds30tb/tez/query41.q.out | 4 +-
.../perf/tpcds30tb/tez/query46.q.out | 8 +-
.../perf/tpcds30tb/tez/query47.q.out | 4 +-
.../perf/tpcds30tb/tez/query48.q.out | 26 +-
.../perf/tpcds30tb/tez/query53.q.out | 42 +-
.../perf/tpcds30tb/tez/query56.q.out | 2 +-
.../perf/tpcds30tb/tez/query57.q.out | 4 +-
.../perf/tpcds30tb/tez/query63.q.out | 42 +-
.../perf/tpcds30tb/tez/query64.q.out | 4 +-
.../perf/tpcds30tb/tez/query66.q.out | 4 +-
.../perf/tpcds30tb/tez/query73.q.out | 4 +-
.../perf/tpcds30tb/tez/query78.q.out | 466 ++---
.../perf/tpcds30tb/tez/query82.q.out | 4 +-
.../perf/tpcds30tb/tez/query85.q.out | 81 +-
.../perf/tpcds30tb/tez/query88.q.out | 4 +-
.../perf/tpcds30tb/tez/query89.q.out | 4 +-
.../perf/tpcds30tb/tez/query91.q.out | 4 +-
.../perf/tpcds30tb/tez/query98.q.out | 4 +-
.../results/clientpositive/smb_mapjoin_47.q.out | 4 +-
.../clientpositive/tez/explainanalyze_4.q.out | 54 +-
333 files changed, 7041 insertions(+), 2499 deletions(-)
diff --git a/common/src/java/org/apache/hadoop/hive/ql/log/PerfLogger.java b/common/src/java/org/apache/hadoop/hive/ql/log/PerfLogger.java
index 15720d03fe1..cd49c5efdb1 100644
--- a/common/src/java/org/apache/hadoop/hive/ql/log/PerfLogger.java
+++ b/common/src/java/org/apache/hadoop/hive/ql/log/PerfLogger.java
@@ -95,6 +95,7 @@ public class PerfLogger {
public static final String LOAD_HASHTABLE = "LoadHashtable";
public static final String TEZ_GET_SESSION = "TezGetSession";
public static final String SAVE_TO_RESULTS_CACHE = "saveToResultsCache";
+ public static final String SEARCH_TRANSFORMER = "transformSearch";
public static final String FILE_MOVES = "FileMoves";
public static final String LOAD_TABLE = "LoadTable";
diff --git a/hbase-handler/src/test/results/positive/hbase_queries.q.out b/hbase-handler/src/test/results/positive/hbase_queries.q.out
index 6745327f846..21e28305697 100644
--- a/hbase-handler/src/test/results/positive/hbase_queries.q.out
+++ b/hbase-handler/src/test/results/positive/hbase_queries.q.out
@@ -326,7 +326,7 @@ STAGE PLANS:
alias: hbase_table_2
Statistics: Num rows: 1 Data size: 188 Basic stats: COMPLETE Column stats: NONE
Filter Operator
- predicate: ((key < 120) and (key > 100)) (type: boolean)
+ predicate: ((key > 100) and (key < 120)) (type: boolean)
Statistics: Num rows: 1 Data size: 188 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: key (type: int), value (type: string)
diff --git a/iceberg/iceberg-handler/src/test/results/positive/delete_iceberg_copy_on_write_unpartitioned.q.out b/iceberg/iceberg-handler/src/test/results/positive/delete_iceberg_copy_on_write_unpartitioned.q.out
index fe2376b4f56..707be189e49 100644
--- a/iceberg/iceberg-handler/src/test/results/positive/delete_iceberg_copy_on_write_unpartitioned.q.out
+++ b/iceberg/iceberg-handler/src/test/results/positive/delete_iceberg_copy_on_write_unpartitioned.q.out
@@ -48,10 +48,10 @@ STAGE PLANS:
Map Operator Tree:
TableScan
alias: tbl_ice
- filterExpr: (((b <> 'one') and (b <> 'four') and (a <> 22)) or ((a = 22) or (b) IN ('one', 'four')) is null or (b) IN ('one', 'four') or (a = 22)) (type: boolean)
+ filterExpr: (((b) IN ('four', 'one') or (a = 22)) is null or (((b < 'four') or ((b > 'four') and (b < 'one')) or (b > 'one')) and (a <> 22)) or (b) IN ('four', 'one') or (a = 22)) (type: boolean)
Statistics: Num rows: 7 Data size: 672 Basic stats: COMPLETE Column stats: COMPLETE
Filter Operator
- predicate: ((((b <> 'one') and (b <> 'four') and (a <> 22)) or ((a = 22) or (b) IN ('one', 'four')) is null) and FILE__PATH is not null) (type: boolean)
+ predicate: ((((b) IN ('four', 'one') or (a = 22)) is null or (((b < 'four') or ((b > 'four') and (b < 'one')) or (b > 'one')) and (a <> 22))) and FILE__PATH is not null) (type: boolean)
Statistics: Num rows: 7 Data size: 672 Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
expressions: a (type: int), b (type: string), c (type: int), PARTITION__SPEC__ID (type: int), PARTITION__HASH (type: bigint), FILE__PATH (type: string), ROW__POSITION (type: bigint), PARTITION__PROJECTION (type: string)
@@ -65,7 +65,7 @@ STAGE PLANS:
Statistics: Num rows: 7 Data size: 3388 Basic stats: COMPLETE Column stats: COMPLETE
value expressions: _col0 (type: int), _col1 (type: string), _col2 (type: int), _col3 (type: int), _col4 (type: bigint), _col6 (type: bigint), _col7 (type: string)
Filter Operator
- predicate: (((b) IN ('one', 'four') or (a = 22)) and FILE__PATH is not null) (type: boolean)
+ predicate: (((b) IN ('four', 'one') or (a = 22)) and FILE__PATH is not null) (type: boolean)
Statistics: Num rows: 4 Data size: 368 Basic stats: COMPLETE Column stats: COMPLETE
Reduce Output Operator
key expressions: FILE__PATH (type: string)
@@ -74,7 +74,7 @@ STAGE PLANS:
Map-reduce partition columns: FILE__PATH (type: string)
Statistics: Num rows: 4 Data size: 368 Basic stats: COMPLETE Column stats: COMPLETE
Filter Operator
- predicate: ((a = 22) or (b) IN ('one', 'four')) (type: boolean)
+ predicate: ((a = 22) or (b) IN ('four', 'one')) (type: boolean)
Statistics: Num rows: 4 Data size: 384 Basic stats: COMPLETE Column stats: COMPLETE
Reduce Output Operator
key expressions: FILE__PATH (type: string)
diff --git a/iceberg/iceberg-handler/src/test/results/positive/delete_iceberg_mixed.q.out b/iceberg/iceberg-handler/src/test/results/positive/delete_iceberg_mixed.q.out
index 240f6559f34..74c4275a47e 100644
--- a/iceberg/iceberg-handler/src/test/results/positive/delete_iceberg_mixed.q.out
+++ b/iceberg/iceberg-handler/src/test/results/positive/delete_iceberg_mixed.q.out
@@ -94,7 +94,7 @@ Stage-4
Select Operator [SEL_56] (rows=7 width=188)
Output:["_col0","_col1","_col2","_col3","_col4","_col5","_col6"]
Filter Operator [FIL_53] (rows=7 width=188)
- predicate:(((id <= 4) or (id <> 2) or ((id > 4) or (id = 2)) is null) and (((id <= 4) and (id <> 2)) or ((id > 4) or (id = 2)) is null) and FILE__PATH is not null)
+ predicate:((((id = 2) or (id > 4)) is null or (id < 2) or ((id > 2) and (id <= 4))) and FILE__PATH is not null)
TableScan [TS_0] (rows=7 width=188)
default@ice01,ice01,Tbl:COMPLETE,Col:NONE,Output:["id","name"]
<-Reducer 4 [SIMPLE_EDGE] vectorized
@@ -114,7 +114,7 @@ Stage-4
SHUFFLE [RS_57]
PartitionCols:FILE__PATH
Filter Operator [FIL_54] (rows=7 width=4)
- predicate:(((id > 4) or (id = 2)) and FILE__PATH is not null)
+ predicate:(((id = 2) or (id > 4)) and FILE__PATH is not null)
Please refer to the previous TableScan [TS_0]
<-Reducer 5 [CONTAINS] vectorized
File Output Operator [FS_70]
diff --git a/iceberg/iceberg-handler/src/test/results/positive/llap/iceberg_bucket_map_join_7.q.out b/iceberg/iceberg-handler/src/test/results/positive/llap/iceberg_bucket_map_join_7.q.out
index 6701fbaf410..94961e581c0 100644
--- a/iceberg/iceberg-handler/src/test/results/positive/llap/iceberg_bucket_map_join_7.q.out
+++ b/iceberg/iceberg-handler/src/test/results/positive/llap/iceberg_bucket_map_join_7.q.out
@@ -150,27 +150,27 @@ Stage-0
File Output Operator [FS_61]
Limit [LIM_60] (rows=20 width=447)
Number of rows:20
- Select Operator [SEL_59] (rows=791 width=447)
+ Select Operator [SEL_59] (rows=473 width=447)
Output:["_col0","_col1","_col2","_col3","_col4"]
<-Map 1 [SIMPLE_EDGE] vectorized, llap
SHUFFLE [RS_58]
- Top N Key Operator [TNK_57] (rows=791 width=447)
+ Top N Key Operator [TNK_57] (rows=473 width=447)
keys:_col0,top n:20
- Map Join Operator [MAPJOIN_56] (rows=791 width=447)
+ Map Join Operator [MAPJOIN_56] (rows=473 width=447)
BucketMapJoin:true,Conds:SEL_55._col0, _col1=RS_53._col0, _col1(Inner),Output:["_col0","_col1","_col2","_col3","_col4"]
<-Map 3 [CUSTOM_EDGE] vectorized, llap
MULTICAST [RS_53]
PartitionCols:_col0, _col1
- Select Operator [SEL_52] (rows=500 width=178)
+ Select Operator [SEL_52] (rows=387 width=178)
Output:["_col0","_col1"]
- Filter Operator [FIL_51] (rows=500 width=178)
- predicate:((key <> '0') and (key <> '100') and value is not null)
+ Filter Operator [FIL_51] (rows=387 width=178)
+ predicate:(((key < '0') or ((key > '0') and (key < '100')) or (key > '100')) and key is not null and value is not null)
TableScan [TS_3] (rows=500 width=178)
default@src,b,Tbl:COMPLETE,Col:COMPLETE,Output:["key","value"]
- <-Select Operator [SEL_55] (rows=500 width=269)
+ <-Select Operator [SEL_55] (rows=387 width=269)
Output:["_col0","_col1","_col2"]
- Filter Operator [FIL_54] (rows=500 width=269)
- predicate:((key1 <> '0') and (key1 <> '100') and key2 is not null)
+ Filter Operator [FIL_54] (rows=387 width=269)
+ predicate:(((key1 < '0') or ((key1 > '0') and (key1 < '100')) or (key1 > '100')) and key1 is not null and key2 is not null)
TableScan [TS_0] (rows=500 width=269)
default@srcbucket_big,a,Tbl:COMPLETE,Col:COMPLETE,Grouping Num Buckets:8,Grouping Partition Columns:["key1","key2"],Output:["key1","key2","value"]
@@ -346,27 +346,27 @@ Stage-0
File Output Operator [FS_41]
Limit [LIM_40] (rows=20 width=447)
Number of rows:20
- Select Operator [SEL_39] (rows=791 width=447)
+ Select Operator [SEL_39] (rows=473 width=447)
Output:["_col0","_col1","_col2","_col3","_col4"]
<-Map 1 [SIMPLE_EDGE] vectorized, llap
SHUFFLE [RS_38]
- Top N Key Operator [TNK_37] (rows=791 width=447)
+ Top N Key Operator [TNK_37] (rows=473 width=447)
keys:_col0,top n:20
- Map Join Operator [MAPJOIN_36] (rows=791 width=447)
+ Map Join Operator [MAPJOIN_36] (rows=473 width=447)
BucketMapJoin:true,Conds:SEL_35._col0=RS_33._col0(Inner),Output:["_col0","_col1","_col2","_col3","_col4"]
<-Map 3 [CUSTOM_EDGE] vectorized, llap
MULTICAST [RS_33]
PartitionCols:_col0
- Select Operator [SEL_32] (rows=500 width=178)
+ Select Operator [SEL_32] (rows=387 width=178)
Output:["_col0","_col1"]
- Filter Operator [FIL_31] (rows=500 width=178)
- predicate:((key <> '0') and (key <> '100'))
+ Filter Operator [FIL_31] (rows=387 width=178)
+ predicate:(((key < '0') or ((key > '0') and (key < '100')) or (key > '100')) and key is not null)
TableScan [TS_3] (rows=500 width=178)
default@src,b,Tbl:COMPLETE,Col:COMPLETE,Output:["key","value"]
- <-Select Operator [SEL_35] (rows=500 width=269)
+ <-Select Operator [SEL_35] (rows=387 width=269)
Output:["_col0","_col1","_col2"]
- Filter Operator [FIL_34] (rows=500 width=269)
- predicate:((key1 <> '0') and (key1 <> '100'))
+ Filter Operator [FIL_34] (rows=387 width=269)
+ predicate:(((key1 < '0') or ((key1 > '0') and (key1 < '100')) or (key1 > '100')) and key1 is not null)
TableScan [TS_0] (rows=500 width=269)
default@srcbucket_big,a,Tbl:COMPLETE,Col:COMPLETE,Grouping Num Buckets:4,Grouping Partition Columns:["key1"],Output:["key1","key2","value"]
@@ -435,40 +435,40 @@ POSTHOOK: Input: default@srcbucket_big
Plan optimized by CBO.
Vertex dependency in root stage
-Map 1 <- Map 3 (CUSTOM_EDGE)
-Reducer 2 <- Map 1 (SIMPLE_EDGE)
+Map 2 <- Map 1 (BROADCAST_EDGE)
+Reducer 3 <- Map 2 (SIMPLE_EDGE)
Stage-0
Fetch Operator
limit:20
Stage-1
- Reducer 2 vectorized, llap
+ Reducer 3 vectorized, llap
File Output Operator [FS_41]
Limit [LIM_40] (rows=20 width=447)
Number of rows:20
- Select Operator [SEL_39] (rows=791 width=447)
+ Select Operator [SEL_39] (rows=612 width=447)
Output:["_col0","_col1","_col2","_col3","_col4"]
- <-Map 1 [SIMPLE_EDGE] vectorized, llap
+ <-Map 2 [SIMPLE_EDGE] vectorized, llap
SHUFFLE [RS_38]
- Top N Key Operator [TNK_37] (rows=791 width=447)
+ Top N Key Operator [TNK_37] (rows=612 width=447)
keys:_col0,top n:20
- Map Join Operator [MAPJOIN_36] (rows=791 width=447)
- BucketMapJoin:true,Conds:SEL_35._col0=RS_33._col0(Inner),Output:["_col0","_col1","_col2","_col3","_col4"]
- <-Map 3 [CUSTOM_EDGE] vectorized, llap
- MULTICAST [RS_33]
+ Map Join Operator [MAPJOIN_36] (rows=612 width=447)
+ Conds:RS_33._col0=SEL_35._col0(Inner),Output:["_col0","_col1","_col2","_col3","_col4"]
+ <-Map 1 [BROADCAST_EDGE] vectorized, llap
+ BROADCAST [RS_33]
PartitionCols:_col0
- Select Operator [SEL_32] (rows=500 width=178)
- Output:["_col0","_col1"]
- Filter Operator [FIL_31] (rows=500 width=178)
- predicate:key is not null
- TableScan [TS_3] (rows=500 width=178)
- default@src,b,Tbl:COMPLETE,Col:COMPLETE,Output:["key","value"]
- <-Select Operator [SEL_35] (rows=500 width=269)
- Output:["_col0","_col1","_col2"]
- Filter Operator [FIL_34] (rows=500 width=269)
- predicate:((key2 <> 'val_0') and (key2 <> 'val_100') and key1 is not null)
- TableScan [TS_0] (rows=500 width=269)
- default@srcbucket_big,a,Tbl:COMPLETE,Col:COMPLETE,Grouping Num Buckets:4,Grouping Partition Columns:["key1"],Output:["key1","key2","value"]
+ Select Operator [SEL_32] (rows=387 width=269)
+ Output:["_col0","_col1","_col2"]
+ Filter Operator [FIL_31] (rows=387 width=269)
+ predicate:(((key2 < 'val_0') or ((key2 > 'val_0') and (key2 < 'val_100')) or (key2 > 'val_100')) and key1 is not null)
+ TableScan [TS_0] (rows=500 width=269)
+ default@srcbucket_big,a,Tbl:COMPLETE,Col:COMPLETE,Output:["key1","key2","value"]
+ <-Select Operator [SEL_35] (rows=500 width=178)
+ Output:["_col0","_col1"]
+ Filter Operator [FIL_34] (rows=500 width=178)
+ predicate:key is not null
+ TableScan [TS_3] (rows=500 width=178)
+ default@src,b,Tbl:COMPLETE,Col:COMPLETE,Output:["key","value"]
PREHOOK: query: SELECT *
FROM srcbucket_big a
diff --git a/iceberg/iceberg-handler/src/test/results/positive/llap/llap_iceberg_read_orc.q.out b/iceberg/iceberg-handler/src/test/results/positive/llap/llap_iceberg_read_orc.q.out
index a1907766f19..4a6770803dc 100644
--- a/iceberg/iceberg-handler/src/test/results/positive/llap/llap_iceberg_read_orc.q.out
+++ b/iceberg/iceberg-handler/src/test/results/positive/llap/llap_iceberg_read_orc.q.out
@@ -359,11 +359,11 @@ STAGE PLANS:
Map Operator Tree:
TableScan
alias: o
- filterExpr: ((quantity NOT BETWEEN 39 AND 0 or quantity NOT BETWEEN 69 AND 39 or (quantity > 70)) and (((quantity > 0) and (quantity < 39)) or ((quantity > 39) and (quantity < 69)) or (quantity > 70)) and itemid is not null) (type: boolean)
+ filterExpr: ((((quantity > 0) and (quantity < 39)) or ((quantity > 39) and (quantity < 69)) or (quantity > 70)) and quantity is not null and itemid is not null) (type: boolean)
probeDecodeDetails: cacheKey:HASH_MAP_MAPJOIN_29_container, bigKeyColName:itemid, smallTablePos:1, keyRatio:0.9523809523809523
Statistics: Num rows: 21 Data size: 168 Basic stats: COMPLETE Column stats: COMPLETE
Filter Operator
- predicate: ((quantity NOT BETWEEN 39 AND 0 or quantity NOT BETWEEN 69 AND 39 or (quantity > 70)) and (((quantity > 0) and (quantity < 39)) or ((quantity > 39) and (quantity < 69)) or (quantity > 70)) and itemid is not null) (type: boolean)
+ predicate: ((((quantity > 0) and (quantity < 39)) or ((quantity > 39) and (quantity < 69)) or (quantity > 70)) and quantity is not null and itemid is not null) (type: boolean)
Statistics: Num rows: 21 Data size: 168 Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
expressions: quantity (type: int), itemid (type: int)
diff --git a/iceberg/iceberg-handler/src/test/results/positive/merge_iceberg_copy_on_write_unpartitioned.q.out b/iceberg/iceberg-handler/src/test/results/positive/merge_iceberg_copy_on_write_unpartitioned.q.out
index 0584c5fc95e..4b413fb80f1 100644
--- a/iceberg/iceberg-handler/src/test/results/positive/merge_iceberg_copy_on_write_unpartitioned.q.out
+++ b/iceberg/iceberg-handler/src/test/results/positive/merge_iceberg_copy_on_write_unpartitioned.q.out
@@ -311,7 +311,7 @@ STAGE PLANS:
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8
Statistics: Num rows: 8 Data size: 3884 Basic stats: COMPLETE Column stats: COMPLETE
Filter Operator
- predicate: ((_col5 <> _col8) or ((_col5 = _col8) or _col5 is null) is null) (type: boolean)
+ predicate: ((_col5 <> _col8) or (_col5 is null or (_col5 = _col8)) is null) (type: boolean)
Statistics: Num rows: 8 Data size: 3884 Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
expressions: _col0 (type: int), _col1 (type: bigint), _col2 (type: string), _col3 (type: bigint), _col4 (type: string), _col5 (type: int), _col6 (type: string), _col7 (type: int)
@@ -621,7 +621,7 @@ STAGE PLANS:
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8
Statistics: Num rows: 8 Data size: 3884 Basic stats: COMPLETE Column stats: COMPLETE
Filter Operator
- predicate: ((((_col5 <> _col8) or (_col5 <= 100)) and _col5 is not null) or (((_col5 = _col8) and (_col5 > 100)) or _col5 is null) is null) (type: boolean)
+ predicate: ((((_col5 <> _col8) or (_col5 <= 100)) and _col5 is not null) or (_col5 is null or ((_col5 = _col8) and (_col5 > 100))) is null) (type: boolean)
Statistics: Num rows: 8 Data size: 3884 Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
expressions: _col0 (type: int), _col1 (type: bigint), _col2 (type: string), _col3 (type: bigint), _col4 (type: string), _col5 (type: int), _col6 (type: string), _col7 (type: int)
diff --git a/iceberg/iceberg-handler/src/test/results/positive/mv_iceberg_orc4.q.out b/iceberg/iceberg-handler/src/test/results/positive/mv_iceberg_orc4.q.out
index bc153b7b867..445cf331b29 100644
--- a/iceberg/iceberg-handler/src/test/results/positive/mv_iceberg_orc4.q.out
+++ b/iceberg/iceberg-handler/src/test/results/positive/mv_iceberg_orc4.q.out
@@ -93,7 +93,7 @@ POSTHOOK: Input: default@tbl_ice_v2
POSTHOOK: Output: default@mat1
CBO PLAN:
HiveAggregate(group=[{0, 1}], agg#0=[sum($2)])
- HiveProject($f0=[$0], $f1=[$1], $f2=[$2])
+ HiveProject(b=[$0], c=[$1], $f2=[$2])
HiveUnion(all=[true])
HiveProject(b=[$0], c=[$1], $f2=[$2])
HiveAggregate(group=[{1, 2}], agg#0=[sum($4)])
diff --git a/iceberg/iceberg-handler/src/test/results/positive/update_iceberg_copy_on_write_partitioned.q.out b/iceberg/iceberg-handler/src/test/results/positive/update_iceberg_copy_on_write_partitioned.q.out
index e1d97e48b33..46ad44e193c 100644
--- a/iceberg/iceberg-handler/src/test/results/positive/update_iceberg_copy_on_write_partitioned.q.out
+++ b/iceberg/iceberg-handler/src/test/results/positive/update_iceberg_copy_on_write_partitioned.q.out
@@ -50,10 +50,10 @@ STAGE PLANS:
Map Operator Tree:
TableScan
alias: tbl_ice
- filterExpr: ((a = 22) or (b) IN ('one', 'four')) (type: boolean)
+ filterExpr: ((a = 22) or (b) IN ('four', 'one')) (type: boolean)
Statistics: Num rows: 3 Data size: 288 Basic stats: COMPLETE Column stats: PARTIAL
Filter Operator
- predicate: ((a = 22) or (b) IN ('one', 'four')) (type: boolean)
+ predicate: ((a = 22) or (b) IN ('four', 'one')) (type: boolean)
Statistics: Num rows: 3 Data size: 288 Basic stats: COMPLETE Column stats: PARTIAL
Select Operator
expressions: PARTITION__SPEC__ID (type: int), PARTITION__HASH (type: bigint), FILE__PATH (type: string), ROW__POSITION (type: bigint), PARTITION__PROJECTION (type: string), a (type: int), 'Changed' (type: string), c (type: int)
@@ -71,10 +71,10 @@ STAGE PLANS:
Map Operator Tree:
TableScan
alias: tbl_ice
- filterExpr: (((b <> 'one') and (b <> 'four') and (a <> 22)) or ((a = 22) or (b) IN ('one', 'four')) is null) (type: boolean)
+ filterExpr: (((b) IN ('four', 'one') or (a = 22)) is null or (((b < 'four') or ((b > 'four') and (b < 'one')) or (b > 'one')) and (a <> 22))) (type: boolean)
Statistics: Num rows: 1 Data size: 84 Basic stats: COMPLETE Column stats: PARTIAL
Filter Operator
- predicate: ((((b <> 'one') and (b <> 'four') and (a <> 22)) or ((a = 22) or (b) IN ('one', 'four')) is null) and FILE__PATH is not null) (type: boolean)
+ predicate: ((((b) IN ('four', 'one') or (a = 22)) is null or (((b < 'four') or ((b > 'four') and (b < 'one')) or (b > 'one')) and (a <> 22))) and FILE__PATH is not null) (type: boolean)
Statistics: Num rows: 1 Data size: 84 Basic stats: COMPLETE Column stats: PARTIAL
Select Operator
expressions: a (type: int), b (type: string), c (type: int), PARTITION__SPEC__ID (type: int), PARTITION__HASH (type: bigint), FILE__PATH (type: string), ROW__POSITION (type: bigint), PARTITION__PROJECTION (type: string)
@@ -92,10 +92,10 @@ STAGE PLANS:
Map Operator Tree:
TableScan
alias: tbl_ice
- filterExpr: ((a = 22) or (b) IN ('one', 'four')) (type: boolean)
+ filterExpr: ((a = 22) or (b) IN ('four', 'one')) (type: boolean)
Statistics: Num rows: 3 Data size: 288 Basic stats: COMPLETE Column stats: PARTIAL
Filter Operator
- predicate: ((a = 22) or (b) IN ('one', 'four')) (type: boolean)
+ predicate: ((a = 22) or (b) IN ('four', 'one')) (type: boolean)
Statistics: Num rows: 3 Data size: 288 Basic stats: COMPLETE Column stats: PARTIAL
Reduce Output Operator
key expressions: FILE__PATH (type: string)
@@ -105,7 +105,7 @@ STAGE PLANS:
Statistics: Num rows: 3 Data size: 288 Basic stats: COMPLETE Column stats: PARTIAL
value expressions: a (type: int), b (type: string), c (type: int), PARTITION__SPEC__ID (type: int), PARTITION__HASH (type: bigint), PARTITION__PROJECTION (type: string)
Filter Operator
- predicate: (((b) IN ('one', 'four') or (a = 22)) and FILE__PATH is not null) (type: boolean)
+ predicate: (((b) IN ('four', 'one') or (a = 22)) and FILE__PATH is not null) (type: boolean)
Statistics: Num rows: 3 Data size: 276 Basic stats: COMPLETE Column stats: PARTIAL
Reduce Output Operator
key expressions: FILE__PATH (type: string)
diff --git a/iceberg/iceberg-handler/src/test/results/positive/update_iceberg_copy_on_write_unpartitioned.q.out b/iceberg/iceberg-handler/src/test/results/positive/update_iceberg_copy_on_write_unpartitioned.q.out
index 627959eee2e..6a149603f73 100644
--- a/iceberg/iceberg-handler/src/test/results/positive/update_iceberg_copy_on_write_unpartitioned.q.out
+++ b/iceberg/iceberg-handler/src/test/results/positive/update_iceberg_copy_on_write_unpartitioned.q.out
@@ -49,10 +49,10 @@ STAGE PLANS:
Map Operator Tree:
TableScan
alias: tbl_ice
- filterExpr: ((a = 22) or (b) IN ('one', 'four')) (type: boolean)
+ filterExpr: ((a = 22) or (b) IN ('four', 'one')) (type: boolean)
Statistics: Num rows: 7 Data size: 672 Basic stats: COMPLETE Column stats: COMPLETE
Filter Operator
- predicate: ((a = 22) or (b) IN ('one', 'four')) (type: boolean)
+ predicate: ((a = 22) or (b) IN ('four', 'one')) (type: boolean)
Statistics: Num rows: 4 Data size: 384 Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
expressions: PARTITION__SPEC__ID (type: int), PARTITION__HASH (type: bigint), FILE__PATH (type: string), ROW__POSITION (type: bigint), PARTITION__PROJECTION (type: string), a (type: int), 'Changed' (type: string), c (type: int)
@@ -71,10 +71,10 @@ STAGE PLANS:
Map Operator Tree:
TableScan
alias: tbl_ice
- filterExpr: ((a = 22) or (b) IN ('one', 'four') or ((b <> 'one') and (b <> 'four') and (a <> 22)) or ((a = 22) or (b) IN ('one', 'four')) is null) (type: boolean)
+ filterExpr: ((a = 22) or (b) IN ('four', 'one') or ((b) IN ('four', 'one') or (a = 22)) is null or (((b < 'four') or ((b > 'four') and (b < 'one')) or (b > 'one')) and (a <> 22))) (type: boolean)
Statistics: Num rows: 7 Data size: 672 Basic stats: COMPLETE Column stats: COMPLETE
Filter Operator
- predicate: ((a = 22) or (b) IN ('one', 'four')) (type: boolean)
+ predicate: ((a = 22) or (b) IN ('four', 'one')) (type: boolean)
Statistics: Num rows: 4 Data size: 384 Basic stats: COMPLETE Column stats: COMPLETE
Reduce Output Operator
key expressions: FILE__PATH (type: string)
@@ -84,7 +84,7 @@ STAGE PLANS:
Statistics: Num rows: 4 Data size: 384 Basic stats: COMPLETE Column stats: COMPLETE
value expressions: a (type: int), b (type: string), c (type: int), PARTITION__SPEC__ID (type: int), PARTITION__HASH (type: bigint), PARTITION__PROJECTION (type: string)
Filter Operator
- predicate: (((b) IN ('one', 'four') or (a = 22)) and FILE__PATH is not null) (type: boolean)
+ predicate: (((b) IN ('four', 'one') or (a = 22)) and FILE__PATH is not null) (type: boolean)
Statistics: Num rows: 4 Data size: 368 Basic stats: COMPLETE Column stats: COMPLETE
Reduce Output Operator
key expressions: FILE__PATH (type: string)
@@ -93,7 +93,7 @@ STAGE PLANS:
Map-reduce partition columns: FILE__PATH (type: string)
Statistics: Num rows: 4 Data size: 368 Basic stats: COMPLETE Column stats: COMPLETE
Filter Operator
- predicate: ((((b <> 'one') and (b <> 'four') and (a <> 22)) or ((a = 22) or (b) IN ('one', 'four')) is null) and FILE__PATH is not null) (type: boolean)
+ predicate: ((((b) IN ('four', 'one') or (a = 22)) is null or (((b < 'four') or ((b > 'four') and (b < 'one')) or (b > 'one')) and (a <> 22))) and FILE__PATH is not null) (type: boolean)
Statistics: Num rows: 7 Data size: 672 Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
expressions: a (type: int), b (type: string), c (type: int), PARTITION__SPEC__ID (type: int), PARTITION__HASH (type: bigint), FILE__PATH (type: string), ROW__POSITION (type: bigint), PARTITION__PROJECTION (type: string)
diff --git a/itests/hive-blobstore/src/test/results/clientpositive/insert_into_dynamic_partitions.q.out b/itests/hive-blobstore/src/test/results/clientpositive/insert_into_dynamic_partitions.q.out
index 3cb4b1dc7d0..4735e1978c1 100644
--- a/itests/hive-blobstore/src/test/results/clientpositive/insert_into_dynamic_partitions.q.out
+++ b/itests/hive-blobstore/src/test/results/clientpositive/insert_into_dynamic_partitions.q.out
@@ -87,7 +87,7 @@ POSTHOOK: type: QUERY
POSTHOOK: Input: _dummy_database@_dummy_table
POSTHOOK: Output: default@table1
OPTIMIZED SQL: SELECT *
-FROM TABLE(INLINE(ARRAY[ (1, '101'), (2, '202'), (3, '303'), (4, '404'), (5, '505')]))
+FROM TABLE(INLINE(ARRAY[(1, '101'), (2, '202'), (3, '303'), (4, '404'), (5, '505')]))
STAGE DEPENDENCIES:
Stage-1 is a root stage
Stage-0 depends on stages: Stage-1
diff --git a/itests/hive-blobstore/src/test/results/clientpositive/insert_into_table.q.out b/itests/hive-blobstore/src/test/results/clientpositive/insert_into_table.q.out
index f7ab417fc62..bb60ea80cec 100644
--- a/itests/hive-blobstore/src/test/results/clientpositive/insert_into_table.q.out
+++ b/itests/hive-blobstore/src/test/results/clientpositive/insert_into_table.q.out
@@ -51,7 +51,7 @@ POSTHOOK: type: QUERY
POSTHOOK: Input: _dummy_database@_dummy_table
POSTHOOK: Output: default@table1
OPTIMIZED SQL: SELECT *
-FROM TABLE(INLINE(ARRAY[ (1)]))
+FROM TABLE(INLINE(ARRAY[(1)]))
STAGE DEPENDENCIES:
Stage-1 is a root stage
Stage-7 depends on stages: Stage-1 , consists of Stage-4, Stage-3, Stage-5
diff --git a/itests/hive-blobstore/src/test/results/clientpositive/insert_overwrite_dynamic_partitions.q.out b/itests/hive-blobstore/src/test/results/clientpositive/insert_overwrite_dynamic_partitions.q.out
index c144f2dfaec..0d804540f98 100644
--- a/itests/hive-blobstore/src/test/results/clientpositive/insert_overwrite_dynamic_partitions.q.out
+++ b/itests/hive-blobstore/src/test/results/clientpositive/insert_overwrite_dynamic_partitions.q.out
@@ -105,7 +105,7 @@ POSTHOOK: type: QUERY
POSTHOOK: Input: _dummy_database@_dummy_table
POSTHOOK: Output: default@table1
OPTIMIZED SQL: SELECT *
-FROM TABLE(INLINE(ARRAY[ (1, '101'), (2, '202'), (3, '303'), (4, '404'), (5, '505')]))
+FROM TABLE(INLINE(ARRAY[(1, '101'), (2, '202'), (3, '303'), (4, '404'), (5, '505')]))
STAGE DEPENDENCIES:
Stage-1 is a root stage
Stage-0 depends on stages: Stage-1
diff --git a/itests/hive-blobstore/src/test/results/clientpositive/insert_overwrite_table.q.out b/itests/hive-blobstore/src/test/results/clientpositive/insert_overwrite_table.q.out
index bdcfaa9451b..17e71e96d71 100644
--- a/itests/hive-blobstore/src/test/results/clientpositive/insert_overwrite_table.q.out
+++ b/itests/hive-blobstore/src/test/results/clientpositive/insert_overwrite_table.q.out
@@ -59,7 +59,7 @@ POSTHOOK: type: QUERY
POSTHOOK: Input: _dummy_database@_dummy_table
POSTHOOK: Output: default@table1
OPTIMIZED SQL: SELECT *
-FROM TABLE(INLINE(ARRAY[ (1)]))
+FROM TABLE(INLINE(ARRAY[(1)]))
STAGE DEPENDENCIES:
Stage-1 is a root stage
Stage-7 depends on stages: Stage-1 , consists of Stage-4, Stage-3, Stage-5
diff --git a/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestMaterializedViewRebuild.java b/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestMaterializedViewRebuild.java
index d38e6695cb4..4453c321d22 100644
--- a/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestMaterializedViewRebuild.java
+++ b/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestMaterializedViewRebuild.java
@@ -55,7 +55,7 @@ public class TestMaterializedViewRebuild extends CompactorOnTezTest {
private static final List<String> INCREMENTAL_REBUILD_PLAN = Arrays.asList(
"CBO PLAN:",
"HiveProject(a=[$0], b=[$1], c=[$2])",
- " HiveFilter(condition=[AND(<(2, $5.writeid), OR(>($0, 0), IS NULL($0)))])",
+ " HiveFilter(condition=[AND(>($5.writeid, 2), OR(IS NULL($0), >($0, 0)))])",
" HiveTableScan(table=[[default, t1]], table:alias=[t1])",
""
);
diff --git a/packaging/pom.xml b/packaging/pom.xml
index f7c6ac7aee9..87360fb5892 100644
--- a/packaging/pom.xml
+++ b/packaging/pom.xml
@@ -173,6 +173,10 @@
\Qhttps://www.eclipse.org/org/documents/epl-v10.php\E
\Qhttp://www.eclipse.org/org/documents/epl-v10.php\E
</EPL-1.0>
+ <EPL-2.0>
+ \Qhttps://raw.githubusercontent.com/locationtech/jts/master/LICENSE_EPLv2.txt\E
+ \Qhttps://www.eclipse.org/org/documents/epl-2.0/EPL-2.0.txt\E
+ </EPL-2.0>
<MIT>
https?://(www\.)?opensource\.org/licenses/mit(-license.php)?
</MIT>
diff --git a/pom.xml b/pom.xml
index ecf70972b30..162afc177f5 100644
--- a/pom.xml
+++ b/pom.xml
@@ -109,10 +109,10 @@
<!-- Make sure to sync it with standalone-metastore/pom.xml -->
<antlr4.version>4.9.3</antlr4.version>
<apache-directory-server.version>2.0.0-M24</apache-directory-server.version>
- <avatica.version>1.12.0</avatica.version>
+ <avatica.version>1.23.0</avatica.version>
<avro.version>1.11.4</avro.version>
<bcprov-jdk18on.version>1.78</bcprov-jdk18on.version>
- <calcite.version>1.25.0</calcite.version>
+ <calcite.version>1.33.0</calcite.version>
<datanucleus-api-jdo.version>5.2.8</datanucleus-api-jdo.version>
<datanucleus-core.version>5.2.10</datanucleus-core.version>
<datanucleus-jdo.version>3.2.0-release</datanucleus-jdo.version>
@@ -149,6 +149,7 @@
<!-- httpcomponents are not always in version sync -->
<httpcomponents.client.version>4.5.13</httpcomponents.client.version>
<httpcomponents.core.version>4.4.13</httpcomponents.core.version>
+ <immutables.version>2.9.2</immutables.version>
<ivy.version>2.5.2</ivy.version>
<jackson.version>2.16.1</jackson.version>
<jamon.plugin.version>2.3.4</jamon.plugin.version>
@@ -163,6 +164,7 @@
<joda.version>2.13.0</joda.version>
<jodd.version>6.0.0</jodd.version>
<json.version>1.8</json.version>
+ <jts.version>1.19.0</jts.version>
<junit.version>4.13.2</junit.version>
<junit.jupiter.version>5.11.2</junit.jupiter.version>
<junit.vintage.version>5.11.2</junit.vintage.version>
@@ -343,6 +345,18 @@
<artifactId>parquet-hadoop-bundle</artifactId>
<version>${parquet.version}</version>
</dependency>
+ <dependency>
+ <groupId>org.immutables</groupId>
+ <artifactId>value-annotations</artifactId>
+ <version>${immutables.version}</version>
+ <scope>provided</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.locationtech.jts</groupId>
+ <artifactId>jts-core</artifactId>
+ <version>${jts.version}</version>
+ <scope>runtime</scope>
+ </dependency>
<dependency>
<groupId>com.sun.jersey</groupId>
<artifactId>jersey-core</artifactId>
diff --git a/ql/pom.xml b/ql/pom.xml
index 39daed65636..ab72ea4ba82 100644
--- a/ql/pom.xml
+++ b/ql/pom.xml
@@ -852,6 +852,14 @@
<groupId>org.codehaus.janino</groupId>
<artifactId>janino</artifactId>
</dependency>
+ <dependency>
+ <groupId>org.immutables</groupId>
+ <artifactId>value-annotations</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.locationtech.jts</groupId>
+ <artifactId>jts-core</artifactId>
+ </dependency>
<dependency>
<groupId>com.sun.jersey.contribs</groupId>
<artifactId>jersey-multipart</artifactId>
diff --git a/ql/src/java/org/apache/hadoop/hive/ql/ddl/view/materialized/alter/rebuild/AlterMaterializedViewRebuildAnalyzer.java b/ql/src/java/org/apache/hadoop/hive/ql/ddl/view/materialized/alter/rebuild/AlterMaterializedViewRebuildAnalyzer.java
index 542a1174ee2..988656d9ffe 100644
--- a/ql/src/java/org/apache/hadoop/hive/ql/ddl/view/materialized/alter/rebuild/AlterMaterializedViewRebuildAnalyzer.java
+++ b/ql/src/java/org/apache/hadoop/hive/ql/ddl/view/materialized/alter/rebuild/AlterMaterializedViewRebuildAnalyzer.java
@@ -52,6 +52,7 @@
import org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveTableScan;
import org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveUnion;
import org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveInBetweenExpandRule;
+import org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveSearchRules;
import org.apache.hadoop.hive.ql.optimizer.calcite.rules.views.HiveAggregateInsertDeleteIncrementalRewritingRule;
import org.apache.hadoop.hive.ql.optimizer.calcite.rules.views.HiveAggregateInsertIncrementalRewritingRule;
import org.apache.hadoop.hive.ql.optimizer.calcite.rules.views.HiveAggregatePartitionIncrementalRewritingRule;
@@ -292,7 +293,11 @@ protected RelNode applyMaterializedViewRewriting(RelOptPlanner planner, RelNode
basePlan = executeProgram(basePlan, program.build(), mdProvider, executorProvider, singletonList(materialization));
program = new HepProgramBuilder();
- generatePartialProgram(program, false, HepMatchOrder.DEPTH_FIRST, HivePushdownSnapshotFilterRule.INSTANCE);
+ generatePartialProgram(program,
+ false,
+ HepMatchOrder.DEPTH_FIRST,
+ HiveSearchRules.FILTER_SEARCH_EXPAND,
+ HivePushdownSnapshotFilterRule.INSTANCE);
basePlan = executeProgram(basePlan, program.build(), mdProvider, executorProvider);
perfLogger.perfLogEnd(this.getClass().getName(), PerfLogger.OPTIMIZER, "Calcite: View-based rewriting");
diff --git a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/Bug.java b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/Bug.java
index c1564ee0d3a..3377e37788c 100644
--- a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/Bug.java
+++ b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/Bug.java
@@ -103,4 +103,14 @@ public final class Bug {
* Whether <a href="https://issues.apache.org/jira/browse/CALCITE-6704">CALCITE-6704</a> is fixed.
*/
public static final boolean CALCITE_6704_FIXED = false;
+
+ /**
+ * Whether <a href="https://issues.apache.org/jira/browse/CALCITE-6513">CALCITE-6737</a> is fixed.
+ */
+ public static final boolean CALCITE_6737_FIXED = false;
+
+ /**
+ * Whether <a href="https://issues.apache.org/jira/browse/CALCITE-6954">CALCITE-6954</a> is fixed.
+ */
+ public static final boolean CALCITE_6954_FIXED = false;
}
diff --git a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveCalciteUtil.java b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveCalciteUtil.java
index 98f45476cd0..4308e45a956 100644
--- a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveCalciteUtil.java
+++ b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveCalciteUtil.java
@@ -258,9 +258,9 @@ public static RexNode projectNonColumnEquiConditions(ProjectFactory factory, Rel
// fields
if (newKeyCount > 0) {
leftRel = factory.createProject(leftRel, Collections.emptyList(), newLeftFields,
- SqlValidatorUtil.uniquify(newLeftFieldNames));
+ SqlValidatorUtil.uniquify(newLeftFieldNames, true), Collections.emptySet());
rightRel = factory.createProject(rightRel, Collections.emptyList(), newRightFields,
- SqlValidatorUtil.uniquify(newRightFieldNames));
+ SqlValidatorUtil.uniquify(newRightFieldNames, true), Collections.emptySet());
}
inputRels[0] = leftRel;
diff --git a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveRelBuilder.java b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveRelBuilder.java
index 8722da515ae..80ee1cb4647 100644
--- a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveRelBuilder.java
+++ b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveRelBuilder.java
@@ -37,7 +37,6 @@
import org.apache.calcite.rex.RexUtil;
import org.apache.calcite.schema.SchemaPlus;
import org.apache.calcite.server.CalciteServerStatement;
-import org.apache.calcite.sql.SqlAggFunction;
import org.apache.calcite.sql.SqlFunction;
import org.apache.calcite.sql.fun.SqlStdOperatorTable;
import org.apache.calcite.tools.FrameworkConfig;
@@ -46,11 +45,6 @@
import org.apache.calcite.tools.RelBuilderFactory;
import org.apache.calcite.util.ImmutableBitSet;
import org.apache.calcite.util.Litmus;
-import org.apache.hadoop.hive.ql.optimizer.calcite.functions.HiveMergeableAggregate;
-import org.apache.hadoop.hive.ql.optimizer.calcite.functions.HiveSqlCountAggFunction;
-import org.apache.hadoop.hive.ql.optimizer.calcite.functions.HiveSqlMinMaxAggFunction;
-import org.apache.hadoop.hive.ql.optimizer.calcite.functions.HiveSqlSumAggFunction;
-import org.apache.hadoop.hive.ql.optimizer.calcite.functions.HiveSqlSumEmptyIsZeroAggFunction;
import org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveFloorDate;
import java.util.ArrayList;
@@ -138,24 +132,6 @@ public static SqlFunction getFloorSqlFunction(TimeUnitRange flag) {
return SqlStdOperatorTable.FLOOR;
}
- public static SqlAggFunction getRollup(SqlAggFunction aggregation) {
- if (aggregation instanceof HiveMergeableAggregate) {
- HiveMergeableAggregate mAgg = (HiveMergeableAggregate) aggregation;
- return mAgg.getMergeAggFunction();
- }
- if (aggregation instanceof HiveSqlSumAggFunction
- || aggregation instanceof HiveSqlMinMaxAggFunction
- || aggregation instanceof HiveSqlSumEmptyIsZeroAggFunction) {
- return aggregation;
- }
- if (aggregation instanceof HiveSqlCountAggFunction) {
- HiveSqlCountAggFunction countAgg = (HiveSqlCountAggFunction) aggregation;
- return new HiveSqlSumEmptyIsZeroAggFunction(countAgg.isDistinct(), countAgg.getReturnTypeInference(),
- countAgg.getOperandTypeInference(), countAgg.getOperandTypeChecker());
- }
- return null;
- }
-
/** Creates a {@link Join} with correlating variables. */
@Override
public RelBuilder join(JoinRelType joinType, RexNode condition,
diff --git a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveRelFactories.java b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveRelFactories.java
index ac2cef60fdd..950e9ca5770 100644
--- a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveRelFactories.java
+++ b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveRelFactories.java
@@ -118,8 +118,11 @@ private HiveRelFactories() {
*/
private static class HiveProjectFactoryImpl implements ProjectFactory {
@Override
- public RelNode createProject(RelNode child, List<RelHint> hints,
- List<? extends RexNode> childExprs, List<String> fieldNames) {
+ public RelNode createProject(RelNode child, List<RelHint> hints, List<? extends RexNode> childExprs,
+ List<? extends String> fieldNames, Set<CorrelationId> variablesSet) {
+ if (!variablesSet.isEmpty()) {
+ throw new IllegalStateException("Setting variables is not supported");
+ }
RelOptCluster cluster = child.getCluster();
RelDataType rowType = RexUtil.createStructType(
cluster.getTypeFactory(), childExprs, fieldNames, SqlValidatorUtil.EXPR_SUGGESTER);
diff --git a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveRexExecutorImpl.java b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveRexExecutorImpl.java
index 66984a3ebde..7639a893fa5 100644
--- a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveRexExecutorImpl.java
+++ b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveRexExecutorImpl.java
@@ -51,7 +51,7 @@ public void reduce(RexBuilder rexBuilder, List<RexNode> constExps, List<RexNode>
for (RexNode rexNode : constExps) {
// initialize the converter
ExprNodeConverter converter = new ExprNodeConverter("", null, null, null,
- new HashSet<>(), rexBuilder);
+ new HashSet<>(), rexBuilder.getTypeFactory());
// convert RexNode to ExprNodeGenericFuncDesc
ExprNodeDesc expr = rexNode.accept(converter);
if (expr instanceof ExprNodeGenericFuncDesc) {
diff --git a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveTypeFactory.java b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveTypeFactory.java
new file mode 100644
index 00000000000..f0a7514b7ed
--- /dev/null
+++ b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveTypeFactory.java
@@ -0,0 +1,54 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.optimizer.calcite;
+
+import org.apache.calcite.jdbc.JavaTypeFactoryImpl;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.sql.type.SqlTypeName;
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+import java.util.List;
+
+public class HiveTypeFactory extends JavaTypeFactoryImpl {
+ public HiveTypeFactory() {
+ super(new HiveTypeSystemImpl());
+ if (Bug.CALCITE_6954_FIXED) {
+ throw new IllegalStateException("Class redundant once fix is merged");
+ }
+ }
+
+ @Override
+ protected @Nullable RelDataType leastRestrictiveArrayMultisetType(final List<RelDataType> types,
+ final SqlTypeName sqlTypeName) {
+ RelDataType type = super.leastRestrictiveArrayMultisetType(types, sqlTypeName);
+ if (type != null) {
+ return canonize(type);
+ }
+ return null;
+ }
+
+ @Override
+ protected @Nullable RelDataType leastRestrictiveMapType(final List<RelDataType> types,
+ final SqlTypeName sqlTypeName) {
+ RelDataType type = super.leastRestrictiveMapType(types, sqlTypeName);
+ if (type != null) {
+ return canonize(type);
+ }
+ return null;
+ }
+}
diff --git a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/RangeConverter.java b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/RangeConverter.java
new file mode 100644
index 00000000000..991e91b89f3
--- /dev/null
+++ b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/RangeConverter.java
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.optimizer.calcite;
+
+import com.google.common.collect.Range;
+
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rex.RexBuilder;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.sql.SqlOperator;
+import org.apache.calcite.sql.fun.SqlStdOperatorTable;
+import org.apache.calcite.util.RangeSets;
+import org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveBetween;
+
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * Converts a {@link Range} to a {@link RexNode} expression based on the range type as defined by the
+ * {@link RangeSets.Consumer}.
+ *
+ * <table>
+ * <tr><th>Range <th>RexNode
+ * <tr><td>{@code (a..b)} <td>{@code x > a AND x < b}
+ * <tr><td>{@code [a..b]} <td>{@code x BETWEEN a AND b}
+ * <tr><td>{@code [a..a]} <td>{@code a}
+ * <tr><td>{@code (a..b]} <td>{@code x > a AND x <= b}
+ * <tr><td>{@code [a..b)} <td>{@code x >= a AND x < b}
+ * <tr><td>{@code (a..+∞)} <td>{@code x > a}
+ * <tr><td>{@code [a..+∞)} <td>{@code x >= a}
+ * <tr><td>{@code (-∞..b)} <td>{@code x < b}
+ * <tr><td>{@code (-∞..b]} <td>{@code x <= b}
+ * <tr><td>{@code (-∞..+∞)}<td>{@code true}
+ * </table>
+ *
+ * @param <C> type of the range
+ */
+class RangeConverter<C extends Comparable<C>> implements RangeSets.Consumer<C> {
+
+ private final RexBuilder rexBuilder;
+ private final RelDataType type;
+ private final RexNode ref;
+ final List<RexNode> inLiterals;
+ final List<RexNode> nodes;
+
+ public RangeConverter(RexBuilder rexBuilder, RelDataType type, RexNode ref) {
+ this.rexBuilder = rexBuilder;
+ this.type = type;
+ this.ref = ref;
+ this.inLiterals = new ArrayList<>();
+ this.nodes = new ArrayList<>();
+ }
+
+ private RexNode op(SqlOperator op, C value) {
+ return rexBuilder.makeCall(op, ref, rexBuilder.makeLiteral(value, type, true, true));
+ }
+
+ private RexNode and(RexNode... nodes) {
+ return rexBuilder.makeCall(SqlStdOperatorTable.AND, nodes);
+ }
+
+ public void all() {
+ nodes.add(rexBuilder.makeLiteral(true));
+ }
+
+ @Override
+ public void atLeast(C lower) {
+ nodes.add(op(SqlStdOperatorTable.GREATER_THAN_OR_EQUAL, lower));
+ }
+
+ @Override
+ public void atMost(C upper) {
+ nodes.add(op(SqlStdOperatorTable.LESS_THAN_OR_EQUAL, upper));
+ }
+
+ @Override
+ public void greaterThan(C lower) {
+ nodes.add(op(SqlStdOperatorTable.GREATER_THAN, lower));
+ }
+
+ @Override
+ public void lessThan(C upper) {
+ nodes.add(op(SqlStdOperatorTable.LESS_THAN, upper));
+ }
+
+ @Override
+ public void singleton(C value) {
+ inLiterals.add(rexBuilder.makeLiteral(value, type, true, true));
+ }
+
+ @Override
+ public void closed(C lower, C upper) {
+ nodes.add(rexBuilder.makeCall(HiveBetween.INSTANCE,
+ rexBuilder.makeLiteral(false),
+ ref,
+ rexBuilder.makeLiteral(lower, type, true, true),
+ rexBuilder.makeLiteral(upper, type, true, true)));
+ }
+
+ @Override
+ public void closedOpen(C lower, C upper) {
+ nodes.add(and(op(SqlStdOperatorTable.GREATER_THAN_OR_EQUAL, lower), op(SqlStdOperatorTable.LESS_THAN, upper)));
+ }
+
+ @Override
+ public void openClosed(C lower, C upper) {
+ nodes.add(and(op(SqlStdOperatorTable.GREATER_THAN, lower), op(SqlStdOperatorTable.LESS_THAN_OR_EQUAL, upper)));
+ }
+
+ @Override
+ public void open(C lower, C upper) {
+ nodes.add(and(op(SqlStdOperatorTable.GREATER_THAN, lower), op(SqlStdOperatorTable.LESS_THAN, upper)));
+ }
+}
diff --git a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/SearchTransformer.java b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/SearchTransformer.java
new file mode 100644
index 00000000000..dc88dade4b2
--- /dev/null
+++ b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/SearchTransformer.java
@@ -0,0 +1,139 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.optimizer.calcite;
+
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rex.RexBuilder;
+import org.apache.calcite.rex.RexCall;
+import org.apache.calcite.rex.RexLiteral;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.rex.RexShuttle;
+import org.apache.calcite.rex.RexUnknownAs;
+import org.apache.calcite.rex.RexUtil;
+import org.apache.calcite.sql.fun.SqlStdOperatorTable;
+import org.apache.calcite.util.RangeSets;
+import org.apache.calcite.util.Sarg;
+import org.apache.hadoop.hive.ql.log.PerfLogger;
+import org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveBetween;
+import org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveIn;
+import org.apache.hadoop.hive.ql.session.SessionState;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Objects;
+
+/**
+ * A class that transforms a call to the internal {@link SqlStdOperatorTable#SEARCH} operator into an equivalent
+ * {@link RexNode} expression. The new expression comprises conjunctions/disjunctions involving the following operators:
+ * <ul>
+ * <li>standard binary comparisons ({@code <,<=,>,>=,=,<>});</li>
+ * <li>{@code IS NULL}, {@code IS NOT NULL};</li>
+ * <li>{@link HiveIn}, {@link HiveBetween}.</li>
+ * </ul>
+ * Below you can find an example from the transformation of a SEARCH expression of INTEGER type.
+ * <pre>{@code
+ * Input: SEARCH($9, Sarg[1, 2, 3, [10..15], (20..+∞)])
+ * Output: OR(IN($9, 1, 2, 3), BETWEEN(false, $9, 10, 15), >($9, 20))
+ * }</pre>
+ *
+ * @param <C> type of the Sarg literal
+ */
+public class SearchTransformer<C extends Comparable<C>> {
+ private final RexBuilder rexBuilder;
+ private final RexNode ref;
+ private final Sarg<C> sarg;
+ protected final RelDataType type;
+
+ public SearchTransformer(RexBuilder rexBuilder, RexCall search) {
+ this.rexBuilder = rexBuilder;
+ ref = search.getOperands().get(0);
+ RexLiteral literal = (RexLiteral) search.operands.get(1);
+ sarg = Objects.requireNonNull(literal.getValueAs(Sarg.class), "Sarg");
+ type = literal.getType();
+ }
+
+ public RexNode transform() {
+ PerfLogger perfLogger = SessionState.getPerfLogger();
+ perfLogger.perfLogBegin(this.getClass().getName(), PerfLogger.SEARCH_TRANSFORMER);
+
+ RangeConverter<C> consumer = new RangeConverter<>(rexBuilder, type, ref);
+ RangeSets.forEach(sarg.rangeSet, consumer);
+
+ List<RexNode> orList = new ArrayList<>();
+ if (sarg.nullAs == RexUnknownAs.TRUE) {
+ orList.add(rexBuilder.makeCall(SqlStdOperatorTable.IS_NULL, ref));
+ }
+ switch (consumer.inLiterals.size()) {
+ case 0:
+ break;
+ case 1:
+ orList.add(rexBuilder.makeCall(SqlStdOperatorTable.EQUALS, ref, consumer.inLiterals.get(0)));
+ break;
+ default:
+ List<RexNode> operands = new ArrayList<>(consumer.inLiterals.size() + 1);
+ operands.add(ref);
+ operands.addAll(consumer.inLiterals);
+ orList.add(rexBuilder.makeCall(HiveIn.INSTANCE, operands));
+ }
+ orList.addAll(consumer.nodes);
+ RexNode x = RexUtil.composeDisjunction(rexBuilder, orList);
+
+ if (sarg.nullAs == RexUnknownAs.FALSE) {
+ RexNode notNull = rexBuilder.makeCall(SqlStdOperatorTable.IS_NOT_NULL, ref);
+ x = RexUtil.composeConjunction(rexBuilder, Arrays.asList(notNull, x));
+ }
+ perfLogger.perfLogEnd(this.getClass().getName(), PerfLogger.SEARCH_TRANSFORMER);
+ return x;
+ }
+
+ public static class Shuttle extends RexShuttle {
+ private final RexBuilder rexBuilder;
+
+ public Shuttle(final RexBuilder rexBuilder) {
+ this.rexBuilder = rexBuilder;
+ }
+
+ @Override public RexNode visitCall(RexCall call) {
+ final boolean[] update = {false};
+ final List<RexNode> clonedOperands;
+ switch (call.getKind()) {
+ // Flatten AND/OR operands.
+ case OR:
+ clonedOperands = visitList(call.operands, update);
+ if (update[0]) {
+ return RexUtil.composeDisjunction(rexBuilder, clonedOperands);
+ } else {
+ return call;
+ }
+ case AND:
+ clonedOperands = visitList(call.operands, update);
+ if (update[0]) {
+ return RexUtil.composeConjunction(rexBuilder, clonedOperands);
+ } else {
+ return call;
+ }
+ case SEARCH:
+ return new SearchTransformer<>(rexBuilder, call).transform();
+ default:
+ return super.visitCall(call);
+ }
+ }
+ }
+}
+
diff --git a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/functions/HiveMergeableAggregate.java b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/functions/HiveMergeableAggregate.java
index 9abd7f73cdd..b7490cf3c0e 100644
--- a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/functions/HiveMergeableAggregate.java
+++ b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/functions/HiveMergeableAggregate.java
@@ -59,7 +59,8 @@ public HiveMergeableAggregate(String string, SqlKind kind, SqlReturnTypeInferenc
}
- public SqlAggFunction getMergeAggFunction() {
+ @Override
+ public SqlAggFunction getRollup() {
return mergeAgg;
}
diff --git a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/functions/HiveSqlCountAggFunction.java b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/functions/HiveSqlCountAggFunction.java
index 615f30d3d12..7318676928f 100644
--- a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/functions/HiveSqlCountAggFunction.java
+++ b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/functions/HiveSqlCountAggFunction.java
@@ -31,13 +31,13 @@
import org.apache.calcite.sql.SqlSplittableAggFunction;
import org.apache.calcite.sql.SqlSyntax;
import org.apache.calcite.sql.SqlSplittableAggFunction.CountSplitter;
-import org.apache.calcite.sql.SqlSplittableAggFunction.Registry;
import org.apache.calcite.sql.fun.SqlStdOperatorTable;
import org.apache.calcite.sql.type.SqlOperandTypeChecker;
import org.apache.calcite.sql.type.SqlOperandTypeInference;
import org.apache.calcite.sql.type.SqlReturnTypeInference;
import org.apache.calcite.sql.type.SqlTypeName;
import org.apache.calcite.util.ImmutableIntList;
+import org.checkerframework.checker.nullness.qual.Nullable;
import com.google.common.collect.ImmutableList;
@@ -126,4 +126,10 @@ public AggregateCall topSplit(RexBuilder rexBuilder,
false, ImmutableList.of(ordinal), -1, aggregateCall.type, aggregateCall.name);
}
}
+
+ @Override
+ public @Nullable SqlAggFunction getRollup() {
+ return new HiveSqlSumEmptyIsZeroAggFunction(isDistinct(), getReturnTypeInference(), getOperandTypeInference(),
+ getOperandTypeChecker());
+ }
}
diff --git a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/functions/HiveSqlMinMaxAggFunction.java b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/functions/HiveSqlMinMaxAggFunction.java
index 8254ea81632..f86e990ed7a 100644
--- a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/functions/HiveSqlMinMaxAggFunction.java
+++ b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/functions/HiveSqlMinMaxAggFunction.java
@@ -46,4 +46,9 @@ public <T> T unwrap(Class<T> clazz) {
}
return super.unwrap(clazz);
}
+
+ @Override
+ public SqlAggFunction getRollup() {
+ return this;
+ }
}
diff --git a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/functions/HiveSqlSumAggFunction.java b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/functions/HiveSqlSumAggFunction.java
index 974dab1f0f6..0b749c3e524 100644
--- a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/functions/HiveSqlSumAggFunction.java
+++ b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/functions/HiveSqlSumAggFunction.java
@@ -124,4 +124,9 @@ public AggregateCall topSplit(RexBuilder rexBuilder,
false, ImmutableList.of(ordinal), -1, aggregateCall.type, aggregateCall.name);
}
}
+
+ @Override
+ public SqlAggFunction getRollup() {
+ return this;
+ }
}
diff --git a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/functions/HiveSqlSumEmptyIsZeroAggFunction.java b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/functions/HiveSqlSumEmptyIsZeroAggFunction.java
index 512ceda38f0..1d41dd2f0cc 100644
--- a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/functions/HiveSqlSumEmptyIsZeroAggFunction.java
+++ b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/functions/HiveSqlSumEmptyIsZeroAggFunction.java
@@ -74,4 +74,9 @@ public RelDataType getReturnType(RelDataTypeFactory typeFactory) {
}
return super.unwrap(clazz);
}
+
+ @Override
+ public SqlAggFunction getRollup() {
+ return this;
+ }
}
diff --git a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveFilter.java b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveFilter.java
index 241d2b09153..1a1f4d3d5dd 100644
--- a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveFilter.java
+++ b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveFilter.java
@@ -22,8 +22,6 @@
import org.apache.calcite.rel.RelNode;
import org.apache.calcite.rel.RelShuttle;
import org.apache.calcite.rel.core.Filter;
-import org.apache.calcite.rel.core.Join;
-import org.apache.calcite.rel.core.TableScan;
import org.apache.calcite.rel.metadata.RelMetadataQuery;
import org.apache.calcite.rex.RexNode;
import org.apache.hadoop.hive.ql.optimizer.calcite.correlation.CorrelationInfoVisitor;
diff --git a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveIn.java b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveIn.java
index 213625567ca..699a436dcee 100644
--- a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveIn.java
+++ b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveIn.java
@@ -29,6 +29,12 @@
import org.apache.calcite.sql.type.InferTypes;
import org.apache.calcite.sql.type.ReturnTypes;
+/**
+ * An operator for representing the {@code <in value list>} expression of SQL standard.
+ * Essentially, it means that this operator must not be used for {@code IN} sub-queries.
+ * Consequently, the {@code SqlKind} of the operator must not be {@link SqlKind#IN} since
+ * the latter only appears in the context of a {@link org.apache.calcite.rex.RexSubQuery}.
+ */
public class HiveIn extends SqlSpecialOperator {
public static final SqlSpecialOperator INSTANCE =
@@ -37,7 +43,7 @@ public class HiveIn extends SqlSpecialOperator {
private HiveIn() {
super(
"IN",
- SqlKind.IN,
+ SqlKind.OTHER_FUNCTION,
30,
true,
ReturnTypes.BOOLEAN_NULLABLE,
@@ -57,7 +63,7 @@ public void unparse(
SqlNode sqlNode = opList.get(0);
sqlNode.unparse(writer, leftPrec, getLeftPrec());
writer.sep("IN");
- Frame frame = writer.startList(FrameTypeEnum.SETOP, "(", ")");
+ Frame frame = writer.startList(FrameTypeEnum.SIMPLE, "(", ")");
for (SqlNode op : opList.subList(1, opList.size())) {
writer.sep(",");
op.unparse(writer, 0, 0);
diff --git a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/jdbc/JdbcHiveTableScan.java b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/jdbc/JdbcHiveTableScan.java
index a59d58a7ec4..379bb3a033e 100644
--- a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/jdbc/JdbcHiveTableScan.java
+++ b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/jdbc/JdbcHiveTableScan.java
@@ -29,6 +29,9 @@
import org.apache.calcite.rel.RelWriter;
import org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveTableScan;
+
+import com.google.common.collect.ImmutableList;
+
/**
* Relational expression representing a scan of a HiveDB collection.
*
@@ -42,7 +45,7 @@ public class JdbcHiveTableScan extends JdbcTableScan {
public JdbcHiveTableScan(RelOptCluster cluster, RelOptTable table, JdbcTable jdbcTable,
JdbcConvention jdbcConvention, HiveTableScan hiveTableScan) {
- super(cluster, table, jdbcTable, jdbcConvention);
+ super(cluster, ImmutableList.of(), table, jdbcTable, jdbcConvention);
this.hiveTableScan= hiveTableScan;
}
diff --git a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/CommonRelSubExprRegisterRule.java b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/CommonRelSubExprRegisterRule.java
index 40b29f9a4ac..9f425c799bf 100644
--- a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/CommonRelSubExprRegisterRule.java
+++ b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/CommonRelSubExprRegisterRule.java
@@ -17,6 +17,7 @@
package org.apache.hadoop.hive.ql.optimizer.calcite.rules;
import org.apache.calcite.plan.CommonRelSubExprRule;
+import org.apache.calcite.plan.RelOptRule;
import org.apache.calcite.plan.RelOptRuleCall;
import org.apache.calcite.rel.RelNode;
import org.apache.calcite.rel.core.Aggregate;
@@ -31,8 +32,6 @@
import java.util.function.Predicate;
-import static org.apache.calcite.plan.RelRule.Config.EMPTY;
-
/**
* Rule for saving relational expressions that appear more than once in a query tree to the planner context.
*/
@@ -47,7 +46,7 @@ public final class CommonRelSubExprRegisterRule extends CommonRelSubExprRule {
o -> o.operand(Project.class).predicate(new InterestingRelNodePredicate()).anyInputs());
private CommonRelSubExprRegisterRule(OperandTransform operands) {
- super(EMPTY.withOperandSupplier(operands).as(Config.class));
+ super(new CommonRelSubExprRegisterRule.Config().withOperandSupplier(operands).as(Config.class));
}
@Override
@@ -81,4 +80,10 @@ public boolean test(final RelNode rel) {
}
}
+ private static class Config extends HiveRuleConfig implements CommonRelSubExprRule.Config {
+ @Override
+ public RelOptRule toRule() {
+ return new CommonRelSubExprRegisterRule(this.operandSupplier());
+ }
+ }
}
diff --git a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/CteRuleConfig.java b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/CteRuleConfig.java
index a33b26303ac..1110cb581ab 100644
--- a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/CteRuleConfig.java
+++ b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/CteRuleConfig.java
@@ -17,25 +17,39 @@
package org.apache.hadoop.hive.ql.optimizer.calcite.rules;
import org.apache.calcite.plan.RelOptRule;
-import org.apache.calcite.plan.RelRule;
import org.apache.calcite.rel.core.TableScan;
-import org.apache.calcite.util.ImmutableBeans;
import java.util.List;
import java.util.Map;
-public interface CteRuleConfig extends RelRule.Config {
- CteRuleConfig DEFAULT = EMPTY.withOperandSupplier(b -> b.operand(TableScan.class).noInputs()).as(CteRuleConfig.class);
+public class CteRuleConfig extends HiveRuleConfig {
+ public static CteRuleConfig config() {
+ return new CteRuleConfig().withOperandSupplier(b -> b.operand(TableScan.class).noInputs()).as(CteRuleConfig.class);
+ }
+
+ private Map<List<String>, Integer> tableOccurences;
+ private int referenceThreshold;
- CteRuleConfig withTableOccurrences(Map<List<String>, Integer> tableOccurrences);
- @ImmutableBeans.Property
- Map<List<String>, Integer> getTableOccurrences();
- CteRuleConfig withReferenceThreshold(int referenceThreshold);
- @ImmutableBeans.Property
- int referenceThreshold();
+ public CteRuleConfig withTableOccurrences(Map<List<String>, Integer> tableOccurrences) {
+ this.tableOccurences = tableOccurrences;
+ return this;
+ }
+
+ public Map<List<String>, Integer> getTableOccurrences() {
+ return this.tableOccurences;
+ }
+
+ public CteRuleConfig withReferenceThreshold(int referenceThreshold) {
+ this.referenceThreshold = referenceThreshold;
+ return this;
+ }
+
+ public int referenceThreshold() {
+ return referenceThreshold;
+ }
@Override
- default RelOptRule toRule() {
+ public RelOptRule toRule() {
throw new IllegalStateException("Must not be called");
}
}
diff --git a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveAggregateJoinTransposeRule.java b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveAggregateJoinTransposeRule.java
index 29533bdad03..35d189e96b2 100644
--- a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveAggregateJoinTransposeRule.java
+++ b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveAggregateJoinTransposeRule.java
@@ -224,8 +224,7 @@ public void onMatch(RelOptRuleCall call) {
}
}
side.newInput = relBuilder.push(joinInput)
- .aggregate(relBuilder.groupKey(belowAggregateKey, null),
- belowAggCalls)
+ .aggregate(relBuilder.groupKey(belowAggregateKey), belowAggCalls)
.build();
}
offset += fieldCount;
diff --git a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveAggregateSplitRule.java b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveAggregateSplitRule.java
index 8534981b19f..a8d21402084 100644
--- a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveAggregateSplitRule.java
+++ b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveAggregateSplitRule.java
@@ -30,7 +30,6 @@
import org.apache.calcite.tools.RelBuilder;
import org.apache.calcite.tools.RelBuilderFactory;
import org.apache.calcite.util.ImmutableBitSet;
-import org.apache.hadoop.hive.ql.optimizer.calcite.HiveRelBuilder;
import org.apache.hadoop.hive.ql.optimizer.calcite.HiveRelFactories;
import org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveAggregate;
import org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveGroupingID;
@@ -73,8 +72,7 @@ public void onMatch(RelOptRuleCall call) {
if (aggregateCall.filterArg >= 0) {
return;
}
- SqlAggFunction aggFunction =
- HiveRelBuilder.getRollup(aggregateCall.getAggregation());
+ SqlAggFunction aggFunction = aggregateCall.getAggregation().getRollup();
if (aggFunction == null) {
return;
}
diff --git a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveDruidRules.java b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveDruidRules.java
index 700f604b67f..c8a3c205b40 100644
--- a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveDruidRules.java
+++ b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveDruidRules.java
@@ -17,9 +17,8 @@
*/
package org.apache.hadoop.hive.ql.optimizer.calcite.rules;
-import com.google.common.base.Preconditions;
-import com.google.common.collect.Lists;
import org.apache.calcite.adapter.druid.DruidQuery;
+import org.apache.calcite.adapter.druid.DruidRules;
import org.apache.calcite.adapter.druid.DruidRules.DruidAggregateProjectRule;
import org.apache.calcite.adapter.druid.DruidRules.DruidAggregateRule;
import org.apache.calcite.adapter.druid.DruidRules.DruidFilterRule;
@@ -29,19 +28,9 @@
import org.apache.calcite.adapter.druid.DruidRules.DruidSortRule;
import org.apache.calcite.plan.RelOptRule;
import org.apache.calcite.plan.RelOptRuleCall;
-import org.apache.calcite.plan.RelRule;
import org.apache.calcite.rel.core.Aggregate;
import org.apache.calcite.rel.core.AggregateCall;
-import org.apache.calcite.rel.core.Filter;
-import org.apache.calcite.rel.core.Project;
-import org.apache.calcite.rel.core.Sort;
-import org.apache.calcite.rel.rules.AggregateFilterTransposeRule;
-import org.apache.calcite.rel.rules.CoreRules;
import org.apache.calcite.rel.rules.DateRangeRules;
-import org.apache.calcite.rel.rules.FilterAggregateTransposeRule;
-import org.apache.calcite.rel.rules.FilterProjectTransposeRule;
-import org.apache.calcite.rel.rules.ProjectFilterTransposeRule;
-import org.apache.calcite.rel.rules.SortProjectTransposeRule;
import org.apache.calcite.sql.SqlKind;
import org.apache.calcite.sql.fun.SqlSumEmptyIsZeroAggFunction;
import org.apache.calcite.tools.RelBuilder;
@@ -50,6 +39,9 @@
import org.apache.calcite.util.Pair;
import org.apache.hadoop.hive.ql.optimizer.calcite.HiveRelFactories;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.Lists;
+
import java.util.ArrayList;
import java.util.HashSet;
import java.util.LinkedHashSet;
@@ -60,102 +52,47 @@
/**
* Druid rules with Hive builder factory.
- *
- * Simplify this class when upgrading to Calcite 1.26 using
- * <a href="https://issues.apache.org/jira/browse/CALCITE-4200">CALCITE-4200</a>
*/
public class HiveDruidRules {
- public static final DruidFilterRule FILTER = RelRule.Config.EMPTY
- .withOperandSupplier(b0 ->
- b0.operand(Filter.class).oneInput(b1 ->
- b1.operand(DruidQuery.class).noInputs()))
- .withRelBuilderFactory(HiveRelFactories.HIVE_BUILDER)
- .as(DruidFilterRule.Config.class)
- .toRule();
+ public static final RelOptRule FILTER =
+ DruidFilterRule.DruidFilterRuleConfig.DEFAULT.withRelBuilderFactory(HiveRelFactories.HIVE_BUILDER).toRule();
- public static final DruidProjectRule PROJECT = RelRule.Config.EMPTY
- .withOperandSupplier(b0 ->
- b0.operand(Project.class).oneInput(b1 ->
- b1.operand(DruidQuery.class).noInputs()))
- .withRelBuilderFactory(HiveRelFactories.HIVE_BUILDER)
- .as(DruidProjectRule.Config.class)
- .toRule();
+ public static final RelOptRule PROJECT =
+ DruidProjectRule.DruidProjectRuleConfig.DEFAULT.withRelBuilderFactory(HiveRelFactories.HIVE_BUILDER).toRule();
- public static final DruidAggregateRule AGGREGATE = RelRule.Config.EMPTY
- .withOperandSupplier(b0 ->
- b0.operand(Aggregate.class).oneInput(b1 ->
- b1.operand(DruidQuery.class).noInputs()))
- .withRelBuilderFactory(HiveRelFactories.HIVE_BUILDER)
- .as(DruidAggregateRule.Config.class)
- .toRule();
+ public static final RelOptRule AGGREGATE =
+ DruidAggregateRule.DruidAggregateRuleConfig.DEFAULT.withRelBuilderFactory(HiveRelFactories.HIVE_BUILDER).toRule();
- public static final DruidAggregateProjectRule AGGREGATE_PROJECT = RelRule.Config.EMPTY
- .withOperandSupplier(b0 ->
- b0.operand(Aggregate.class).oneInput(b1 ->
- b1.operand(Project.class).oneInput(b2 ->
- b2.operand(DruidQuery.class).noInputs())))
- .withRelBuilderFactory(HiveRelFactories.HIVE_BUILDER)
- .as(DruidAggregateProjectRule.Config.class)
- .toRule();
+ public static final RelOptRule AGGREGATE_PROJECT =
+ DruidAggregateProjectRule.DruidAggregateProjectRuleConfig.DEFAULT.withRelBuilderFactory(
+ HiveRelFactories.HIVE_BUILDER).toRule();
- public static final DruidSortRule SORT = RelRule.Config.EMPTY
- .withOperandSupplier(b0 ->
- b0.operand(Sort.class).oneInput(b1 ->
- b1.operand(DruidQuery.class).noInputs()))
- .withRelBuilderFactory(HiveRelFactories.HIVE_BUILDER)
- .as(DruidSortRule.Config.class)
- .toRule();
+ public static final RelOptRule SORT =
+ DruidSortRule.DruidSortRuleConfig.DEFAULT.withRelBuilderFactory(HiveRelFactories.HIVE_BUILDER).toRule();
- public static final SortProjectTransposeRule SORT_PROJECT_TRANSPOSE = SortProjectTransposeRule.Config.DEFAULT
- .withOperandFor(Sort.class, Project.class, DruidQuery.class)
- .withRelBuilderFactory(HiveRelFactories.HIVE_BUILDER)
- .as(SortProjectTransposeRule.Config.class)
- .toRule();
+ public static final RelOptRule SORT_PROJECT_TRANSPOSE =
+ DruidRules.SORT_PROJECT_TRANSPOSE.config.withRelBuilderFactory(HiveRelFactories.HIVE_BUILDER).toRule();
- public static final ProjectFilterTransposeRule PROJECT_FILTER_TRANSPOSE = ProjectFilterTransposeRule.Config.DEFAULT
- .withOperandFor(Project.class, Filter.class, DruidQuery.class)
- .withRelBuilderFactory(HiveRelFactories.HIVE_BUILDER)
- .as(ProjectFilterTransposeRule.Config.class)
- .toRule();
+ public static final RelOptRule PROJECT_FILTER_TRANSPOSE =
+ DruidRules.PROJECT_FILTER_TRANSPOSE.config.withRelBuilderFactory(HiveRelFactories.HIVE_BUILDER).toRule();
- public static final FilterProjectTransposeRule FILTER_PROJECT_TRANSPOSE = CoreRules.FILTER_PROJECT_TRANSPOSE.config
- .withOperandFor(Filter.class, Project.class, DruidQuery.class)
- .withCopyFilter(true)
- .withCopyProject(true)
- .withRelBuilderFactory(HiveRelFactories.HIVE_BUILDER)
- .as(FilterProjectTransposeRule.Config.class)
- .toRule();
+ public static final RelOptRule FILTER_PROJECT_TRANSPOSE =
+ DruidRules.FILTER_PROJECT_TRANSPOSE.config.withRelBuilderFactory(HiveRelFactories.HIVE_BUILDER).toRule();
- public static final AggregateFilterTransposeRule AGGREGATE_FILTER_TRANSPOSE =
- CoreRules.AGGREGATE_FILTER_TRANSPOSE.config
- .withOperandFor(Aggregate.class, Filter.class, DruidQuery.class)
- .withRelBuilderFactory(HiveRelFactories.HIVE_BUILDER)
- .as(AggregateFilterTransposeRule.Config.class)
- .toRule();
+ public static final RelOptRule AGGREGATE_FILTER_TRANSPOSE =
+ DruidRules.AGGREGATE_FILTER_TRANSPOSE.config.withRelBuilderFactory(HiveRelFactories.HIVE_BUILDER).toRule();
- public static final FilterAggregateTransposeRule FILTER_AGGREGATE_TRANSPOSE =
- CoreRules.FILTER_AGGREGATE_TRANSPOSE.config
- .withOperandFor(Filter.class, Aggregate.class, DruidQuery.class)
- .withRelBuilderFactory(HiveRelFactories.HIVE_BUILDER)
- .as(FilterAggregateTransposeRule.Config.class)
- .toRule();
+ public static final RelOptRule FILTER_AGGREGATE_TRANSPOSE =
+ DruidRules.FILTER_AGGREGATE_TRANSPOSE.config.withRelBuilderFactory(HiveRelFactories.HIVE_BUILDER).toRule();
- public static final DruidPostAggregationProjectRule POST_AGGREGATION_PROJECT = RelRule.Config.EMPTY
- .withOperandSupplier(b0 ->
- b0.operand(Project.class).oneInput(b1 ->
- b1.operand(DruidQuery.class).noInputs()))
- .withRelBuilderFactory(HiveRelFactories.HIVE_BUILDER)
- .as(DruidPostAggregationProjectRule.Config.class)
- .toRule();
+ public static final RelOptRule POST_AGGREGATION_PROJECT =
+ DruidPostAggregationProjectRule.DruidPostAggregationProjectRuleConfig.DEFAULT.withRelBuilderFactory(
+ HiveRelFactories.HIVE_BUILDER).toRule();
- public static final DruidHavingFilterRule HAVING_FILTER_RULE = RelRule.Config.EMPTY
- .withOperandSupplier(b0 ->
- b0.operand(Filter.class).oneInput(b1 ->
- b1.operand(DruidQuery.class).noInputs()))
- .withRelBuilderFactory(HiveRelFactories.HIVE_BUILDER)
- .as(DruidHavingFilterRule.Config.class)
- .toRule();
+ public static final RelOptRule HAVING_FILTER_RULE =
+ DruidHavingFilterRule.DruidHavingFilterRuleConfig.DEFAULT.withRelBuilderFactory(HiveRelFactories.HIVE_BUILDER)
+ .toRule();
public static final AggregateExpandDistinctAggregatesDruidRule EXPAND_SINGLE_DISTINCT_AGGREGATES_DRUID_RULE =
new AggregateExpandDistinctAggregatesDruidRule(HiveRelFactories.HIVE_BUILDER);
diff --git a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveFilterJoinRule.java b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveFilterJoinRule.java
index bf08c9805ff..caa0b6456f4 100644
--- a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveFilterJoinRule.java
+++ b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveFilterJoinRule.java
@@ -20,7 +20,6 @@
import com.google.common.collect.ImmutableList;
import com.google.common.collect.Sets;
import java.util.ArrayList;
-import java.util.BitSet;
import java.util.List;
import org.apache.calcite.plan.RelOptRule;
@@ -30,7 +29,6 @@
import org.apache.calcite.plan.RelOptUtil.InputFinder;
import org.apache.calcite.plan.RelOptUtil.RexInputConverter;
import org.apache.calcite.rel.RelNode;
-import org.apache.calcite.plan.RelRule;
import org.apache.calcite.rel.core.Filter;
import org.apache.calcite.rel.core.Join;
import org.apache.calcite.rel.core.JoinRelType;
@@ -42,7 +40,6 @@
import org.apache.calcite.rex.RexNode;
import org.apache.calcite.rex.RexUtil;
import org.apache.calcite.tools.RelBuilder;
-import org.apache.calcite.tools.RelBuilderFactory;
import org.apache.calcite.util.ImmutableBitSet;
import org.apache.hadoop.hive.ql.optimizer.calcite.Bug;
import org.apache.hadoop.hive.ql.optimizer.calcite.HiveCalciteUtil;
@@ -66,18 +63,12 @@ public abstract class HiveFilterJoinRule extends FilterJoinRule {
/**
* Creates a PushFilterPastJoinRule with an explicit root operand.
*/
- protected HiveFilterJoinRule(RelOptRuleOperand operand, String id, boolean smart,
- RelBuilderFactory relBuilderFactory) {
-
- super(
- (Config) RelRule.Config.EMPTY
- .withDescription("HiveFilterJoinRule(" + id + ")")
- .withOperandSupplier(b0 ->
- b0.exactly(operand))
- .as(FilterJoinRule.Config.class)
- .withSmart(smart)
- .withPredicate((join, joinType, exp) -> true)
- .withRelBuilderFactory(relBuilderFactory)
+ protected HiveFilterJoinRule(RelOptRuleOperand operand, String id){
+ super(FilterIntoJoinRule.FilterIntoJoinRuleConfig.DEFAULT
+ .withDescription("HiveFilterJoinRule(" + id + ")")
+ .withOperandSupplier(b0 -> b0.exactly(operand))
+ .withRelBuilderFactory(HiveRelFactories.HIVE_BUILDER)
+ .as(Config.class)
);
}
@@ -108,7 +99,7 @@ public boolean matches(RelOptRuleCall call) {
public static class HiveFilterJoinMergeRule extends HiveFilterJoinRule {
public HiveFilterJoinMergeRule() {
super(operand(Filter.class, operand(Join.class, any())),
- HiveFilterJoinMergeRule.class.getSimpleName(), true, HiveRelFactories.HIVE_BUILDER);
+ HiveFilterJoinMergeRule.class.getSimpleName());
}
@Override
@@ -131,7 +122,7 @@ public void onMatch(RelOptRuleCall call) {
public static class HiveFilterJoinTransposeRule extends HiveFilterJoinRule {
public HiveFilterJoinTransposeRule() {
super(RelOptRule.operand(Join.class, RelOptRule.any()),
- HiveFilterJoinTransposeRule.class.getSimpleName(), true, HiveRelFactories.HIVE_BUILDER);
+ HiveFilterJoinTransposeRule.class.getSimpleName());
}
@Override
diff --git a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveFilterSortPredicates.java b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveFilterSortPredicates.java
index 6ecf94b5f63..34961baf857 100644
--- a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveFilterSortPredicates.java
+++ b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveFilterSortPredicates.java
@@ -34,6 +34,7 @@
import org.apache.calcite.rex.RexVisitorImpl;
import org.apache.calcite.sql.SqlKind;
import org.apache.calcite.util.Pair;
+import org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveIn;
import org.apache.hadoop.hive.ql.optimizer.calcite.stats.FilterSelectivityEstimator;
import org.apache.hadoop.hive.ql.optimizer.calcite.stats.HiveRelMdSize;
import org.slf4j.Logger;
@@ -240,9 +241,6 @@ private static Double functionCost(RexCall call) {
case BETWEEN:
return 3d;
- case IN:
- return 2d * (call.getOperands().size() - 1);
-
case AND:
case OR:
return 1d * call.getOperands().size();
@@ -253,6 +251,9 @@ private static Double functionCost(RexCall call) {
return 8d;
default:
+ if (HiveIn.INSTANCE.equals(call.op)) {
+ return 2d * (call.getOperands().size() - 1);
+ }
// By default, we give this heuristic value to unrecognized functions.
// The idea is that those functions will be more expensive to evaluate
// than the simple functions considered above.
diff --git a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveInBetweenExpandRule.java b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveInBetweenExpandRule.java
index 49b67f19757..a00f0566d28 100644
--- a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveInBetweenExpandRule.java
+++ b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveInBetweenExpandRule.java
@@ -32,6 +32,8 @@
import org.apache.calcite.rex.RexUtil;
import org.apache.calcite.sql.fun.SqlStdOperatorTable;
import org.apache.hadoop.hive.ql.optimizer.calcite.HiveRelFactories;
+import org.apache.hadoop.hive.ql.optimizer.calcite.SearchTransformer;
+import org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveIn;
import org.apache.hadoop.hive.ql.optimizer.calcite.translator.RexNodeConverter;
/**
@@ -149,6 +151,9 @@ private RexInBetweenExpander(RexBuilder rexBuilder) {
@Override
public RexNode visitCall(final RexCall call) {
switch (call.getKind()) {
+ case SEARCH: {
+ return new SearchTransformer<>(rexBuilder, call).transform().accept(this);
+ }
case AND: {
boolean[] update = {false};
List<RexNode> newOperands = visitList(call.operands, update);
@@ -165,19 +170,6 @@ public RexNode visitCall(final RexCall call) {
}
return call;
}
- case IN: {
- List<RexNode> newOperands = RexNodeConverter.transformInToOrOperands(
- call.getOperands(), rexBuilder);
- if (newOperands == null) {
- // We could not execute transformation, return expression
- return call;
- }
- modified = true;
- if (newOperands.size() > 1) {
- return rexBuilder.makeCall(SqlStdOperatorTable.OR, newOperands);
- }
- return newOperands.get(0);
- }
case BETWEEN: {
List<RexNode> newOperands = RexNodeConverter.rewriteBetweenChildren(
call.getOperands(), rexBuilder);
@@ -188,6 +180,18 @@ public RexNode visitCall(final RexCall call) {
return rexBuilder.makeCall(SqlStdOperatorTable.AND, newOperands);
}
default:
+ if (HiveIn.INSTANCE.equals(call.op)) {
+ List<RexNode> newOperands = RexNodeConverter.transformInToOrOperands(call.getOperands(), rexBuilder);
+ if (newOperands == null) {
+ // We could not execute transformation, return expression
+ return call;
+ }
+ modified = true;
+ if (newOperands.size() > 1) {
+ return rexBuilder.makeCall(SqlStdOperatorTable.OR, newOperands);
+ }
+ return newOperands.get(0);
+ }
return super.visitCall(call);
}
}
diff --git a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveLoptJoinTree.java b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveLoptJoinTree.java
new file mode 100644
index 00000000000..4c07246da87
--- /dev/null
+++ b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveLoptJoinTree.java
@@ -0,0 +1,223 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.optimizer.calcite.rules;
+
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.Join;
+import org.apache.hadoop.hive.ql.optimizer.calcite.Bug;
+import org.checkerframework.checker.initialization.qual.NotOnlyInitialized;
+import org.checkerframework.checker.initialization.qual.UnderInitialization;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Objects;
+
+public class HiveLoptJoinTree {
+ //~ Instance fields --------------------------------------------------------
+
+ @NotOnlyInitialized
+ private final BinaryTree factorTree;
+ private final RelNode joinTree;
+ private final boolean removableSelfJoin;
+
+ //~ Constructors -----------------------------------------------------------
+
+ /**
+ * Creates a join-tree consisting of a single node.
+ *
+ * @param joinTree RelNode corresponding to the single node
+ * @param factorId factor id of the node
+ */
+ @SuppressWarnings("argument.type.incompatible")
+ public HiveLoptJoinTree(RelNode joinTree, int factorId) {
+ if (Bug.CALCITE_6737_FIXED) {
+ throw new IllegalStateException("This class is redundant when the fix for CALCITE-6737 is merged into Calcite. " +
+ "Remove it and use LoptOptimizeJoinRule instead of HiveLoptOptimizeJoinRule");
+ }
+
+ this.joinTree = joinTree;
+ this.factorTree = new Leaf(factorId, this);
+ this.removableSelfJoin = false;
+ }
+
+ /**
+ * Associates the factor ids with a join-tree.
+ *
+ * @param joinTree RelNodes corresponding to the join tree
+ * @param factorTree tree of the factor ids
+ * @param removableSelfJoin whether the join corresponds to a removable
+ * self-join
+ */
+ public HiveLoptJoinTree(
+ RelNode joinTree,
+ BinaryTree factorTree,
+ boolean removableSelfJoin) {
+ if (Bug.CALCITE_6737_FIXED) {
+ throw new IllegalStateException("This class is redundant when the fix for CALCITE-6737 is merged into Calcite. " +
+ "Remove it and use LoptOptimizeJoinRule instead of HiveLoptOptimizeJoinRule");
+ }
+
+ this.joinTree = joinTree;
+ this.factorTree = factorTree;
+ this.removableSelfJoin = removableSelfJoin;
+ }
+
+ /**
+ * Associates the factor ids with a join-tree given the factors corresponding
+ * to the left and right subtrees of the join.
+ *
+ * @param joinTree RelNodes corresponding to the join tree
+ * @param leftFactorTree tree of the factor ids for left subtree
+ * @param rightFactorTree tree of the factor ids for the right subtree
+ */
+ public HiveLoptJoinTree(
+ RelNode joinTree,
+ BinaryTree leftFactorTree,
+ BinaryTree rightFactorTree) {
+ this(joinTree, leftFactorTree, rightFactorTree, false);
+ }
+
+ /**
+ * Associates the factor ids with a join-tree given the factors corresponding
+ * to the left and right subtrees of the join. Also indicates whether the
+ * join is a removable self-join.
+ *
+ * @param joinTree RelNodes corresponding to the join tree
+ * @param leftFactorTree tree of the factor ids for left subtree
+ * @param rightFactorTree tree of the factor ids for the right subtree
+ * @param removableSelfJoin true if the join is a removable self-join
+ */
+ public HiveLoptJoinTree(
+ RelNode joinTree,
+ BinaryTree leftFactorTree,
+ BinaryTree rightFactorTree,
+ boolean removableSelfJoin) {
+ if (Bug.CALCITE_6737_FIXED) {
+ throw new IllegalStateException("This class is redundant when the fix for CALCITE-6737 is merged into Calcite. "
+ + "Remove it and use LoptOptimizeJoinRule instead of HiveLoptOptimizeJoinRule");
+ }
+
+ factorTree = new HiveLoptJoinTree.Node(leftFactorTree, rightFactorTree, this);
+ this.joinTree = joinTree;
+ this.removableSelfJoin = removableSelfJoin;
+ }
+
+ //~ Methods ----------------------------------------------------------------
+
+ public RelNode getJoinTree() {
+ return joinTree;
+ }
+
+ public HiveLoptJoinTree getLeft() {
+ final HiveLoptJoinTree.Node node = (HiveLoptJoinTree.Node) factorTree;
+ return new HiveLoptJoinTree(
+ ((Join) joinTree).getLeft(),
+ node.getLeft(),
+ node.getLeft().getParent().isRemovableSelfJoin());
+ }
+
+ public HiveLoptJoinTree getRight() {
+ final HiveLoptJoinTree.Node node = (HiveLoptJoinTree.Node) factorTree;
+ return new HiveLoptJoinTree(
+ ((Join) joinTree).getRight(),
+ node.getRight(),
+ node.getRight().getParent().isRemovableSelfJoin());
+ }
+
+ public BinaryTree getFactorTree() {
+ return factorTree;
+ }
+
+ public List<Integer> getTreeOrder() {
+ List<Integer> treeOrder = new ArrayList<>();
+ getTreeOrder(treeOrder);
+ return treeOrder;
+ }
+
+ public void getTreeOrder(List<Integer> treeOrder) {
+ factorTree.getTreeOrder(treeOrder);
+ }
+
+ public boolean isRemovableSelfJoin() {
+ return removableSelfJoin;
+ }
+
+ //~ Inner Classes ----------------------------------------------------------
+
+ /**
+ * Simple binary tree class that stores an id in the leaf nodes and keeps
+ * track of the parent HiveLoptJoinTree object associated with the binary tree.
+ */
+ protected abstract static class BinaryTree {
+ @NotOnlyInitialized
+ private final HiveLoptJoinTree parent;
+
+ protected BinaryTree(@UnderInitialization HiveLoptJoinTree parent) {
+ this.parent = parent;
+ }
+
+ public HiveLoptJoinTree getParent() {
+ return parent;
+ }
+
+ public abstract void getTreeOrder(List<Integer> treeOrder);
+ }
+
+ /** Binary tree node that has no children. */
+ protected static class Leaf extends BinaryTree {
+ private final int id;
+
+ public Leaf(int rootId, @UnderInitialization HiveLoptJoinTree parent) {
+ super(parent);
+ this.id = rootId;
+ }
+
+ /** Returns the id associated with a leaf node in a binary tree. */
+ public int getId() {
+ return id;
+ }
+
+ @Override public void getTreeOrder(List<Integer> treeOrder) {
+ treeOrder.add(id);
+ }
+ }
+
+ /** Binary tree node that has two children. */
+ protected static class Node extends BinaryTree {
+ private final BinaryTree left;
+ private final BinaryTree right;
+
+ public Node(BinaryTree left, BinaryTree right, @UnderInitialization HiveLoptJoinTree parent) {
+ super(parent);
+ this.left = Objects.requireNonNull(left, "left");
+ this.right = Objects.requireNonNull(right, "right");
+ }
+
+ public BinaryTree getLeft() {
+ return left;
+ }
+
+ public BinaryTree getRight() {
+ return right;
+ }
+
+ @Override public void getTreeOrder(List<Integer> treeOrder) {
+ left.getTreeOrder(treeOrder);
+ right.getTreeOrder(treeOrder);
+ }
+ }
+}
diff --git a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveLoptMultiJoin.java b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveLoptMultiJoin.java
new file mode 100644
index 00000000000..912ceb60921
--- /dev/null
+++ b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveLoptMultiJoin.java
@@ -0,0 +1,871 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.optimizer.calcite.rules;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.Lists;
+import org.apache.calcite.plan.RelOptUtil;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.JoinRelType;
+import org.apache.calcite.rel.logical.LogicalJoin;
+import org.apache.calcite.rel.metadata.RelColumnOrigin;
+import org.apache.calcite.rel.metadata.RelMetadataQuery;
+import org.apache.calcite.rel.rules.MultiJoin;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.calcite.rel.type.RelDataTypeField;
+import org.apache.calcite.rex.RexCall;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.util.BitSets;
+import org.apache.calcite.util.ImmutableBitSet;
+import org.apache.calcite.util.ImmutableIntList;
+import org.apache.hadoop.hive.ql.optimizer.calcite.Bug;
+import org.checkerframework.checker.initialization.qual.UnderInitialization;
+import org.checkerframework.checker.initialization.qual.UnknownInitialization;
+import org.checkerframework.checker.nullness.qual.MonotonicNonNull;
+import org.checkerframework.checker.nullness.qual.Nullable;
+import org.checkerframework.checker.nullness.qual.RequiresNonNull;
+
+import java.util.ArrayList;
+import java.util.BitSet;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.ListIterator;
+import java.util.Map;
+import java.util.Set;
+
+import static java.util.Objects.requireNonNull;
+
+public class HiveLoptMultiJoin {
+ //~ Instance fields --------------------------------------------------------
+
+ /** The MultiJoin being optimized. */
+ MultiJoin multiJoin;
+
+ /**
+ * Join filters associated with the MultiJoin, decomposed into a list.
+ * Excludes left/right outer join filters.
+ */
+ private List<RexNode> joinFilters;
+
+ /**
+ * All join filters associated with the MultiJoin, decomposed into a
+ * list. Includes left/right outer join filters.
+ */
+ private List<RexNode> allJoinFilters;
+
+ /** Number of factors into the MultiJoin. */
+ private final int nJoinFactors;
+
+ /** Total number of fields in the MultiJoin. */
+ private int nTotalFields;
+
+ /** Original inputs into the MultiJoin. */
+ private final ImmutableList<RelNode> joinFactors;
+
+ /**
+ * If a join factor is null-generating in a left or right outer join,
+ * joinTypes indicates the join type corresponding to the factor. Otherwise,
+ * it is set to INNER.
+ */
+ private final ImmutableList<JoinRelType> joinTypes;
+
+ /**
+ * If a join factor is null-generating in a left or right outer join, the
+ * bitmap contains the non-null generating factors that the null-generating
+ * factor is dependent upon.
+ */
+ private final ImmutableBitSet[] outerJoinFactors;
+
+ /**
+ * Bitmap corresponding to the fields projected from each join factor, after
+ * row scan processing has completed. This excludes fields referenced in
+ * join conditions, unless the field appears in the final projection list.
+ */
+ private List<@Nullable ImmutableBitSet> projFields;
+
+ /**
+ * Map containing reference counts of the fields referenced in join
+ * conditions for each join factor. If a field is only required for a
+ * semijoin, then it is removed from the reference count. (Hence the need
+ * for reference counts instead of simply a bitmap.) The map is indexed by
+ * the factor number.
+ */
+ private Map<Integer, int[]> joinFieldRefCountsMap;
+
+ /**
+ * For each join filter, associates a bitmap indicating all factors
+ * referenced by the filter.
+ */
+ private final Map<RexNode, ImmutableBitSet> factorsRefByJoinFilter = new HashMap<>();
+
+ /**
+ * For each join filter, associates a bitmap indicating all fields
+ * referenced by the filter.
+ */
+ private final Map<RexNode, ImmutableBitSet> fieldsRefByJoinFilter = new HashMap<>();
+
+ /**
+ * Starting RexInputRef index corresponding to each join factor.
+ */
+ int [] joinStart;
+
+ /**
+ * Number of fields in each join factor.
+ */
+ int [] nFieldsInJoinFactor;
+
+ /**
+ * Bitmap indicating which factors each factor references in join filters
+ * that correspond to comparisons.
+ */
+ ImmutableBitSet @MonotonicNonNull [] factorsRefByFactor;
+
+ /**
+ * Weights of each factor combination.
+ */
+ int @MonotonicNonNull [][] factorWeights;
+
+ /**
+ * Type factory.
+ */
+ final RelDataTypeFactory factory;
+
+ /**
+ * Indicates for each factor whether its join can be removed because it is
+ * the dimension table in a semijoin. If it can be, the entry indicates the
+ * factor id of the fact table (corresponding to the dimension table) in the
+ * semijoin that allows the factor to be removed. If the factor cannot be
+ * removed, the entry corresponding to the factor is null.
+ */
+ @Nullable Integer [] joinRemovalFactors;
+
+ /**
+ * The semijoins that allow the join of a dimension table to be removed.
+ */
+ LogicalJoin[] joinRemovalSemiJoins;
+
+ /**
+ * Set of null-generating factors whose corresponding outer join can be
+ * removed from the query plan.
+ */
+ Set<Integer> removableOuterJoinFactors;
+
+ /**
+ * Map consisting of all pairs of self-joins where the self-join can be
+ * removed because the join between the identical factors is an equality
+ * join on the same set of unique keys. The map is keyed by either factor in
+ * the self join.
+ */
+ Map<Integer, HiveLoptMultiJoin.RemovableSelfJoin> removableSelfJoinPairs;
+
+ //~ Constructors -----------------------------------------------------------
+
+ public HiveLoptMultiJoin(MultiJoin multiJoin) {
+ if (Bug.CALCITE_6737_FIXED) {
+ throw new IllegalStateException("This class is redundant when the fix for CALCITE-6737 is merged into Calcite. "
+ + "Remove it and use LoptOptimizeJoinRule instead of HiveLoptOptimizeJoinRule");
+ }
+
+ this.multiJoin = multiJoin;
+ joinFactors = ImmutableList.copyOf(multiJoin.getInputs());
+ nJoinFactors = joinFactors.size();
+ projFields = multiJoin.getProjFields();
+ joinFieldRefCountsMap = multiJoin.getCopyJoinFieldRefCountsMap();
+
+ joinFilters =
+ Lists.newArrayList(RelOptUtil.conjunctions(multiJoin.getJoinFilter()));
+
+ allJoinFilters = new ArrayList<>(joinFilters);
+ List<@Nullable RexNode> outerJoinFilters = multiJoin.getOuterJoinConditions();
+ for (int i = 0; i < nJoinFactors; i++) {
+ allJoinFilters.addAll(RelOptUtil.conjunctions(outerJoinFilters.get(i)));
+ }
+
+ int start = 0;
+ nTotalFields = multiJoin.getRowType().getFieldCount();
+ joinStart = new int[nJoinFactors];
+ nFieldsInJoinFactor = new int[nJoinFactors];
+ for (int i = 0; i < nJoinFactors; i++) {
+ joinStart[i] = start;
+ nFieldsInJoinFactor[i] =
+ joinFactors.get(i).getRowType().getFieldCount();
+ start += nFieldsInJoinFactor[i];
+ }
+
+ // Extract outer join information from the join factors, including the type
+ // of outer join and the factors that a null-generating factor is dependent
+ // upon.
+ joinTypes = ImmutableList.copyOf(multiJoin.getJoinTypes());
+ List<@Nullable RexNode> outerJoinConds = this.multiJoin.getOuterJoinConditions();
+ outerJoinFactors = new ImmutableBitSet[nJoinFactors];
+ for (int i = 0; i < nJoinFactors; i++) {
+ RexNode outerJoinCond = outerJoinConds.get(i);
+ if (outerJoinCond != null) {
+ // set a bitmap containing the factors referenced in the
+ // ON condition of the outer join; mask off the factor
+ // corresponding to the factor itself
+ ImmutableBitSet dependentFactors =
+ getJoinFilterFactorBitmap(outerJoinCond, false);
+ dependentFactors = dependentFactors.clear(i);
+ outerJoinFactors[i] = dependentFactors;
+ }
+ }
+
+ // determine which join factors each join filter references
+ setJoinFilterRefs();
+
+ factory = multiJoin.getCluster().getTypeFactory();
+
+ joinRemovalFactors = new Integer[nJoinFactors];
+ joinRemovalSemiJoins = new LogicalJoin[nJoinFactors];
+
+ removableOuterJoinFactors = new HashSet<>();
+ removableSelfJoinPairs = new HashMap<>();
+ }
+
+ //~ Methods ----------------------------------------------------------------
+
+ /**
+ * Returns the MultiJoin corresponding to this multi-join.
+ */
+ public MultiJoin getMultiJoinRel() {
+ return multiJoin;
+ }
+
+ /**
+ * Returns the number of factors in this multi-join.
+ */
+ public int getNumJoinFactors() {
+ return nJoinFactors;
+ }
+
+ /**
+ * Returns the factor corresponding to the given factor index.
+ *
+ * @param factIdx Factor to be returned
+ */
+ public RelNode getJoinFactor(int factIdx) {
+ return joinFactors.get(factIdx);
+ }
+
+ /**
+ * Returns the total number of fields in the multi-join.
+ */
+ public int getNumTotalFields() {
+ return nTotalFields;
+ }
+
+ /**
+ * Returns the number of fields in a given factor.
+ *
+ * @param factIdx Desired factor
+ */
+ public int getNumFieldsInJoinFactor(int factIdx) {
+ return nFieldsInJoinFactor[factIdx];
+ }
+
+ /**
+ * Returns all non-outer join filters in this multi-join.
+ */
+ public List<RexNode> getJoinFilters() {
+ return joinFilters;
+ }
+
+ /**
+ * Returns a bitmap corresponding to the factors referenced within
+ * the specified join filter.
+ *
+ * @param joinFilter Filter for which information will be returned
+ */
+ public ImmutableBitSet getFactorsRefByJoinFilter(RexNode joinFilter) {
+ return requireNonNull(
+ factorsRefByJoinFilter.get(joinFilter),
+ () -> "joinFilter is not found in factorsRefByJoinFilter: " + joinFilter);
+ }
+
+ /**
+ * Returns an array of fields contained within the multi-join.
+ */
+ public List<RelDataTypeField> getMultiJoinFields() {
+ return multiJoin.getRowType().getFieldList();
+ }
+
+ /**
+ * Returns a bitmap corresponding to the fields referenced by a join filter.
+ *
+ * @param joinFilter the filter for which information will be returned
+ */
+ public ImmutableBitSet getFieldsRefByJoinFilter(RexNode joinFilter) {
+ return requireNonNull(
+ fieldsRefByJoinFilter.get(joinFilter),
+ () -> "joinFilter is not found in fieldsRefByJoinFilter: " + joinFilter);
+ }
+
+ /**
+ * Returns weights of the different factors relative to one another.
+ */
+ public int @Nullable [][] getFactorWeights() {
+ return factorWeights;
+ }
+
+ /**
+ * Returns a bitmap corresponding to the factors referenced by the specified
+ * factor in the various join filters that correspond to comparisons.
+ *
+ * @param factIdx Factor for which information will be returned
+ */
+ public ImmutableBitSet getFactorsRefByFactor(int factIdx) {
+ return requireNonNull(factorsRefByFactor, "factorsRefByFactor")[factIdx];
+ }
+
+ /**
+ * Returns the starting offset within the multi-join for the specified factor.
+ *
+ * @param factIdx Factor for which information will be returned
+ */
+ public int getJoinStart(int factIdx) {
+ return joinStart[factIdx];
+ }
+
+ /**
+ * Returns whether the factor corresponds to a null-generating factor
+ * in a left or right outer join.
+ *
+ * @param factIdx Factor for which information will be returned
+ */
+ public boolean isNullGenerating(int factIdx) {
+ return joinTypes.get(factIdx).isOuterJoin();
+ }
+
+ /**
+ * Returns a bitmap containing the factors that a null-generating factor is
+ * dependent upon, if the factor is null-generating in a left or right outer
+ * join; otherwise null is returned.
+ *
+ * @param factIdx Factor for which information will be returned
+ */
+ public ImmutableBitSet getOuterJoinFactors(int factIdx) {
+ return outerJoinFactors[factIdx];
+ }
+
+ /**
+ * Returns outer join conditions associated with the specified null-generating
+ * factor.
+ *
+ * @param factIdx Factor for which information will be returned
+ */
+ public @Nullable RexNode getOuterJoinCond(int factIdx) {
+ return multiJoin.getOuterJoinConditions().get(factIdx);
+ }
+
+ /**
+ * Returns a bitmap containing the fields that are projected from a factor.
+ *
+ * @param factIdx Factor for which information will be returned
+ */
+ public @Nullable ImmutableBitSet getProjFields(int factIdx) {
+ return projFields.get(factIdx);
+ }
+
+ /**
+ * Returns the join field reference counts for a factor.
+ *
+ * @param factIdx Factor for which information will be returned
+ */
+ public int [] getJoinFieldRefCounts(int factIdx) {
+ return requireNonNull(
+ joinFieldRefCountsMap.get(factIdx),
+ () -> "no entry in joinFieldRefCountsMap found for " + factIdx);
+ }
+
+ /**
+ * Returns the factor id of the fact table corresponding to a dimension
+ * table in a semi-join, in the case where the join with the dimension table
+ * can be removed.
+ *
+ * @param dimIdx Dimension factor for which information will be returned
+ */
+ public @Nullable Integer getJoinRemovalFactor(int dimIdx) {
+ return joinRemovalFactors[dimIdx];
+ }
+
+ /**
+ * Returns the semi-join that allows the join of a dimension table to be
+ * removed.
+ *
+ * @param dimIdx Dimension factor for which information will be returned
+ */
+ public LogicalJoin getJoinRemovalSemiJoin(int dimIdx) {
+ return joinRemovalSemiJoins[dimIdx];
+ }
+
+ /**
+ * Indicates that a dimension factor's join can be removed because of a
+ * semijoin with a fact table.
+ *
+ * @param dimIdx Dimension factor
+ * @param factIdx Fact factor
+ */
+ public void setJoinRemovalFactor(int dimIdx, int factIdx) {
+ joinRemovalFactors[dimIdx] = factIdx;
+ }
+
+ /**
+ * Indicates the semi-join that allows the join of a dimension table to be
+ * removed.
+ *
+ * @param dimIdx Dimension factor
+ * @param semiJoin the semijoin
+ */
+ public void setJoinRemovalSemiJoin(int dimIdx, LogicalJoin semiJoin) {
+ joinRemovalSemiJoins[dimIdx] = semiJoin;
+ }
+
+ /**
+ * Returns a bitmap representing the factors referenced in a join filter.
+ *
+ * @param joinFilter the join filter
+ * @param setFields if true, add the fields referenced by the join filter
+ * into a map
+ *
+ * @return the bitmap containing the factor references
+ */
+ @RequiresNonNull({"joinStart", "nFieldsInJoinFactor"})
+ ImmutableBitSet getJoinFilterFactorBitmap(
+ @UnderInitialization HiveLoptMultiJoin this,
+ RexNode joinFilter,
+ boolean setFields) {
+ ImmutableBitSet fieldRefBitmap = fieldBitmap(joinFilter);
+ if (setFields) {
+ fieldsRefByJoinFilter.put(joinFilter, fieldRefBitmap);
+ }
+
+ return factorBitmap(fieldRefBitmap);
+ }
+
+ private static ImmutableBitSet fieldBitmap(RexNode joinFilter) {
+ final RelOptUtil.InputFinder inputFinder = new RelOptUtil.InputFinder();
+ joinFilter.accept(inputFinder);
+ return inputFinder.build();
+ }
+
+ /**
+ * Sets bitmaps indicating which factors and fields each join filter
+ * references.
+ */
+ @RequiresNonNull({"allJoinFilters", "joinStart", "nFieldsInJoinFactor"})
+ private void setJoinFilterRefs(
+ @UnderInitialization HiveLoptMultiJoin this) {
+ ListIterator<RexNode> filterIter = allJoinFilters.listIterator();
+ while (filterIter.hasNext()) {
+ RexNode joinFilter = filterIter.next();
+
+ // ignore the literal filter; if necessary, we'll add it back
+ // later
+ if (joinFilter.isAlwaysTrue()) {
+ filterIter.remove();
+ }
+ ImmutableBitSet factorRefBitmap =
+ getJoinFilterFactorBitmap(joinFilter, true);
+ factorsRefByJoinFilter.put(joinFilter, factorRefBitmap);
+ }
+ }
+
+ /**
+ * Sets the bitmap indicating which factors a filter references based on
+ * which fields it references.
+ *
+ * @param fieldRefBitmap bitmap representing fields referenced
+ * @return bitmap representing factors referenced that will
+ * be set by this method
+ */
+ @RequiresNonNull({"joinStart", "nFieldsInJoinFactor"})
+ private ImmutableBitSet factorBitmap(
+ @UnknownInitialization HiveLoptMultiJoin this,
+ ImmutableBitSet fieldRefBitmap) {
+ ImmutableBitSet.Builder factorRefBitmap = ImmutableBitSet.builder();
+ for (int field : fieldRefBitmap) {
+ int factor = findRef(field);
+ factorRefBitmap.set(factor);
+ }
+ return factorRefBitmap.build();
+ }
+
+ /**
+ * Determines the join factor corresponding to a RexInputRef.
+ *
+ * @param rexInputRef rexInputRef index
+ *
+ * @return index corresponding to join factor
+ */
+ @RequiresNonNull({"joinStart", "nFieldsInJoinFactor"})
+ public int findRef(
+ @UnknownInitialization HiveLoptMultiJoin this,
+ int rexInputRef) {
+ for (int i = 0; i < nJoinFactors; i++) {
+ if ((rexInputRef >= joinStart[i])
+ && (rexInputRef < (joinStart[i] + nFieldsInJoinFactor[i]))) {
+ return i;
+ }
+ }
+ throw new AssertionError();
+ }
+
+ /**
+ * Sets weighting for each combination of factors, depending on which join
+ * filters reference which factors. Greater weight is given to equality
+ * conditions. Also, sets bitmaps indicating which factors are referenced by
+ * each factor within join filters that are comparisons.
+ */
+ public void setFactorWeights() {
+ factorWeights = new int[nJoinFactors][nJoinFactors];
+ factorsRefByFactor = new ImmutableBitSet[nJoinFactors];
+ for (int i = 0; i < nJoinFactors; i++) {
+ factorsRefByFactor[i] = ImmutableBitSet.of();
+ }
+
+ for (RexNode joinFilter : allJoinFilters) {
+ ImmutableBitSet factorRefs = factorsRefByJoinFilter.get(joinFilter);
+
+ // don't give weights to non-comparison expressions
+ if (!(joinFilter instanceof RexCall)) {
+ continue;
+ }
+ if (!joinFilter.isA(SqlKind.COMPARISON)) {
+ continue;
+ }
+
+ // OR the factors referenced in this join filter into the
+ // bitmaps corresponding to each of the factors; however,
+ // exclude the bit corresponding to the factor itself
+ for (int factor : requireNonNull(factorRefs, "factorRefs")) {
+ factorsRefByFactor[factor] =
+ factorsRefByFactor[factor]
+ .rebuild()
+ .addAll(factorRefs)
+ .clear(factor)
+ .build();
+ }
+
+ if (factorRefs.cardinality() == 2) {
+ int leftFactor = factorRefs.nextSetBit(0);
+ int rightFactor = factorRefs.nextSetBit(leftFactor + 1);
+
+ final RexCall call = (RexCall) joinFilter;
+ ImmutableBitSet leftFields = fieldBitmap(call.getOperands().get(0));
+ ImmutableBitSet leftBitmap = factorBitmap(leftFields);
+
+ // filter contains only two factor references, one on each
+ // side of the operator
+ int weight;
+ if (leftBitmap.cardinality() == 1) {
+ // give higher weight to equi-joins
+ switch (joinFilter.getKind()) {
+ case EQUALS:
+ weight = 3;
+ break;
+ default:
+ weight = 2;
+ }
+ } else {
+ // cross product of two tables
+ weight = 1;
+ }
+ setFactorWeight(weight, leftFactor, rightFactor);
+ } else {
+ // multiple factor references -- set a weight for each
+ // combination of factors referenced within the filter
+ final List<Integer> list = ImmutableIntList.copyOf(factorRefs);
+ for (int outer : list) {
+ for (int inner : list) {
+ if (outer != inner) {
+ setFactorWeight(1, outer, inner);
+ }
+ }
+ }
+ }
+ }
+ }
+
+ /**
+ * Sets an individual weight if the new weight is better than the current
+ * one.
+ *
+ * @param weight weight to be set
+ * @param leftFactor index of left factor
+ * @param rightFactor index of right factor
+ */
+ @RequiresNonNull("factorWeights")
+ private void setFactorWeight(int weight, int leftFactor, int rightFactor) {
+ if (factorWeights[leftFactor][rightFactor] < weight) {
+ factorWeights[leftFactor][rightFactor] = weight;
+ factorWeights[rightFactor][leftFactor] = weight;
+ }
+ }
+
+ /**
+ * Returns whether if a join tree contains all factors required.
+ *
+ * @param joinTree Join tree to be examined
+ * @param factorsNeeded Bitmap of factors required
+ *
+ * @return true if join tree contains all required factors
+ */
+ public boolean hasAllFactors(
+ HiveLoptJoinTree joinTree,
+ BitSet factorsNeeded) {
+ return BitSets.contains(BitSets.of(joinTree.getTreeOrder()), factorsNeeded);
+ }
+
+ /**
+ * Sets a bitmap indicating all child RelNodes in a join tree.
+ *
+ * @param joinTree join tree to be examined
+ * @param childFactors bitmap to be set
+ */
+ @Deprecated // to be removed before 2.0
+ public void getChildFactors(HiveLoptJoinTree joinTree,
+ ImmutableBitSet.Builder childFactors) {
+ for (int child : joinTree.getTreeOrder()) {
+ childFactors.set(child);
+ }
+ }
+
+ /**
+ * Retrieves the fields corresponding to a join between a left and right
+ * tree.
+ *
+ * @param left left hand side of the join
+ * @param right right hand side of the join
+ *
+ * @return fields of the join
+ */
+ public List<RelDataTypeField> getJoinFields(
+ HiveLoptJoinTree left,
+ HiveLoptJoinTree right) {
+ RelDataType rowType =
+ factory.createJoinType(
+ left.getJoinTree().getRowType(),
+ right.getJoinTree().getRowType());
+ return rowType.getFieldList();
+ }
+
+ /**
+ * Adds a join factor to the set of factors that can be removed because the
+ * factor is the null-generating factor in an outer join, its join keys are
+ * unique, and the factor is not projected in the query.
+ *
+ * @param factIdx Join factor
+ */
+ public void addRemovableOuterJoinFactor(int factIdx) {
+ removableOuterJoinFactors.add(factIdx);
+ }
+
+ /**
+ * Return whether the factor corresponds to the null-generating factor in
+ * an outer join that can be removed.
+ *
+ * @param factIdx Factor in question
+ */
+ public boolean isRemovableOuterJoinFactor(int factIdx) {
+ return removableOuterJoinFactors.contains(factIdx);
+ }
+
+ /**
+ * Adds to a map that keeps track of removable self-join pairs.
+ *
+ * @param factor1 one of the factors in the self-join
+ * @param factor2 the second factor in the self-join
+ */
+ public void addRemovableSelfJoinPair(int factor1, int factor2) {
+ int leftFactor;
+ int rightFactor;
+
+ // Put the factor with more fields on the left so it will be
+ // preserved after the self-join is removed.
+ if (getNumFieldsInJoinFactor(factor1)
+ > getNumFieldsInJoinFactor(factor2)) {
+ leftFactor = factor1;
+ rightFactor = factor2;
+ } else {
+ leftFactor = factor2;
+ rightFactor = factor1;
+ }
+
+ // Compute a column mapping such that if a column from the right
+ // factor is also referenced in the left factor, we will map the
+ // right reference to the left to avoid redundant references.
+ final Map<Integer, Integer> columnMapping = new HashMap<>();
+
+ // First, locate the originating column for all simple column
+ // references in the left factor.
+ final RelNode left = getJoinFactor(leftFactor);
+ final RelMetadataQuery mq = left.getCluster().getMetadataQuery();
+ final Map<Integer, Integer> leftFactorColMapping = new HashMap<>();
+ for (int i = 0; i < left.getRowType().getFieldCount(); i++) {
+ final RelColumnOrigin colOrigin = mq.getColumnOrigin(left, i);
+ if (colOrigin != null && !colOrigin.isDerived()) {
+ leftFactorColMapping.put(
+ colOrigin.getOriginColumnOrdinal(),
+ i);
+ }
+ }
+
+ // Then, see if the right factor references any of the same columns
+ // by locating their originating columns. If there are matches,
+ // then we want to store the corresponding offset into the left
+ // factor.
+ RelNode right = getJoinFactor(rightFactor);
+ for (int i = 0; i < right.getRowType().getFieldCount(); i++) {
+ final RelColumnOrigin colOrigin = mq.getColumnOrigin(right, i);
+ if (colOrigin == null || colOrigin.isDerived()) {
+ continue;
+ }
+ Integer leftOffset =
+ leftFactorColMapping.get(colOrigin.getOriginColumnOrdinal());
+ if (leftOffset == null) {
+ continue;
+ }
+ columnMapping.put(i, leftOffset);
+ }
+
+ HiveLoptMultiJoin.RemovableSelfJoin selfJoin =
+ new HiveLoptMultiJoin.RemovableSelfJoin(leftFactor, rightFactor, columnMapping);
+
+ removableSelfJoinPairs.put(leftFactor, selfJoin);
+ removableSelfJoinPairs.put(rightFactor, selfJoin);
+ }
+
+ /**
+ * Returns the other factor in a self-join pair if the factor passed in is
+ * a factor in a removable self-join; otherwise, returns null.
+ *
+ * @param factIdx one of the factors in a self-join pair
+ */
+ public @Nullable Integer getOtherSelfJoinFactor(int factIdx) {
+ HiveLoptMultiJoin.RemovableSelfJoin selfJoin = removableSelfJoinPairs.get(factIdx);
+ if (selfJoin == null) {
+ return null;
+ } else if (selfJoin.rightFactor == factIdx) {
+ return selfJoin.leftFactor;
+ } else {
+ return selfJoin.rightFactor;
+ }
+ }
+
+ /**
+ * Returns whether the factor is the left factor in a self-join.
+ *
+ * @param factIdx Factor in a self-join
+ */
+ public boolean isLeftFactorInRemovableSelfJoin(int factIdx) {
+ HiveLoptMultiJoin.RemovableSelfJoin selfJoin = removableSelfJoinPairs.get(factIdx);
+ if (selfJoin == null) {
+ return false;
+ }
+ return selfJoin.leftFactor == factIdx;
+ }
+
+ /**
+ * Returns whether the factor is the right factor in a self-join.
+ *
+ * @param factIdx Factor in a self-join
+ */
+ public boolean isRightFactorInRemovableSelfJoin(int factIdx) {
+ HiveLoptMultiJoin.RemovableSelfJoin selfJoin = removableSelfJoinPairs.get(factIdx);
+ if (selfJoin == null) {
+ return false;
+ }
+ return selfJoin.rightFactor == factIdx;
+ }
+
+ /**
+ * Determines whether there is a mapping from a column in the right factor
+ * of a self-join to a column from the left factor. Assumes that the right
+ * factor is a part of a self-join.
+ *
+ * @param rightFactor the index of the right factor
+ * @param rightOffset the column offset of the right factor
+ *
+ * @return the offset of the corresponding column in the left factor, if
+ * such a column mapping exists; otherwise, null is returned
+ */
+ public @Nullable Integer getRightColumnMapping(int rightFactor, int rightOffset) {
+ HiveLoptMultiJoin.RemovableSelfJoin selfJoin = requireNonNull(removableSelfJoinPairs.get(rightFactor),
+ () -> "removableSelfJoinPairs.get(rightFactor) is null for " + rightFactor
+ + ", map=" + removableSelfJoinPairs);
+ assert selfJoin.rightFactor == rightFactor;
+ return selfJoin.columnMapping.get(rightOffset);
+ }
+
+ public HiveLoptMultiJoin.Edge createEdge(RexNode condition) {
+ ImmutableBitSet fieldRefBitmap = fieldBitmap(condition);
+ ImmutableBitSet factorRefBitmap = factorBitmap(fieldRefBitmap);
+ return new HiveLoptMultiJoin.Edge(condition, factorRefBitmap, fieldRefBitmap);
+ }
+
+ /** Information about a join-condition. */
+ static class Edge {
+ final ImmutableBitSet factors;
+ final ImmutableBitSet columns;
+ final RexNode condition;
+
+ Edge(RexNode condition, ImmutableBitSet factors, ImmutableBitSet columns) {
+ this.condition = condition;
+ this.factors = factors;
+ this.columns = columns;
+ }
+
+ @Override public String toString() {
+ return "Edge(condition: " + condition
+ + ", factors: " + factors
+ + ", columns: " + columns + ")";
+ }
+ }
+
+ //~ Inner Classes ----------------------------------------------------------
+
+ /**
+ * Utility class used to keep track of the factors in a removable self-join.
+ * The right factor in the self-join is the one that will be removed.
+ */
+ private static class RemovableSelfJoin {
+ /** The left factor in a removable self-join. */
+ private final int leftFactor;
+
+ /** The right factor in a removable self-join, namely the factor that will
+ * be removed. */
+ private final int rightFactor;
+
+ /** A mapping that maps references to columns from the right factor to
+ * columns in the left factor, if the column is referenced in both
+ * factors. */
+ private final Map<Integer, Integer> columnMapping;
+
+ RemovableSelfJoin(
+ int leftFactor,
+ int rightFactor,
+ Map<Integer, Integer> columnMapping) {
+ this.leftFactor = leftFactor;
+ this.rightFactor = rightFactor;
+ this.columnMapping = columnMapping;
+ }
+ }
+}
diff --git a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveLoptOptimizeJoinRule.java b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveLoptOptimizeJoinRule.java
new file mode 100644
index 00000000000..09a3dee8bf1
--- /dev/null
+++ b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveLoptOptimizeJoinRule.java
@@ -0,0 +1,2082 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.optimizer.calcite.rules;
+
+import org.apache.calcite.plan.RelOptCost;
+import org.apache.calcite.plan.RelOptRule;
+import org.apache.calcite.plan.RelOptRuleCall;
+import org.apache.calcite.plan.RelOptTable;
+import org.apache.calcite.plan.RelOptUtil;
+import org.apache.calcite.plan.RelRule;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.Join;
+import org.apache.calcite.rel.core.JoinInfo;
+import org.apache.calcite.rel.core.JoinRelType;
+import org.apache.calcite.rel.logical.LogicalJoin;
+import org.apache.calcite.rel.metadata.RelColumnOrigin;
+import org.apache.calcite.rel.metadata.RelMdUtil;
+import org.apache.calcite.rel.metadata.RelMetadataQuery;
+import org.apache.calcite.rel.rules.MultiJoin;
+import org.apache.calcite.rel.rules.TransformationRule;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.calcite.rel.type.RelDataTypeField;
+import org.apache.calcite.rex.RexBuilder;
+import org.apache.calcite.rex.RexCall;
+import org.apache.calcite.rex.RexInputRef;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.rex.RexUtil;
+import org.apache.calcite.sql.fun.SqlStdOperatorTable;
+import org.apache.calcite.tools.RelBuilder;
+import org.apache.calcite.util.BitSets;
+import org.apache.calcite.util.ImmutableBitSet;
+import org.apache.calcite.util.ImmutableIntList;
+import org.apache.calcite.util.Pair;
+import org.apache.calcite.util.mapping.IntPair;
+import org.apache.hadoop.hive.ql.optimizer.calcite.Bug;
+import org.apache.hadoop.hive.ql.optimizer.calcite.HiveRelFactories;
+import org.checkerframework.checker.nullness.qual.KeyFor;
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+import java.util.ArrayList;
+import java.util.BitSet;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.ListIterator;
+import java.util.Map;
+import java.util.Set;
+import java.util.TreeSet;
+
+import static java.util.Objects.requireNonNull;
+
+public class HiveLoptOptimizeJoinRule extends RelRule<HiveLoptOptimizeJoinRule.Config>
+ implements TransformationRule {
+
+ /** Creates an HiveLoptOptimizeJoinRule. */
+ protected HiveLoptOptimizeJoinRule(HiveLoptOptimizeJoinRule.Config config) {
+ super(config);
+
+ if (Bug.CALCITE_6737_FIXED) {
+ throw new IllegalStateException("This class is redundant when the fix for CALCITE-6737 is merged into Calcite. "
+ + "Remove it and use LoptOptimizeJoinRule instead of HiveLoptOptimizeJoinRule");
+ }
+ }
+
+ //~ Methods ----------------------------------------------------------------
+
+ @Override public void onMatch(RelOptRuleCall call) {
+ final MultiJoin multiJoinRel = call.rel(0);
+ final HiveLoptMultiJoin multiJoin = new HiveLoptMultiJoin(multiJoinRel);
+ final RelMetadataQuery mq = call.getMetadataQuery();
+
+ findRemovableOuterJoins(mq, multiJoin);
+
+ final RexBuilder rexBuilder = multiJoinRel.getCluster().getRexBuilder();
+ final HiveLoptSemiJoinOptimizer semiJoinOpt =
+ new HiveLoptSemiJoinOptimizer(call.getMetadataQuery(), multiJoin, rexBuilder);
+
+ // determine all possible semijoins
+ semiJoinOpt.makePossibleSemiJoins(multiJoin);
+
+ // select the optimal join filters for semijoin filtering by
+ // iteratively calling chooseBestSemiJoin; chooseBestSemiJoin will
+ // apply semijoins in sort order, based on the cost of scanning each
+ // factor; as it selects semijoins to apply and iterates through the
+ // loop, the cost of scanning a factor will decrease in accordance
+ // with the semijoins selected
+ int iterations = 0;
+ do {
+ if (!semiJoinOpt.chooseBestSemiJoin(multiJoin)) {
+ break;
+ }
+ if (iterations++ > 10) {
+ break;
+ }
+ } while (true);
+
+ multiJoin.setFactorWeights();
+
+ findRemovableSelfJoins(mq, multiJoin);
+
+ findBestOrderings(mq, call.builder(), multiJoin, semiJoinOpt, call);
+ }
+
+ /**
+ * Locates all null generating factors whose outer join can be removed. The
+ * outer join can be removed if the join keys corresponding to the null
+ * generating factor are unique and no columns are projected from it.
+ *
+ * @param multiJoin join factors being optimized
+ */
+ private static void findRemovableOuterJoins(RelMetadataQuery mq, HiveLoptMultiJoin multiJoin) {
+ final List<Integer> removalCandidates = new ArrayList<>();
+ for (int factIdx = 0;
+ factIdx < multiJoin.getNumJoinFactors();
+ factIdx++) {
+ if (multiJoin.isNullGenerating(factIdx)) {
+ removalCandidates.add(factIdx);
+ }
+ }
+
+ while (!removalCandidates.isEmpty()) {
+ final Set<Integer> retryCandidates = new HashSet<>();
+
+ outerForLoop:
+ for (int factIdx : removalCandidates) {
+ // reject the factor if it is referenced in the projection list
+ ImmutableBitSet projFields = multiJoin.getProjFields(factIdx);
+ if ((projFields == null) || (projFields.cardinality() > 0)) {
+ continue;
+ }
+
+ // setup a bitmap containing the equi-join keys corresponding to
+ // the null generating factor; both operands in the filter must
+ // be RexInputRefs and only one side corresponds to the null
+ // generating factor
+ RexNode outerJoinCond = multiJoin.getOuterJoinCond(factIdx);
+ final List<RexNode> ojFilters = new ArrayList<>();
+ RelOptUtil.decomposeConjunction(outerJoinCond, ojFilters);
+ int numFields = multiJoin.getNumFieldsInJoinFactor(factIdx);
+ final ImmutableBitSet.Builder joinKeyBuilder =
+ ImmutableBitSet.builder();
+ final ImmutableBitSet.Builder otherJoinKeyBuilder =
+ ImmutableBitSet.builder();
+ int firstFieldNum = multiJoin.getJoinStart(factIdx);
+ int lastFieldNum = firstFieldNum + numFields;
+ for (RexNode filter : ojFilters) {
+ if (!(filter instanceof RexCall)) {
+ continue;
+ }
+ RexCall filterCall = (RexCall) filter;
+ if ((filterCall.getOperator() != SqlStdOperatorTable.EQUALS)
+ || !(filterCall.getOperands().get(0)
+ instanceof RexInputRef)
+ || !(filterCall.getOperands().get(1)
+ instanceof RexInputRef)) {
+ continue;
+ }
+ int leftRef =
+ ((RexInputRef) filterCall.getOperands().get(0)).getIndex();
+ int rightRef =
+ ((RexInputRef) filterCall.getOperands().get(1)).getIndex();
+ setJoinKey(
+ joinKeyBuilder,
+ otherJoinKeyBuilder,
+ leftRef,
+ rightRef,
+ firstFieldNum,
+ lastFieldNum,
+ true);
+ }
+
+ if (joinKeyBuilder.cardinality() == 0) {
+ continue;
+ }
+
+ // make sure the only join fields referenced are the ones in
+ // the current outer join
+ final ImmutableBitSet joinKeys = joinKeyBuilder.build();
+ int [] joinFieldRefCounts =
+ multiJoin.getJoinFieldRefCounts(factIdx);
+ for (int i = 0; i < joinFieldRefCounts.length; i++) {
+ if ((joinFieldRefCounts[i] > 1)
+ || (!joinKeys.get(i) && (joinFieldRefCounts[i] == 1))) {
+ continue outerForLoop;
+ }
+ }
+
+ // See if the join keys are unique. Because the keys are
+ // part of an equality join condition, nulls are filtered out
+ // by the join. So, it's ok if there are nulls in the join
+ // keys.
+ if (RelMdUtil.areColumnsDefinitelyUniqueWhenNullsFiltered(mq,
+ multiJoin.getJoinFactor(factIdx), joinKeys)) {
+ multiJoin.addRemovableOuterJoinFactor(factIdx);
+
+ // Since we are no longer joining this factor,
+ // decrement the reference counters corresponding to
+ // the join keys from the other factors that join with
+ // this one. Later, in the outermost loop, we'll have
+ // the opportunity to retry removing those factors.
+ final ImmutableBitSet otherJoinKeys = otherJoinKeyBuilder.build();
+ for (int otherKey : otherJoinKeys) {
+ int otherFactor = multiJoin.findRef(otherKey);
+ if (multiJoin.isNullGenerating(otherFactor)) {
+ retryCandidates.add(otherFactor);
+ }
+ int [] otherJoinFieldRefCounts =
+ multiJoin.getJoinFieldRefCounts(otherFactor);
+ int offset = multiJoin.getJoinStart(otherFactor);
+ --otherJoinFieldRefCounts[otherKey - offset];
+ }
+ }
+ }
+ removalCandidates.clear();
+ removalCandidates.addAll(retryCandidates);
+ }
+ }
+
+ /**
+ * Sets a join key if only one of the specified input references corresponds
+ * to a specified factor as determined by its field numbers. Also keeps
+ * track of the keys from the other factor.
+ *
+ * @param joinKeys join keys to be set if a key is found
+ * @param otherJoinKeys join keys for the other join factor
+ * @param ref1 first input reference
+ * @param ref2 second input reference
+ * @param firstFieldNum first field number of the factor
+ * @param lastFieldNum last field number + 1 of the factor
+ * @param swap if true, check for the desired input reference in the second
+ * input reference parameter if the first input reference isn't the correct
+ * one
+ */
+ private static void setJoinKey(
+ ImmutableBitSet.Builder joinKeys,
+ ImmutableBitSet.Builder otherJoinKeys,
+ int ref1,
+ int ref2,
+ int firstFieldNum,
+ int lastFieldNum,
+ boolean swap) {
+ if ((ref1 >= firstFieldNum) && (ref1 < lastFieldNum)) {
+ if (!((ref2 >= firstFieldNum) && (ref2 < lastFieldNum))) {
+ joinKeys.set(ref1 - firstFieldNum);
+ otherJoinKeys.set(ref2);
+ }
+ return;
+ }
+ if (swap) {
+ setJoinKey(
+ joinKeys,
+ otherJoinKeys,
+ ref2,
+ ref1,
+ firstFieldNum,
+ lastFieldNum,
+ false);
+ }
+ }
+
+ /**
+ * Locates pairs of joins that are self-joins where the join can be removed
+ * because the join condition between the two factors is an equality join on
+ * unique keys.
+ *
+ * @param multiJoin join factors being optimized
+ */
+ private static void findRemovableSelfJoins(RelMetadataQuery mq, HiveLoptMultiJoin multiJoin) {
+ // Candidates for self-joins must be simple factors
+ Map<Integer, RelOptTable> simpleFactors = getSimpleFactors(mq, multiJoin);
+
+ // See if a simple factor is repeated and therefore potentially is
+ // part of a self-join. Restrict each factor to at most one
+ // self-join.
+ final List<RelOptTable> repeatedTables = new ArrayList<>();
+ final Map<Integer, Integer> selfJoinPairs = new HashMap<>();
+ @KeyFor("simpleFactors") Integer [] factors =
+ new TreeSet<>(simpleFactors.keySet()).toArray(new Integer[0]);
+ for (int i = 0; i < factors.length; i++) {
+ if (repeatedTables.contains(simpleFactors.get(factors[i]))) {
+ continue;
+ }
+ for (int j = i + 1; j < factors.length; j++) {
+ @KeyFor("simpleFactors") int leftFactor = factors[i];
+ @KeyFor("simpleFactors") int rightFactor = factors[j];
+ if (simpleFactors.get(leftFactor).getQualifiedName().equals(
+ simpleFactors.get(rightFactor).getQualifiedName())) {
+ selfJoinPairs.put(leftFactor, rightFactor);
+ repeatedTables.add(simpleFactors.get(leftFactor));
+ break;
+ }
+ }
+ }
+
+ // From the candidate self-join pairs, determine if there is
+ // the appropriate join condition between the two factors that will
+ // allow the join to be removed.
+ for (Integer factor1 : selfJoinPairs.keySet()) {
+ final int factor2 = selfJoinPairs.get(factor1);
+ final List<RexNode> selfJoinFilters = new ArrayList<>();
+ for (RexNode filter : multiJoin.getJoinFilters()) {
+ ImmutableBitSet joinFactors =
+ multiJoin.getFactorsRefByJoinFilter(filter);
+ if ((joinFactors.cardinality() == 2)
+ && joinFactors.get(factor1)
+ && joinFactors.get(factor2)) {
+ selfJoinFilters.add(filter);
+ }
+ }
+ if ((selfJoinFilters.size() > 0)
+ && isSelfJoinFilterUnique(
+ mq,
+ multiJoin,
+ factor1,
+ factor2,
+ selfJoinFilters)) {
+ multiJoin.addRemovableSelfJoinPair(factor1, factor2);
+ }
+ }
+ }
+
+ /**
+ * Retrieves join factors that correspond to simple table references. A
+ * simple table reference is a single table reference with no grouping or
+ * aggregation.
+ *
+ * @param multiJoin join factors being optimized
+ *
+ * @return map consisting of the simple factors and the tables they
+ * correspond
+ */
+ private static Map<Integer, RelOptTable> getSimpleFactors(RelMetadataQuery mq,
+ HiveLoptMultiJoin multiJoin) {
+ final Map<Integer, RelOptTable> returnList = new HashMap<>();
+
+ // Loop through all join factors and locate the ones where each
+ // column referenced from the factor is not derived and originates
+ // from the same underlying table. Also, discard factors that
+ // are null-generating or will be removed because of semijoins.
+ if (multiJoin.getMultiJoinRel().isFullOuterJoin()) {
+ return returnList;
+ }
+ for (int factIdx = 0; factIdx < multiJoin.getNumJoinFactors(); factIdx++) {
+ if (multiJoin.isNullGenerating(factIdx)
+ || (multiJoin.getJoinRemovalFactor(factIdx) != null)) {
+ continue;
+ }
+ final RelNode rel = multiJoin.getJoinFactor(factIdx);
+ final RelOptTable table = mq.getTableOrigin(rel);
+ if (table != null) {
+ returnList.put(factIdx, table);
+ }
+ }
+
+ return returnList;
+ }
+
+ /**
+ * Determines if the equality join filters between two factors that map to
+ * the same table consist of unique, identical keys.
+ *
+ * @param multiJoin join factors being optimized
+ * @param leftFactor left factor in the join
+ * @param rightFactor right factor in the join
+ * @param joinFilterList list of join filters between the two factors
+ *
+ * @return true if the criteria are met
+ */
+ private static boolean isSelfJoinFilterUnique(
+ RelMetadataQuery mq,
+ HiveLoptMultiJoin multiJoin,
+ int leftFactor,
+ int rightFactor,
+ List<RexNode> joinFilterList) {
+ RexBuilder rexBuilder =
+ multiJoin.getMultiJoinRel().getCluster().getRexBuilder();
+ RelNode leftRel = multiJoin.getJoinFactor(leftFactor);
+ RelNode rightRel = multiJoin.getJoinFactor(rightFactor);
+ RexNode joinFilters =
+ RexUtil.composeConjunction(rexBuilder, joinFilterList);
+
+ // Adjust the offsets in the filter by shifting the left factor
+ // to the left and shifting the right factor to the left and then back
+ // to the right by the number of fields in the left
+ int [] adjustments = new int[multiJoin.getNumTotalFields()];
+ int leftAdjust = multiJoin.getJoinStart(leftFactor);
+ int nLeftFields = leftRel.getRowType().getFieldCount();
+ for (int i = 0; i < nLeftFields; i++) {
+ adjustments[leftAdjust + i] = -leftAdjust;
+ }
+ int rightAdjust = multiJoin.getJoinStart(rightFactor);
+ for (int i = 0; i < rightRel.getRowType().getFieldCount(); i++) {
+ adjustments[rightAdjust + i] = -rightAdjust + nLeftFields;
+ }
+ joinFilters =
+ joinFilters.accept(
+ new RelOptUtil.RexInputConverter(
+ rexBuilder,
+ multiJoin.getMultiJoinFields(),
+ leftRel.getRowType().getFieldList(),
+ rightRel.getRowType().getFieldList(),
+ adjustments));
+
+ return areSelfJoinKeysUnique(mq, leftRel, rightRel, joinFilters);
+ }
+
+ /**
+ * Generates N optimal join orderings. Each ordering contains each factor as
+ * the first factor in the ordering.
+ *
+ * @param multiJoin join factors being optimized
+ * @param semiJoinOpt optimal semijoins for each factor
+ * @param call RelOptRuleCall associated with this rule
+ */
+ private static void findBestOrderings(
+ RelMetadataQuery mq,
+ RelBuilder relBuilder,
+ HiveLoptMultiJoin multiJoin,
+ HiveLoptSemiJoinOptimizer semiJoinOpt,
+ RelOptRuleCall call) {
+ final List<RelNode> plans = new ArrayList<>();
+
+ final List<String> fieldNames =
+ multiJoin.getMultiJoinRel().getRowType().getFieldNames();
+
+ // generate the N join orderings
+ for (int i = 0; i < multiJoin.getNumJoinFactors(); i++) {
+ // first factor cannot be null generating
+ if (multiJoin.isNullGenerating(i)) {
+ continue;
+ }
+ HiveLoptJoinTree joinTree =
+ createOrdering(
+ mq,
+ relBuilder,
+ multiJoin,
+ semiJoinOpt,
+ i);
+ if (joinTree == null) {
+ continue;
+ }
+
+ RelNode newProject =
+ createTopProject(call.builder(), multiJoin, joinTree, fieldNames);
+ plans.add(newProject);
+ }
+
+ // transform the selected plans; note that we wait till then the end to
+ // transform everything so any intermediate RelNodes we create are not
+ // converted to RelSubsets The HEP planner will choose the join subtree
+ // with the best cumulative cost. Volcano planner keeps the alternative
+ // join subtrees and cost the final plan to pick the best one.
+ for (RelNode plan : plans) {
+ call.transformTo(plan);
+ }
+ }
+
+ /**
+ * Creates the topmost projection that will sit on top of the selected join
+ * ordering. The projection needs to match the original join ordering. Also,
+ * places any post-join filters on top of the project.
+ *
+ * @param multiJoin join factors being optimized
+ * @param joinTree selected join ordering
+ * @param fieldNames field names corresponding to the projection expressions
+ *
+ * @return created projection
+ */
+ private static RelNode createTopProject(
+ RelBuilder relBuilder,
+ HiveLoptMultiJoin multiJoin,
+ HiveLoptJoinTree joinTree,
+ List<String> fieldNames) {
+ List<RexNode> newProjExprs = new ArrayList<>();
+ RexBuilder rexBuilder =
+ multiJoin.getMultiJoinRel().getCluster().getRexBuilder();
+
+ // create a projection on top of the joins, matching the original
+ // join order
+ final List<Integer> newJoinOrder = joinTree.getTreeOrder();
+ int nJoinFactors = multiJoin.getNumJoinFactors();
+ List<RelDataTypeField> fields = multiJoin.getMultiJoinFields();
+
+ // create a mapping from each factor to its field offset in the join
+ // ordering
+ final Map<Integer, Integer> factorToOffsetMap = new HashMap<>();
+ for (int pos = 0, fieldStart = 0; pos < nJoinFactors; pos++) {
+ factorToOffsetMap.put(newJoinOrder.get(pos), fieldStart);
+ fieldStart +=
+ multiJoin.getNumFieldsInJoinFactor(newJoinOrder.get(pos));
+ }
+
+ for (int currFactor = 0; currFactor < nJoinFactors; currFactor++) {
+ // if the factor is the right factor in a removable self-join,
+ // then where possible, remap references to the right factor to
+ // the corresponding reference in the left factor
+ Integer leftFactor = null;
+ if (multiJoin.isRightFactorInRemovableSelfJoin(currFactor)) {
+ leftFactor = multiJoin.getOtherSelfJoinFactor(currFactor);
+ }
+ for (int fieldPos = 0;
+ fieldPos < multiJoin.getNumFieldsInJoinFactor(currFactor);
+ fieldPos++) {
+ int newOffset = requireNonNull(factorToOffsetMap.get(currFactor),
+ () -> "factorToOffsetMap.get(currFactor)") + fieldPos;
+ if (leftFactor != null) {
+ Integer leftOffset =
+ multiJoin.getRightColumnMapping(currFactor, fieldPos);
+ if (leftOffset != null) {
+ newOffset =
+ requireNonNull(factorToOffsetMap.get(leftFactor),
+ "factorToOffsetMap.get(leftFactor)") + leftOffset;
+ }
+ }
+ newProjExprs.add(
+ rexBuilder.makeInputRef(
+ fields.get(newProjExprs.size()).getType(),
+ newOffset));
+ }
+ }
+
+ relBuilder.push(joinTree.getJoinTree());
+ relBuilder.project(newProjExprs, fieldNames);
+
+ // Place the post-join filter (if it exists) on top of the final
+ // projection.
+ RexNode postJoinFilter =
+ multiJoin.getMultiJoinRel().getPostJoinFilter();
+ if (postJoinFilter != null) {
+ relBuilder.filter(postJoinFilter);
+ }
+ return relBuilder.build();
+ }
+
+ /**
+ * Computes the cardinality of the join columns from a particular factor,
+ * when that factor is joined with another join tree.
+ *
+ * @param multiJoin join factors being optimized
+ * @param semiJoinOpt optimal semijoins chosen for each factor
+ * @param joinTree the join tree that the factor is being joined with
+ * @param filters possible join filters to select from
+ * @param factor the factor being added
+ *
+ * @return computed cardinality
+ */
+ private static @Nullable Double computeJoinCardinality(
+ RelMetadataQuery mq,
+ HiveLoptMultiJoin multiJoin,
+ HiveLoptSemiJoinOptimizer semiJoinOpt,
+ HiveLoptJoinTree joinTree,
+ List<RexNode> filters,
+ int factor) {
+ final ImmutableBitSet childFactors =
+ ImmutableBitSet.builder()
+ .addAll(joinTree.getTreeOrder())
+ .set(factor)
+ .build();
+
+ int factorStart = multiJoin.getJoinStart(factor);
+ int nFields = multiJoin.getNumFieldsInJoinFactor(factor);
+ final ImmutableBitSet.Builder joinKeys = ImmutableBitSet.builder();
+
+ // first loop through the inner join filters, picking out the ones
+ // that reference only the factors in either the join tree or the
+ // factor that will be added
+ setFactorJoinKeys(
+ multiJoin,
+ filters,
+ childFactors,
+ factorStart,
+ nFields,
+ joinKeys);
+
+ // then loop through the outer join filters where the factor being
+ // added is the null generating factor in the outer join
+ setFactorJoinKeys(
+ multiJoin,
+ RelOptUtil.conjunctions(multiJoin.getOuterJoinCond(factor)),
+ childFactors,
+ factorStart,
+ nFields,
+ joinKeys);
+
+ // if the join tree doesn't contain all the necessary factors in
+ // any of the join filters, then joinKeys will be empty, so return
+ // null in that case
+ if (joinKeys.isEmpty()) {
+ return null;
+ } else {
+ return mq.getDistinctRowCount(semiJoinOpt.getChosenSemiJoin(factor),
+ joinKeys.build(), null);
+ }
+ }
+
+ /**
+ * Locates from a list of filters those that correspond to a particular join
+ * tree. Then, for each of those filters, extracts the fields corresponding
+ * to a particular factor, setting them in a bitmap.
+ *
+ * @param multiJoin join factors being optimized
+ * @param filters list of join filters
+ * @param joinFactors bitmap containing the factors in a particular join
+ * tree
+ * @param factorStart the initial offset of the factor whose join keys will
+ * be extracted
+ * @param nFields the number of fields in the factor whose join keys will be
+ * extracted
+ * @param joinKeys the bitmap that will be set with the join keys
+ */
+ private static void setFactorJoinKeys(
+ HiveLoptMultiJoin multiJoin,
+ List<RexNode> filters,
+ ImmutableBitSet joinFactors,
+ int factorStart,
+ int nFields,
+ ImmutableBitSet.Builder joinKeys) {
+ for (RexNode joinFilter : filters) {
+ ImmutableBitSet filterFactors =
+ multiJoin.getFactorsRefByJoinFilter(joinFilter);
+
+ // if all factors in the join filter are in the bitmap containing
+ // the factors in a join tree, then from that filter, add the
+ // fields corresponding to the specified factor to the join key
+ // bitmap; in doing so, adjust the join keys so they start at
+ // offset 0
+ if (joinFactors.contains(filterFactors)) {
+ ImmutableBitSet joinFields =
+ multiJoin.getFieldsRefByJoinFilter(joinFilter);
+ for (int field = joinFields.nextSetBit(factorStart);
+ (field >= 0)
+ && (field < (factorStart + nFields));
+ field = joinFields.nextSetBit(field + 1)) {
+ joinKeys.set(field - factorStart);
+ }
+ }
+ }
+ }
+
+ /**
+ * Generates a join tree with a specific factor as the first factor in the
+ * join tree.
+ *
+ * @param multiJoin join factors being optimized
+ * @param semiJoinOpt optimal semijoins for each factor
+ * @param firstFactor first factor in the tree
+ *
+ * @return constructed join tree or null if it is not possible for
+ * firstFactor to appear as the first factor in the join
+ */
+ private static @Nullable HiveLoptJoinTree createOrdering(
+ RelMetadataQuery mq,
+ RelBuilder relBuilder,
+ HiveLoptMultiJoin multiJoin,
+ HiveLoptSemiJoinOptimizer semiJoinOpt,
+ int firstFactor) {
+ HiveLoptJoinTree joinTree = null;
+ final int nJoinFactors = multiJoin.getNumJoinFactors();
+ final BitSet factorsToAdd = BitSets.range(0, nJoinFactors);
+ final BitSet factorsAdded = new BitSet(nJoinFactors);
+ final List<RexNode> filtersToAdd =
+ new ArrayList<>(multiJoin.getJoinFilters());
+
+ int prevFactor = -1;
+ while (factorsToAdd.cardinality() > 0) {
+ int nextFactor;
+ boolean selfJoin = false;
+ if (factorsAdded.cardinality() == 0) {
+ nextFactor = firstFactor;
+ } else {
+ // If the factor just added is part of a removable self-join
+ // and the other half of the self-join hasn't been added yet,
+ // then add it next. Otherwise, look for the optimal factor
+ // to add next.
+ Integer selfJoinFactor =
+ multiJoin.getOtherSelfJoinFactor(prevFactor);
+ if ((selfJoinFactor != null)
+ && !factorsAdded.get(selfJoinFactor)) {
+ nextFactor = selfJoinFactor;
+ selfJoin = true;
+ } else {
+ nextFactor =
+ getBestNextFactor(
+ mq,
+ multiJoin,
+ factorsToAdd,
+ factorsAdded,
+ semiJoinOpt,
+ joinTree,
+ filtersToAdd);
+ }
+ }
+
+ // add the factor; pass in a bitmap representing the factors
+ // this factor joins with that have already been added to
+ // the tree
+ BitSet factorsNeeded =
+ multiJoin.getFactorsRefByFactor(nextFactor).toBitSet();
+ if (multiJoin.isNullGenerating(nextFactor)) {
+ factorsNeeded.or(multiJoin.getOuterJoinFactors(nextFactor).toBitSet());
+ }
+ factorsNeeded.and(factorsAdded);
+ joinTree =
+ addFactorToTree(
+ mq,
+ relBuilder,
+ multiJoin,
+ semiJoinOpt,
+ joinTree,
+ nextFactor,
+ factorsNeeded,
+ filtersToAdd,
+ selfJoin);
+ if (joinTree == null) {
+ return null;
+ }
+ factorsToAdd.clear(nextFactor);
+ factorsAdded.set(nextFactor);
+ prevFactor = nextFactor;
+ }
+
+ assert filtersToAdd.size() == 0;
+ return joinTree;
+ }
+
+ /**
+ * Determines the best factor to be added next into a join tree.
+ *
+ * @param multiJoin join factors being optimized
+ * @param factorsToAdd factors to choose from to add next
+ * @param factorsAdded factors that have already been added to the join tree
+ * @param semiJoinOpt optimal semijoins for each factor
+ * @param joinTree join tree constructed thus far
+ * @param filtersToAdd remaining filters that need to be added
+ *
+ * @return index of the best factor to add next
+ */
+ private static int getBestNextFactor(
+ RelMetadataQuery mq,
+ HiveLoptMultiJoin multiJoin,
+ BitSet factorsToAdd,
+ BitSet factorsAdded,
+ HiveLoptSemiJoinOptimizer semiJoinOpt,
+ @Nullable HiveLoptJoinTree joinTree,
+ List<RexNode> filtersToAdd) {
+ // iterate through the remaining factors and determine the
+ // best one to add next
+ int nextFactor = -1;
+ int bestWeight = 0;
+ Double bestCardinality = null;
+ int [][] factorWeights = multiJoin.getFactorWeights();
+ for (int factor : BitSets.toIter(factorsToAdd)) {
+ // if the factor corresponds to a dimension table whose
+ // join we can remove, make sure the corresponding fact
+ // table is in the current join tree
+ Integer factIdx = multiJoin.getJoinRemovalFactor(factor);
+ if (factIdx != null) {
+ if (!factorsAdded.get(factIdx)) {
+ continue;
+ }
+ }
+
+ // can't add a null-generating factor if its dependent,
+ // non-null generating factors haven't been added yet
+ if (multiJoin.isNullGenerating(factor)
+ && !BitSets.contains(factorsAdded,
+ multiJoin.getOuterJoinFactors(factor))) {
+ continue;
+ }
+
+ // determine the best weight between the current factor
+ // under consideration and the factors that have already
+ // been added to the tree
+ int dimWeight = 0;
+ for (int prevFactor : BitSets.toIter(factorsAdded)) {
+ int[] factorWeight = requireNonNull(factorWeights, "factorWeights")[prevFactor];
+ if (factorWeight[factor] > dimWeight) {
+ dimWeight = factorWeight[factor];
+ }
+ }
+
+ // only compute the join cardinality if we know that
+ // this factor joins with some part of the current join
+ // tree and is potentially better than other factors
+ // already considered
+ Double cardinality = null;
+ if ((dimWeight > 0)
+ && ((dimWeight > bestWeight) || (dimWeight == bestWeight))) {
+ cardinality =
+ computeJoinCardinality(
+ mq,
+ multiJoin,
+ semiJoinOpt,
+ requireNonNull(joinTree, "joinTree"),
+ filtersToAdd,
+ factor);
+ }
+
+ // if two factors have the same weight, pick the one
+ // with the higher cardinality join key, relative to
+ // the join being considered
+ if ((dimWeight > bestWeight)
+ || ((dimWeight == bestWeight)
+ && ((bestCardinality == null)
+ || ((cardinality != null)
+ && (cardinality > bestCardinality))))) {
+ nextFactor = factor;
+ bestWeight = dimWeight;
+ bestCardinality = cardinality;
+ }
+ }
+
+ return nextFactor;
+ }
+
+ /**
+ * Returns whether a RelNode corresponds to a Join that wasn't one of the
+ * original MultiJoin input factors.
+ */
+ private static boolean isJoinTree(RelNode rel) {
+ // full outer joins were already optimized in a prior instantiation
+ // of this rule; therefore we should never see a join input that's
+ // a full outer join
+ if (rel instanceof Join) {
+ assert ((Join) rel).getJoinType() != JoinRelType.FULL;
+ return true;
+ } else {
+ return false;
+ }
+ }
+
+ /**
+ * Adds a new factor into the current join tree. The factor is either pushed
+ * down into one of the subtrees of the join recursively, or it is added to
+ * the top of the current tree, whichever yields a better ordering.
+ *
+ * @param multiJoin join factors being optimized
+ * @param semiJoinOpt optimal semijoins for each factor
+ * @param joinTree current join tree
+ * @param factorToAdd new factor to be added
+ * @param factorsNeeded factors that must precede the factor to be added
+ * @param filtersToAdd filters remaining to be added; filters added to the
+ * new join tree are removed from the list
+ * @param selfJoin true if the join being created is a self-join that's
+ * removable
+ *
+ * @return optimal join tree with the new factor added if it is possible to
+ * add the factor; otherwise, null is returned
+ */
+ private static @Nullable HiveLoptJoinTree addFactorToTree(
+ RelMetadataQuery mq,
+ RelBuilder relBuilder,
+ HiveLoptMultiJoin multiJoin,
+ HiveLoptSemiJoinOptimizer semiJoinOpt,
+ @Nullable HiveLoptJoinTree joinTree,
+ int factorToAdd,
+ BitSet factorsNeeded,
+ List<RexNode> filtersToAdd,
+ boolean selfJoin) {
+
+ // if the factor corresponds to the null generating factor in an outer
+ // join that can be removed, then create a replacement join
+ if (multiJoin.isRemovableOuterJoinFactor(factorToAdd)) {
+ return createReplacementJoin(
+ relBuilder,
+ multiJoin,
+ semiJoinOpt,
+ requireNonNull(joinTree, "joinTree"),
+ -1,
+ factorToAdd,
+ ImmutableIntList.of(),
+ null,
+ filtersToAdd);
+ }
+
+ // if the factor corresponds to a dimension table whose join we
+ // can remove, create a replacement join if the corresponding fact
+ // table is in the current join tree
+ if (multiJoin.getJoinRemovalFactor(factorToAdd) != null) {
+ return createReplacementSemiJoin(
+ relBuilder,
+ multiJoin,
+ semiJoinOpt,
+ joinTree,
+ factorToAdd,
+ filtersToAdd);
+ }
+
+ // if this is the first factor in the tree, create a join tree with
+ // the single factor
+ if (joinTree == null) {
+ return new HiveLoptJoinTree(
+ semiJoinOpt.getChosenSemiJoin(factorToAdd),
+ factorToAdd);
+ }
+
+ // Create a temporary copy of the filter list as we may need the
+ // original list to pass into addToTop(). However, if no tree was
+ // created by addToTop() because the factor being added is part of
+ // a self-join, then pass the original filter list so the added
+ // filters will still be removed from the list.
+ final List<RexNode> tmpFilters = new ArrayList<>(filtersToAdd);
+ HiveLoptJoinTree topTree =
+ addToTop(
+ mq,
+ relBuilder,
+ multiJoin,
+ semiJoinOpt,
+ joinTree,
+ factorToAdd,
+ filtersToAdd,
+ selfJoin);
+ HiveLoptJoinTree pushDownTree =
+ pushDownFactor(
+ mq,
+ relBuilder,
+ multiJoin,
+ semiJoinOpt,
+ joinTree,
+ factorToAdd,
+ factorsNeeded,
+ (topTree == null) ? filtersToAdd : tmpFilters,
+ selfJoin);
+
+ // pick the lower cost option, and replace the join ordering with
+ // the ordering associated with the best option
+ HiveLoptJoinTree bestTree;
+ RelOptCost costPushDown = null;
+ RelOptCost costTop = null;
+ if (pushDownTree != null) {
+ costPushDown = mq.getCumulativeCost(pushDownTree.getJoinTree());
+ }
+ if (topTree != null) {
+ costTop = mq.getCumulativeCost(topTree.getJoinTree());
+ }
+
+ if (pushDownTree == null) {
+ bestTree = topTree;
+ } else if (topTree == null) {
+ bestTree = pushDownTree;
+ } else {
+ requireNonNull(costPushDown, "costPushDown");
+ requireNonNull(costTop, "costTop");
+ if (costPushDown.isEqWithEpsilon(costTop)) {
+ // if both plans cost the same (with an allowable round-off
+ // margin of error), favor the one that passes
+ // around the wider rows further up in the tree
+ if (rowWidthCost(pushDownTree.getJoinTree())
+ < rowWidthCost(topTree.getJoinTree())) {
+ bestTree = pushDownTree;
+ } else {
+ bestTree = topTree;
+ }
+ } else if (costPushDown.isLt(costTop)) {
+ bestTree = pushDownTree;
+ } else {
+ bestTree = topTree;
+ }
+ }
+
+ return bestTree;
+ }
+
+ /**
+ * Computes a cost for a join tree based on the row widths of the inputs
+ * into the join. Joins where the inputs have the fewest number of columns
+ * lower in the tree are better than equivalent joins where the inputs with
+ * the larger number of columns are lower in the tree.
+ *
+ * @param tree a tree of RelNodes
+ *
+ * @return the cost associated with the width of the tree
+ */
+ private static int rowWidthCost(RelNode tree) {
+ // The width cost is the width of the tree itself plus the widths
+ // of its children. Hence, skinnier rows are better when they're
+ // lower in the tree since the width of a RelNode contributes to
+ // the cost of each LogicalJoin that appears above that RelNode.
+ int width = tree.getRowType().getFieldCount();
+ if (isJoinTree(tree)) {
+ Join joinRel = (Join) tree;
+ width +=
+ rowWidthCost(joinRel.getLeft())
+ + rowWidthCost(joinRel.getRight());
+ }
+ return width;
+ }
+
+ /**
+ * Creates a join tree where the new factor is pushed down one of the
+ * operands of the current join tree.
+ *
+ * @param multiJoin join factors being optimized
+ * @param semiJoinOpt optimal semijoins for each factor
+ * @param joinTree current join tree
+ * @param factorToAdd new factor to be added
+ * @param factorsNeeded factors that must precede the factor to be added
+ * @param filtersToAdd filters remaining to be added; filters that are added
+ * to the join tree are removed from the list
+ * @param selfJoin true if the factor being added is part of a removable
+ * self-join
+ *
+ * @return optimal join tree with the new factor pushed down the current
+ * join tree if it is possible to do the pushdown; otherwise, null is
+ * returned
+ */
+ private static @Nullable HiveLoptJoinTree pushDownFactor(
+ RelMetadataQuery mq,
+ RelBuilder relBuilder,
+ HiveLoptMultiJoin multiJoin,
+ HiveLoptSemiJoinOptimizer semiJoinOpt,
+ HiveLoptJoinTree joinTree,
+ int factorToAdd,
+ BitSet factorsNeeded,
+ List<RexNode> filtersToAdd,
+ boolean selfJoin) {
+ // pushdown option only works if we already have a join tree
+ if (!isJoinTree(joinTree.getJoinTree())) {
+ return null;
+ }
+ int childNo = -1;
+ HiveLoptJoinTree left = joinTree.getLeft();
+ HiveLoptJoinTree right = joinTree.getRight();
+ Join joinRel = (Join) joinTree.getJoinTree();
+ JoinRelType joinType = joinRel.getJoinType();
+
+ // can't push factors pass self-joins because in order to later remove
+ // them, we need to keep the factors together
+ if (joinTree.isRemovableSelfJoin()) {
+ return null;
+ }
+
+ // If there are no constraints as to which side the factor must
+ // be pushed, arbitrarily push to the left. In the case of a
+ // self-join, always push to the input that contains the other
+ // half of the self-join.
+ if (selfJoin) {
+ BitSet selfJoinFactor = new BitSet(multiJoin.getNumJoinFactors());
+ Integer factor = requireNonNull(multiJoin.getOtherSelfJoinFactor(factorToAdd),
+ () -> "multiJoin.getOtherSelfJoinFactor(" + factorToAdd + ") is null");
+ selfJoinFactor.set(factor);
+ if (multiJoin.hasAllFactors(left, selfJoinFactor)) {
+ childNo = 0;
+ } else {
+ assert multiJoin.hasAllFactors(right, selfJoinFactor);
+ childNo = 1;
+ }
+ } else if (
+ (factorsNeeded.cardinality() == 0)
+ && !joinType.generatesNullsOnLeft()) {
+ childNo = 0;
+ } else {
+ // push to the left if the LHS contains all factors that the
+ // current factor needs and that side is not null-generating;
+ // same check for RHS
+ if (multiJoin.hasAllFactors(left, factorsNeeded)
+ && !joinType.generatesNullsOnLeft()) {
+ childNo = 0;
+ } else if (
+ multiJoin.hasAllFactors(right, factorsNeeded)
+ && !joinType.generatesNullsOnRight()) {
+ childNo = 1;
+ }
+ // if it couldn't be pushed down to either side, then it can
+ // only be put on top
+ }
+ if (childNo == -1) {
+ return null;
+ }
+
+ // remember the original join order before the pushdown so we can
+ // appropriately adjust any filters already attached to the join
+ // node
+ final List<Integer> origJoinOrder = joinTree.getTreeOrder();
+
+ // recursively pushdown the factor
+ HiveLoptJoinTree subTree = (childNo == 0) ? left : right;
+ subTree =
+ addFactorToTree(
+ mq,
+ relBuilder,
+ multiJoin,
+ semiJoinOpt,
+ subTree,
+ factorToAdd,
+ factorsNeeded,
+ filtersToAdd,
+ selfJoin);
+
+ if (childNo == 0) {
+ left = subTree;
+ } else {
+ right = subTree;
+ }
+
+ // adjust the join condition from the original join tree to reflect
+ // pushdown of the new factor as well as any swapping that may have
+ // been done during the pushdown
+ RexNode newCondition =
+ ((Join) joinTree.getJoinTree()).getCondition();
+ newCondition =
+ adjustFilter(
+ multiJoin,
+ requireNonNull(left, "left"),
+ requireNonNull(right, "right"),
+ newCondition,
+ factorToAdd,
+ origJoinOrder,
+ joinTree.getJoinTree().getRowType().getFieldList());
+
+ // determine if additional filters apply as a result of adding the
+ // new factor, provided this isn't a left or right outer join; for
+ // those cases, the additional filters will be added on top of the
+ // join in createJoinSubtree
+ if ((joinType != JoinRelType.LEFT) && (joinType != JoinRelType.RIGHT)) {
+ RexNode condition =
+ addFilters(
+ multiJoin,
+ left,
+ -1,
+ right,
+ filtersToAdd,
+ true);
+ RexBuilder rexBuilder =
+ multiJoin.getMultiJoinRel().getCluster().getRexBuilder();
+ newCondition =
+ RelOptUtil.andJoinFilters(
+ rexBuilder,
+ newCondition,
+ condition);
+ }
+
+ // create the new join tree with the factor pushed down
+ return createJoinSubtree(
+ mq,
+ relBuilder,
+ multiJoin,
+ left,
+ right,
+ newCondition,
+ joinType,
+ filtersToAdd,
+ false,
+ false);
+ }
+
+ /**
+ * Creates a join tree with the new factor added to the top of the tree.
+ *
+ * @param multiJoin join factors being optimized
+ * @param semiJoinOpt optimal semijoins for each factor
+ * @param joinTree current join tree
+ * @param factorToAdd new factor to be added
+ * @param filtersToAdd filters remaining to be added; modifies the list to
+ * remove filters that can be added to the join tree
+ * @param selfJoin true if the join being created is a self-join that's
+ * removable
+ *
+ * @return new join tree
+ */
+ private static @Nullable HiveLoptJoinTree addToTop(
+ RelMetadataQuery mq,
+ RelBuilder relBuilder,
+ HiveLoptMultiJoin multiJoin,
+ HiveLoptSemiJoinOptimizer semiJoinOpt,
+ HiveLoptJoinTree joinTree,
+ int factorToAdd,
+ List<RexNode> filtersToAdd,
+ boolean selfJoin) {
+ // self-joins can never be created at the top of an existing
+ // join tree because it needs to be paired directly with the
+ // other self-join factor
+ if (selfJoin && isJoinTree(joinTree.getJoinTree())) {
+ return null;
+ }
+
+ // if the factor being added is null-generating, create the join
+ // as a left outer join since it's being added to the RHS side of
+ // the join; createJoinSubTree may swap the inputs and therefore
+ // convert the left outer join to a right outer join; if the original
+ // MultiJoin was a full outer join, these should be the only
+ // factors in the join, so create the join as a full outer join
+ JoinRelType joinType;
+ if (multiJoin.getMultiJoinRel().isFullOuterJoin()) {
+ assert multiJoin.getNumJoinFactors() == 2;
+ joinType = JoinRelType.FULL;
+ } else if (multiJoin.isNullGenerating(factorToAdd)) {
+ joinType = JoinRelType.LEFT;
+ } else {
+ joinType = JoinRelType.INNER;
+ }
+
+ HiveLoptJoinTree rightTree =
+ new HiveLoptJoinTree(
+ semiJoinOpt.getChosenSemiJoin(factorToAdd),
+ factorToAdd);
+
+ // in the case of a left or right outer join, use the specific
+ // outer join condition
+ RexNode condition;
+ if ((joinType == JoinRelType.LEFT) || (joinType == JoinRelType.RIGHT)) {
+ condition = requireNonNull(multiJoin.getOuterJoinCond(factorToAdd),
+ "multiJoin.getOuterJoinCond(factorToAdd)");
+ } else {
+ condition =
+ addFilters(
+ multiJoin,
+ joinTree,
+ -1,
+ rightTree,
+ filtersToAdd,
+ false);
+ }
+
+ return createJoinSubtree(
+ mq,
+ relBuilder,
+ multiJoin,
+ joinTree,
+ rightTree,
+ condition,
+ joinType,
+ filtersToAdd,
+ true,
+ selfJoin);
+ }
+
+ /**
+ * Determines which join filters can be added to the current join tree. Note
+ * that the join filter still reflects the original join ordering. It will
+ * only be adjusted to reflect the new join ordering if the "adjust"
+ * parameter is set to true.
+ *
+ * @param multiJoin join factors being optimized
+ * @param leftTree left subtree of the join tree
+ * @param leftIdx if ≥ 0, only consider filters that reference leftIdx in
+ * leftTree; otherwise, consider all filters that reference any factor in
+ * leftTree
+ * @param rightTree right subtree of the join tree
+ * @param filtersToAdd remaining join filters that need to be added; those
+ * that are added are removed from the list
+ * @param adjust if true, adjust filter to reflect new join ordering
+ *
+ * @return AND'd expression of the join filters that can be added to the
+ * current join tree
+ */
+ private static RexNode addFilters(
+ HiveLoptMultiJoin multiJoin,
+ HiveLoptJoinTree leftTree,
+ int leftIdx,
+ HiveLoptJoinTree rightTree,
+ List<RexNode> filtersToAdd,
+ boolean adjust) {
+ // loop through the remaining filters to be added and pick out the
+ // ones that reference only the factors in the new join tree
+ final RexBuilder rexBuilder =
+ multiJoin.getMultiJoinRel().getCluster().getRexBuilder();
+ final ImmutableBitSet.Builder childFactorBuilder =
+ ImmutableBitSet.builder();
+ childFactorBuilder.addAll(rightTree.getTreeOrder());
+ if (leftIdx >= 0) {
+ childFactorBuilder.set(leftIdx);
+ } else {
+ childFactorBuilder.addAll(leftTree.getTreeOrder());
+ }
+ for (int child : rightTree.getTreeOrder()) {
+ childFactorBuilder.set(child);
+ }
+
+ final ImmutableBitSet childFactor = childFactorBuilder.build();
+ RexNode condition = null;
+ final ListIterator<RexNode> filterIter = filtersToAdd.listIterator();
+ while (filterIter.hasNext()) {
+ RexNode joinFilter = filterIter.next();
+ ImmutableBitSet filterBitmap =
+ multiJoin.getFactorsRefByJoinFilter(joinFilter);
+
+ // if all factors in the join filter are in the join tree,
+ // AND the filter to the current join condition
+ if (childFactor.contains(filterBitmap)) {
+ if (condition == null) {
+ condition = joinFilter;
+ } else {
+ condition =
+ rexBuilder.makeCall(
+ SqlStdOperatorTable.AND,
+ condition,
+ joinFilter);
+ }
+ filterIter.remove();
+ }
+ }
+
+ if (adjust && (condition != null)) {
+ int [] adjustments = new int[multiJoin.getNumTotalFields()];
+ if (needsAdjustment(
+ multiJoin,
+ adjustments,
+ leftTree,
+ rightTree,
+ false)) {
+ condition =
+ condition.accept(
+ new RelOptUtil.RexInputConverter(
+ rexBuilder,
+ multiJoin.getMultiJoinFields(),
+ leftTree.getJoinTree().getRowType().getFieldList(),
+ rightTree.getJoinTree().getRowType().getFieldList(),
+ adjustments));
+ }
+ }
+
+ if (condition == null) {
+ condition = rexBuilder.makeLiteral(true);
+ }
+
+ return condition;
+ }
+
+ /**
+ * Adjusts a filter to reflect a newly added factor in the middle of an
+ * existing join tree.
+ *
+ * @param multiJoin join factors being optimized
+ * @param left left subtree of the join
+ * @param right right subtree of the join
+ * @param condition current join condition
+ * @param factorAdded index corresponding to the newly added factor
+ * @param origJoinOrder original join order, before factor was pushed into
+ * the tree
+ * @param origFields fields from the original join before the factor was
+ * added
+ *
+ * @return modified join condition reflecting addition of the new factor
+ */
+ private static RexNode adjustFilter(
+ HiveLoptMultiJoin multiJoin,
+ HiveLoptJoinTree left,
+ HiveLoptJoinTree right,
+ RexNode condition,
+ int factorAdded,
+ List<Integer> origJoinOrder,
+ List<RelDataTypeField> origFields) {
+ final List<Integer> newJoinOrder = new ArrayList<>();
+ left.getTreeOrder(newJoinOrder);
+ right.getTreeOrder(newJoinOrder);
+
+ int totalFields =
+ left.getJoinTree().getRowType().getFieldCount()
+ + right.getJoinTree().getRowType().getFieldCount()
+ - multiJoin.getNumFieldsInJoinFactor(factorAdded);
+ int [] adjustments = new int[totalFields];
+
+ // go through each factor and adjust relative to the original
+ // join order
+ boolean needAdjust = false;
+ int nFieldsNew = 0;
+ for (int newPos = 0; newPos < newJoinOrder.size(); newPos++) {
+ int nFieldsOld = 0;
+
+ // no need to make any adjustments on the newly added factor
+ int factor = newJoinOrder.get(newPos);
+ if (factor != factorAdded) {
+ // locate the position of the factor in the original join
+ // ordering
+ for (int pos : origJoinOrder) {
+ if (factor == pos) {
+ break;
+ }
+ nFieldsOld += multiJoin.getNumFieldsInJoinFactor(pos);
+ }
+
+ // fill in the adjustment array for this factor
+ if (remapJoinReferences(
+ multiJoin,
+ factor,
+ newJoinOrder,
+ newPos,
+ adjustments,
+ nFieldsOld,
+ nFieldsNew,
+ false)) {
+ needAdjust = true;
+ }
+ }
+ nFieldsNew += multiJoin.getNumFieldsInJoinFactor(factor);
+ }
+
+ if (needAdjust) {
+ RexBuilder rexBuilder =
+ multiJoin.getMultiJoinRel().getCluster().getRexBuilder();
+ condition =
+ condition.accept(
+ new RelOptUtil.RexInputConverter(
+ rexBuilder,
+ origFields,
+ left.getJoinTree().getRowType().getFieldList(),
+ right.getJoinTree().getRowType().getFieldList(),
+ adjustments));
+ }
+
+ return condition;
+ }
+
+ /**
+ * Sets an adjustment array based on where column references for a
+ * particular factor end up as a result of a new join ordering.
+ *
+ * <p>If the factor is not the right factor in a removable self-join, then
+ * it needs to be adjusted as follows:
+ *
+ * <ul>
+ * <li>First subtract, based on where the factor was in the original join
+ * ordering.
+ * <li>Then add on the number of fields in the factors that now precede this
+ * factor in the new join ordering.
+ * </ul>
+ *
+ * <p>If the factor is the right factor in a removable self-join and its
+ * column reference can be mapped to the left factor in the self-join, then:
+ *
+ * <ul>
+ * <li>First subtract, based on where the column reference is in the new
+ * join ordering.
+ * <li>Then, add on the number of fields up to the start of the left factor
+ * in the self-join in the new join ordering.
+ * <li>Then, finally add on the offset of the corresponding column from the
+ * left factor.
+ * </ul>
+ *
+ * <p>Note that this only applies if both factors in the self-join are in the
+ * join ordering. If they are, then the left factor always precedes the
+ * right factor in the join ordering.
+ *
+ * @param multiJoin join factors being optimized
+ * @param factor the factor whose references are being adjusted
+ * @param newJoinOrder the new join ordering containing the factor
+ * @param newPos the position of the factor in the new join ordering
+ * @param adjustments the adjustments array that will be set
+ * @param offset the starting offset within the original join ordering for
+ * the columns of the factor being adjusted
+ * @param newOffset the new starting offset in the new join ordering for the
+ * columns of the factor being adjusted
+ * @param alwaysUseDefault always use the default adjustment value
+ * regardless of whether the factor is the right factor in a removable
+ * self-join
+ *
+ * @return true if at least one column from the factor requires adjustment
+ */
+ private static boolean remapJoinReferences(
+ HiveLoptMultiJoin multiJoin,
+ int factor,
+ List<Integer> newJoinOrder,
+ int newPos,
+ int[] adjustments,
+ int offset,
+ int newOffset,
+ boolean alwaysUseDefault) {
+ boolean needAdjust = false;
+ int defaultAdjustment = -offset + newOffset;
+ if (!alwaysUseDefault
+ && multiJoin.isRightFactorInRemovableSelfJoin(factor)
+ && (newPos != 0)
+ && newJoinOrder.get(newPos - 1).equals(
+ multiJoin.getOtherSelfJoinFactor(factor))) {
+ int nLeftFields =
+ multiJoin.getNumFieldsInJoinFactor(
+ newJoinOrder.get(
+ newPos - 1));
+ for (int i = 0;
+ i < multiJoin.getNumFieldsInJoinFactor(factor);
+ i++) {
+ Integer leftOffset = multiJoin.getRightColumnMapping(factor, i);
+
+ // if the left factor doesn't reference the column, then
+ // use the default adjustment value
+ if (leftOffset == null) {
+ adjustments[i + offset] = defaultAdjustment;
+ } else {
+ adjustments[i + offset] =
+ -(offset + i) + (newOffset - nLeftFields)
+ + leftOffset;
+ }
+ if (adjustments[i + offset] != 0) {
+ needAdjust = true;
+ }
+ }
+ } else {
+ if (defaultAdjustment != 0) {
+ needAdjust = true;
+ for (int i = 0;
+ i < multiJoin.getNumFieldsInJoinFactor(
+ newJoinOrder.get(newPos));
+ i++) {
+ adjustments[i + offset] = defaultAdjustment;
+ }
+ }
+ }
+
+ return needAdjust;
+ }
+
+ /**
+ * In the event that a dimension table does not need to be joined because of
+ * a semijoin, this method creates a join tree that consists of a projection
+ * on top of an existing join tree. The existing join tree must contain the
+ * fact table in the semijoin that allows the dimension table to be removed.
+ *
+ * <p>The projection created on top of the join tree mimics a join of the
+ * fact and dimension tables. In order for the dimension table to have been
+ * removed, the only fields referenced from the dimension table are its
+ * dimension keys. Therefore, we can replace these dimension fields with the
+ * fields corresponding to the semijoin keys from the fact table in the
+ * projection.
+ *
+ * @param multiJoin join factors being optimized
+ * @param semiJoinOpt optimal semijoins for each factor
+ * @param factTree existing join tree containing the fact table
+ * @param dimIdx dimension table factor id
+ * @param filtersToAdd filters remaining to be added; filters added to the
+ * new join tree are removed from the list
+ *
+ * @return created join tree or null if the corresponding fact table has not
+ * been joined in yet
+ */
+ private static @Nullable HiveLoptJoinTree createReplacementSemiJoin(
+ RelBuilder relBuilder,
+ HiveLoptMultiJoin multiJoin,
+ HiveLoptSemiJoinOptimizer semiJoinOpt,
+ @Nullable HiveLoptJoinTree factTree,
+ int dimIdx,
+ List<RexNode> filtersToAdd) {
+ // if the current join tree doesn't contain the fact table, then
+ // don't bother trying to create the replacement join just yet
+ if (factTree == null) {
+ return null;
+ }
+
+ int factIdx = requireNonNull(multiJoin.getJoinRemovalFactor(dimIdx),
+ () -> "multiJoin.getJoinRemovalFactor(dimIdx) for " + dimIdx + ", " + multiJoin);
+ final List<Integer> joinOrder = factTree.getTreeOrder();
+ assert joinOrder.contains(factIdx);
+
+ // figure out the position of the fact table in the current jointree
+ int adjustment = 0;
+ for (Integer factor : joinOrder) {
+ if (factor == factIdx) {
+ break;
+ }
+ adjustment += multiJoin.getNumFieldsInJoinFactor(factor);
+ }
+
+ // map the dimension keys to the corresponding keys from the fact
+ // table, based on the fact table's position in the current jointree
+ List<RelDataTypeField> dimFields =
+ multiJoin.getJoinFactor(dimIdx).getRowType().getFieldList();
+ int nDimFields = dimFields.size();
+ Integer [] replacementKeys = new Integer[nDimFields];
+ LogicalJoin semiJoin = multiJoin.getJoinRemovalSemiJoin(dimIdx);
+ ImmutableIntList dimKeys = semiJoin.analyzeCondition().leftKeys;
+ ImmutableIntList factKeys = semiJoin.analyzeCondition().rightKeys;
+ for (int i = 0; i < dimKeys.size(); i++) {
+ replacementKeys[dimKeys.get(i)] = factKeys.get(i) + adjustment;
+ }
+
+ return createReplacementJoin(
+ relBuilder,
+ multiJoin,
+ semiJoinOpt,
+ factTree,
+ factIdx,
+ dimIdx,
+ dimKeys,
+ replacementKeys,
+ filtersToAdd);
+ }
+
+ /**
+ * Creates a replacement join, projecting either dummy columns or
+ * replacement keys from the factor that doesn't actually need to be joined.
+ *
+ * @param multiJoin join factors being optimized
+ * @param semiJoinOpt optimal semijoins for each factor
+ * @param currJoinTree current join tree being added to
+ * @param leftIdx if ≥ 0, when creating the replacement join, only consider
+ * filters that reference leftIdx in currJoinTree; otherwise, consider all
+ * filters that reference any factor in currJoinTree
+ * @param factorToAdd new factor whose join can be removed
+ * @param newKeys join keys that need to be replaced
+ * @param replacementKeys the keys that replace the join keys; null if we're
+ * removing the null generating factor in an outer join
+ * @param filtersToAdd filters remaining to be added; filters added to the
+ * new join tree are removed from the list
+ *
+ * @return created join tree with an appropriate projection for the factor
+ * that can be removed
+ */
+ private static HiveLoptJoinTree createReplacementJoin(
+ RelBuilder relBuilder,
+ HiveLoptMultiJoin multiJoin,
+ HiveLoptSemiJoinOptimizer semiJoinOpt,
+ HiveLoptJoinTree currJoinTree,
+ int leftIdx,
+ int factorToAdd,
+ ImmutableIntList newKeys,
+ Integer @Nullable [] replacementKeys,
+ List<RexNode> filtersToAdd) {
+ // create a projection, projecting the fields from the join tree
+ // containing the current joinRel and the new factor; for fields
+ // corresponding to join keys, replace them with the corresponding key
+ // from the replacementKeys passed in; for other fields, just create a
+ // null expression as a placeholder for the column; this is done so we
+ // don't have to adjust the offsets of other expressions that reference
+ // the new factor; the placeholder expression values should never be
+ // referenced, so that's why it's ok to create these possibly invalid
+ // expressions
+ RelNode currJoinRel = currJoinTree.getJoinTree();
+ List<RelDataTypeField> currFields = currJoinRel.getRowType().getFieldList();
+ final int nCurrFields = currFields.size();
+ List<RelDataTypeField> newFields =
+ multiJoin.getJoinFactor(factorToAdd).getRowType().getFieldList();
+ final int nNewFields = newFields.size();
+ List<Pair<RexNode, String>> projects = new ArrayList<>();
+ RexBuilder rexBuilder = currJoinRel.getCluster().getRexBuilder();
+ RelDataTypeFactory typeFactory = rexBuilder.getTypeFactory();
+
+ for (int i = 0; i < nCurrFields; i++) {
+ projects.add(
+ Pair.of(rexBuilder.makeInputRef(currFields.get(i).getType(), i),
+ currFields.get(i).getName()));
+ }
+ for (int i = 0; i < nNewFields; i++) {
+ RexNode projExpr;
+ RelDataType newType = newFields.get(i).getType();
+ if (!newKeys.contains(i)) {
+ if (replacementKeys == null) {
+ // null generating factor in an outer join; so make the
+ // type nullable
+ newType =
+ typeFactory.createTypeWithNullability(newType, true);
+ }
+ projExpr = rexBuilder.makeNullLiteral(newType);
+ } else {
+ // TODO: is the above if (replacementKeys==null) check placed properly?
+ requireNonNull(replacementKeys, "replacementKeys");
+ RelDataTypeField mappedField = currFields.get(replacementKeys[i]);
+ RexNode mappedInput =
+ rexBuilder.makeInputRef(
+ mappedField.getType(),
+ replacementKeys[i]);
+
+ // if the types aren't the same, create a cast
+ if (mappedField.getType() == newType) {
+ projExpr = mappedInput;
+ } else {
+ projExpr =
+ rexBuilder.makeCast(
+ newFields.get(i).getType(),
+ mappedInput);
+ }
+ }
+ projects.add(Pair.of(projExpr, newFields.get(i).getName()));
+ }
+
+ relBuilder.push(currJoinRel);
+ relBuilder.project(Pair.left(projects), Pair.right(projects));
+
+ // remove the join conditions corresponding to the join we're removing;
+ // we don't actually need to use them, but we need to remove them
+ // from the list since they're no longer needed
+ HiveLoptJoinTree newTree =
+ new HiveLoptJoinTree(
+ semiJoinOpt.getChosenSemiJoin(factorToAdd),
+ factorToAdd);
+ addFilters(
+ multiJoin,
+ currJoinTree,
+ leftIdx,
+ newTree,
+ filtersToAdd,
+ false);
+
+ // Filters referencing factors other than leftIdx and factorToAdd
+ // still do need to be applied. So, add them into a separate
+ // LogicalFilter placed on top off the projection created above.
+ if (leftIdx >= 0) {
+ addAdditionalFilters(
+ relBuilder,
+ multiJoin,
+ currJoinTree,
+ newTree,
+ filtersToAdd);
+ }
+
+ // finally, create a join tree consisting of the current join's join
+ // tree with the newly created projection; note that in the factor
+ // tree, we act as if we're joining in the new factor, even
+ // though we really aren't; this is needed so we can map the columns
+ // from the new factor as we go up in the join tree
+ return new HiveLoptJoinTree(
+ relBuilder.build(),
+ currJoinTree.getFactorTree(),
+ newTree.getFactorTree());
+ }
+
+ /**
+ * Creates a LogicalJoin given left and right operands and a join condition.
+ * Swaps the operands if beneficial.
+ *
+ * @param multiJoin join factors being optimized
+ * @param left left operand
+ * @param right right operand
+ * @param condition join condition
+ * @param joinType the join type
+ * @param fullAdjust true if the join condition reflects the original join
+ * ordering and therefore has not gone through any type of adjustment yet;
+ * otherwise, the condition has already been partially adjusted and only
+ * needs to be further adjusted if swapping is done
+ * @param filtersToAdd additional filters that may be added on top of the
+ * resulting LogicalJoin, if the join is a left or right outer join
+ * @param selfJoin true if the join being created is a self-join that's
+ * removable
+ *
+ * @return created LogicalJoin
+ */
+ private static HiveLoptJoinTree createJoinSubtree(
+ RelMetadataQuery mq,
+ RelBuilder relBuilder,
+ HiveLoptMultiJoin multiJoin,
+ HiveLoptJoinTree left,
+ HiveLoptJoinTree right,
+ RexNode condition,
+ JoinRelType joinType,
+ List<RexNode> filtersToAdd,
+ boolean fullAdjust,
+ boolean selfJoin) {
+ RexBuilder rexBuilder =
+ multiJoin.getMultiJoinRel().getCluster().getRexBuilder();
+
+ // swap the inputs if beneficial
+ if (swapInputs(mq, multiJoin, left, right, selfJoin)) {
+ HiveLoptJoinTree tmp = right;
+ right = left;
+ left = tmp;
+ if (!fullAdjust) {
+ condition =
+ swapFilter(
+ rexBuilder,
+ multiJoin,
+ right,
+ left,
+ condition);
+ }
+ if ((joinType != JoinRelType.INNER)
+ && (joinType != JoinRelType.FULL)) {
+ joinType =
+ (joinType == JoinRelType.LEFT) ? JoinRelType.RIGHT
+ : JoinRelType.LEFT;
+ }
+ }
+
+ if (fullAdjust) {
+ int [] adjustments = new int[multiJoin.getNumTotalFields()];
+ if (needsAdjustment(
+ multiJoin,
+ adjustments,
+ left,
+ right,
+ selfJoin)) {
+ condition =
+ condition.accept(
+ new RelOptUtil.RexInputConverter(
+ rexBuilder,
+ multiJoin.getMultiJoinFields(),
+ left.getJoinTree().getRowType().getFieldList(),
+ right.getJoinTree().getRowType().getFieldList(),
+ adjustments));
+ }
+ }
+
+ relBuilder.push(left.getJoinTree())
+ .push(right.getJoinTree())
+ .join(joinType, condition);
+
+ // if this is a left or right outer join, and additional filters can
+ // be applied to the resulting join, then they need to be applied
+ // as a filter on top of the outer join result
+ if ((joinType == JoinRelType.LEFT) || (joinType == JoinRelType.RIGHT)) {
+ assert !selfJoin;
+ addAdditionalFilters(
+ relBuilder,
+ multiJoin,
+ left,
+ right,
+ filtersToAdd);
+ }
+
+ return new HiveLoptJoinTree(
+ relBuilder.build(),
+ left.getFactorTree(),
+ right.getFactorTree(),
+ selfJoin);
+ }
+
+ /**
+ * Determines whether any additional filters are applicable to a join tree.
+ * If there are any, creates a filter node on top of the join tree with the
+ * additional filters.
+ *
+ * @param relBuilder Builder holding current join tree
+ * @param multiJoin join factors being optimized
+ * @param left left side of join tree
+ * @param right right side of join tree
+ * @param filtersToAdd remaining filters
+ */
+ private static void addAdditionalFilters(
+ RelBuilder relBuilder,
+ HiveLoptMultiJoin multiJoin,
+ HiveLoptJoinTree left,
+ HiveLoptJoinTree right,
+ List<RexNode> filtersToAdd) {
+ RexNode filterCond =
+ addFilters(multiJoin, left, -1, right, filtersToAdd, false);
+ if (!filterCond.isAlwaysTrue()) {
+ // adjust the filter to reflect the outer join output
+ int [] adjustments = new int[multiJoin.getNumTotalFields()];
+ if (needsAdjustment(multiJoin, adjustments, left, right, false)) {
+ RexBuilder rexBuilder =
+ multiJoin.getMultiJoinRel().getCluster().getRexBuilder();
+ filterCond =
+ filterCond.accept(
+ new RelOptUtil.RexInputConverter(
+ rexBuilder,
+ multiJoin.getMultiJoinFields(),
+ relBuilder.peek().getRowType().getFieldList(),
+ adjustments));
+ relBuilder.filter(filterCond);
+ }
+ }
+ }
+
+ /**
+ * Swaps the operands to a join, so the smaller input is on the right. Or,
+ * if this is a removable self-join, swap so the factor that should be
+ * preserved when the self-join is removed is put on the left.
+ *
+ * @param multiJoin join factors being optimized
+ * @param left left side of join tree
+ * @param right right hand side of join tree
+ * @param selfJoin true if the join is a removable self-join
+ *
+ * @return true if swapping should be done
+ */
+ private static boolean swapInputs(
+ RelMetadataQuery mq,
+ HiveLoptMultiJoin multiJoin,
+ HiveLoptJoinTree left,
+ HiveLoptJoinTree right,
+ boolean selfJoin) {
+ boolean swap = false;
+
+ if (selfJoin) {
+ return !multiJoin.isLeftFactorInRemovableSelfJoin(
+ ((HiveLoptJoinTree.Leaf) left.getFactorTree()).getId());
+ }
+
+ final Double leftRowCount = mq.getRowCount(left.getJoinTree());
+ final Double rightRowCount = mq.getRowCount(right.getJoinTree());
+
+ // The left side is smaller than the right if it has fewer rows,
+ // or if it has the same number of rows as the right (excluding
+ // roundoff), but fewer columns.
+ if ((leftRowCount != null)
+ && (rightRowCount != null)
+ && ((leftRowCount < rightRowCount)
+ || ((Math.abs(leftRowCount - rightRowCount) < RelOptUtil.EPSILON)
+ && (rowWidthCost(left.getJoinTree()) < rowWidthCost(right.getJoinTree()))))) {
+ swap = true;
+ }
+ return swap;
+ }
+
+ /**
+ * Adjusts a filter to reflect swapping of join inputs.
+ *
+ * @param rexBuilder rexBuilder
+ * @param multiJoin join factors being optimized
+ * @param origLeft original LHS of the join tree (before swap)
+ * @param origRight original RHS of the join tree (before swap)
+ * @param condition original join condition
+ *
+ * @return join condition reflect swap of join inputs
+ */
+ private static RexNode swapFilter(
+ RexBuilder rexBuilder,
+ HiveLoptMultiJoin multiJoin,
+ HiveLoptJoinTree origLeft,
+ HiveLoptJoinTree origRight,
+ RexNode condition) {
+ int nFieldsOnLeft =
+ origLeft.getJoinTree().getRowType().getFieldCount();
+ int nFieldsOnRight =
+ origRight.getJoinTree().getRowType().getFieldCount();
+ int [] adjustments = new int[nFieldsOnLeft + nFieldsOnRight];
+
+ for (int i = 0; i < nFieldsOnLeft; i++) {
+ adjustments[i] = nFieldsOnRight;
+ }
+ for (int i = nFieldsOnLeft; i < (nFieldsOnLeft + nFieldsOnRight); i++) {
+ adjustments[i] = -nFieldsOnLeft;
+ }
+
+ condition =
+ condition.accept(
+ new RelOptUtil.RexInputConverter(
+ rexBuilder,
+ multiJoin.getJoinFields(origLeft, origRight),
+ multiJoin.getJoinFields(origRight, origLeft),
+ adjustments));
+
+ return condition;
+ }
+
+ /**
+ * Sets an array indicating how much each factor in a join tree needs to be
+ * adjusted to reflect the tree's join ordering.
+ *
+ * @param multiJoin join factors being optimized
+ * @param adjustments array to be filled out
+ * @param joinTree join tree
+ * @param otherTree null unless joinTree only represents the left side of
+ * the join tree
+ * @param selfJoin true if no adjustments need to be made for self-joins
+ *
+ * @return true if some adjustment is required; false otherwise
+ */
+ private static boolean needsAdjustment(
+ HiveLoptMultiJoin multiJoin,
+ int[] adjustments,
+ HiveLoptJoinTree joinTree,
+ HiveLoptJoinTree otherTree,
+ boolean selfJoin) {
+ boolean needAdjustment = false;
+
+ final List<Integer> joinOrder = new ArrayList<>();
+ joinTree.getTreeOrder(joinOrder);
+ if (otherTree != null) {
+ otherTree.getTreeOrder(joinOrder);
+ }
+
+ int nFields = 0;
+ for (int newPos = 0; newPos < joinOrder.size(); newPos++) {
+ int origPos = joinOrder.get(newPos);
+ int joinStart = multiJoin.getJoinStart(origPos);
+
+ // Determine the adjustments needed for join references. Note
+ // that if the adjustment is being done for a self-join filter,
+ // we always use the default adjustment value rather than
+ // remapping the right factor to reference the left factor.
+ // Otherwise, we have no way of later identifying that the join is
+ // self-join.
+ if (remapJoinReferences(
+ multiJoin,
+ origPos,
+ joinOrder,
+ newPos,
+ adjustments,
+ joinStart,
+ nFields,
+ selfJoin)) {
+ needAdjustment = true;
+ }
+ nFields += multiJoin.getNumFieldsInJoinFactor(origPos);
+ }
+
+ return needAdjustment;
+ }
+
+ /**
+ * Determines whether a join is a removable self-join. It is if it's an
+ * inner join between identical, simple factors and the equality portion of
+ * the join condition consists of the same set of unique keys.
+ *
+ * @param joinRel the join
+ *
+ * @return true if the join is removable
+ */
+ public static boolean isRemovableSelfJoin(Join joinRel) {
+ final RelNode left = joinRel.getLeft();
+ final RelNode right = joinRel.getRight();
+
+ if (joinRel.getJoinType().isOuterJoin()) {
+ return false;
+ }
+
+ // Make sure the join is between the same simple factor
+ final RelMetadataQuery mq = joinRel.getCluster().getMetadataQuery();
+ final RelOptTable leftTable = mq.getTableOrigin(left);
+ if (leftTable == null) {
+ return false;
+ }
+ final RelOptTable rightTable = mq.getTableOrigin(right);
+ if (rightTable == null) {
+ return false;
+ }
+ if (!leftTable.getQualifiedName().equals(rightTable.getQualifiedName())) {
+ return false;
+ }
+
+ // Determine if the join keys are identical and unique
+ return areSelfJoinKeysUnique(mq, left, right, joinRel.getCondition());
+ }
+
+ /**
+ * Determines if the equality portion of a self-join condition is between
+ * identical keys that are unique.
+ *
+ * @param mq Metadata query
+ * @param leftRel left side of the join
+ * @param rightRel right side of the join
+ * @param joinFilters the join condition
+ *
+ * @return true if the equality join keys are the same and unique
+ */
+ private static boolean areSelfJoinKeysUnique(RelMetadataQuery mq,
+ RelNode leftRel, RelNode rightRel, RexNode joinFilters) {
+ final JoinInfo joinInfo = JoinInfo.of(leftRel, rightRel, joinFilters);
+
+ // Make sure each key on the left maps to the same simple column as the
+ // corresponding key on the right
+ for (IntPair pair : joinInfo.pairs()) {
+ final RelColumnOrigin leftOrigin =
+ mq.getColumnOrigin(leftRel, pair.source);
+ if (leftOrigin == null || leftOrigin.isDerived()) {
+ return false;
+ }
+ final RelColumnOrigin rightOrigin =
+ mq.getColumnOrigin(rightRel, pair.target);
+ if (rightOrigin == null || rightOrigin.isDerived()) {
+ return false;
+ }
+ if (leftOrigin.getOriginColumnOrdinal()
+ != rightOrigin.getOriginColumnOrdinal()) {
+ return false;
+ }
+ }
+
+ // Now that we've verified that the keys are the same, see if they
+ // are unique. When removing self-joins, if needed, we'll later add an
+ // IS NOT NULL filter on the join keys that are nullable. Therefore,
+ // it's ok if there are nulls in the unique key.
+ return RelMdUtil.areColumnsDefinitelyUniqueWhenNullsFiltered(mq, leftRel,
+ joinInfo.leftSet());
+ }
+
+ public static final RelOptRule INSTANCE =
+ new Config().withOperandSupplier(b -> b.operand(MultiJoin.class).anyInputs())
+ .withDescription("HiveLoptOptimizeJoinRule")
+ .toRule();
+
+ public static class Config extends HiveRuleConfig {
+ @Override
+ public HiveLoptOptimizeJoinRule toRule() {
+ return new HiveLoptOptimizeJoinRule(this);
+ }
+ }
+}
diff --git a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveLoptSemiJoinOptimizer.java b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveLoptSemiJoinOptimizer.java
new file mode 100644
index 00000000000..22b70053fc1
--- /dev/null
+++ b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveLoptSemiJoinOptimizer.java
@@ -0,0 +1,861 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.optimizer.calcite.rules;
+
+import org.apache.calcite.plan.RelOptCost;
+import org.apache.calcite.plan.RelOptTable;
+import org.apache.calcite.plan.RelOptUtil;
+import org.apache.calcite.plan.ViewExpanders;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.JoinInfo;
+import org.apache.calcite.rel.core.JoinRelType;
+import org.apache.calcite.rel.logical.LogicalJoin;
+import org.apache.calcite.rel.metadata.RelColumnOrigin;
+import org.apache.calcite.rel.metadata.RelMdUtil;
+import org.apache.calcite.rel.metadata.RelMetadataQuery;
+import org.apache.calcite.rex.RexBuilder;
+import org.apache.calcite.rex.RexCall;
+import org.apache.calcite.rex.RexInputRef;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.rex.RexUtil;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.fun.SqlStdOperatorTable;
+import org.apache.calcite.util.ImmutableBitSet;
+import org.apache.calcite.util.Util;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableSet;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Ordering;
+
+import org.apache.hadoop.hive.ql.optimizer.calcite.Bug;
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.List;
+import java.util.ListIterator;
+import java.util.Map;
+import java.util.Set;
+
+import static java.util.Objects.requireNonNull;
+
+/**
+ * Implements the logic for determining the optimal
+ * semi-joins to be used in processing joins in a query.
+ */
+public class HiveLoptSemiJoinOptimizer {
+ //~ Static fields/initializers ---------------------------------------------
+
+ // minimum score required for a join filter to be considered
+ private static final int THRESHOLD_SCORE = 10;
+
+ //~ Instance fields --------------------------------------------------------
+
+ private final RexBuilder rexBuilder;
+
+ private final RelMetadataQuery mq;
+
+ /**
+ * Semijoins corresponding to each join factor, if they are going to be
+ * filtered by semijoins. Otherwise, the entry is the original join factor.
+ */
+ private RelNode [] chosenSemiJoins;
+
+ /**
+ * Associates potential semijoins with each fact table factor. The first
+ * parameter in the map corresponds to the fact table. The second
+ * corresponds to the dimension table and a SemiJoin that captures all
+ * the necessary semijoin data between that fact and dimension table
+ */
+ private final Map<Integer, Map<Integer, LogicalJoin>> possibleSemiJoins = new HashMap<>();
+
+ private final Ordering<Integer> factorCostOrdering =
+ Ordering.from(new FactorCostComparator());
+
+ //~ Constructors -----------------------------------------------------------
+
+ public HiveLoptSemiJoinOptimizer(
+ RelMetadataQuery mq,
+ HiveLoptMultiJoin multiJoin,
+ RexBuilder rexBuilder) {
+ if (Bug.CALCITE_6737_FIXED) {
+ throw new IllegalStateException("This class is redundant when the fix for CALCITE-6737 is merged into Calcite. "
+ + "Remove it and use LoptSemiJoinOptimizer instead of HiveLoptSemiJoinOptimizer");
+ }
+
+ // there are no semijoins yet, so initialize to the original
+ // factors
+ this.mq = mq;
+ int nJoinFactors = multiJoin.getNumJoinFactors();
+ chosenSemiJoins = new RelNode[nJoinFactors];
+ for (int i = 0; i < nJoinFactors; i++) {
+ chosenSemiJoins[i] = multiJoin.getJoinFactor(i);
+ }
+
+ this.rexBuilder = rexBuilder;
+ }
+
+ //~ Methods ----------------------------------------------------------------
+
+ /**
+ * Determines all possible semijoins that can be used by dimension tables to
+ * filter fact tables. Constructs SemiJoinRels corresponding to potential
+ * dimension table filters and stores them in the member field
+ * "possibleSemiJoins"
+ *
+ * @param multiJoin join factors being optimized
+ */
+ public void makePossibleSemiJoins(HiveLoptMultiJoin multiJoin) {
+ possibleSemiJoins.clear();
+
+ // semijoins can't be used with any type of outer join, including full
+ if (multiJoin.getMultiJoinRel().isFullOuterJoin()) {
+ return;
+ }
+
+ int nJoinFactors = multiJoin.getNumJoinFactors();
+ for (int factIdx = 0; factIdx < nJoinFactors; factIdx++) {
+ final Map<Integer, List<RexNode>> dimFilters = new HashMap<>();
+ final Map<Integer, LogicalJoin> semiJoinMap = new HashMap<>();
+
+ // loop over all filters and find equality filters that reference
+ // this factor and one other factor
+ for (RexNode joinFilter : multiJoin.getJoinFilters()) {
+ int dimIdx = isSuitableFilter(multiJoin, joinFilter, factIdx);
+ if (dimIdx == -1) {
+ continue;
+ }
+
+ // if either the fact or dimension table is null generating,
+ // we cannot use semijoins
+ if (multiJoin.isNullGenerating(factIdx)
+ || multiJoin.isNullGenerating(dimIdx)) {
+ continue;
+ }
+
+ // if we've already matched against this dimension factor,
+ // then add the filter to the list associated with
+ // that dimension factor; otherwise, create a new entry
+ List<RexNode> currDimFilters = dimFilters.get(dimIdx);
+ if (currDimFilters == null) {
+ currDimFilters = new ArrayList<>();
+ }
+ currDimFilters.add(joinFilter);
+ dimFilters.put(dimIdx, currDimFilters);
+ }
+
+ // if there are potential dimension filters, determine if there
+ // are appropriate indexes
+ Set<Integer> dimIdxes = dimFilters.keySet();
+ for (Integer dimIdx : dimIdxes) {
+ List<RexNode> joinFilters = dimFilters.get(dimIdx);
+ if (joinFilters != null) {
+ LogicalJoin semiJoin =
+ findSemiJoinIndexByCost(
+ multiJoin,
+ joinFilters,
+ factIdx,
+ dimIdx);
+
+ // if an index is available, keep track of it as a
+ // possible semijoin
+ if (semiJoin != null) {
+ semiJoinMap.put(dimIdx, semiJoin);
+ possibleSemiJoins.put(factIdx, semiJoinMap);
+ }
+ }
+ }
+ }
+ }
+
+ /**
+ * Determines if a join filter can be used with a semijoin against a
+ * specified fact table. A suitable filter is of the form "factable.col1 =
+ * dimTable.col2".
+ *
+ * @param multiJoin join factors being optimized
+ * @param joinFilter filter to be analyzed
+ * @param factIdx index corresponding to the fact table
+ *
+ * @return index of corresponding dimension table if the filter is
+ * appropriate; otherwise -1 is returned
+ */
+ private static int isSuitableFilter(
+ HiveLoptMultiJoin multiJoin,
+ RexNode joinFilter,
+ int factIdx) {
+ // ignore non-equality filters where the operands are not
+ // RexInputRefs
+ switch (joinFilter.getKind()) {
+ case EQUALS:
+ break;
+ default:
+ return -1;
+ }
+ List<RexNode> operands = ((RexCall) joinFilter).getOperands();
+ if (!(operands.get(0) instanceof RexInputRef)
+ || !(operands.get(1) instanceof RexInputRef)) {
+ return -1;
+ }
+
+ // filter is suitable if each side of the filter only contains a
+ // single factor reference and one side references the fact table and
+ // the other references the dimension table; since we know this is
+ // a join filter and we've already verified that the operands are
+ // RexInputRefs, verify that the factors belong to the fact and
+ // dimension table
+ ImmutableBitSet joinRefs = multiJoin.getFactorsRefByJoinFilter(joinFilter);
+ assert joinRefs.cardinality() == 2;
+ int factor1 = joinRefs.nextSetBit(0);
+ int factor2 = joinRefs.nextSetBit(factor1 + 1);
+ if (factor1 == factIdx) {
+ return factor2;
+ }
+ if (factor2 == factIdx) {
+ return factor1;
+ }
+ return -1;
+ }
+
+ /**
+ * Given a list of possible filters on a fact table, determine if there is
+ * an index that can be used, provided all the fact table keys originate
+ * from the same underlying table.
+ *
+ * @param multiJoin join factors being optimized
+ * @param joinFilters filters to be used on the fact table
+ * @param factIdx index in join factors corresponding to the fact table
+ * @param dimIdx index in join factors corresponding to the dimension table
+ *
+ * @return SemiJoin containing information regarding the semijoin that
+ * can be used to filter the fact table
+ */
+ private @Nullable LogicalJoin findSemiJoinIndexByCost(
+ HiveLoptMultiJoin multiJoin,
+ List<RexNode> joinFilters,
+ int factIdx,
+ int dimIdx) {
+ // create a SemiJoin with the semi-join condition and keys
+ RexNode semiJoinCondition =
+ RexUtil.composeConjunction(rexBuilder, joinFilters);
+
+ int leftAdjustment = 0;
+ for (int i = 0; i < factIdx; i++) {
+ leftAdjustment -= multiJoin.getNumFieldsInJoinFactor(i);
+ }
+
+ semiJoinCondition =
+ adjustSemiJoinCondition(
+ multiJoin,
+ leftAdjustment,
+ semiJoinCondition,
+ factIdx,
+ dimIdx);
+
+ RelNode factRel = multiJoin.getJoinFactor(factIdx);
+ RelNode dimRel = multiJoin.getJoinFactor(dimIdx);
+ final JoinInfo joinInfo = JoinInfo.of(factRel, dimRel, semiJoinCondition);
+ assert joinInfo.leftKeys.size() > 0;
+
+ // mutable copies
+ final List<Integer> leftKeys = Lists.newArrayList(joinInfo.leftKeys);
+ final List<Integer> rightKeys = Lists.newArrayList(joinInfo.rightKeys);
+
+ // make sure all the fact table keys originate from the same table
+ // and are simple column references
+ final List<Integer> actualLeftKeys = new ArrayList<>();
+ LcsTable factTable =
+ validateKeys(
+ factRel,
+ leftKeys,
+ rightKeys,
+ actualLeftKeys);
+ if (factTable == null) {
+ return null;
+ }
+
+ // find the best index
+ final List<Integer> bestKeyOrder = new ArrayList<>();
+ LcsTableScan tmpFactRel =
+ (LcsTableScan) factTable.toRel(
+ ViewExpanders.simpleContext(factRel.getCluster()));
+
+ LcsIndexOptimizer indexOptimizer = new LcsIndexOptimizer(tmpFactRel);
+ FemLocalIndex bestIndex =
+ indexOptimizer.findSemiJoinIndexByCost(
+ dimRel,
+ actualLeftKeys,
+ rightKeys,
+ bestKeyOrder);
+
+ if (bestIndex == null) {
+ return null;
+ }
+
+ // if necessary, truncate the keys to reflect the ones that match
+ // the index and remove the corresponding, unnecessary filters from
+ // the condition; note that we don't save the actual keys here because
+ // later when the semijoin is pushed past other RelNodes, the keys will
+ // be converted
+ final List<Integer> truncatedLeftKeys;
+ final List<Integer> truncatedRightKeys;
+ if (actualLeftKeys.size() == bestKeyOrder.size()) {
+ truncatedLeftKeys = leftKeys;
+ truncatedRightKeys = rightKeys;
+ } else {
+ truncatedLeftKeys = new ArrayList<>();
+ truncatedRightKeys = new ArrayList<>();
+ for (int key : bestKeyOrder) {
+ truncatedLeftKeys.add(leftKeys.get(key));
+ truncatedRightKeys.add(rightKeys.get(key));
+ }
+ semiJoinCondition =
+ removeExtraFilters(
+ truncatedLeftKeys,
+ multiJoin.getNumFieldsInJoinFactor(factIdx),
+ semiJoinCondition);
+ }
+ return LogicalJoin.create(factRel, dimRel, ImmutableList.of(),
+ requireNonNull(semiJoinCondition, "semiJoinCondition"),
+ ImmutableSet.of(), JoinRelType.SEMI);
+ }
+
+ /**
+ * Modifies the semijoin condition to reflect the fact that the RHS is now
+ * the second factor into a join and the LHS is the first.
+ *
+ * @param multiJoin join factors being optimized
+ * @param leftAdjustment amount the left RexInputRefs need to be adjusted by
+ * @param semiJoinCondition condition to be adjusted
+ * @param leftIdx index of the join factor corresponding to the LHS of the
+ * semijoin,
+ * @param rightIdx index of the join factor corresponding to the RHS of the
+ * semijoin
+ *
+ * @return modified semijoin condition
+ */
+ private RexNode adjustSemiJoinCondition(
+ HiveLoptMultiJoin multiJoin,
+ int leftAdjustment,
+ RexNode semiJoinCondition,
+ int leftIdx,
+ int rightIdx) {
+ // adjust the semijoin condition to reflect the fact that the
+ // RHS is now the second factor into the semijoin and the LHS
+ // is the first
+ int rightAdjustment = 0;
+ for (int i = 0; i < rightIdx; i++) {
+ rightAdjustment -= multiJoin.getNumFieldsInJoinFactor(i);
+ }
+ int rightStart = -rightAdjustment;
+ int numFieldsLeftIdx = multiJoin.getNumFieldsInJoinFactor(leftIdx);
+ int numFieldsRightIdx = multiJoin.getNumFieldsInJoinFactor(rightIdx);
+ rightAdjustment += numFieldsLeftIdx;
+
+ // only adjust the filter if adjustments are required
+ if ((leftAdjustment != 0) || (rightAdjustment != 0)) {
+ int [] adjustments = new int[multiJoin.getNumTotalFields()];
+ if (leftAdjustment != 0) {
+ for (int i = -leftAdjustment;
+ i < (-leftAdjustment + numFieldsLeftIdx);
+ i++) {
+ adjustments[i] = leftAdjustment;
+ }
+ }
+ if (rightAdjustment != 0) {
+ for (int i = rightStart;
+ i < (rightStart + numFieldsRightIdx);
+ i++) {
+ adjustments[i] = rightAdjustment;
+ }
+ }
+ return semiJoinCondition.accept(
+ new RelOptUtil.RexInputConverter(
+ rexBuilder,
+ multiJoin.getMultiJoinFields(),
+ adjustments));
+ }
+
+ return semiJoinCondition;
+ }
+
+ /**
+ * Validates the candidate semijoin keys corresponding to the fact table.
+ * Ensure the keys all originate from the same underlying table, and they
+ * all correspond to simple column references. If unsuitable keys are found,
+ * they're removed from the key list and a new list corresponding to the
+ * remaining valid keys is returned.
+ *
+ * @param factRel fact table RelNode
+ * @param leftKeys fact table semijoin keys
+ * @param rightKeys dimension table semijoin keys
+ * @param actualLeftKeys the remaining valid fact table semijoin keys
+ *
+ * @return the underlying fact table if the semijoin keys are valid;
+ * otherwise null
+ */
+ private @Nullable LcsTable validateKeys(
+ RelNode factRel,
+ List<Integer> leftKeys,
+ List<Integer> rightKeys,
+ List<Integer> actualLeftKeys) {
+ int keyIdx = 0;
+ RelOptTable theTable = null;
+ ListIterator<Integer> keyIter = leftKeys.listIterator();
+ while (keyIter.hasNext()) {
+ boolean removeKey = false;
+ final RelColumnOrigin colOrigin =
+ mq.getColumnOrigin(factRel, keyIter.next());
+
+ // can't use the rid column as a semijoin key
+ if ((colOrigin == null || !colOrigin.isDerived())
+ || LucidDbSpecialOperators.isLcsRidColumnId(
+ colOrigin.getOriginColumnOrdinal())) {
+ removeKey = true;
+ } else {
+ RelOptTable table = colOrigin.getOriginTable();
+ if (theTable == null) {
+ if (!(table instanceof LcsTable)) {
+ // not a column store table
+ removeKey = true;
+ } else {
+ theTable = table;
+ }
+ } else {
+ // the tables must match because the column has
+ // a simple origin
+ assert table == theTable;
+ }
+ }
+ if (colOrigin != null && !removeKey) {
+ actualLeftKeys.add(colOrigin.getOriginColumnOrdinal());
+ keyIdx++;
+ } else {
+ keyIter.remove();
+ rightKeys.remove(keyIdx);
+ }
+ }
+
+ // if all keys have been removed, then we don't have any valid semijoin
+ // keys
+ if (actualLeftKeys.isEmpty()) {
+ return null;
+ } else {
+ return (LcsTable) theTable;
+ }
+ }
+
+ /**
+ * Removes from an expression any sub-expressions that reference key values
+ * that aren't contained in a key list passed in. The keys represent join
+ * keys on one side of a join. The subexpressions are all assumed to be of
+ * the form "tab1.col1 = tab2.col2".
+ *
+ * @param keys join keys from one side of the join
+ * @param nFields number of fields in the side of the join for which the
+ * keys correspond
+ * @param condition original expression
+ *
+ * @return modified expression with filters that don't reference specified
+ * keys removed
+ */
+ private @Nullable RexNode removeExtraFilters(
+ List<Integer> keys,
+ int nFields,
+ RexNode condition) {
+ // recursively walk the expression; if all sub-expressions are
+ // removed from one side of the expression, just return what remains
+ // from the other side
+ assert condition instanceof RexCall;
+ RexCall call = (RexCall) condition;
+ if (condition.isA(SqlKind.AND)) {
+ List<RexNode> operands = call.getOperands();
+ RexNode left =
+ removeExtraFilters(
+ keys,
+ nFields,
+ operands.get(0));
+ RexNode right =
+ removeExtraFilters(
+ keys,
+ nFields,
+ operands.get(1));
+ if (left == null) {
+ return right;
+ }
+ if (right == null) {
+ return left;
+ }
+ return rexBuilder.makeCall(
+ SqlStdOperatorTable.AND,
+ left,
+ right);
+ }
+
+ // determine which side of the equality filter references the join
+ // operand we're interested in; then, check if it is contained in
+ // our key list
+ assert call.getOperator() == SqlStdOperatorTable.EQUALS;
+ List<RexNode> operands = call.getOperands();
+ assert operands.get(0) instanceof RexInputRef;
+ assert operands.get(1) instanceof RexInputRef;
+ int idx = ((RexInputRef) operands.get(0)).getIndex();
+ if (idx < nFields) {
+ if (!keys.contains(idx)) {
+ return null;
+ }
+ } else {
+ idx = ((RexInputRef) operands.get(1)).getIndex();
+ if (!keys.contains(idx)) {
+ return null;
+ }
+ }
+ return condition;
+ }
+
+ /**
+ * Finds the optimal semijoin for filtering the least costly fact table from
+ * among the remaining possible semijoins to choose from. The chosen
+ * semijoin is stored in the chosenSemiJoins array
+ *
+ * @param multiJoin join factors being optimized
+ *
+ * @return true if a suitable semijoin is found; false otherwise
+ */
+ public boolean chooseBestSemiJoin(HiveLoptMultiJoin multiJoin) {
+ // sort the join factors based on the cost of each factor filtered by
+ // semijoins, if semijoins have been chosen
+ int nJoinFactors = multiJoin.getNumJoinFactors();
+ List<Integer> sortedFactors =
+ factorCostOrdering.immutableSortedCopy(Util.range(nJoinFactors));
+
+ // loop through the factors in sort order, treating the factor as
+ // a fact table; analyze the possible semijoins associated with
+ // that fact table
+ for (int i = 0; i < nJoinFactors; i++) {
+ Integer factIdx = sortedFactors.get(i);
+ RelNode factRel = chosenSemiJoins[factIdx];
+ Map<Integer, LogicalJoin> possibleDimensions =
+ possibleSemiJoins.get(factIdx);
+ if (possibleDimensions == null) {
+ continue;
+ }
+ double bestScore = 0.0;
+ int bestDimIdx = -1;
+
+ // loop through each dimension table associated with the current
+ // fact table and analyze the ones that have semijoins with this
+ // fact table
+ Set<Integer> dimIdxes = possibleDimensions.keySet();
+ for (Integer dimIdx : dimIdxes) {
+ LogicalJoin semiJoin = possibleDimensions.get(dimIdx);
+ if (semiJoin == null) {
+ continue;
+ }
+
+ // keep track of the dimension table that has the best score
+ // for filtering this fact table
+ double score =
+ computeScore(
+ factRel,
+ chosenSemiJoins[dimIdx],
+ semiJoin);
+ if ((score > THRESHOLD_SCORE) && (score > bestScore)) {
+ bestDimIdx = dimIdx;
+ bestScore = score;
+ }
+ }
+
+ // if a suitable dimension table has been found, associate it
+ // with the fact table in the chosenSemiJoins array; also remove
+ // the entry from possibleSemiJoins so we won't chose it again;
+ // note that we create the SemiJoin using the chosen semijoins
+ // already created for each factor so any chaining of filters will
+ // be accounted for
+ if (bestDimIdx != -1) {
+ int bestDimIdxFinal = bestDimIdx;
+ LogicalJoin semiJoin = requireNonNull(possibleDimensions.get(bestDimIdxFinal),
+ () -> "possibleDimensions.get(" + bestDimIdxFinal + ") is null");
+ LogicalJoin chosenSemiJoin =
+ LogicalJoin.create(factRel,
+ chosenSemiJoins[bestDimIdx],
+ ImmutableList.of(),
+ semiJoin.getCondition(),
+ ImmutableSet.of(),
+ JoinRelType.SEMI);
+ chosenSemiJoins[factIdx] = chosenSemiJoin;
+
+ // determine if the dimension table doesn't need to be joined
+ // as a result of this semijoin
+ removeJoin(multiJoin, chosenSemiJoin, factIdx, bestDimIdx);
+
+ removePossibleSemiJoin(
+ possibleDimensions,
+ factIdx,
+ bestDimIdx);
+
+ // need to also remove the semijoin from the possible
+ // semijoins associated with this dimension table, as the
+ // semijoin can only be used to filter one table, not both
+ removePossibleSemiJoin(
+ possibleSemiJoins.get(bestDimIdx),
+ bestDimIdx,
+ factIdx);
+ return true;
+ }
+
+ // continue searching on the next fact table if we couldn't find
+ // a semijoin for the current fact table
+ }
+
+ return false;
+ }
+
+ /**
+ * Computes a score relevant to applying a set of semijoins on a fact table.
+ * The higher the score, the better.
+ *
+ * @param factRel fact table being filtered
+ * @param dimRel dimension table that participates in semijoin
+ * @param semiJoin semijoin between fact and dimension tables
+ *
+ * @return computed score of applying the dimension table filters on the
+ * fact table
+ */
+ private double computeScore(
+ RelNode factRel,
+ RelNode dimRel,
+ LogicalJoin semiJoin) {
+ // Estimate savings as a result of applying semijoin filter on fact
+ // table. As a heuristic, the selectivity of the semijoin needs to
+ // be less than half. There may be instances where an even smaller
+ // selectivity value is required because of the overhead of
+ // index lookups on a very large fact table. Half was chosen as
+ // a middle ground based on testing that was done with a large
+ // data set.
+ final ImmutableBitSet dimCols = ImmutableBitSet.of(semiJoin.analyzeCondition().rightKeys);
+ final double selectivity =
+ RelMdUtil.computeSemiJoinSelectivity(mq, factRel, dimRel, semiJoin);
+ if (selectivity > .5) {
+ return 0;
+ }
+
+ final RelOptCost factCost = mq.getCumulativeCost(factRel);
+
+ // if not enough information, return a low score
+ if (factCost == null) {
+ return 0;
+ }
+ double savings =
+ (1.0 - Math.sqrt(selectivity))
+ * Math.max(1.0, factCost.getRows());
+
+ // Additional savings if the dimension columns are unique. We can
+ // ignore nulls since they will be filtered out by the semijoin.
+ boolean uniq =
+ RelMdUtil.areColumnsDefinitelyUniqueWhenNullsFiltered(mq,
+ dimRel, dimCols);
+ if (uniq) {
+ savings *= 2.0;
+ }
+
+ // compute the cost of doing an extra scan on the dimension table,
+ // including the distinct sort on top of the scan; if the dimension
+ // columns are already unique, no need to add on the dup removal cost
+ final Double dimSortCost = mq.getRowCount(dimRel);
+ final Double dupRemCost = uniq ? 0 : dimSortCost;
+ final RelOptCost dimCost = mq.getCumulativeCost(dimRel);
+ if ((dimSortCost == null)
+ || (dupRemCost == null)
+ || (dimCost == null)) {
+ return 0;
+ }
+
+ double dimRows = dimCost.getRows();
+ if (dimRows < 1.0) {
+ dimRows = 1.0;
+ }
+ return savings / dimRows;
+ }
+
+ /**
+ * Determines whether a join of the dimension table in a semijoin can be
+ * removed. It can be if the dimension keys are unique and the only fields
+ * referenced from the dimension table are its semijoin keys. The semijoin
+ * keys can be mapped to the corresponding keys from the fact table (because
+ * of the equality condition associated with the semijoin keys). Therefore,
+ * that's why the dimension table can be removed even though those fields
+ * are referenced elsewhere in the query tree.
+ *
+ * @param multiJoin join factors being optimized
+ * @param semiJoin semijoin under consideration
+ * @param factIdx id of the fact table in the semijoin
+ * @param dimIdx id of the dimension table in the semijoin
+ */
+ private void removeJoin(
+ HiveLoptMultiJoin multiJoin,
+ LogicalJoin semiJoin,
+ int factIdx,
+ int dimIdx) {
+ // if the dimension can be removed because of another semijoin, then
+ // no need to proceed any further
+ if (multiJoin.getJoinRemovalFactor(dimIdx) != null) {
+ return;
+ }
+
+ // Check if the semijoin keys corresponding to the dimension table
+ // are unique. The semijoin will filter out the nulls.
+ final ImmutableBitSet dimKeys = ImmutableBitSet.of(semiJoin.analyzeCondition().rightKeys);
+ final RelNode dimRel = multiJoin.getJoinFactor(dimIdx);
+ if (!RelMdUtil.areColumnsDefinitelyUniqueWhenNullsFiltered(mq, dimRel,
+ dimKeys)) {
+ return;
+ }
+
+ // check that the only fields referenced from the dimension table
+ // in either its projection or join conditions are the dimension
+ // keys
+ ImmutableBitSet dimProjRefs = multiJoin.getProjFields(dimIdx);
+ if (dimProjRefs == null) {
+ int nDimFields = multiJoin.getNumFieldsInJoinFactor(dimIdx);
+ dimProjRefs = ImmutableBitSet.range(0, nDimFields);
+ }
+ if (!dimKeys.contains(dimProjRefs)) {
+ return;
+ }
+ int [] dimJoinRefCounts = multiJoin.getJoinFieldRefCounts(dimIdx);
+ for (int i = 0; i < dimJoinRefCounts.length; i++) {
+ if (dimJoinRefCounts[i] > 0) {
+ if (!dimKeys.get(i)) {
+ return;
+ }
+ }
+ }
+
+ // criteria met; keep track of the fact table and the semijoin that
+ // allow the join of this dimension table to be removed
+ multiJoin.setJoinRemovalFactor(dimIdx, factIdx);
+ multiJoin.setJoinRemovalSemiJoin(dimIdx, semiJoin);
+
+ // if the dimension table doesn't reference anything in its projection
+ // and the only fields referenced in its joins are the dimension keys
+ // of this semijoin, then we can decrement the join reference counts
+ // corresponding to the fact table's semijoin keys, since the
+ // dimension table doesn't need to use those keys
+ if (dimProjRefs.cardinality() != 0) {
+ return;
+ }
+ for (int i = 0; i < dimJoinRefCounts.length; i++) {
+ if (dimJoinRefCounts[i] > 1) {
+ return;
+ } else if (dimJoinRefCounts[i] == 1) {
+ if (!dimKeys.get(i)) {
+ return;
+ }
+ }
+ }
+ int [] factJoinRefCounts = multiJoin.getJoinFieldRefCounts(factIdx);
+ for (Integer key : semiJoin.analyzeCondition().leftKeys) {
+ factJoinRefCounts[key]--;
+ }
+ }
+
+ /**
+ * Removes a dimension table from a fact table's list of possible semi-joins.
+ *
+ * @param possibleDimensions possible dimension tables associated with the
+ * fact table
+ * @param factIdx index corresponding to fact table
+ * @param dimIdx index corresponding to dimension table
+ */
+ private void removePossibleSemiJoin(
+ @Nullable Map<Integer, LogicalJoin> possibleDimensions,
+ Integer factIdx,
+ Integer dimIdx) {
+ // dimension table may not have a corresponding semijoin if it
+ // wasn't indexable
+ if (possibleDimensions == null) {
+ return;
+ }
+ possibleDimensions.remove(dimIdx);
+ if (possibleDimensions.isEmpty()) {
+ possibleSemiJoins.remove(factIdx);
+ } else {
+ possibleSemiJoins.put(factIdx, possibleDimensions);
+ }
+ }
+
+ /**
+ * Returns the optimal semijoin for the specified factor; may be the factor
+ * itself if semijoins are not chosen for the factor.
+ *
+ * @param factIdx Index corresponding to the desired factor
+ */
+ public RelNode getChosenSemiJoin(int factIdx) {
+ return chosenSemiJoins[factIdx];
+ }
+
+ //~ Inner Classes ----------------------------------------------------------
+
+ /** Compares factors. */
+ private class FactorCostComparator
+ implements Comparator<Integer> {
+ @Override public int compare(Integer rel1Idx, Integer rel2Idx) {
+ RelOptCost c1 =
+ mq.getCumulativeCost(chosenSemiJoins[rel1Idx]);
+ RelOptCost c2 =
+ mq.getCumulativeCost(chosenSemiJoins[rel2Idx]);
+
+ // nulls are arbitrarily sorted
+ if ((c1 == null) || (c2 == null)) {
+ return -1;
+ }
+ return c1.isLt(c2) ? -1 : (c1.equals(c2) ? 0 : 1);
+ }
+ }
+
+ /** Dummy class to allow code to compile. */
+ private abstract static class LcsTable implements RelOptTable {
+ }
+
+ /** Dummy class to allow code to compile. */
+ private static class LcsTableScan {
+ }
+
+ /** Dummy class to allow code to compile. */
+ private static class LcsIndexOptimizer {
+ LcsIndexOptimizer(LcsTableScan rel) {}
+
+ public @Nullable FemLocalIndex findSemiJoinIndexByCost(RelNode dimRel,
+ List<Integer> actualLeftKeys, List<Integer> rightKeys,
+ List<Integer> bestKeyOrder) {
+ return null;
+ }
+ }
+
+ /** Dummy class to allow code to compile. */
+ private static class FemLocalIndex { }
+
+ /** Dummy class to allow code to compile. */
+ private static class LucidDbSpecialOperators {
+ public static boolean isLcsRidColumnId(int originColumnOrdinal) {
+ return false;
+ }
+ }
+}
diff --git a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HivePointLookupOptimizerRule.java b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HivePointLookupOptimizerRule.java
index bd43b7d6437..92dd1790cb9 100644
--- a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HivePointLookupOptimizerRule.java
+++ b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HivePointLookupOptimizerRule.java
@@ -602,7 +602,7 @@ private static RexNode handleAND(RexBuilder rexBuilder, RexCall call) {
for (int i = 0; i < operands.size(); i++) {
RexNode operand = operands.get(i);
- if (operand.getKind() == SqlKind.IN) {
+ if (operand instanceof RexCall && HiveIn.INSTANCE.equals(((RexCall) operand).op)) {
RexCall inCall = (RexCall) operand;
if (!HiveCalciteUtil.isDeterministic(inCall.getOperands().get(0))) {
continue;
@@ -737,7 +737,7 @@ private static RexNode handleOR(RexBuilder rexBuilder, RexCall call) {
final Multimap<RexNode,SimilarRexNodeElement> inLHSExprToRHSExprs = LinkedHashMultimap.create();
for (int i = 0; i < operands.size(); i++) {
RexNode operand = operands.get(i);
- if (operand.getKind() == SqlKind.IN) {
+ if (operand instanceof RexCall && HiveIn.INSTANCE.equals(((RexCall) operand).op)) {
RexCall inCall = (RexCall) operand;
if (!HiveCalciteUtil.isDeterministic(inCall.getOperands().get(0))) {
continue;
diff --git a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HivePreFilteringRule.java b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HivePreFilteringRule.java
index c78e7a42346..e33908cc6b5 100644
--- a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HivePreFilteringRule.java
+++ b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HivePreFilteringRule.java
@@ -23,13 +23,14 @@
import java.util.List;
import java.util.Set;
-import org.apache.calcite.plan.RelOptRule;
+import org.apache.calcite.plan.RelOptPredicateList;
import org.apache.calcite.plan.RelOptRuleCall;
import org.apache.calcite.plan.RelOptUtil;
import org.apache.calcite.rel.RelNode;
import org.apache.calcite.rel.core.Filter;
import org.apache.calcite.rel.core.RelFactories.FilterFactory;
import org.apache.calcite.rel.core.TableScan;
+import org.apache.calcite.rel.rules.ReduceExpressionsRule;
import org.apache.calcite.rex.RexBuilder;
import org.apache.calcite.rex.RexCall;
import org.apache.calcite.rex.RexNode;
@@ -45,7 +46,15 @@
import com.google.common.collect.Multimap;
import com.google.common.collect.Sets;
-public class HivePreFilteringRule extends RelOptRule {
+import static org.apache.calcite.rel.rules.ReduceExpressionsRule.FilterReduceExpressionsRule.FilterReduceExpressionsRuleConfig;
+
+/**
+ * A rule that splits a {@code Filter} operator into two, deterministic and non-deterministic, to facilitate push down
+ * of the latter by other rules.
+ * It is a reduction rule cause along with splitting it needs to reduce some of the new conditions to ensure that it
+ * can reach a fix point.
+ */
+public class HivePreFilteringRule extends ReduceExpressionsRule<ReduceExpressionsRule.Config> {
protected static final Logger LOG = LoggerFactory.getLogger(HivePreFilteringRule.class);
@@ -55,7 +64,10 @@ public class HivePreFilteringRule extends RelOptRule {
private final int maxCNFNodeCount;
public HivePreFilteringRule(int maxCNFNodeCount) {
- super(operand(Filter.class, operand(RelNode.class, any())));
+ super(FilterReduceExpressionsRuleConfig.DEFAULT.withMatchNullability(false)
+ .withOperandSupplier(o -> o.operand(Filter.class).oneInput(c -> c.operand(RelNode.class).anyInputs()))
+ .withDescription("HivePreFilteringRule")
+ .as(ReduceExpressionsRule.Config.class));
this.filterFactory = HiveRelFactories.HIVE_FILTER_FACTORY;
this.maxCNFNodeCount = maxCNFNodeCount;
}
@@ -158,6 +170,15 @@ public void onMatch(RelOptRuleCall call) {
return;
}
+ // The created expressions must be reduced/folded before we check if they are pushed already below.
+ // Comparing reduced with no reduced predicates may cause this rule to fire indefinitely failing to reach a
+ // fixpoint.
+ reduceExpressions(filter,
+ operandsToPushDown,
+ RelOptPredicateList.EMPTY,
+ true,
+ config.matchNullability(),
+ config.treatDynamicCallsAsConstant());
// 3. If the new conjuncts are already present in the plan, we bail out
final List<RexNode> newConjuncts = HiveCalciteUtil.getPredsNotPushedAlready(filter.getInput(),
operandsToPushDown);
diff --git a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveReduceExpressionsRule.java b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveReduceExpressionsRule.java
index 0521dc3e32a..919bd07eba6 100644
--- a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveReduceExpressionsRule.java
+++ b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveReduceExpressionsRule.java
@@ -46,11 +46,10 @@ private HiveReduceExpressionsRule() {
* {@link org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveFilter}.
*/
public static final RelOptRule FILTER_INSTANCE =
- ReduceExpressionsRule.FilterReduceExpressionsRule.Config.DEFAULT
+ ReduceExpressionsRule.FilterReduceExpressionsRule.FilterReduceExpressionsRuleConfig.DEFAULT
.withOperandFor(HiveFilter.class)
.withMatchNullability(false)
.withRelBuilderFactory(HiveRelFactories.HIVE_BUILDER)
- .as(ReduceExpressionsRule.FilterReduceExpressionsRule.Config.class)
.toRule();
/**
@@ -58,10 +57,9 @@ private HiveReduceExpressionsRule() {
* {@link org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveProject}.
*/
public static final RelOptRule PROJECT_INSTANCE =
- ReduceExpressionsRule.ProjectReduceExpressionsRule.Config.DEFAULT
+ ReduceExpressionsRule.ProjectReduceExpressionsRule.ProjectReduceExpressionsRuleConfig.DEFAULT
.withOperandFor(HiveProject.class)
.withRelBuilderFactory(HiveRelFactories.HIVE_BUILDER)
- .as(ReduceExpressionsRule.ProjectReduceExpressionsRule.Config.class)
.toRule();
/**
@@ -69,11 +67,10 @@ private HiveReduceExpressionsRule() {
* {@link org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveJoin}.
*/
public static final RelOptRule JOIN_INSTANCE =
- ReduceExpressionsRule.JoinReduceExpressionsRule.Config.DEFAULT
+ ReduceExpressionsRule.JoinReduceExpressionsRule.JoinReduceExpressionsRuleConfig.DEFAULT
.withOperandFor(HiveJoin.class)
.withMatchNullability(false)
.withRelBuilderFactory(HiveRelFactories.HIVE_BUILDER)
- .as(ReduceExpressionsRule.JoinReduceExpressionsRule.Config.class)
.toRule();
/**
@@ -81,11 +78,10 @@ private HiveReduceExpressionsRule() {
* {@link org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveSemiJoin}.
*/
public static final RelOptRule SEMIJOIN_INSTANCE =
- ReduceExpressionsRule.JoinReduceExpressionsRule.Config.DEFAULT
+ ReduceExpressionsRule.JoinReduceExpressionsRule.JoinReduceExpressionsRuleConfig.DEFAULT
.withOperandFor(HiveSemiJoin.class)
.withMatchNullability(false)
.withRelBuilderFactory(HiveRelFactories.HIVE_BUILDER)
- .as(ReduceExpressionsRule.JoinReduceExpressionsRule.Config.class)
.toRule();
}
diff --git a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveReduceExpressionsWithStatsRule.java b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveReduceExpressionsWithStatsRule.java
index 407896826f1..74de4c58660 100644
--- a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveReduceExpressionsWithStatsRule.java
+++ b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveReduceExpressionsWithStatsRule.java
@@ -156,7 +156,7 @@ public RexNode visitCall(RexCall call) {
// We cannot apply the reduction
return call;
- } else if (call.getOperator().getKind() == SqlKind.IN) {
+ } else if (HiveIn.INSTANCE.equals(call.op)) {
if (call.getOperands().get(0) instanceof RexInputRef) {
// Ref
RexInputRef ref = (RexInputRef) call.getOperands().get(0);
diff --git a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRemoveEmptySingleRules.java b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRemoveEmptySingleRules.java
index 2cdc0d2ff65..44414112f2f 100644
--- a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRemoveEmptySingleRules.java
+++ b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRemoveEmptySingleRules.java
@@ -20,7 +20,6 @@
import org.apache.calcite.plan.RelOptRule;
import org.apache.calcite.plan.RelOptRuleCall;
import org.apache.calcite.plan.RelOptUtil;
-import org.apache.calcite.plan.RelRule;
import org.apache.calcite.plan.hep.HepRelVertex;
import org.apache.calcite.plan.volcano.RelSubset;
import org.apache.calcite.rel.RelNode;
@@ -35,7 +34,6 @@
import org.apache.calcite.rex.RexNode;
import org.apache.calcite.tools.RelBuilder;
import org.apache.hadoop.hive.ql.optimizer.calcite.Bug;
-import org.apache.hadoop.hive.ql.optimizer.calcite.HiveRelFactories;
import org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveAggregate;
import org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveAntiJoin;
import org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveFilter;
@@ -57,34 +55,28 @@
public class HiveRemoveEmptySingleRules extends PruneEmptyRules {
public static final RelOptRule PROJECT_INSTANCE =
- RelRule.Config.EMPTY
- .withDescription("HivePruneEmptyProject")
- .as(PruneEmptyRules.RemoveEmptySingleRule.Config.class)
+ new RemoveEmptySingleRuleConfig()
.withOperandFor(HiveProject.class, project -> true)
- .withRelBuilderFactory(HiveRelFactories.HIVE_BUILDER)
+ .withDescription("HivePruneEmptyProject")
.toRule();
public static final RelOptRule FILTER_INSTANCE =
- RelRule.Config.EMPTY
- .withDescription("HivePruneEmptyFilter")
- .as(PruneEmptyRules.RemoveEmptySingleRule.Config.class)
+ new RemoveEmptySingleRuleConfig()
.withOperandFor(HiveFilter.class, singleRel -> true)
- .withRelBuilderFactory(HiveRelFactories.HIVE_BUILDER)
+ .withDescription("HivePruneEmptyFilter")
.toRule();
public static final RelOptRule JOIN_LEFT_INSTANCE = getJoinLeftInstance(HiveJoin.class);
public static final RelOptRule SEMI_JOIN_LEFT_INSTANCE = getJoinLeftInstance(HiveSemiJoin.class);
private static <R extends RelNode> RelOptRule getJoinLeftInstance(Class<R> clazz) {
- return RelRule.Config.EMPTY
+ return new JoinLeftEmptyRuleConfig()
.withOperandSupplier(b0 ->
b0.operand(clazz).inputs(
b1 -> b1.operand(Values.class)
.predicate(Values::isEmpty).noInputs(),
b2 -> b2.operand(RelNode.class).anyInputs()))
.withDescription("HivePruneEmptyJoin(left)")
- .as(JoinLeftEmptyRuleConfig.class)
- .withRelBuilderFactory(HiveRelFactories.HIVE_BUILDER)
.toRule();
}
@@ -97,8 +89,9 @@ private static <R extends RelNode> RelOptRule getJoinLeftInstance(Class<R> clazz
* to
* select null as emp.col0 ... null as emp.coln, dept.* from dept
*/
- public interface JoinLeftEmptyRuleConfig extends PruneEmptyRule.Config {
- @Override default PruneEmptyRule toRule() {
+ private static final class JoinLeftEmptyRuleConfig extends HiveRuleConfig implements PruneEmptyRule.Config {
+ @Override
+ public PruneEmptyRule toRule() {
return new PruneEmptyRule(this) {
@Override public void onMatch(RelOptRuleCall call) {
if (Bug.CALCITE_5294_FIXED) {
@@ -127,15 +120,13 @@ public interface JoinLeftEmptyRuleConfig extends PruneEmptyRule.Config {
public static final RelOptRule SEMI_JOIN_RIGHT_INSTANCE = getJoinRightInstance(HiveSemiJoin.class);
private static <R extends RelNode> RelOptRule getJoinRightInstance(Class<R> clazz) {
- return RelRule.Config.EMPTY
+ return new JoinRightEmptyRuleConfig()
.withOperandSupplier(b0 ->
b0.operand(clazz).inputs(
b1 -> b1.operand(RelNode.class).anyInputs(),
b2 -> b2.operand(Values.class).predicate(Values::isEmpty)
.noInputs()))
.withDescription("HivePruneEmptyJoin(right)")
- .as(JoinRightEmptyRuleConfig.class)
- .withRelBuilderFactory(HiveRelFactories.HIVE_BUILDER)
.toRule();
}
@@ -148,8 +139,10 @@ private static <R extends RelNode> RelOptRule getJoinRightInstance(Class<R> claz
* to
* select emp.*, null as dept.col0 ... null as dept.coln from emp
*/
- public interface JoinRightEmptyRuleConfig extends PruneEmptyRule.Config {
- @Override default PruneEmptyRule toRule() {
+ private static final class JoinRightEmptyRuleConfig extends HiveRuleConfig implements
+ PruneEmptyRules.JoinRightEmptyRuleConfig {
+ @Override
+ public PruneEmptyRule toRule() {
return new PruneEmptyRule(this) {
@Override public void onMatch(RelOptRuleCall call) {
if (Bug.CALCITE_5294_FIXED) {
@@ -191,29 +184,25 @@ private static RelNode padWithNulls(RelBuilder builder, RelNode input, RelDataTy
return builder.convert(resultType, true).build();
}
- public static final RelOptRule CORRELATE_RIGHT_INSTANCE = RelRule.Config.EMPTY
+ public static final RelOptRule CORRELATE_RIGHT_INSTANCE = new CorrelateRightEmptyRuleConfig()
.withOperandSupplier(b0 ->
b0.operand(Correlate.class).inputs(
b1 -> b1.operand(RelNode.class).anyInputs(),
b2 -> b2.operand(Values.class).predicate(Values::isEmpty).noInputs()))
.withDescription("PruneEmptyCorrelate(right)")
- .withRelBuilderFactory(HiveRelFactories.HIVE_BUILDER)
- .as(CorrelateRightEmptyRuleConfig.class)
.toRule();
- public static final RelOptRule CORRELATE_LEFT_INSTANCE = RelRule.Config.EMPTY
+ public static final RelOptRule CORRELATE_LEFT_INSTANCE = new CorrelateLeftEmptyRuleConfig()
.withOperandSupplier(b0 ->
b0.operand(Correlate.class).inputs(
b1 -> b1.operand(Values.class).predicate(Values::isEmpty).noInputs(),
b2 -> b2.operand(RelNode.class).anyInputs()))
.withDescription("PruneEmptyCorrelate(left)")
- .withRelBuilderFactory(HiveRelFactories.HIVE_BUILDER)
- .as(CorrelateLeftEmptyRuleConfig.class)
.toRule();
/** Configuration for rule that prunes a correlate if left input is empty. */
- public interface CorrelateLeftEmptyRuleConfig extends PruneEmptyRule.Config {
+ public static class CorrelateLeftEmptyRuleConfig extends HiveRuleConfig implements PruneEmptyRule.Config {
@Override
- default PruneEmptyRule toRule() {
+ public PruneEmptyRule toRule() {
return new PruneEmptyRule(this) {
@Override
public void onMatch(RelOptRuleCall call) {
@@ -228,9 +217,9 @@ public void onMatch(RelOptRuleCall call) {
}
/** Configuration for rule that prunes a correlate if right input is empty. */
- public interface CorrelateRightEmptyRuleConfig extends PruneEmptyRule.Config {
+ public static class CorrelateRightEmptyRuleConfig extends HiveRuleConfig implements PruneEmptyRule.Config {
@Override
- default PruneEmptyRule toRule() {
+ public PruneEmptyRule toRule() {
return new PruneEmptyRule(this) {
@Override
public void onMatch(RelOptRuleCall call) {
@@ -262,46 +251,38 @@ public void onMatch(RelOptRuleCall call) {
}
public static final RelOptRule SORT_INSTANCE =
- RelRule.Config.EMPTY
- .withDescription("HivePruneEmptySort")
- .as(PruneEmptyRules.RemoveEmptySingleRule.Config.class)
+ new RemoveEmptySingleRuleConfig()
.withOperandFor(HiveSortLimit.class, singleRel -> true)
- .withRelBuilderFactory(HiveRelFactories.HIVE_BUILDER)
+ .withDescription("HivePruneEmptySort")
.toRule();
public static final RelOptRule SORT_FETCH_ZERO_INSTANCE =
- RelRule.Config.EMPTY
- .withOperandSupplier(b ->
- b.operand(HiveSortLimit.class).anyInputs())
+ new SortFetchZeroRuleConfig()
+ .withOperandSupplier(b -> b.operand(HiveSortLimit.class).anyInputs())
.withDescription("HivePruneSortLimit0")
- .as(PruneEmptyRules.SortFetchZeroRuleConfig.class)
- .withRelBuilderFactory(HiveRelFactories.HIVE_BUILDER)
.toRule();
public static final RelOptRule AGGREGATE_INSTANCE =
- RelRule.Config.EMPTY
- .withDescription("HivePruneEmptyAggregate")
- .as(PruneEmptyRules.RemoveEmptySingleRule.Config.class)
+ new RemoveEmptySingleRuleConfig()
.withOperandFor(HiveAggregate.class, Aggregate::isNotGrandTotal)
- .withRelBuilderFactory(HiveRelFactories.HIVE_BUILDER)
+ .withDescription("HivePruneEmptyAggregate")
.toRule();
public static final RelOptRule UNION_INSTANCE =
- RelRule.Config.EMPTY
+ new HiveUnionEmptyPruneRuleConfig()
.withOperandSupplier(b0 ->
b0.operand(HiveUnion.class).unorderedInputs(b1 ->
b1.operand(Values.class)
.predicate(Values::isEmpty).noInputs()))
.withDescription("HivePruneEmptyUnionBranch")
- .as(HiveUnionEmptyPruneRuleConfig.class)
- .withRelBuilderFactory(HiveRelFactories.HIVE_BUILDER)
.toRule();
/**
* Copy of {@link PruneEmptyRules.UnionEmptyPruneRuleConfig} but this version expects {@link Union}.
*/
- public interface HiveUnionEmptyPruneRuleConfig extends PruneEmptyRules.PruneEmptyRule.Config {
- @Override default PruneEmptyRules.PruneEmptyRule toRule() {
+ private static final class HiveUnionEmptyPruneRuleConfig extends HiveRuleConfig implements PruneEmptyRules.PruneEmptyRule.Config {
+ @Override
+ public PruneEmptyRules.PruneEmptyRule toRule() {
return new PruneEmptyRules.PruneEmptyRule(this) {
@Override public void onMatch(RelOptRuleCall call) {
if (Bug.CALCITE_5293_FIXED) {
@@ -360,4 +341,13 @@ private static boolean isEmpty(RelNode node) {
}
return false;
}
+
+ private static final class RemoveEmptySingleRuleConfig extends HiveRuleConfig
+ implements RemoveEmptySingleRule.RemoveEmptySingleRuleConfig {
+ }
+
+ private static final class SortFetchZeroRuleConfig extends HiveRuleConfig
+ implements PruneEmptyRules.SortFetchZeroRuleConfig {
+ }
+
}
diff --git a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRexShuttleTransformRule.java b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRexShuttleTransformRule.java
new file mode 100644
index 00000000000..bf287359058
--- /dev/null
+++ b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRexShuttleTransformRule.java
@@ -0,0 +1,57 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.optimizer.calcite.rules;
+
+import org.apache.calcite.plan.RelOptRuleCall;
+import org.apache.calcite.plan.RelRule;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rex.RexBuilder;
+import org.apache.calcite.rex.RexShuttle;
+
+import java.util.Objects;
+import java.util.function.Function;
+
+public class HiveRexShuttleTransformRule extends RelRule<HiveRexShuttleTransformRule.Config> {
+ private HiveRexShuttleTransformRule(Config config) {
+ super(config);
+ }
+
+ @Override
+ public void onMatch(final RelOptRuleCall call) {
+ RelNode startNode = call.rel(0);
+ RelNode rewriteNode = startNode.accept(config.shuttleFactory.apply(startNode.getCluster().getRexBuilder()));
+ if (startNode != rewriteNode) {
+ call.transformTo(rewriteNode);
+ }
+ }
+
+ public static class Config extends HiveRuleConfig {
+ private Function<RexBuilder, RexShuttle> shuttleFactory;
+
+ public Config withRexShuttle(Function<RexBuilder, RexShuttle> shuttleFactory) {
+ this.shuttleFactory = Objects.requireNonNull(shuttleFactory);
+ return this;
+ }
+
+ @Override
+ public HiveRexShuttleTransformRule toRule() {
+ return new HiveRexShuttleTransformRule(this);
+ }
+ }
+
+}
diff --git a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRuleConfig.java b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRuleConfig.java
new file mode 100644
index 00000000000..e33d4f1079a
--- /dev/null
+++ b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRuleConfig.java
@@ -0,0 +1,66 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.optimizer.calcite.rules;
+
+import org.apache.calcite.plan.RelRule;
+import org.apache.calcite.tools.RelBuilderFactory;
+import org.apache.hadoop.hive.ql.optimizer.calcite.HiveRelFactories;
+
+import java.util.Objects;
+
+public abstract class HiveRuleConfig implements RelRule.Config {
+ private RelBuilderFactory factory = HiveRelFactories.HIVE_BUILDER;
+ private String description;
+ private RelRule.OperandTransform operandSupplier;
+
+ @Override
+ public RelBuilderFactory relBuilderFactory() {
+ return factory;
+ }
+
+ @Override
+ public RelRule.Config withRelBuilderFactory(RelBuilderFactory factory) {
+ this.factory = Objects.requireNonNull(factory);
+ return this;
+ }
+
+ @Override
+ public String description() {
+ return this.description;
+ }
+
+ @Override
+ public RelRule.Config withDescription(String description) {
+ if (description == null || description.isEmpty()) {
+ throw new IllegalArgumentException("Description can not be null/empty");
+ }
+ this.description = description;
+ return this;
+ }
+
+ @Override
+ public RelRule.OperandTransform operandSupplier() {
+ return operandSupplier;
+ }
+
+ @Override
+ public RelRule.Config withOperandSupplier(RelRule.OperandTransform transform) {
+ this.operandSupplier = Objects.requireNonNull(transform);
+ return this;
+ }
+
+}
diff --git a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSearchRules.java b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSearchRules.java
new file mode 100644
index 00000000000..86f2a20d553
--- /dev/null
+++ b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSearchRules.java
@@ -0,0 +1,50 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.optimizer.calcite.rules;
+
+import org.apache.calcite.plan.RelOptRule;
+import org.apache.hadoop.hive.ql.optimizer.calcite.SearchTransformer;
+import org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveFilter;
+import org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveJoin;
+import org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveProject;
+
+/**
+ * A holder class for rules related to the SEARCH operator.
+ */
+public final class HiveSearchRules {
+ private HiveSearchRules() {
+ throw new IllegalStateException();
+ }
+
+ public static final RelOptRule PROJECT_SEARCH_EXPAND =
+ new HiveRexShuttleTransformRule.Config().withRexShuttle(SearchTransformer.Shuttle::new)
+ .withDescription("HiveProjectSearchExpandRule")
+ .withOperandSupplier(o -> o.operand(HiveProject.class).anyInputs())
+ .toRule();
+ public static final RelOptRule FILTER_SEARCH_EXPAND =
+ new HiveRexShuttleTransformRule.Config().withRexShuttle(SearchTransformer.Shuttle::new)
+ .withDescription("HiveFilterSearchExpandRule")
+ .withOperandSupplier(o -> o.operand(HiveFilter.class).anyInputs())
+ .toRule();
+ public static final RelOptRule JOIN_SEARCH_EXPAND =
+ new HiveRexShuttleTransformRule.Config().withRexShuttle(SearchTransformer.Shuttle::new)
+ .withDescription("HiveJoinSearchExpandRule")
+ .withOperandSupplier(o -> o.operand(HiveJoin.class).anyInputs())
+ .toRule();
+
+}
diff --git a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/jdbc/HiveJdbcImplementor.java b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/jdbc/HiveJdbcImplementor.java
index 5524c58d58d..096c96e7ec1 100644
--- a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/jdbc/HiveJdbcImplementor.java
+++ b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/jdbc/HiveJdbcImplementor.java
@@ -22,6 +22,7 @@
import java.util.List;
import org.apache.calcite.adapter.java.JavaTypeFactory;
import org.apache.calcite.adapter.jdbc.JdbcImplementor;
+import org.apache.calcite.adapter.jdbc.JdbcTableScan;
import org.apache.calcite.linq4j.tree.Expressions;
import org.apache.calcite.rel.RelFieldCollation;
import org.apache.calcite.rel.RelNode;
@@ -93,6 +94,10 @@ public HiveJdbcImplementor(SqlDialect dialect, JavaTypeFactory typeFactory) {
return result(join, leftResult, rightResult);
}
+ public Result visit(JdbcTableScan scan) {
+ return result(scan.jdbcTable.tableName(), ImmutableList.of(Clause.FROM), scan, null);
+ }
+
/**
* Converts a {@link RexNode} condition into a {@link SqlNode}.
*
diff --git a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/jdbc/JDBCExpandExpressionsRule.java b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/jdbc/JDBCExpandExpressionsRule.java
index 2ed2e5c70e8..5e8b3c078e9 100644
--- a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/jdbc/JDBCExpandExpressionsRule.java
+++ b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/jdbc/JDBCExpandExpressionsRule.java
@@ -35,6 +35,7 @@
import org.apache.calcite.rex.RexUtil;
import org.apache.calcite.sql.SqlKind;
import org.apache.calcite.sql.fun.SqlStdOperatorTable;
+import org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveIn;
import org.apache.hadoop.hive.ql.optimizer.calcite.translator.RexNodeConverter;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -179,18 +180,15 @@ public RexNode visitCall(RexCall inputCall) {
RexNode node = super.visitCall(inputCall);
if (node instanceof RexCall) {
RexCall call = (RexCall) node;
- switch (call.getKind()) {
- case IN:
- return transformIntoOrAndClause(rexBuilder, call);
- default:
- break;
+ if (HiveIn.INSTANCE.equals(call.op)) {
+ return transformIntoOrAndClause(rexBuilder, call);
}
}
return RexUtil.isFlat(node) ? node : RexUtil.flatten(rexBuilder, node);
}
private RexNode transformIntoOrAndClause(RexBuilder rexBuilder, RexCall expression) {
- assert expression.getKind() == SqlKind.IN;
+ assert HiveIn.INSTANCE.equals(expression.op);
if (expression.getOperands().get(0).getKind() != SqlKind.ROW) {
// Nothing to do, return expression
diff --git a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/jdbc/JDBCExtractJoinFilterRule.java b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/jdbc/JDBCExtractJoinFilterRule.java
index 32c486a584b..9f5f5767d73 100644
--- a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/jdbc/JDBCExtractJoinFilterRule.java
+++ b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/jdbc/JDBCExtractJoinFilterRule.java
@@ -20,6 +20,7 @@
import org.apache.calcite.plan.RelOptRuleCall;
import org.apache.calcite.rel.core.Join;
import org.apache.calcite.rel.rules.AbstractJoinExtractFilterRule;
+import org.apache.calcite.rel.rules.JoinExtractFilterRule;
import org.apache.hadoop.hive.ql.optimizer.calcite.HiveRelFactories;
import org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.jdbc.HiveJdbcConverter;
import org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveJoin;
@@ -35,6 +36,13 @@
public final class JDBCExtractJoinFilterRule extends AbstractJoinExtractFilterRule {
//~ Static fields/initializers ---------------------------------------------
+ private static final AbstractJoinExtractFilterRule.Config DEFAULT =
+ JoinExtractFilterRule.Config.DEFAULT.withOperandSupplier(b0 -> b0.operand(HiveJoin.class)
+ .inputs(b1 -> b1.operand(HiveJdbcConverter.class).anyInputs(),
+ b2 -> b2.operand(HiveJdbcConverter.class).anyInputs()))
+ .withRelBuilderFactory(HiveRelFactories.HIVE_BUILDER)
+ .withDescription("JDBCExtractJoinFilterRule")
+ .as(AbstractJoinExtractFilterRule.Config.class);
public static final JDBCExtractJoinFilterRule INSTANCE = new JDBCExtractJoinFilterRule();
//~ Constructors -----------------------------------------------------------
@@ -43,10 +51,7 @@ public final class JDBCExtractJoinFilterRule extends AbstractJoinExtractFilterRu
* Creates an JoinExtractFilterRule.
*/
public JDBCExtractJoinFilterRule() {
- super(operand(HiveJoin.class,
- operand(HiveJdbcConverter.class, any()),
- operand(HiveJdbcConverter.class, any())),
- HiveRelFactories.HIVE_BUILDER, null);
+ super(DEFAULT);
}
//~ Methods ----------------------------------------------------------------
diff --git a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/jdbc/JDBCFilterJoinRule.java b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/jdbc/JDBCFilterJoinRule.java
index 6fcb3a58ee7..63b6506ac4d 100644
--- a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/jdbc/JDBCFilterJoinRule.java
+++ b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/jdbc/JDBCFilterJoinRule.java
@@ -21,7 +21,6 @@
import org.apache.calcite.plan.RelOptRuleCall;
import org.apache.calcite.rel.core.Filter;
import org.apache.calcite.rel.core.Join;
-import org.apache.hadoop.hive.ql.optimizer.calcite.HiveRelFactories;
import org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveFilter;
import org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.jdbc.HiveJdbcConverter;
import org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveJoin;
@@ -39,9 +38,9 @@ public class JDBCFilterJoinRule extends HiveFilterJoinRule {
public JDBCFilterJoinRule() {
super(RelOptRule.operand(HiveFilter.class,
RelOptRule.operand(HiveJoin.class,
- RelOptRule.operand(HiveJdbcConverter.class, RelOptRule.any()),
- RelOptRule.operand(HiveJdbcConverter.class, RelOptRule.any()))),
- "JDBCFilterJoinRule", true, HiveRelFactories.HIVE_BUILDER);
+ RelOptRule.operand(HiveJdbcConverter.class, RelOptRule.any()),
+ RelOptRule.operand(HiveJdbcConverter.class, RelOptRule.any()))),
+ "JDBCFilterJoinRule");
}
@Override
diff --git a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/jdbc/JDBCRexCallValidator.java b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/jdbc/JDBCRexCallValidator.java
index 7c72bd3f03d..91e81280e24 100644
--- a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/jdbc/JDBCRexCallValidator.java
+++ b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/jdbc/JDBCRexCallValidator.java
@@ -27,6 +27,7 @@
import org.apache.calcite.rex.RexVisitorImpl;
import org.apache.calcite.sql.SqlDialect;
import org.apache.calcite.sql.SqlOperator;
+import org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveIn;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -54,6 +55,9 @@ private boolean validRexCall(RexCall call) {
return false;
}
final SqlOperator operator = call.getOperator();
+ if (HiveIn.INSTANCE.equals(operator)) {
+ return true;
+ }
List <RexNode> operands = call.getOperands();
RelDataType resType = call.getType();
ArrayList<RelDataType> paramsListType = new ArrayList<RelDataType>();
diff --git a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/views/HiveAugmentSnapshotMaterializationRule.java b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/views/HiveAugmentSnapshotMaterializationRule.java
index 253d9b6c996..f5f2e46c0f1 100644
--- a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/views/HiveAugmentSnapshotMaterializationRule.java
+++ b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/views/HiveAugmentSnapshotMaterializationRule.java
@@ -29,12 +29,11 @@
import org.apache.calcite.rex.RexNode;
import org.apache.calcite.sql.fun.SqlStdOperatorTable;
import org.apache.calcite.tools.RelBuilder;
-import org.apache.calcite.util.ImmutableBeans;
import org.apache.hadoop.hive.common.type.SnapshotContext;
import org.apache.hadoop.hive.ql.metadata.Table;
import org.apache.hadoop.hive.ql.metadata.VirtualColumn;
-import org.apache.hadoop.hive.ql.optimizer.calcite.HiveRelFactories;
import org.apache.hadoop.hive.ql.optimizer.calcite.RelOptHiveTable;
+import org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveRuleConfig;
import org.apache.hadoop.hive.ql.optimizer.calcite.translator.TypeConverter;
import java.util.HashSet;
@@ -68,23 +67,28 @@
public class HiveAugmentSnapshotMaterializationRule extends RelRule<HiveAugmentSnapshotMaterializationRule.Config> {
public static RelOptRule with(Map<String, SnapshotContext> mvMetaStoredSnapshot) {
- return RelRule.Config.EMPTY.as(HiveAugmentSnapshotMaterializationRule.Config.class)
- .withMvMetaStoredSnapshot(mvMetaStoredSnapshot)
- .withRelBuilderFactory(HiveRelFactories.HIVE_BUILDER)
- .withOperandSupplier(operandBuilder -> operandBuilder.operand(TableScan.class).anyInputs())
- .withDescription("HiveAugmentSnapshotMaterializationRule")
- .toRule();
+ return new Config().withMvMetaStoredSnapshot(mvMetaStoredSnapshot)
+ .withOperandSupplier(operandBuilder -> operandBuilder.operand(TableScan.class).anyInputs())
+ .withDescription("HiveAugmentSnapshotMaterializationRule")
+ .toRule();
}
- public interface Config extends RelRule.Config {
+ public static class Config extends HiveRuleConfig {
- HiveAugmentSnapshotMaterializationRule.Config withMvMetaStoredSnapshot(
- Map<String, SnapshotContext> mvMetaStoredSnapshot);
+ private Map<String, SnapshotContext> mvMetaStoredSnapshot;
- @ImmutableBeans.Property
- Map<String, SnapshotContext> getMvMetaStoredSnapshot();
+ public HiveAugmentSnapshotMaterializationRule.Config withMvMetaStoredSnapshot(
+ Map<String, SnapshotContext> mvMetaStoredSnapshot) {
+ this.mvMetaStoredSnapshot = mvMetaStoredSnapshot;
+ return this;
+ }
+
+ public Map<String, SnapshotContext> getMvMetaStoredSnapshot() {
+ return mvMetaStoredSnapshot;
+ }
- @Override default HiveAugmentSnapshotMaterializationRule toRule() {
+ @Override
+ public HiveAugmentSnapshotMaterializationRule toRule() {
return new HiveAugmentSnapshotMaterializationRule(this);
}
}
diff --git a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/views/HiveMaterializedViewRule.java b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/views/HiveMaterializedViewRule.java
index 6cf9eee1a5e..4c71f7534e3 100644
--- a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/views/HiveMaterializedViewRule.java
+++ b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/views/HiveMaterializedViewRule.java
@@ -37,7 +37,6 @@
import org.apache.calcite.rel.rules.materialize.MaterializedViewOnlyAggregateRule;
import org.apache.calcite.rel.rules.ProjectRemoveRule;
import org.apache.calcite.rex.RexNode;
-import org.apache.calcite.sql.SqlAggFunction;
import org.apache.calcite.sql.SqlFunction;
import org.apache.calcite.tools.RelBuilder;
import org.apache.calcite.tools.RelBuilderFactory;
@@ -138,11 +137,6 @@ public HiveMaterializedViewProjectAggregateRule(
protected SqlFunction getFloorSqlFunction(TimeUnitRange flag) {
return HiveRelBuilder.getFloorSqlFunction(flag);
}
-
- @Override
- public SqlAggFunction getRollup(SqlAggFunction aggregation) {
- return HiveRelBuilder.getRollup(aggregation);
- }
}
protected static class HiveMaterializedViewOnlyAggregateRule extends MaterializedViewOnlyAggregateRule {
@@ -155,11 +149,6 @@ public HiveMaterializedViewOnlyAggregateRule(
protected SqlFunction getFloorSqlFunction(TimeUnitRange flag) {
return HiveRelBuilder.getFloorSqlFunction(flag);
}
-
- @Override
- public SqlAggFunction getRollup(SqlAggFunction aggregation) {
- return HiveRelBuilder.getRollup(aggregation);
- }
}
/**
diff --git a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/views/HivePushdownSnapshotFilterRule.java b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/views/HivePushdownSnapshotFilterRule.java
index 055ff91d510..0965c81977c 100644
--- a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/views/HivePushdownSnapshotFilterRule.java
+++ b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/views/HivePushdownSnapshotFilterRule.java
@@ -32,11 +32,11 @@
import org.apache.calcite.rex.RexShuttle;
import org.apache.calcite.rex.RexTableInputRef;
import org.apache.calcite.sql.SqlKind;
-import org.apache.calcite.sql.type.SqlTypeFamily;
+import org.apache.calcite.sql.type.SqlTypeName;
import org.apache.hadoop.hive.ql.metadata.VirtualColumn;
-import org.apache.hadoop.hive.ql.optimizer.calcite.HiveRelFactories;
import org.apache.hadoop.hive.ql.optimizer.calcite.RelOptHiveTable;
import org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveFilter;
+import org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveRuleConfig;
import java.util.Objects;
import java.util.Set;
@@ -60,15 +60,13 @@
public class HivePushdownSnapshotFilterRule extends RelRule<HivePushdownSnapshotFilterRule.Config> {
public static final RelOptRule INSTANCE =
- RelRule.Config.EMPTY.as(HivePushdownSnapshotFilterRule.Config.class)
- .withRelBuilderFactory(HiveRelFactories.HIVE_BUILDER)
- .withOperandSupplier(operandBuilder -> operandBuilder.operand(HiveFilter.class).anyInputs())
- .withDescription("HivePushdownSnapshotFilterRule")
- .toRule();
+ new Config().withOperandSupplier(operandBuilder -> operandBuilder.operand(HiveFilter.class).anyInputs())
+ .withDescription("HivePushdownSnapshotFilterRule")
+ .toRule();
- public interface Config extends RelRule.Config {
+ public static class Config extends HiveRuleConfig {
@Override
- default HivePushdownSnapshotFilterRule toRule() {
+ public HivePushdownSnapshotFilterRule toRule() {
return new HivePushdownSnapshotFilterRule(this);
}
}
@@ -113,7 +111,7 @@ private boolean setSnapShotId(RexNode op1, RexNode op2) {
}
RexLiteral literal = (RexLiteral) op1;
- if (literal.getType().getSqlTypeName().getFamily() != SqlTypeFamily.NUMERIC) {
+ if (literal.getTypeName() != SqlTypeName.DECIMAL) {
return false;
}
diff --git a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/views/HiveRowIsDeletedPropagator.java b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/views/HiveRowIsDeletedPropagator.java
index afb1ef36fb3..0441bec6fab 100644
--- a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/views/HiveRowIsDeletedPropagator.java
+++ b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/views/HiveRowIsDeletedPropagator.java
@@ -30,7 +30,6 @@
import org.apache.calcite.rex.RexVisitorImpl;
import org.apache.calcite.sql.SqlKind;
import org.apache.calcite.sql.fun.SqlStdOperatorTable;
-import org.apache.calcite.sql.type.SqlTypeName;
import org.apache.calcite.tools.RelBuilder;
import org.apache.calcite.util.ReflectUtil;
import org.apache.calcite.util.ReflectiveVisitor;
@@ -264,9 +263,17 @@ public RelNode visit(HiveJoin join, Context context) {
projectNames.add(ANY_INSERTED_COLUMN_NAME);
// Create input refs to derived expressions in project
- RelDataType boolIntType = relBuilder.getTypeFactory().createSqlType(SqlTypeName.BOOLEAN);
- RexNode anyDeleted = rexBuilder.makeInputRef(boolIntType, projects.size() - 2);
- RexNode anyInserted = rexBuilder.makeInputRef(boolIntType, projects.size() - 1);
+ int anyDeletedIndex = projects.size() - 2;
+ int anyInsertedIndex = projects.size() - 1;
+ // Get types from corresponding projects, otherwise we may see a Type mismatch AssertionError
+ // from Calcite's Litmus. For example,
+ // type mismatch:
+ // ref:
+ // BOOLEAN NOT NULL
+ // input:
+ // BOOLEAN
+ RexNode anyDeleted = rexBuilder.makeInputRef(projects.get(anyDeletedIndex).getType(), anyDeletedIndex);
+ RexNode anyInserted = rexBuilder.makeInputRef(projects.get(anyInsertedIndex).getType(), anyInsertedIndex);
// Create filter condition: NOT( (leftDeleted OR rightDeleted) AND (leftInserted OR rightInserted) )
// We exploit that a row can not be deleted and inserted at the same time.
diff --git a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/FilterSelectivityEstimator.java b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/FilterSelectivityEstimator.java
index 8a1e80d885d..9bc6cad909d 100644
--- a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/FilterSelectivityEstimator.java
+++ b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/FilterSelectivityEstimator.java
@@ -30,6 +30,7 @@
import org.apache.calcite.rel.core.Filter;
import org.apache.calcite.rel.core.Project;
import org.apache.calcite.rel.metadata.RelMetadataQuery;
+import org.apache.calcite.rex.RexBuilder;
import org.apache.calcite.rex.RexCall;
import org.apache.calcite.rex.RexInputRef;
import org.apache.calcite.rex.RexLiteral;
@@ -37,23 +38,21 @@
import org.apache.calcite.rex.RexUtil;
import org.apache.calcite.rex.RexVisitorImpl;
import org.apache.calcite.sql.SqlKind;
-import org.apache.calcite.sql.SqlOperator;
import org.apache.calcite.sql.type.SqlTypeName;
-import org.apache.calcite.sql.type.SqlTypeUtil;
import org.apache.calcite.util.ImmutableBitSet;
import org.apache.datasketches.kll.KllFloatsSketch;
import org.apache.datasketches.memory.Memory;
import org.apache.hadoop.hive.ql.optimizer.calcite.HiveCalciteUtil;
import org.apache.hadoop.hive.ql.optimizer.calcite.HiveConfPlannerContext;
import org.apache.hadoop.hive.ql.optimizer.calcite.RelOptHiveTable;
+import org.apache.hadoop.hive.ql.optimizer.calcite.SearchTransformer;
+import org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveIn;
import org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveTableScan;
import org.apache.hadoop.hive.ql.plan.ColStatistics;
import org.apache.hadoop.hive.ql.session.SessionState;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
-import static org.apache.calcite.sql.fun.SqlStdOperatorTable.NOT_BETWEEN;
-
public class FilterSelectivityEstimator extends RexVisitorImpl<Double> {
protected static final Logger LOG = LoggerFactory.getLogger(FilterSelectivityEstimator.class);
@@ -61,12 +60,14 @@ public class FilterSelectivityEstimator extends RexVisitorImpl<Double> {
private final RelNode childRel;
private final double childCardinality;
private final RelMetadataQuery mq;
+ private final RexBuilder rexBuilder;
public FilterSelectivityEstimator(RelNode childRel, RelMetadataQuery mq) {
super(true);
this.mq = mq;
this.childRel = childRel;
this.childCardinality = mq.getRowCount(childRel);
+ this.rexBuilder = childRel.getCluster().getRexBuilder();
}
public Double estimateSelectivity(RexNode predicate) {
@@ -98,20 +99,24 @@ public Double visitCall(RexCall call) {
}
Double selectivity;
- SqlKind op = getOp(call);
- switch (op) {
+ switch (call.getKind()) {
case AND: {
selectivity = computeConjunctionSelectivity(call);
break;
}
-
+ case SEARCH:
+ return new SearchTransformer<>(rexBuilder, call).transform().accept(this);
case OR: {
selectivity = computeDisjunctionSelectivity(call);
break;
}
- case NOT:
+ case NOT: {
+ Double opSelectivity = call.getOperands().get(0).accept(this);
+ assert (opSelectivity >= 0 && opSelectivity <= 1);
+ return 1.0 - opSelectivity;
+ }
case NOT_EQUALS: {
selectivity = computeNotEqualitySelectivity(call);
break;
@@ -146,7 +151,7 @@ public Double visitCall(RexCall call) {
case GREATER_THAN_OR_EQUAL:
case LESS_THAN:
case GREATER_THAN: {
- selectivity = computeRangePredicateSelectivity(call, op);
+ selectivity = computeRangePredicateSelectivity(call, call.getKind());
break;
}
@@ -154,24 +159,23 @@ public Double visitCall(RexCall call) {
selectivity = computeBetweenPredicateSelectivity(call);
break;
- case IN: {
- // TODO: 1) check for duplicates 2) We assume in clause values to be
- // present in NDV which may not be correct (Range check can find it) 3) We
- // assume values in NDV set is uniformly distributed over col values
- // (account for skewness - histogram).
- selectivity = computeFunctionSelectivity(call);
- if (selectivity != null) {
- selectivity = selectivity * (call.operands.size() - 1);
- if (selectivity <= 0.0) {
- selectivity = 0.10;
- } else if (selectivity >= 1.0) {
- selectivity = 1.0;
+ default:
+ if (HiveIn.INSTANCE.equals(call.op)) {
+ // TODO: 1) check for duplicates 2) We assume in clause values to be
+ // present in NDV which may not be correct (Range check can find it) 3) We
+ // assume values in NDV set is uniformly distributed over col values
+ // (account for skewness - histogram).
+ selectivity = computeFunctionSelectivity(call);
+ if (selectivity != null) {
+ selectivity = selectivity * (call.operands.size() - 1);
+ if (selectivity <= 0.0) {
+ selectivity = 0.10;
+ } else if (selectivity >= 1.0) {
+ selectivity = 1.0;
+ }
}
+ break;
}
- break;
- }
-
- default:
selectivity = computeFunctionSelectivity(call);
}
@@ -471,21 +475,6 @@ private boolean isPartitionPredicate(RexNode expr, RelNode r) {
return false;
}
- private SqlKind getOp(RexCall call) {
- SqlKind op = call.getKind();
-
- if (call.getKind().equals(SqlKind.OTHER_FUNCTION)
- && SqlTypeUtil.inBooleanFamily(call.getType())) {
- SqlOperator sqlOp = call.getOperator();
- String opName = (sqlOp != null) ? sqlOp.getName() : "";
- if (opName.equalsIgnoreCase("in")) {
- op = SqlKind.IN;
- }
- }
-
- return op;
- }
-
@Override
public Double visitLiteral(RexLiteral literal) {
if (literal.isAlwaysFalse() || RexUtil.isNull(literal)) {
diff --git a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/ExprNodeConverter.java b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/ExprNodeConverter.java
index d559a8745f4..e07092b9867 100644
--- a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/ExprNodeConverter.java
+++ b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/ExprNodeConverter.java
@@ -61,6 +61,7 @@
import org.apache.hadoop.hive.ql.metadata.Hive;
import org.apache.hadoop.hive.ql.metadata.HiveException;
import org.apache.hadoop.hive.ql.optimizer.ConstantPropagateProcFactory;
+import org.apache.hadoop.hive.ql.optimizer.calcite.SearchTransformer;
import org.apache.hadoop.hive.ql.optimizer.calcite.translator.ASTConverter.RexVisitor;
import org.apache.hadoop.hive.ql.optimizer.calcite.translator.ASTConverter.Schema;
import org.apache.hadoop.hive.ql.parse.ASTNode;
@@ -108,7 +109,7 @@ public class ExprNodeConverter extends RexVisitorImpl<ExprNodeDesc> {
private final RelDataTypeFactory dTFactory;
protected final Logger LOG = LoggerFactory.getLogger(this.getClass().getName());
private static long uniqueCounter = 0;
- private RexBuilder rexBuilder = null;
+ private final RexBuilder rexBuilder;
public ExprNodeConverter(String tabAlias, RelDataType inputRowType,
Set<Integer> vCols, RelDataTypeFactory dTFactory) {
@@ -125,12 +126,6 @@ public ExprNodeConverter(String tabAlias, String columnAlias, RelDataType inputR
this(tabAlias, columnAlias, inputRowType, outputRowType, inputVCols, dTFactory, false);
}
- public ExprNodeConverter(String tabAlias, String columnAlias, RelDataType inputRowType,
- RelDataType outputRowType, Set<Integer> inputVCols, RexBuilder rexBuilder) {
- this(tabAlias, columnAlias, inputRowType, outputRowType, inputVCols, rexBuilder.getTypeFactory(), false);
- this.rexBuilder = rexBuilder;
- }
-
public ExprNodeConverter(String tabAlias, String columnAlias, RelDataType inputRowType,
RelDataType outputRowType, Set<Integer> inputVCols, RelDataTypeFactory dTFactory,
boolean foldExpr) {
@@ -140,6 +135,7 @@ public ExprNodeConverter(String tabAlias, String columnAlias, RelDataType inputR
this.inputVCols = ImmutableSet.copyOf(inputVCols);
this.dTFactory = dTFactory;
this.foldExpr = foldExpr;
+ this.rexBuilder = new RexBuilder(dTFactory);
}
public List<WindowFunctionSpec> getWindowFunctionSpec() {
@@ -203,6 +199,8 @@ public ExprNodeDesc visitCall(RexCall call) {
for (RexNode operand : call.operands) {
args.add(operand.accept(this));
}
+ } else if (call.getKind() == SqlKind.SEARCH) {
+ return new SearchTransformer<>(rexBuilder, call).transform().accept(this);
} else {
for (RexNode operand : call.operands) {
args.add(operand.accept(this));
diff --git a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/RexNodeConverter.java b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/RexNodeConverter.java
index cddc7cb9ca2..ab9eb330b3a 100644
--- a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/RexNodeConverter.java
+++ b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/RexNodeConverter.java
@@ -29,6 +29,7 @@
import org.apache.calcite.rel.type.RelDataTypeFactory;
import org.apache.calcite.rex.RexBuilder;
import org.apache.calcite.rex.RexCall;
+import org.apache.calcite.rex.RexLiteral;
import org.apache.calcite.rex.RexNode;
import org.apache.calcite.rex.RexUtil;
import org.apache.calcite.sql.SqlBinaryOperator;
@@ -60,6 +61,7 @@
import org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveComponentAccess;
import org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveExtractDate;
import org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveFloorDate;
+import org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveIn;
import org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveToDateSqlOperator;
import org.apache.hadoop.hive.ql.parse.SemanticException;
import org.apache.hadoop.hive.ql.parse.type.ExprNodeTypeCheck;
@@ -256,7 +258,7 @@ private RexNode convert(ExprNodeGenericFuncDesc func) throws SemanticException {
} else if (HiveFloorDate.ALL_FUNCTIONS.contains(calciteOp)) {
// If it is a floor <date> operator, we need to rewrite it
childRexNodeLst = rewriteFloorDateChildren(calciteOp, childRexNodeLst, rexBuilder);
- } else if (calciteOp.getKind() == SqlKind.IN && isAllPrimitive) {
+ } else if (HiveIn.INSTANCE.equals(calciteOp) && isAllPrimitive) {
if (childRexNodeLst.size() == 2) {
// if it is a single item in an IN clause, transform A IN (B) to A = B
// from IN [A,B] => EQUALS [A,B]
@@ -544,16 +546,24 @@ public static List<RexNode> transformInToOrOperands(List<RexNode> operands, RexB
}
for (int i = 1; i < operands.size(); i++) {
List<RexNode> conjuncts = new ArrayList<>(columnExpressions.getOperands().size() - 1);
- RexCall valueExpressions = (RexCall) operands.get(i);
- if (!HiveCalciteUtil.isDeterministic(valueExpressions)) {
- // Bail out
+ List<?> valueExpressions = null;
+ if (operands.get(i) instanceof RexLiteral) {
+ RexLiteral literal = (RexLiteral) operands.get(i);
+ valueExpressions = literal.getValueAs(List.class);
+ } else if (operands.get(i) instanceof RexCall) {
+ RexCall call = (RexCall) operands.get(i);
+ if (HiveCalciteUtil.isDeterministic(call)) {
+ valueExpressions = call.getOperands();
+ }
+ }
+ if (valueExpressions == null) {
return null;
}
for (int j = 0; j < columnExpressions.getOperands().size(); j++) {
conjuncts.add(rexBuilder.makeCall(
SqlStdOperatorTable.EQUALS,
columnExpressions.getOperands().get(j),
- valueExpressions.getOperands().get(j)));
+ (RexNode) valueExpressions.get(j)));
}
if (conjuncts.size() > 1) {
disjuncts.add(rexBuilder.makeCall(
@@ -569,7 +579,7 @@ public static List<RexNode> transformInToOrOperands(List<RexNode> operands, RexB
public static List<RexNode> rewriteInClauseChildren(SqlOperator op, List<RexNode> childRexNodeLst,
RexBuilder rexBuilder) throws SemanticException {
- assert op.getKind() == SqlKind.IN;
+ assert op == HiveIn.INSTANCE;
RexNode firstPred = childRexNodeLst.get(0);
List<RexNode> newChildRexNodeLst = new ArrayList<RexNode>();
for (int i = 1; i < childRexNodeLst.size(); i++) {
diff --git a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/SqlFunctionConverter.java b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/SqlFunctionConverter.java
index 2ce99b06a9b..156cc9d8ac0 100644
--- a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/SqlFunctionConverter.java
+++ b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/SqlFunctionConverter.java
@@ -241,7 +241,6 @@ public static ASTNode buildAST(SqlOperator op, List<ASTNode> children, RelDataTy
ASTNode node;
if (hToken != null) {
switch (op.kind) {
- case IN:
case BETWEEN:
case ROW:
case ARRAY_VALUE_CONSTRUCTOR:
diff --git a/ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java b/ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java
index 9e0a222047e..71be8d8b70d 100644
--- a/ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java
+++ b/ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java
@@ -95,7 +95,6 @@
import org.apache.calcite.rel.metadata.RelMetadataQuery;
import org.apache.calcite.rel.rules.FilterMergeRule;
import org.apache.calcite.rel.rules.JoinToMultiJoinRule;
-import org.apache.calcite.rel.rules.LoptOptimizeJoinRule;
import org.apache.calcite.rel.rules.ProjectMergeRule;
import org.apache.calcite.rel.rules.ProjectRemoveRule;
import org.apache.calcite.rel.type.RelDataType;
@@ -169,6 +168,7 @@
import org.apache.hadoop.hive.ql.optimizer.calcite.CommonTableExpressionSuggesterFactory;
import org.apache.hadoop.hive.ql.optimizer.calcite.HiveCalciteUtil;
import org.apache.hadoop.hive.ql.optimizer.calcite.HiveConfPlannerContext;
+import org.apache.hadoop.hive.ql.optimizer.calcite.HiveTypeFactory;
import org.apache.hadoop.hive.ql.optimizer.calcite.HiveDefaultRelMetadataProvider;
import org.apache.hadoop.hive.ql.optimizer.calcite.HiveMaterializedViewASTSubQueryRewriteShuttle;
import org.apache.hadoop.hive.ql.optimizer.calcite.HiveSqlTypeUtil;
@@ -177,7 +177,9 @@
import org.apache.hadoop.hive.ql.optimizer.calcite.rules.CteRuleConfig;
import org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveAggregateSortLimitRule;
import org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveJoinSwapConstraintsRule;
+import org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveLoptOptimizeJoinRule;
import org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveRemoveEmptySingleRules;
+import org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveSearchRules;
import org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveSemiJoinProjectTransposeRule;
import org.apache.hadoop.hive.ql.optimizer.calcite.rules.RemoveInfrequentCteRule;
import org.apache.hadoop.hive.ql.optimizer.calcite.rules.jdbc.JDBCAggregateProjectMergeRule;
@@ -1622,7 +1624,7 @@ public RelNode apply(RelOptCluster cluster, RelOptSchema relOptSchema, SchemaPlu
* recreate cluster, so that it picks up the additional traitDef
*/
RelOptPlanner planner = createPlanner(conf, statsSource, ctx.isExplainPlan());
- final RexBuilder rexBuilder = cluster.getRexBuilder();
+ final RexBuilder rexBuilder = new RexBuilder(new HiveTypeFactory());
final RelOptCluster optCluster = RelOptCluster.create(planner, rexBuilder);
this.cluster = optCluster;
@@ -1754,6 +1756,8 @@ public RelNode apply(RelOptCluster cluster, RelOptSchema relOptSchema, SchemaPlu
LOG.debug("Plan after CTE rewriting:\n{}", RelOptUtil.toString(calcitePlan));
}
}
+ calcitePlan = applySearchExpandTransforms(calcitePlan, mdProvider.getMetadataProvider(), executorProvider);
+
perfLogger.perfLogBegin(this.getClass().getName(), PerfLogger.HIVE_SORT_PREDICATES);
if (conf.getBoolVar(HiveConf.ConfVars.HIVE_OPTIMIZE_SORT_PREDS_WITH_STATS)) {
calcitePlan = calcitePlan.accept(new HiveFilterSortPredicates(noColsMissingStats));
@@ -1765,6 +1769,15 @@ public RelNode apply(RelOptCluster cluster, RelOptSchema relOptSchema, SchemaPlu
return calcitePlan;
}
+ private RelNode applySearchExpandTransforms(RelNode basePlan, RelMetadataProvider mdProvider,
+ RexExecutor executor) {
+ HepProgramBuilder searchProgram = new HepProgramBuilder();
+ searchProgram.addRuleCollection(ImmutableList.of(HiveSearchRules.FILTER_SEARCH_EXPAND,
+ HiveSearchRules.PROJECT_SEARCH_EXPAND,
+ HiveSearchRules.JOIN_SEARCH_EXPAND));
+ return executeProgram(basePlan, searchProgram.build(), mdProvider, executor);
+ }
+
/**
* Perform all optimizations before Join Ordering.
*
@@ -2154,7 +2167,7 @@ private RelNode applyCteRewriting(RelOptPlanner planner, RelNode basePlan, RelM
Map<List<String>, Integer> tableOccurrences = RelOptUtil.findAllTables(ctePlan).stream()
.map(RelOptTable::getQualifiedName)
.collect(Collectors.toMap(Function.identity(), v -> 1, Integer::sum));
- CteRuleConfig cteConfig = CteRuleConfig.DEFAULT
+ CteRuleConfig cteConfig = CteRuleConfig.config()
.withReferenceThreshold(referenceThreshold)
.withTableOccurrences(tableOccurrences);
HepProgram spoolProgram = HepProgram.builder()
@@ -2277,7 +2290,7 @@ private RelNode applyJoinOrderingTransform(RelNode basePlan, RelMetadataProvider
rules.toArray(new RelOptRule[0]));
// Join reordering
generatePartialProgram(program, false, HepMatchOrder.BOTTOM_UP,
- new JoinToMultiJoinRule(HiveJoin.class), new LoptOptimizeJoinRule(HiveRelFactories.HIVE_BUILDER));
+ new JoinToMultiJoinRule(HiveJoin.class), HiveLoptOptimizeJoinRule.INSTANCE);
RelNode calciteOptimizedPlan;
try {
@@ -2358,6 +2371,14 @@ private RelNode applyPostJoinOrderingTransform(RelNode basePlan, RelMetadataProv
HiveWindowingLastValueRewrite.INSTANCE);
}
+ // Expand SEARCH since JDBC (and potentially other federation rules) do not know how to handle it.
+ generatePartialProgram(program,
+ true,
+ HepMatchOrder.DEPTH_FIRST,
+ HiveSearchRules.PROJECT_SEARCH_EXPAND,
+ HiveSearchRules.FILTER_SEARCH_EXPAND,
+ HiveSearchRules.JOIN_SEARCH_EXPAND);
+
// 7. Apply Druid transformation rules
generatePartialProgram(program, false, HepMatchOrder.DEPTH_FIRST,
HiveDruidRules.FILTER_DATE_RANGE_RULE,
diff --git a/ql/src/java/org/apache/hadoop/hive/ql/parse/type/HiveFunctionHelper.java b/ql/src/java/org/apache/hadoop/hive/ql/parse/type/HiveFunctionHelper.java
index 32c1f0d8c46..dbf9741d378 100644
--- a/ql/src/java/org/apache/hadoop/hive/ql/parse/type/HiveFunctionHelper.java
+++ b/ql/src/java/org/apache/hadoop/hive/ql/parse/type/HiveFunctionHelper.java
@@ -47,6 +47,7 @@
import org.apache.hadoop.hive.ql.optimizer.calcite.HiveRexExecutorImpl;
import org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveExtractDate;
import org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveFloorDate;
+import org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveIn;
import org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveToDateSqlOperator;
import org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveToUnixTimestampSqlOperator;
import org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveUnixTimestampSqlOperator;
@@ -265,7 +266,7 @@ public RexNode getExpression(String functionText, FunctionInfo fi,
} else if (HiveFloorDate.ALL_FUNCTIONS.contains(calciteOp)) {
// If it is a floor <date> operator, we need to rewrite it
inputs = RexNodeConverter.rewriteFloorDateChildren(calciteOp, inputs, rexBuilder);
- } else if (calciteOp.getKind() == SqlKind.IN) {
+ } else if (HiveIn.INSTANCE.equals(calciteOp)) {
// if it is a single item in an IN clause, transform A IN (B) to A = B
// from IN [A,B] => EQUALS [A,B]
// if it is more than an single item in an IN clause,
diff --git a/ql/src/java/org/apache/hadoop/hive/ql/parse/type/RexNodeExprFactory.java b/ql/src/java/org/apache/hadoop/hive/ql/parse/type/RexNodeExprFactory.java
index 8cc75d7e1fc..7f6877ced1f 100644
--- a/ql/src/java/org/apache/hadoop/hive/ql/parse/type/RexNodeExprFactory.java
+++ b/ql/src/java/org/apache/hadoop/hive/ql/parse/type/RexNodeExprFactory.java
@@ -48,6 +48,7 @@
import org.apache.calcite.sql.fun.SqlQuantifyOperator;
import org.apache.calcite.sql.fun.SqlStdOperatorTable;
import org.apache.calcite.sql.parser.SqlParserPos;
+import org.apache.calcite.sql.type.SqlTypeFamily;
import org.apache.calcite.sql.type.SqlTypeName;
import org.apache.calcite.sql.type.ArraySqlType;
import org.apache.calcite.util.ConversionUtil;
@@ -615,8 +616,9 @@ protected RexNode createConstantExpr(TypeInfo typeInfo, Object constantValue)
SqlStdOperatorTable.ROW,
operands);
}
- return rexBuilder.makeLiteral(constantValue,
- TypeConverter.convert(typeInfo, rexBuilder.getTypeFactory()), false);
+ RelDataType finalType = TypeConverter.convert(typeInfo, rexBuilder.getTypeFactory());
+ boolean allowCast = finalType.getFamily() == SqlTypeFamily.CHARACTER;
+ return rexBuilder.makeLiteral(constantValue, finalType, allowCast);
}
/**
diff --git a/ql/src/test/org/apache/hadoop/hive/ql/io/TestSymlinkTextInputFormat.java b/ql/src/test/org/apache/hadoop/hive/ql/io/TestSymlinkTextInputFormat.java
index 4e7aaa826e2..f4c856f7e7a 100644
--- a/ql/src/test/org/apache/hadoop/hive/ql/io/TestSymlinkTextInputFormat.java
+++ b/ql/src/test/org/apache/hadoop/hive/ql/io/TestSymlinkTextInputFormat.java
@@ -158,7 +158,7 @@ public void testCombine() throws Exception {
drv.run(loadFileCommand);
- String cmd = "select key*1 from " + tblName;
+ String cmd = "select key*2 from " + tblName;
ecode = drv.compile(cmd, true);
if (ecode != 0) {
throw new Exception("Select compile: " + cmd
diff --git a/ql/src/test/org/apache/hadoop/hive/ql/optimizer/calcite/rules/TestHivePointLookupOptimizerRule.java b/ql/src/test/org/apache/hadoop/hive/ql/optimizer/calcite/rules/TestHivePointLookupOptimizerRule.java
index 8e06699e316..199a0673c8c 100644
--- a/ql/src/test/org/apache/hadoop/hive/ql/optimizer/calcite/rules/TestHivePointLookupOptimizerRule.java
+++ b/ql/src/test/org/apache/hadoop/hive/ql/optimizer/calcite/rules/TestHivePointLookupOptimizerRule.java
@@ -25,13 +25,13 @@
import org.apache.calcite.rex.RexBuilder;
import org.apache.calcite.rex.RexNode;
import org.apache.calcite.sql.SqlCollation;
-import org.apache.calcite.sql.fun.SqlStdOperatorTable;
import org.apache.calcite.sql.type.SqlTypeName;
import org.apache.calcite.tools.RelBuilder;
import org.apache.calcite.util.ConversionUtil;
import org.apache.hadoop.hive.ql.metadata.Table;
import org.apache.hadoop.hive.ql.optimizer.calcite.RelOptHiveTable;
import org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveFilter;
+import org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveIn;
import org.apache.hadoop.hive.ql.parse.type.RexNodeExprFactory;
import org.junit.Before;
import org.junit.Test;
@@ -75,6 +75,7 @@ private static class MyRecord {
public void before() {
planner = buildPlanner(Collections.singletonList(new HivePointLookupOptimizerRule.FilterCondition(2)));
relBuilder = buildRelBuilder(planner, schemaMock, tableMock, hiveTableMDMock, MyRecord.class);
+ relBuilder = relBuilder.transform(c -> c.withSimplify(false));
}
@Test
@@ -375,8 +376,8 @@ public void testSameVarcharLiteralDifferentPrecision() {
final RelNode basePlan = relBuilder
.scan("t")
.filter(and(relBuilder,
- relBuilder.call(SqlStdOperatorTable.IN, relBuilder.field("f2"), lita30, litb30),
- relBuilder.call(SqlStdOperatorTable.IN, relBuilder.field("f2"), lita14, litb14)))
+ relBuilder.call(HiveIn.INSTANCE, relBuilder.field("f2"), lita30, litb30),
+ relBuilder.call(HiveIn.INSTANCE, relBuilder.field("f2"), lita14, litb14)))
.build();
planner.setRoot(basePlan);
@@ -412,8 +413,8 @@ public void testSameVarcharLiteralDifferentPrecisionValueOverflow() {
final RelNode basePlan = relBuilder
.scan("t")
.filter(and(relBuilder,
- relBuilder.call(SqlStdOperatorTable.IN, relBuilder.field("f2"), lita30, litb30),
- relBuilder.call(SqlStdOperatorTable.IN, relBuilder.field("f2"), litaOverflow, litbOverflow)))
+ relBuilder.call(HiveIn.INSTANCE, relBuilder.field("f2"), lita30, litb30),
+ relBuilder.call(HiveIn.INSTANCE, relBuilder.field("f2"), litaOverflow, litbOverflow)))
.build();
planner.setRoot(basePlan);
@@ -444,8 +445,8 @@ public void testSameVarcharAndNullLiterals() {
final RelNode basePlan = relBuilder
.scan("t")
.filter(and(relBuilder,
- relBuilder.call(SqlStdOperatorTable.IN, relBuilder.field("f2"), lita30, litb30),
- relBuilder.call(SqlStdOperatorTable.IN, relBuilder.field("f2"), lita14, litb14)))
+ relBuilder.call(HiveIn.INSTANCE, relBuilder.field("f2"), lita30, litb30),
+ relBuilder.call(HiveIn.INSTANCE, relBuilder.field("f2"), lita14, litb14)))
.build();
planner.setRoot(basePlan);
@@ -476,8 +477,8 @@ public void testSameVarcharLiteralsDifferentPrecisionInOrExpression() {
final RelNode basePlan = relBuilder
.scan("t")
.filter(or(relBuilder,
- relBuilder.call(SqlStdOperatorTable.IN, relBuilder.field("f2"), lita30, litb30),
- relBuilder.call(SqlStdOperatorTable.IN, relBuilder.field("f2"), lita14, litb14)))
+ relBuilder.call(HiveIn.INSTANCE, relBuilder.field("f2"), lita30, litb30),
+ relBuilder.call(HiveIn.INSTANCE, relBuilder.field("f2"), lita14, litb14)))
.build();
planner.setRoot(basePlan);
@@ -507,8 +508,8 @@ public void testSameDecimalLiteralDifferentPrecision() {
final RelNode basePlan = relBuilder
.scan("t")
.filter(and(relBuilder,
- relBuilder.call(SqlStdOperatorTable.IN, relBuilder.field("f2"), lita30, litb30),
- relBuilder.call(SqlStdOperatorTable.IN, relBuilder.field("f2"), lita14, litb14)))
+ relBuilder.call(HiveIn.INSTANCE, relBuilder.field("f2"), lita30, litb30),
+ relBuilder.call(HiveIn.INSTANCE, relBuilder.field("f2"), lita14, litb14)))
.build();
planner.setRoot(basePlan);
diff --git a/ql/src/test/org/apache/hadoop/hive/ql/optimizer/calcite/rules/views/TestHivePushdownSnapshotFilterRule.java b/ql/src/test/org/apache/hadoop/hive/ql/optimizer/calcite/rules/views/TestHivePushdownSnapshotFilterRule.java
index a0fced83e29..2d96777f181 100644
--- a/ql/src/test/org/apache/hadoop/hive/ql/optimizer/calcite/rules/views/TestHivePushdownSnapshotFilterRule.java
+++ b/ql/src/test/org/apache/hadoop/hive/ql/optimizer/calcite/rules/views/TestHivePushdownSnapshotFilterRule.java
@@ -21,7 +21,6 @@
import org.apache.calcite.plan.RelOptUtil;
import org.apache.calcite.rel.RelNode;
import org.apache.calcite.sql.fun.SqlStdOperatorTable;
-import org.apache.calcite.sql.type.SqlTypeName;
import org.apache.calcite.tools.RelBuilder;
import org.apache.hadoop.hive.ql.optimizer.calcite.HiveRelFactories;
import org.apache.hadoop.hive.ql.optimizer.calcite.RelOptHiveTable;
@@ -47,9 +46,7 @@ public void testFilterIsRemovedAndVersionIntervalFromIsSetWhenFilterHasSnapshotI
RelBuilder relBuilder = HiveRelFactories.HIVE_BUILDER.create(relOptCluster, schemaMock);
RelNode root = relBuilder.push(tableScan)
- .filter(REX_BUILDER.makeCall(SqlStdOperatorTable.LESS_THAN_OR_EQUAL,
- REX_BUILDER.makeInputRef(HiveAugmentSnapshotMaterializationRule.snapshotIdType(TYPE_FACTORY), 3),
- REX_BUILDER.makeLiteral(42, TYPE_FACTORY.createSqlType(SqlTypeName.INTEGER), false)))
+ .filter(relBuilder.call(SqlStdOperatorTable.LESS_THAN_OR_EQUAL, relBuilder.field(3), relBuilder.literal(42)))
.build();
System.out.println(RelOptUtil.toString(root));
@@ -68,9 +65,7 @@ public void testFilterLeftIntactWhenItDoesNotHaveSnapshotIdPredicate() {
RelBuilder relBuilder = HiveRelFactories.HIVE_BUILDER.create(relOptCluster, schemaMock);
RelNode root = relBuilder.push(tableScan)
- .filter(REX_BUILDER.makeCall(SqlStdOperatorTable.LESS_THAN_OR_EQUAL,
- REX_BUILDER.makeInputRef(TYPE_FACTORY.createSqlType(SqlTypeName.INTEGER), 1),
- REX_BUILDER.makeLiteral(42, TYPE_FACTORY.createSqlType(SqlTypeName.INTEGER), false)))
+ .filter(relBuilder.call(SqlStdOperatorTable.LESS_THAN_OR_EQUAL, relBuilder.field(0), relBuilder.literal(42)))
.build();
System.out.println(RelOptUtil.toString(root));
diff --git a/ql/src/test/org/apache/hadoop/hive/ql/optimizer/calcite/stats/TestFilterSelectivityEstimator.java b/ql/src/test/org/apache/hadoop/hive/ql/optimizer/calcite/stats/TestFilterSelectivityEstimator.java
index b3dc6a602f9..4255c756e07 100644
--- a/ql/src/test/org/apache/hadoop/hive/ql/optimizer/calcite/stats/TestFilterSelectivityEstimator.java
+++ b/ql/src/test/org/apache/hadoop/hive/ql/optimizer/calcite/stats/TestFilterSelectivityEstimator.java
@@ -38,6 +38,7 @@
import org.apache.hadoop.hive.ql.optimizer.calcite.HiveRelFactories;
import org.apache.hadoop.hive.ql.optimizer.calcite.HiveTypeSystemImpl;
import org.apache.hadoop.hive.ql.optimizer.calcite.RelOptHiveTable;
+import org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveBetween;
import org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveRelNode;
import org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveTableScan;
import org.apache.hadoop.hive.ql.parse.CalcitePlanner;
@@ -364,7 +365,7 @@ public void testComputeRangePredicateSelectivityGreaterThanOrEqualWhenHigherThan
@Test
public void testComputeRangePredicateSelectivityBetween() {
doReturn(Collections.singletonList(stats)).when(tableMock).getColStat(Collections.singletonList(0));
- RexNode filter = REX_BUILDER.makeCall(SqlStdOperatorTable.BETWEEN, boolFalse, inputRef0, int1, int3);
+ RexNode filter = REX_BUILDER.makeCall(HiveBetween.INSTANCE, boolFalse, inputRef0, int1, int3);
FilterSelectivityEstimator estimator = new FilterSelectivityEstimator(scan, mq);
Assert.assertEquals(0.6923076923076923, estimator.estimateSelectivity(filter), DELTA);
}
@@ -372,7 +373,7 @@ public void testComputeRangePredicateSelectivityBetween() {
@Test
public void testComputeRangePredicateSelectivityBetweenFromMinToMax() {
doReturn(Collections.singletonList(stats)).when(tableMock).getColStat(Collections.singletonList(0));
- RexNode filter = REX_BUILDER.makeCall(SqlStdOperatorTable.BETWEEN, boolFalse, inputRef0, int1, int7);
+ RexNode filter = REX_BUILDER.makeCall(HiveBetween.INSTANCE, boolFalse, inputRef0, int1, int7);
FilterSelectivityEstimator estimator = new FilterSelectivityEstimator(scan, mq);
Assert.assertEquals(1, estimator.estimateSelectivity(filter), DELTA);
}
@@ -380,7 +381,7 @@ public void testComputeRangePredicateSelectivityBetweenFromMinToMax() {
@Test
public void testComputeRangePredicateSelectivityBetweenFromLowerThanMinToHigherThanMax() {
doReturn(Collections.singletonList(stats)).when(tableMock).getColStat(Collections.singletonList(0));
- RexNode filter = REX_BUILDER.makeCall(SqlStdOperatorTable.BETWEEN, boolFalse, inputRef0, int0, int8);
+ RexNode filter = REX_BUILDER.makeCall(HiveBetween.INSTANCE, boolFalse, inputRef0, int0, int8);
FilterSelectivityEstimator estimator = new FilterSelectivityEstimator(scan, mq);
Assert.assertEquals(1, estimator.estimateSelectivity(filter), DELTA);
}
@@ -388,7 +389,7 @@ public void testComputeRangePredicateSelectivityBetweenFromLowerThanMinToHigherT
@Test
public void testComputeRangePredicateSelectivityBetweenLeftLowerThanMin() {
doReturn(Collections.singletonList(stats)).when(tableMock).getColStat(Collections.singletonList(0));
- RexNode filter = REX_BUILDER.makeCall(SqlStdOperatorTable.BETWEEN, boolFalse, inputRef0, int0, int3);
+ RexNode filter = REX_BUILDER.makeCall(HiveBetween.INSTANCE, boolFalse, inputRef0, int0, int3);
FilterSelectivityEstimator estimator = new FilterSelectivityEstimator(scan, mq);
Assert.assertEquals(0.6923076923076923, estimator.estimateSelectivity(filter), DELTA);
}
@@ -396,7 +397,7 @@ public void testComputeRangePredicateSelectivityBetweenLeftLowerThanMin() {
@Test
public void testComputeRangePredicateSelectivityBetweenRightLowerThanMin() {
doReturn(Collections.singletonList(stats)).when(tableMock).getColStat(Collections.singletonList(0));
- RexNode filter = REX_BUILDER.makeCall(SqlStdOperatorTable.BETWEEN, boolFalse, inputRef0, intMinus1, int0);
+ RexNode filter = REX_BUILDER.makeCall(HiveBetween.INSTANCE, boolFalse, inputRef0, intMinus1, int0);
FilterSelectivityEstimator estimator = new FilterSelectivityEstimator(scan, mq);
Assert.assertEquals(0, estimator.estimateSelectivity(filter), DELTA);
}
@@ -404,7 +405,7 @@ public void testComputeRangePredicateSelectivityBetweenRightLowerThanMin() {
@Test
public void testComputeRangePredicateSelectivityBetweenLeftHigherThanMax() {
doReturn(Collections.singletonList(stats)).when(tableMock).getColStat(Collections.singletonList(0));
- RexNode filter = REX_BUILDER.makeCall(SqlStdOperatorTable.BETWEEN, boolFalse, inputRef0, int10, int11);
+ RexNode filter = REX_BUILDER.makeCall(HiveBetween.INSTANCE, boolFalse, inputRef0, int10, int11);
FilterSelectivityEstimator estimator = new FilterSelectivityEstimator(scan, mq);
Assert.assertEquals(0, estimator.estimateSelectivity(filter), DELTA);
}
@@ -412,7 +413,7 @@ public void testComputeRangePredicateSelectivityBetweenLeftHigherThanMax() {
@Test
public void testComputeRangePredicateSelectivityBetweenLeftLowerThanRight() {
doReturn(Collections.singletonList(stats)).when(tableMock).getColStat(Collections.singletonList(0));
- RexNode filter = REX_BUILDER.makeCall(SqlStdOperatorTable.BETWEEN, boolFalse, inputRef0, int4, int2);
+ RexNode filter = REX_BUILDER.makeCall(HiveBetween.INSTANCE, boolFalse, inputRef0, int4, int2);
FilterSelectivityEstimator estimator = new FilterSelectivityEstimator(scan, mq);
Assert.assertEquals(0, estimator.estimateSelectivity(filter), DELTA);
}
@@ -421,7 +422,7 @@ public void testComputeRangePredicateSelectivityBetweenLeftLowerThanRight() {
public void testComputeRangePredicateSelectivityBetweenLeftEqualsRight() {
doReturn(Collections.singletonList(stats)).when(tableMock).getColStat(Collections.singletonList(0));
doReturn(10.0).when(mq).getDistinctRowCount(scan, ImmutableBitSet.of(0), REX_BUILDER.makeLiteral(true));
- RexNode filter = REX_BUILDER.makeCall(SqlStdOperatorTable.BETWEEN, boolFalse, inputRef0, int3, int3);
+ RexNode filter = REX_BUILDER.makeCall(HiveBetween.INSTANCE, boolFalse, inputRef0, int3, int3);
FilterSelectivityEstimator estimator = new FilterSelectivityEstimator(scan, mq);
// this is what FilterSelectivityEstimator returns for a generic "function" based on NDV values, in this case 1 / 10
Assert.assertEquals(0.1, estimator.estimateSelectivity(filter), DELTA);
@@ -430,7 +431,7 @@ public void testComputeRangePredicateSelectivityBetweenLeftEqualsRight() {
@Test
public void testComputeRangePredicateSelectivityNotBetween() {
doReturn(Collections.singletonList(stats)).when(tableMock).getColStat(Collections.singletonList(0));
- RexNode filter = REX_BUILDER.makeCall(SqlStdOperatorTable.BETWEEN, boolTrue, inputRef0, int3, int5);
+ RexNode filter = REX_BUILDER.makeCall(HiveBetween.INSTANCE, boolTrue, inputRef0, int3, int5);
FilterSelectivityEstimator estimator = new FilterSelectivityEstimator(scan, mq);
Assert.assertEquals(0.7692307692307693, estimator.estimateSelectivity(filter), DELTA);
}
@@ -438,7 +439,7 @@ public void testComputeRangePredicateSelectivityNotBetween() {
@Test
public void testComputeRangePredicateSelectivityNotBetweenLowerThanMinHigherThanMax() {
doReturn(Collections.singletonList(stats)).when(tableMock).getColStat(Collections.singletonList(0));
- RexNode filter = REX_BUILDER.makeCall(SqlStdOperatorTable.BETWEEN, boolTrue, inputRef0, int0, int10);
+ RexNode filter = REX_BUILDER.makeCall(HiveBetween.INSTANCE, boolTrue, inputRef0, int0, int10);
FilterSelectivityEstimator estimator = new FilterSelectivityEstimator(scan, mq);
Assert.assertEquals(0, estimator.estimateSelectivity(filter), DELTA);
}
@@ -446,7 +447,7 @@ public void testComputeRangePredicateSelectivityNotBetweenLowerThanMinHigherThan
@Test
public void testComputeRangePredicateSelectivityNotBetweenRightLowerThanLeft() {
doReturn(Collections.singletonList(stats)).when(tableMock).getColStat(Collections.singletonList(0));
- RexNode filter = REX_BUILDER.makeCall(SqlStdOperatorTable.BETWEEN, boolTrue, inputRef0, int5, int3);
+ RexNode filter = REX_BUILDER.makeCall(HiveBetween.INSTANCE, boolTrue, inputRef0, int5, int3);
FilterSelectivityEstimator estimator = new FilterSelectivityEstimator(scan, mq);
Assert.assertEquals(1, estimator.estimateSelectivity(filter), DELTA);
}
@@ -454,7 +455,7 @@ public void testComputeRangePredicateSelectivityNotBetweenRightLowerThanLeft() {
@Test
public void testComputeRangePredicateSelectivityNotBetweenLeftEqualsRight() {
doReturn(Collections.singletonList(stats)).when(tableMock).getColStat(Collections.singletonList(0));
- RexNode filter = REX_BUILDER.makeCall(SqlStdOperatorTable.BETWEEN, boolTrue, inputRef0, int3, int3);
+ RexNode filter = REX_BUILDER.makeCall(HiveBetween.INSTANCE, boolTrue, inputRef0, int3, int3);
FilterSelectivityEstimator estimator = new FilterSelectivityEstimator(scan, mq);
Assert.assertEquals(1, estimator.estimateSelectivity(filter), DELTA);
}
@@ -499,7 +500,7 @@ public void testComputeRangePredicateSelectivityLessThanOrEqualWithNULLS() {
public void testComputeRangePredicateSelectivityBetweenWithNULLS() {
doReturn((double) 20).when(tableMock).getRowCount();
doReturn(Collections.singletonList(stats)).when(tableMock).getColStat(Collections.singletonList(0));
- RexNode filter = REX_BUILDER.makeCall(SqlStdOperatorTable.BETWEEN, boolFalse, inputRef0, int1, int3);
+ RexNode filter = REX_BUILDER.makeCall(HiveBetween.INSTANCE, boolFalse, inputRef0, int1, int3);
FilterSelectivityEstimator estimator = new FilterSelectivityEstimator(scan, mq);
Assert.assertEquals(0.45, estimator.estimateSelectivity(filter), DELTA);
}
@@ -508,7 +509,7 @@ public void testComputeRangePredicateSelectivityBetweenWithNULLS() {
public void testComputeRangePredicateSelectivityNotBetweenWithNULLS() {
doReturn((double) 20).when(tableMock).getRowCount();
doReturn(Collections.singletonList(stats)).when(tableMock).getColStat(Collections.singletonList(0));
- RexNode filter = REX_BUILDER.makeCall(SqlStdOperatorTable.BETWEEN, boolTrue, inputRef0, int1, int3);
+ RexNode filter = REX_BUILDER.makeCall(HiveBetween.INSTANCE, boolTrue, inputRef0, int1, int3);
FilterSelectivityEstimator estimator = new FilterSelectivityEstimator(scan, mq);
Assert.assertEquals(0.55, estimator.estimateSelectivity(filter), DELTA);
}
diff --git a/ql/src/test/org/apache/hadoop/hive/ql/plan/mapping/TestCounterMapping.java b/ql/src/test/org/apache/hadoop/hive/ql/plan/mapping/TestCounterMapping.java
index 66f508e215a..299c6af24ff 100644
--- a/ql/src/test/org/apache/hadoop/hive/ql/plan/mapping/TestCounterMapping.java
+++ b/ql/src/test/org/apache/hadoop/hive/ql/plan/mapping/TestCounterMapping.java
@@ -156,7 +156,7 @@ public void testInConversion() throws ParseException, CommandProcessorException
OpTreeSignature filterSig = pm.lookup(OpTreeSignature.class, fos.get(0));
Object pred = filterSig.getSig().getSigMap().get("getPredicateString");
- assertEquals("((u = 1) or (u = 2)) (type: boolean)", pred);
+ assertEquals("(u) IN (1, 2) (type: boolean)", pred);
}
diff --git a/ql/src/test/results/clientpositive/encrypted/encryption_join_unencrypted_tbl.q.out b/ql/src/test/results/clientpositive/encrypted/encryption_join_unencrypted_tbl.q.out
index d8da3dc6393..9ad7433814b 100644
--- a/ql/src/test/results/clientpositive/encrypted/encryption_join_unencrypted_tbl.q.out
+++ b/ql/src/test/results/clientpositive/encrypted/encryption_join_unencrypted_tbl.q.out
@@ -545,12 +545,12 @@ POSTHOOK: Input: default@encrypted_table
POSTHOOK: Input: default@src
POSTHOOK: Output: hdfs://### HDFS PATH ###
OPTIMIZED SQL: SELECT `t0`.`key`, `t0`.`value`, `t2`.`key` AS `key1`, `t2`.`value` AS `value1`
-FROM (SELECT `key`, `value`, CAST(`key` AS DOUBLE) AS `CAST`
+FROM (SELECT `key`, `value`, CAST(`key` AS DOUBLE) AS `EXPR$0`
FROM `default`.`src`
WHERE CAST(`key` AS DOUBLE) IS NOT NULL) AS `t0`
-INNER JOIN (SELECT `key`, `value`, CAST(`key` AS DOUBLE) AS `CAST`
+INNER JOIN (SELECT `key`, `value`, CAST(`key` AS DOUBLE) AS `EXPR$0`
FROM `default`.`encrypted_table`
-WHERE CAST(`key` AS DOUBLE) IS NOT NULL) AS `t2` ON `t0`.`CAST` = `t2`.`CAST`
+WHERE CAST(`key` AS DOUBLE) IS NOT NULL) AS `t2` ON `t0`.`EXPR$0` = `t2`.`EXPR$0`
STAGE DEPENDENCIES:
Stage-1 is a root stage
Stage-0 depends on stages: Stage-1
diff --git a/ql/src/test/results/clientpositive/llap/acid_no_buckets.q.out b/ql/src/test/results/clientpositive/llap/acid_no_buckets.q.out
index a8c8e0ab9eb..f9feae7ca0d 100644
--- a/ql/src/test/results/clientpositive/llap/acid_no_buckets.q.out
+++ b/ql/src/test/results/clientpositive/llap/acid_no_buckets.q.out
@@ -144,10 +144,10 @@ STAGE PLANS:
Map Operator Tree:
TableScan
alias: srcpart_acid
- filterExpr: ((UDFToInteger(key)) IN (413, 43) and (hr = '11')) (type: boolean)
+ filterExpr: ((UDFToInteger(key)) IN (43, 413) and (hr = '11')) (type: boolean)
Statistics: Num rows: 1000 Data size: 362000 Basic stats: COMPLETE Column stats: PARTIAL
Filter Operator
- predicate: (UDFToInteger(key)) IN (413, 43) (type: boolean)
+ predicate: (UDFToInteger(key)) IN (43, 413) (type: boolean)
Statistics: Num rows: 500 Data size: 181000 Basic stats: COMPLETE Column stats: PARTIAL
Select Operator
expressions: ROW__ID (type: struct<writeid:bigint,bucketid:int,rowid:bigint>), ds (type: string), key (type: string), concat(value, 'updated') (type: string)
@@ -852,10 +852,10 @@ STAGE PLANS:
Map Operator Tree:
TableScan
alias: srcpart_acidb
- filterExpr: ((UDFToInteger(key)) IN (413, 43) and (hr = '11')) (type: boolean)
+ filterExpr: ((UDFToInteger(key)) IN (43, 413) and (hr = '11')) (type: boolean)
Statistics: Num rows: 1000 Data size: 362000 Basic stats: COMPLETE Column stats: PARTIAL
Filter Operator
- predicate: (UDFToInteger(key)) IN (413, 43) (type: boolean)
+ predicate: (UDFToInteger(key)) IN (43, 413) (type: boolean)
Statistics: Num rows: 500 Data size: 181000 Basic stats: COMPLETE Column stats: PARTIAL
Select Operator
expressions: ROW__ID (type: struct<writeid:bigint,bucketid:int,rowid:bigint>), ds (type: string), key (type: string), concat(value, 'updated') (type: string)
@@ -1461,7 +1461,7 @@ STAGE PLANS:
Filter Vectorization:
className: VectorFilterOperator
native: true
- predicateExpression: FilterLongColumnInList(col 6:int, values [413, 43])(children: CastStringToLong(col 0:string) -> 6:int)
+ predicateExpression: FilterLongColumnInList(col 6:int, values [43, 413])(children: CastStringToLong(col 0:string) -> 6:int)
Select Vectorization:
className: VectorSelectOperator
native: true
@@ -2336,7 +2336,7 @@ STAGE PLANS:
Filter Vectorization:
className: VectorFilterOperator
native: true
- predicateExpression: FilterLongColumnInList(col 6:int, values [413, 43])(children: CastStringToLong(col 0:string) -> 6:int)
+ predicateExpression: FilterLongColumnInList(col 6:int, values [43, 413])(children: CastStringToLong(col 0:string) -> 6:int)
Select Vectorization:
className: VectorSelectOperator
native: true
diff --git a/ql/src/test/results/clientpositive/llap/acid_nullscan.q.out b/ql/src/test/results/clientpositive/llap/acid_nullscan.q.out
index c1b59389137..67f143f9813 100644
--- a/ql/src/test/results/clientpositive/llap/acid_nullscan.q.out
+++ b/ql/src/test/results/clientpositive/llap/acid_nullscan.q.out
@@ -40,7 +40,7 @@ POSTHOOK: Input: default@acid_vectorized_n1
#### A masked pattern was here ####
OPTIMIZED SQL: SELECT SUM(`$f0`) AS `$f0`
FROM (SELECT *
-FROM (VALUES (NULL)) AS `t` (`$f0`)
+FROM (SELECT NULL AS `$f0`) AS `t`
WHERE 1 = 0) AS `t`
STAGE DEPENDENCIES:
Stage-1 is a root stage
diff --git a/ql/src/test/results/clientpositive/llap/alter_partition_coltype.q.out b/ql/src/test/results/clientpositive/llap/alter_partition_coltype.q.out
index e6c1340008a..77062447ab1 100644
--- a/ql/src/test/results/clientpositive/llap/alter_partition_coltype.q.out
+++ b/ql/src/test/results/clientpositive/llap/alter_partition_coltype.q.out
@@ -548,7 +548,7 @@ POSTHOOK: Input: pt@alterdynamic_part_table@partcol1=2/partcol2=1
#### A masked pattern was here ####
OPTIMIZED SQL: SELECT `intcol`
FROM `pt`.`alterdynamic_part_table`
-WHERE (`partcol1`, `partcol2`) IN ( (2, '1'), (1, '__HIVE_DEFAULT_PARTITION__'))
+WHERE (`partcol1`, `partcol2`) IN ((2, '1'), (1, '__HIVE_DEFAULT_PARTITION__'))
STAGE DEPENDENCIES:
Stage-0 is a root stage
diff --git a/ql/src/test/results/clientpositive/llap/ambiguitycheck.q.out b/ql/src/test/results/clientpositive/llap/ambiguitycheck.q.out
index e3b931075df..6b7fa8cb677 100644
--- a/ql/src/test/results/clientpositive/llap/ambiguitycheck.q.out
+++ b/ql/src/test/results/clientpositive/llap/ambiguitycheck.q.out
@@ -709,7 +709,7 @@ POSTHOOK: query: explain extended select int(1.2) from src limit 1
POSTHOOK: type: QUERY
POSTHOOK: Input: default@src
#### A masked pattern was here ####
-OPTIMIZED SQL: SELECT CAST(1.2 AS INT) AS `_o__c0`
+OPTIMIZED SQL: SELECT CAST(1 AS INT) AS `_o__c0`
FROM `default`.`src`
LIMIT 1
STAGE DEPENDENCIES:
diff --git a/ql/src/test/results/clientpositive/llap/annotate_stats_filter.q.out b/ql/src/test/results/clientpositive/llap/annotate_stats_filter.q.out
index 41f21f625b8..3ddbcfc696d 100644
--- a/ql/src/test/results/clientpositive/llap/annotate_stats_filter.q.out
+++ b/ql/src/test/results/clientpositive/llap/annotate_stats_filter.q.out
@@ -487,9 +487,9 @@ STAGE PLANS:
Processor Tree:
TableScan
alias: loc_orc
- filterExpr: (state) IN ('OH', 'CA') (type: boolean)
+ filterExpr: (state) IN ('CA', 'OH') (type: boolean)
Filter Operator
- predicate: (state) IN ('OH', 'CA') (type: boolean)
+ predicate: (state) IN ('CA', 'OH') (type: boolean)
Select Operator
expressions: state (type: string), locid (type: int), zip (type: bigint), year (type: int)
outputColumnNames: _col0, _col1, _col2, _col3
@@ -575,9 +575,9 @@ STAGE PLANS:
Processor Tree:
TableScan
alias: loc_orc
- filterExpr: (((year = 2001) or year is null) and (state = 'CA')) (type: boolean)
+ filterExpr: ((year is null or (year = 2001)) and (state = 'CA')) (type: boolean)
Filter Operator
- predicate: (((year = 2001) or year is null) and (state = 'CA')) (type: boolean)
+ predicate: ((year is null or (year = 2001)) and (state = 'CA')) (type: boolean)
Select Operator
expressions: 'CA' (type: string), locid (type: int), zip (type: bigint), year (type: int)
outputColumnNames: _col0, _col1, _col2, _col3
diff --git a/ql/src/test/results/clientpositive/llap/autoColumnStats_5a.q.out b/ql/src/test/results/clientpositive/llap/autoColumnStats_5a.q.out
index a6218663cb6..9c94c3e6179 100644
--- a/ql/src/test/results/clientpositive/llap/autoColumnStats_5a.q.out
+++ b/ql/src/test/results/clientpositive/llap/autoColumnStats_5a.q.out
@@ -18,7 +18,7 @@ POSTHOOK: Input: _dummy_database@_dummy_table
POSTHOOK: Output: default@partitioned1@part=1
Explain
OPTIMIZED SQL: SELECT *
-FROM TABLE(INLINE(ARRAY[ (1, 'original')]))
+FROM TABLE(INLINE(ARRAY[(1, 'original')]))
STAGE DEPENDENCIES:
Stage-1 is a root stage
Stage-2 depends on stages: Stage-1
@@ -279,7 +279,7 @@ POSTHOOK: Input: _dummy_database@_dummy_table
POSTHOOK: Output: default@partitioned1@part=1
Explain
OPTIMIZED SQL: SELECT *
-FROM TABLE(INLINE(ARRAY[ (2, 'original'), (3, 'original'), (4, 'original')]))
+FROM TABLE(INLINE(ARRAY[(2, 'original'), (3, 'original'), (4, 'original')]))
STAGE DEPENDENCIES:
Stage-1 is a root stage
Stage-2 depends on stages: Stage-1
diff --git a/ql/src/test/results/clientpositive/llap/auto_join21.q.out b/ql/src/test/results/clientpositive/llap/auto_join21.q.out
index eb62055400d..055c1b8ecd5 100644
--- a/ql/src/test/results/clientpositive/llap/auto_join21.q.out
+++ b/ql/src/test/results/clientpositive/llap/auto_join21.q.out
@@ -26,11 +26,11 @@ STAGE PLANS:
Map Operator Tree:
TableScan
alias: src1
- filterExpr: ((UDFToDouble(key) < 11.0D) and (UDFToDouble(key) > 9.0D)) (type: boolean)
+ filterExpr: ((UDFToDouble(key) > 9.0D) and (UDFToDouble(key) < 11.0D)) (type: boolean)
probeDecodeDetails: cacheKey:HASH_MAP_MAPJOIN_40_container, bigKeyColName:key, smallTablePos:1, keyRatio:0.11
Statistics: Num rows: 500 Data size: 89000 Basic stats: COMPLETE Column stats: COMPLETE
Filter Operator
- predicate: ((UDFToDouble(key) < 11.0D) and (UDFToDouble(key) > 9.0D)) (type: boolean)
+ predicate: ((UDFToDouble(key) > 9.0D) and (UDFToDouble(key) < 11.0D)) (type: boolean)
Statistics: Num rows: 55 Data size: 9790 Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
expressions: key (type: string), value (type: string)
diff --git a/ql/src/test/results/clientpositive/llap/auto_join28.q.out b/ql/src/test/results/clientpositive/llap/auto_join28.q.out
index 448aff7a9d2..3ddd4293ffa 100644
--- a/ql/src/test/results/clientpositive/llap/auto_join28.q.out
+++ b/ql/src/test/results/clientpositive/llap/auto_join28.q.out
@@ -26,11 +26,11 @@ STAGE PLANS:
Map Operator Tree:
TableScan
alias: src1
- filterExpr: ((UDFToDouble(key) < 11.0D) and (UDFToDouble(key) > 9.0D)) (type: boolean)
+ filterExpr: ((UDFToDouble(key) > 9.0D) and (UDFToDouble(key) < 11.0D)) (type: boolean)
probeDecodeDetails: cacheKey:HASH_MAP_MAPJOIN_40_container, bigKeyColName:key, smallTablePos:1, keyRatio:0.11
Statistics: Num rows: 500 Data size: 89000 Basic stats: COMPLETE Column stats: COMPLETE
Filter Operator
- predicate: ((UDFToDouble(key) < 11.0D) and (UDFToDouble(key) > 9.0D)) (type: boolean)
+ predicate: ((UDFToDouble(key) > 9.0D) and (UDFToDouble(key) < 11.0D)) (type: boolean)
Statistics: Num rows: 55 Data size: 9790 Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
expressions: key (type: string), value (type: string)
diff --git a/ql/src/test/results/clientpositive/llap/auto_join29.q.out b/ql/src/test/results/clientpositive/llap/auto_join29.q.out
index 14182967c86..de5f7b1ea0a 100644
--- a/ql/src/test/results/clientpositive/llap/auto_join29.q.out
+++ b/ql/src/test/results/clientpositive/llap/auto_join29.q.out
@@ -26,11 +26,11 @@ STAGE PLANS:
Map Operator Tree:
TableScan
alias: src1
- filterExpr: ((UDFToDouble(key) < 11.0D) and (UDFToDouble(key) > 9.0D)) (type: boolean)
+ filterExpr: ((UDFToDouble(key) > 9.0D) and (UDFToDouble(key) < 11.0D)) (type: boolean)
probeDecodeDetails: cacheKey:HASH_MAP_MAPJOIN_40_container, bigKeyColName:key, smallTablePos:1, keyRatio:0.11
Statistics: Num rows: 500 Data size: 89000 Basic stats: COMPLETE Column stats: COMPLETE
Filter Operator
- predicate: ((UDFToDouble(key) < 11.0D) and (UDFToDouble(key) > 9.0D)) (type: boolean)
+ predicate: ((UDFToDouble(key) > 9.0D) and (UDFToDouble(key) < 11.0D)) (type: boolean)
Statistics: Num rows: 55 Data size: 9790 Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
expressions: key (type: string), value (type: string)
@@ -2605,11 +2605,11 @@ STAGE PLANS:
Map Operator Tree:
TableScan
alias: src1
- filterExpr: ((UDFToDouble(key) < 11.0D) and (UDFToDouble(key) > 9.0D)) (type: boolean)
+ filterExpr: ((UDFToDouble(key) > 9.0D) and (UDFToDouble(key) < 11.0D)) (type: boolean)
probeDecodeDetails: cacheKey:HASH_MAP_MAPJOIN_40_container, bigKeyColName:key, smallTablePos:1, keyRatio:0.11
Statistics: Num rows: 500 Data size: 89000 Basic stats: COMPLETE Column stats: COMPLETE
Filter Operator
- predicate: ((UDFToDouble(key) < 11.0D) and (UDFToDouble(key) > 9.0D)) (type: boolean)
+ predicate: ((UDFToDouble(key) > 9.0D) and (UDFToDouble(key) < 11.0D)) (type: boolean)
Statistics: Num rows: 55 Data size: 9790 Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
expressions: key (type: string), value (type: string)
@@ -2646,7 +2646,7 @@ STAGE PLANS:
Map Operator Tree:
TableScan
alias: src2
- filterExpr: (((UDFToDouble(key) > 9.0D) and (UDFToDouble(key) < 11.0D)) or ((UDFToDouble(key) < 10.0D) and (UDFToDouble(key) > 9.0D))) (type: boolean)
+ filterExpr: (((UDFToDouble(key) > 9.0D) and (UDFToDouble(key) < 11.0D)) or ((UDFToDouble(key) > 9.0D) and (UDFToDouble(key) < 10.0D))) (type: boolean)
Statistics: Num rows: 500 Data size: 89000 Basic stats: COMPLETE Column stats: COMPLETE
Filter Operator
predicate: ((UDFToDouble(key) > 9.0D) and (UDFToDouble(key) < 11.0D)) (type: boolean)
@@ -2663,7 +2663,7 @@ STAGE PLANS:
Statistics: Num rows: 55 Data size: 9790 Basic stats: COMPLETE Column stats: COMPLETE
value expressions: _col1 (type: string)
Filter Operator
- predicate: ((UDFToDouble(key) < 10.0D) and (UDFToDouble(key) > 9.0D)) (type: boolean)
+ predicate: ((UDFToDouble(key) > 9.0D) and (UDFToDouble(key) < 10.0D)) (type: boolean)
Statistics: Num rows: 55 Data size: 9790 Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
expressions: key (type: string), value (type: string)
diff --git a/ql/src/test/results/clientpositive/llap/auto_join30.q.out b/ql/src/test/results/clientpositive/llap/auto_join30.q.out
index 681148fc5fc..5424a231676 100644
--- a/ql/src/test/results/clientpositive/llap/auto_join30.q.out
+++ b/ql/src/test/results/clientpositive/llap/auto_join30.q.out
@@ -624,36 +624,37 @@ STAGE PLANS:
TableScan
alias: src
filterExpr: key is not null (type: boolean)
- Statistics: Num rows: 500 Data size: 43500 Basic stats: COMPLETE Column stats: COMPLETE
+ probeDecodeDetails: cacheKey:HASH_MAP_MAPJOIN_43_container, bigKeyColName:key, smallTablePos:1, keyRatio:0.632
+ Statistics: Num rows: 500 Data size: 89000 Basic stats: COMPLETE Column stats: COMPLETE
Filter Operator
predicate: key is not null (type: boolean)
- Statistics: Num rows: 500 Data size: 43500 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 500 Data size: 89000 Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
- expressions: key (type: string)
- outputColumnNames: _col0
- Statistics: Num rows: 500 Data size: 43500 Basic stats: COMPLETE Column stats: COMPLETE
+ expressions: key (type: string), value (type: string)
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 500 Data size: 89000 Basic stats: COMPLETE Column stats: COMPLETE
Map Join Operator
condition map:
- Left Outer Join 0 to 1
+ Inner Join 0 to 1
keys:
0 _col0 (type: string)
1 _col0 (type: string)
- outputColumnNames: _col0
+ outputColumnNames: _col0, _col1, _col2
input vertices:
1 Reducer 4
- Statistics: Num rows: 791 Data size: 68817 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 791 Data size: 209615 Basic stats: COMPLETE Column stats: COMPLETE
Map Join Operator
condition map:
- Inner Join 0 to 1
+ Left Outer Join 0 to 1
keys:
- 0 _col0 (type: string)
+ 0 _col2 (type: string)
1 _col0 (type: string)
- outputColumnNames: _col2, _col3
+ outputColumnNames: _col0, _col1
input vertices:
1 Map 3
Statistics: Num rows: 1251 Data size: 222678 Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
- expressions: hash(_col2,_col3) (type: int)
+ expressions: hash(_col0,_col1) (type: int)
outputColumnNames: _col0
Statistics: Num rows: 1251 Data size: 222678 Basic stats: COMPLETE Column stats: COMPLETE
Group By Operator
@@ -674,25 +675,20 @@ STAGE PLANS:
TableScan
alias: src
filterExpr: key is not null (type: boolean)
- Statistics: Num rows: 500 Data size: 89000 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 500 Data size: 43500 Basic stats: COMPLETE Column stats: COMPLETE
Filter Operator
predicate: key is not null (type: boolean)
- Statistics: Num rows: 500 Data size: 89000 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 500 Data size: 43500 Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
- expressions: key (type: string), value (type: string)
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 500 Data size: 89000 Basic stats: COMPLETE Column stats: COMPLETE
+ expressions: key (type: string)
+ outputColumnNames: _col0
+ Statistics: Num rows: 500 Data size: 43500 Basic stats: COMPLETE Column stats: COMPLETE
Reduce Output Operator
key expressions: _col0 (type: string)
null sort order: z
sort order: +
Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 500 Data size: 89000 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions: _col1 (type: string)
- Select Operator
- expressions: key (type: string)
- outputColumnNames: _col0
- Statistics: Num rows: 500 Data size: 43500 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 500 Data size: 43500 Basic stats: COMPLETE Column stats: COMPLETE
Reduce Output Operator
key expressions: _col0 (type: string)
null sort order: z
diff --git a/ql/src/test/results/clientpositive/llap/auto_join5.q.out b/ql/src/test/results/clientpositive/llap/auto_join5.q.out
index d2202ee91b6..81c2de5dc35 100644
--- a/ql/src/test/results/clientpositive/llap/auto_join5.q.out
+++ b/ql/src/test/results/clientpositive/llap/auto_join5.q.out
@@ -59,10 +59,10 @@ STAGE PLANS:
Map Operator Tree:
TableScan
alias: src1
- filterExpr: ((UDFToDouble(key) < 20.0D) and (UDFToDouble(key) > 15.0D)) (type: boolean)
+ filterExpr: ((UDFToDouble(key) > 15.0D) and (UDFToDouble(key) < 20.0D)) (type: boolean)
Statistics: Num rows: 500 Data size: 89000 Basic stats: COMPLETE Column stats: COMPLETE
Filter Operator
- predicate: ((UDFToDouble(key) < 20.0D) and (UDFToDouble(key) > 15.0D)) (type: boolean)
+ predicate: ((UDFToDouble(key) > 15.0D) and (UDFToDouble(key) < 20.0D)) (type: boolean)
Statistics: Num rows: 55 Data size: 9790 Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
expressions: key (type: string), value (type: string)
diff --git a/ql/src/test/results/clientpositive/llap/auto_join_reordering_values.q.out b/ql/src/test/results/clientpositive/llap/auto_join_reordering_values.q.out
index 892ac5cdb2c..bdd2ba97061 100644
--- a/ql/src/test/results/clientpositive/llap/auto_join_reordering_values.q.out
+++ b/ql/src/test/results/clientpositive/llap/auto_join_reordering_values.q.out
@@ -96,7 +96,7 @@ FROM `default`.`user_small`
WHERE `userid` IS NOT NULL) AS `t0`
INNER JOIN ((SELECT `dealid`, `date`, `cityid`, `userid`
FROM `default`.`orderpayment_small`
-WHERE `date` IS NOT NULL AND `dealid` IS NOT NULL AND (`cityid` IS NOT NULL AND `userid` IS NOT NULL)) AS `t2` INNER JOIN (SELECT `date`
+WHERE `date` IS NOT NULL AND `dealid` IS NOT NULL AND `cityid` IS NOT NULL AND `userid` IS NOT NULL) AS `t2` INNER JOIN (SELECT `date`
FROM `default`.`orderpayment_small`
WHERE `date` IS NOT NULL) AS `t4` ON `t2`.`date` = `t4`.`date` INNER JOIN (SELECT `dealid`
FROM `default`.`orderpayment_small`
diff --git a/ql/src/test/results/clientpositive/llap/bucket_map_join_tez2.q.out b/ql/src/test/results/clientpositive/llap/bucket_map_join_tez2.q.out
index 4757c32979f..1cd945de8d8 100644
--- a/ql/src/test/results/clientpositive/llap/bucket_map_join_tez2.q.out
+++ b/ql/src/test/results/clientpositive/llap/bucket_map_join_tez2.q.out
@@ -2349,7 +2349,7 @@ POSTHOOK: Input: default@my_fact
OPTIMIZED SQL: SELECT `t0`.`bucket_col`, `t2`.`join_col` AS `account1`, `t0`.`accounting_period`
FROM (SELECT `bucket_col`, `join_col`, `accounting_period`
FROM `default`.`my_fact`
-WHERE `fiscal_year` = '2015' AND (`accounting_period` = 10 AND `join_col` IS NOT NULL)) AS `t0`
+WHERE `fiscal_year` = '2015' AND `accounting_period` = 10 AND `join_col` IS NOT NULL) AS `t0`
INNER JOIN (SELECT `join_col`
FROM `default`.`my_dim`
WHERE `filter_col` IN ('VAL1', 'VAL2') AND `join_col` IS NOT NULL) AS `t2` ON `t0`.`join_col` = `t2`.`join_col`
diff --git a/ql/src/test/results/clientpositive/llap/bucket_map_join_tez3.q.out b/ql/src/test/results/clientpositive/llap/bucket_map_join_tez3.q.out
index 273295e916b..1330de2ce39 100644
--- a/ql/src/test/results/clientpositive/llap/bucket_map_join_tez3.q.out
+++ b/ql/src/test/results/clientpositive/llap/bucket_map_join_tez3.q.out
@@ -109,10 +109,10 @@ POSTHOOK: Input: default@target_table
OPTIMIZED SQL: SELECT `t0`.`date_col`, CAST('pipeline' AS STRING) AS `string_col`, `t0`.`decimal_col`, `t2`.`date_col` AS `date_col1`, 'pipeline' AS `string_col1`, `t2`.`decimal_col` AS `decimal_col1`
FROM (SELECT `date_col`, `decimal_col`
FROM `default`.`target_table`
-WHERE `string_col` = 'pipeline' AND CASE WHEN `decimal_col` IS NOT NULL THEN CAST(`decimal_col` AS STRING) = '50000000000000000005905545593' ELSE FALSE END AND (`date_col` IS NOT NULL AND `decimal_col` IS NOT NULL)) AS `t0`
+WHERE `string_col` = 'pipeline' AND CASE WHEN `decimal_col` IS NOT NULL THEN CAST(`decimal_col` AS STRING) = '50000000000000000005905545593' ELSE FALSE END AND `date_col` IS NOT NULL AND `decimal_col` IS NOT NULL) AS `t0`
INNER JOIN (SELECT `date_col`, `decimal_col`
FROM `default`.`source_table`
-WHERE CASE WHEN `decimal_col` IS NOT NULL THEN CAST(`decimal_col` AS STRING) = '50000000000000000005905545593' ELSE FALSE END AND (`date_col` IS NOT NULL AND `decimal_col` IS NOT NULL)) AS `t2` ON `t0`.`date_col` = `t2`.`date_col` AND `t0`.`decimal_col` = `t2`.`decimal_col`
+WHERE CASE WHEN `decimal_col` IS NOT NULL THEN CAST(`decimal_col` AS STRING) = '50000000000000000005905545593' ELSE FALSE END AND `date_col` IS NOT NULL AND `decimal_col` IS NOT NULL) AS `t2` ON `t0`.`date_col` = `t2`.`date_col` AND `t0`.`decimal_col` = `t2`.`decimal_col`
STAGE DEPENDENCIES:
Stage-1 is a root stage
Stage-0 depends on stages: Stage-1
@@ -336,10 +336,10 @@ POSTHOOK: Input: default@target_table
OPTIMIZED SQL: SELECT `t0`.`date_col`, CAST('pipeline' AS STRING) AS `string_col`, `t0`.`decimal_col`, `t3`.`date_col` AS `date_col1`, 'pipeline' AS `string_col1`, `t3`.`decimal_col` AS `decimal_col1`
FROM (SELECT `date_col`, `decimal_col`
FROM `default`.`target_table`
-WHERE `string_col` = 'pipeline' AND CASE WHEN `decimal_col` IS NOT NULL THEN CAST(`decimal_col` AS STRING) = '50000000000000000005905545593' ELSE FALSE END AND (`date_col` IS NOT NULL AND `decimal_col` IS NOT NULL)) AS `t0`
+WHERE `string_col` = 'pipeline' AND CASE WHEN `decimal_col` IS NOT NULL THEN CAST(`decimal_col` AS STRING) = '50000000000000000005905545593' ELSE FALSE END AND `date_col` IS NOT NULL AND `decimal_col` IS NOT NULL) AS `t0`
INNER JOIN (SELECT `date_col`, `decimal_col`
FROM `default`.`source_table`
-WHERE CASE WHEN `decimal_col` IS NOT NULL THEN CAST(`decimal_col` AS STRING) = '50000000000000000005905545593' ELSE FALSE END AND (`date_col` IS NOT NULL AND `decimal_col` IS NOT NULL)
+WHERE CASE WHEN `decimal_col` IS NOT NULL THEN CAST(`decimal_col` AS STRING) = '50000000000000000005905545593' ELSE FALSE END AND `date_col` IS NOT NULL AND `decimal_col` IS NOT NULL
GROUP BY `date_col`, `decimal_col`) AS `t3` ON `t0`.`date_col` = `t3`.`date_col` AND `t0`.`decimal_col` = `t3`.`decimal_col`
STAGE DEPENDENCIES:
Stage-1 is a root stage
@@ -586,10 +586,10 @@ POSTHOOK: Input: default@target_table
OPTIMIZED SQL: SELECT `t0`.`date_col`, CAST('pipeline' AS STRING) AS `string_col`, `t0`.`decimal_col`, `t2`.`date_col` AS `date_col1`, 'pipeline' AS `string_col1`, `t2`.`decimal_col` AS `decimal_col1`
FROM (SELECT `date_col`, `decimal_col`
FROM `default`.`target_table`
-WHERE `string_col` = 'pipeline' AND CASE WHEN `decimal_col` IS NOT NULL THEN CAST(`decimal_col` AS STRING) = '50000000000000000005905545593' ELSE FALSE END AND (`date_col` IS NOT NULL AND `decimal_col` IS NOT NULL)) AS `t0`
+WHERE `string_col` = 'pipeline' AND CASE WHEN `decimal_col` IS NOT NULL THEN CAST(`decimal_col` AS STRING) = '50000000000000000005905545593' ELSE FALSE END AND `date_col` IS NOT NULL AND `decimal_col` IS NOT NULL) AS `t0`
INNER JOIN (SELECT `date_col`, `decimal_col`
FROM `default`.`source_table`
-WHERE CASE WHEN `decimal_col` IS NOT NULL THEN CAST(`decimal_col` AS STRING) = '50000000000000000005905545593' ELSE FALSE END AND (`date_col` IS NOT NULL AND `decimal_col` IS NOT NULL)) AS `t2` ON `t0`.`date_col` = `t2`.`date_col` AND `t0`.`decimal_col` = `t2`.`decimal_col`
+WHERE CASE WHEN `decimal_col` IS NOT NULL THEN CAST(`decimal_col` AS STRING) = '50000000000000000005905545593' ELSE FALSE END AND `date_col` IS NOT NULL AND `decimal_col` IS NOT NULL) AS `t2` ON `t0`.`date_col` = `t2`.`date_col` AND `t0`.`decimal_col` = `t2`.`decimal_col`
STAGE DEPENDENCIES:
Stage-1 is a root stage
Stage-0 depends on stages: Stage-1
@@ -812,10 +812,10 @@ POSTHOOK: Input: default@target_table
OPTIMIZED SQL: SELECT `t0`.`date_col`, CAST('pipeline' AS STRING) AS `string_col`, `t0`.`decimal_col`, `t3`.`date_col` AS `date_col1`, 'pipeline' AS `string_col1`, `t3`.`decimal_col` AS `decimal_col1`
FROM (SELECT `date_col`, `decimal_col`
FROM `default`.`target_table`
-WHERE `string_col` = 'pipeline' AND CASE WHEN `decimal_col` IS NOT NULL THEN CAST(`decimal_col` AS STRING) = '50000000000000000005905545593' ELSE FALSE END AND (`date_col` IS NOT NULL AND `decimal_col` IS NOT NULL)) AS `t0`
+WHERE `string_col` = 'pipeline' AND CASE WHEN `decimal_col` IS NOT NULL THEN CAST(`decimal_col` AS STRING) = '50000000000000000005905545593' ELSE FALSE END AND `date_col` IS NOT NULL AND `decimal_col` IS NOT NULL) AS `t0`
INNER JOIN (SELECT `date_col`, `decimal_col`
FROM `default`.`source_table`
-WHERE CASE WHEN `decimal_col` IS NOT NULL THEN CAST(`decimal_col` AS STRING) = '50000000000000000005905545593' ELSE FALSE END AND (`date_col` IS NOT NULL AND `decimal_col` IS NOT NULL)
+WHERE CASE WHEN `decimal_col` IS NOT NULL THEN CAST(`decimal_col` AS STRING) = '50000000000000000005905545593' ELSE FALSE END AND `date_col` IS NOT NULL AND `decimal_col` IS NOT NULL
GROUP BY `date_col`, `decimal_col`) AS `t3` ON `t0`.`date_col` = `t3`.`date_col` AND `t0`.`decimal_col` = `t3`.`decimal_col`
STAGE DEPENDENCIES:
Stage-1 is a root stage
@@ -1061,10 +1061,10 @@ POSTHOOK: Input: default@target_table
OPTIMIZED SQL: SELECT `t0`.`date_col`, CAST('pipeline' AS STRING) AS `string_col`, `t0`.`decimal_col`, `t2`.`date_col` AS `date_col1`, 'pipeline' AS `string_col1`, `t2`.`decimal_col` AS `decimal_col1`
FROM (SELECT `date_col`, `decimal_col`
FROM `default`.`target_table`
-WHERE `string_col` = 'pipeline' AND CASE WHEN `decimal_col` IS NOT NULL THEN CAST(`decimal_col` AS STRING) = '50000000000000000005905545593' ELSE FALSE END AND (`date_col` IS NOT NULL AND `decimal_col` IS NOT NULL)) AS `t0`
+WHERE `string_col` = 'pipeline' AND CASE WHEN `decimal_col` IS NOT NULL THEN CAST(`decimal_col` AS STRING) = '50000000000000000005905545593' ELSE FALSE END AND `date_col` IS NOT NULL AND `decimal_col` IS NOT NULL) AS `t0`
INNER JOIN (SELECT `date_col`, `decimal_col`
FROM `default`.`source_table`
-WHERE CASE WHEN `decimal_col` IS NOT NULL THEN CAST(`decimal_col` AS STRING) = '50000000000000000005905545593' ELSE FALSE END AND (`date_col` IS NOT NULL AND `decimal_col` IS NOT NULL)) AS `t2` ON `t0`.`date_col` = `t2`.`date_col` AND `t0`.`decimal_col` = `t2`.`decimal_col`
+WHERE CASE WHEN `decimal_col` IS NOT NULL THEN CAST(`decimal_col` AS STRING) = '50000000000000000005905545593' ELSE FALSE END AND `date_col` IS NOT NULL AND `decimal_col` IS NOT NULL) AS `t2` ON `t0`.`date_col` = `t2`.`date_col` AND `t0`.`decimal_col` = `t2`.`decimal_col`
STAGE DEPENDENCIES:
Stage-1 is a root stage
Stage-0 depends on stages: Stage-1
@@ -1288,10 +1288,10 @@ POSTHOOK: Input: default@target_table
OPTIMIZED SQL: SELECT `t0`.`date_col`, CAST('pipeline' AS STRING) AS `string_col`, `t0`.`decimal_col`, `t3`.`date_col` AS `date_col1`, 'pipeline' AS `string_col1`, `t3`.`decimal_col` AS `decimal_col1`
FROM (SELECT `date_col`, `decimal_col`
FROM `default`.`target_table`
-WHERE `string_col` = 'pipeline' AND CASE WHEN `decimal_col` IS NOT NULL THEN CAST(`decimal_col` AS STRING) = '50000000000000000005905545593' ELSE FALSE END AND (`date_col` IS NOT NULL AND `decimal_col` IS NOT NULL)) AS `t0`
+WHERE `string_col` = 'pipeline' AND CASE WHEN `decimal_col` IS NOT NULL THEN CAST(`decimal_col` AS STRING) = '50000000000000000005905545593' ELSE FALSE END AND `date_col` IS NOT NULL AND `decimal_col` IS NOT NULL) AS `t0`
INNER JOIN (SELECT `date_col`, `decimal_col`
FROM `default`.`source_table`
-WHERE CASE WHEN `decimal_col` IS NOT NULL THEN CAST(`decimal_col` AS STRING) = '50000000000000000005905545593' ELSE FALSE END AND (`date_col` IS NOT NULL AND `decimal_col` IS NOT NULL)
+WHERE CASE WHEN `decimal_col` IS NOT NULL THEN CAST(`decimal_col` AS STRING) = '50000000000000000005905545593' ELSE FALSE END AND `date_col` IS NOT NULL AND `decimal_col` IS NOT NULL
GROUP BY `date_col`, `decimal_col`) AS `t3` ON `t0`.`date_col` = `t3`.`date_col` AND `t0`.`decimal_col` = `t3`.`decimal_col`
STAGE DEPENDENCIES:
Stage-1 is a root stage
@@ -1538,10 +1538,10 @@ POSTHOOK: Input: default@target_table
OPTIMIZED SQL: SELECT `t0`.`date_col`, CAST('pipeline' AS STRING) AS `string_col`, `t0`.`decimal_col`, `t2`.`date_col` AS `date_col1`, 'pipeline' AS `string_col1`, `t2`.`decimal_col` AS `decimal_col1`
FROM (SELECT `date_col`, `decimal_col`
FROM `default`.`target_table`
-WHERE `string_col` = 'pipeline' AND CASE WHEN `decimal_col` IS NOT NULL THEN CAST(`decimal_col` AS STRING) = '50000000000000000005905545593' ELSE FALSE END AND (`date_col` IS NOT NULL AND `decimal_col` IS NOT NULL)) AS `t0`
+WHERE `string_col` = 'pipeline' AND CASE WHEN `decimal_col` IS NOT NULL THEN CAST(`decimal_col` AS STRING) = '50000000000000000005905545593' ELSE FALSE END AND `date_col` IS NOT NULL AND `decimal_col` IS NOT NULL) AS `t0`
INNER JOIN (SELECT `date_col`, `decimal_col`
FROM `default`.`source_table`
-WHERE CASE WHEN `decimal_col` IS NOT NULL THEN CAST(`decimal_col` AS STRING) = '50000000000000000005905545593' ELSE FALSE END AND (`date_col` IS NOT NULL AND `decimal_col` IS NOT NULL)) AS `t2` ON `t0`.`date_col` = `t2`.`date_col` AND `t0`.`decimal_col` = `t2`.`decimal_col`
+WHERE CASE WHEN `decimal_col` IS NOT NULL THEN CAST(`decimal_col` AS STRING) = '50000000000000000005905545593' ELSE FALSE END AND `date_col` IS NOT NULL AND `decimal_col` IS NOT NULL) AS `t2` ON `t0`.`date_col` = `t2`.`date_col` AND `t0`.`decimal_col` = `t2`.`decimal_col`
STAGE DEPENDENCIES:
Stage-1 is a root stage
Stage-0 depends on stages: Stage-1
@@ -1764,10 +1764,10 @@ POSTHOOK: Input: default@target_table
OPTIMIZED SQL: SELECT `t0`.`date_col`, CAST('pipeline' AS STRING) AS `string_col`, `t0`.`decimal_col`, `t3`.`date_col` AS `date_col1`, 'pipeline' AS `string_col1`, `t3`.`decimal_col` AS `decimal_col1`
FROM (SELECT `date_col`, `decimal_col`
FROM `default`.`target_table`
-WHERE `string_col` = 'pipeline' AND CASE WHEN `decimal_col` IS NOT NULL THEN CAST(`decimal_col` AS STRING) = '50000000000000000005905545593' ELSE FALSE END AND (`date_col` IS NOT NULL AND `decimal_col` IS NOT NULL)) AS `t0`
+WHERE `string_col` = 'pipeline' AND CASE WHEN `decimal_col` IS NOT NULL THEN CAST(`decimal_col` AS STRING) = '50000000000000000005905545593' ELSE FALSE END AND `date_col` IS NOT NULL AND `decimal_col` IS NOT NULL) AS `t0`
INNER JOIN (SELECT `date_col`, `decimal_col`
FROM `default`.`source_table`
-WHERE CASE WHEN `decimal_col` IS NOT NULL THEN CAST(`decimal_col` AS STRING) = '50000000000000000005905545593' ELSE FALSE END AND (`date_col` IS NOT NULL AND `decimal_col` IS NOT NULL)
+WHERE CASE WHEN `decimal_col` IS NOT NULL THEN CAST(`decimal_col` AS STRING) = '50000000000000000005905545593' ELSE FALSE END AND `date_col` IS NOT NULL AND `decimal_col` IS NOT NULL
GROUP BY `date_col`, `decimal_col`) AS `t3` ON `t0`.`date_col` = `t3`.`date_col` AND `t0`.`decimal_col` = `t3`.`decimal_col`
STAGE DEPENDENCIES:
Stage-1 is a root stage
diff --git a/ql/src/test/results/clientpositive/llap/bucketpruning1.q.out b/ql/src/test/results/clientpositive/llap/bucketpruning1.q.out
index 405053ba5ff..538017c538e 100644
--- a/ql/src/test/results/clientpositive/llap/bucketpruning1.q.out
+++ b/ql/src/test/results/clientpositive/llap/bucketpruning1.q.out
@@ -523,7 +523,7 @@ POSTHOOK: Input: default@srcbucket_pruned
#### A masked pattern was here ####
OPTIMIZED SQL: SELECT CAST(1 AS INT) AS `$f0`, CAST('One' AS STRING) AS `$f1`, CAST('2008-04-08' AS STRING) AS `$f2`
FROM `default`.`srcbucket_pruned`
-WHERE `key` = 1 AND (`ds` = '2008-04-08' AND `value` = 'One')
+WHERE `key` = 1 AND `ds` = '2008-04-08' AND `value` = 'One'
STAGE DEPENDENCIES:
Stage-1 is a root stage
Stage-0 depends on stages: Stage-1
@@ -594,7 +594,7 @@ POSTHOOK: Input: default@srcbucket_pruned
#### A masked pattern was here ####
OPTIMIZED SQL: SELECT CAST(1 AS INT) AS `$f0`, CAST('One' AS STRING) AS `$f1`, CAST('2008-04-08' AS STRING) AS `$f2`
FROM `default`.`srcbucket_pruned`
-WHERE `value` = 'One' AND (`key` = 1 AND `ds` = '2008-04-08')
+WHERE `value` = 'One' AND `key` = 1 AND `ds` = '2008-04-08'
STAGE DEPENDENCIES:
Stage-1 is a root stage
Stage-0 depends on stages: Stage-1
@@ -736,7 +736,7 @@ POSTHOOK: Input: default@srcbucket_pruned
#### A masked pattern was here ####
OPTIMIZED SQL: SELECT `key`, `value`, CAST('2008-04-08' AS STRING) AS `ds`
FROM `default`.`srcbucket_pruned`
-WHERE `key` IN (2, 3) AND `ds` = '2008-04-08'
+WHERE `ds` = '2008-04-08' AND `key` IN (2, 3)
STAGE DEPENDENCIES:
Stage-1 is a root stage
Stage-0 depends on stages: Stage-1
@@ -750,13 +750,13 @@ STAGE PLANS:
Map Operator Tree:
TableScan
alias: srcbucket_pruned
- filterExpr: ((key) IN (2, 3) and (ds = '2008-04-08')) (type: boolean)
+ filterExpr: ((ds = '2008-04-08') and (key) IN (2, 3)) (type: boolean)
buckets included: [4,6,] of 16
Statistics: Num rows: 1 Data size: 268 Basic stats: COMPLETE Column stats: COMPLETE
GatherStats: false
Filter Operator
isSamplingPred: false
- predicate: ((key) IN (2, 3) and (ds = '2008-04-08')) (type: boolean)
+ predicate: ((ds = '2008-04-08') and (key) IN (2, 3)) (type: boolean)
Statistics: Num rows: 1 Data size: 268 Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
expressions: key (type: int), value (type: string), '2008-04-08' (type: string)
@@ -807,7 +807,7 @@ POSTHOOK: Input: default@srcbucket_pruned
#### A masked pattern was here ####
OPTIMIZED SQL: SELECT `key`, CAST('One' AS STRING) AS `value`, CAST('2008-04-08' AS STRING) AS `ds`
FROM `default`.`srcbucket_pruned`
-WHERE `key` IN (2, 3) AND (`ds` = '2008-04-08' AND `value` = 'One')
+WHERE `ds` = '2008-04-08' AND `value` = 'One' AND `key` IN (2, 3)
STAGE DEPENDENCIES:
Stage-1 is a root stage
Stage-0 depends on stages: Stage-1
@@ -821,13 +821,13 @@ STAGE PLANS:
Map Operator Tree:
TableScan
alias: srcbucket_pruned
- filterExpr: ((key) IN (2, 3) and (ds = '2008-04-08') and (value = 'One')) (type: boolean)
+ filterExpr: ((ds = '2008-04-08') and (value = 'One') and (key) IN (2, 3)) (type: boolean)
buckets included: [4,6,] of 16
Statistics: Num rows: 1 Data size: 268 Basic stats: COMPLETE Column stats: COMPLETE
GatherStats: false
Filter Operator
isSamplingPred: false
- predicate: ((key) IN (2, 3) and (ds = '2008-04-08') and (value = 'One')) (type: boolean)
+ predicate: ((ds = '2008-04-08') and (value = 'One') and (key) IN (2, 3)) (type: boolean)
Statistics: Num rows: 1 Data size: 268 Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
expressions: key (type: int), 'One' (type: string), '2008-04-08' (type: string)
@@ -878,7 +878,7 @@ POSTHOOK: Input: default@srcbucket_pruned
#### A masked pattern was here ####
OPTIMIZED SQL: SELECT `key`, CAST('One' AS STRING) AS `value`, CAST('2008-04-08' AS STRING) AS `ds`
FROM `default`.`srcbucket_pruned`
-WHERE `value` = 'One' AND (`key` IN (2, 3) AND `ds` = '2008-04-08')
+WHERE `value` = 'One' AND `ds` = '2008-04-08' AND `key` IN (2, 3)
STAGE DEPENDENCIES:
Stage-1 is a root stage
Stage-0 depends on stages: Stage-1
@@ -892,13 +892,13 @@ STAGE PLANS:
Map Operator Tree:
TableScan
alias: srcbucket_pruned
- filterExpr: ((value = 'One') and (key) IN (2, 3) and (ds = '2008-04-08')) (type: boolean)
+ filterExpr: ((value = 'One') and (ds = '2008-04-08') and (key) IN (2, 3)) (type: boolean)
buckets included: [4,6,] of 16
Statistics: Num rows: 1 Data size: 268 Basic stats: COMPLETE Column stats: COMPLETE
GatherStats: false
Filter Operator
isSamplingPred: false
- predicate: ((value = 'One') and (key) IN (2, 3) and (ds = '2008-04-08')) (type: boolean)
+ predicate: ((value = 'One') and (ds = '2008-04-08') and (key) IN (2, 3)) (type: boolean)
Statistics: Num rows: 1 Data size: 268 Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
expressions: key (type: int), 'One' (type: string), '2008-04-08' (type: string)
@@ -949,7 +949,7 @@ POSTHOOK: Input: default@srcbucket_pruned
#### A masked pattern was here ####
OPTIMIZED SQL: SELECT `key`, `value`, CAST('2008-04-08' AS STRING) AS `ds`
FROM `default`.`srcbucket_pruned`
-WHERE `key` IN (1, 2) AND `ds` = '2008-04-08'
+WHERE `ds` = '2008-04-08' AND `key` IN (1, 2)
STAGE DEPENDENCIES:
Stage-1 is a root stage
Stage-0 depends on stages: Stage-1
@@ -963,13 +963,13 @@ STAGE PLANS:
Map Operator Tree:
TableScan
alias: srcbucket_pruned
- filterExpr: ((key) IN (1, 2) and (ds = '2008-04-08')) (type: boolean)
+ filterExpr: ((ds = '2008-04-08') and (key) IN (1, 2)) (type: boolean)
buckets included: [4,13,] of 16
Statistics: Num rows: 1 Data size: 268 Basic stats: COMPLETE Column stats: COMPLETE
GatherStats: false
Filter Operator
isSamplingPred: false
- predicate: ((key) IN (1, 2) and (ds = '2008-04-08')) (type: boolean)
+ predicate: ((ds = '2008-04-08') and (key) IN (1, 2)) (type: boolean)
Statistics: Num rows: 1 Data size: 268 Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
expressions: key (type: int), value (type: string), '2008-04-08' (type: string)
@@ -1020,7 +1020,7 @@ POSTHOOK: Input: default@srcbucket_pruned
#### A masked pattern was here ####
OPTIMIZED SQL: SELECT `key`, CAST('One' AS STRING) AS `value`, CAST('2008-04-08' AS STRING) AS `ds`
FROM `default`.`srcbucket_pruned`
-WHERE `key` IN (1, 2) AND (`value` = 'One' AND `ds` = '2008-04-08')
+WHERE `value` = 'One' AND `ds` = '2008-04-08' AND `key` IN (1, 2)
STAGE DEPENDENCIES:
Stage-1 is a root stage
Stage-0 depends on stages: Stage-1
@@ -1034,13 +1034,13 @@ STAGE PLANS:
Map Operator Tree:
TableScan
alias: srcbucket_pruned
- filterExpr: ((key) IN (1, 2) and (value = 'One') and (ds = '2008-04-08')) (type: boolean)
+ filterExpr: ((value = 'One') and (ds = '2008-04-08') and (key) IN (1, 2)) (type: boolean)
buckets included: [4,13,] of 16
Statistics: Num rows: 1 Data size: 268 Basic stats: COMPLETE Column stats: COMPLETE
GatherStats: false
Filter Operator
isSamplingPred: false
- predicate: ((key) IN (1, 2) and (value = 'One') and (ds = '2008-04-08')) (type: boolean)
+ predicate: ((value = 'One') and (ds = '2008-04-08') and (key) IN (1, 2)) (type: boolean)
Statistics: Num rows: 1 Data size: 268 Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
expressions: key (type: int), 'One' (type: string), '2008-04-08' (type: string)
@@ -1304,7 +1304,7 @@ POSTHOOK: Input: default@srcbucket_pruned
#### A masked pattern was here ####
OPTIMIZED SQL: SELECT `key`, CAST('One' AS STRING) AS `value`, CAST('2008-04-08' AS STRING) AS `ds`
FROM `default`.`srcbucket_pruned`
-WHERE `key` IN (1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17) AND (`ds` = '2008-04-08' AND `value` = 'One')
+WHERE `key` IN (1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17) AND `ds` = '2008-04-08' AND `value` = 'One'
STAGE DEPENDENCIES:
Stage-1 is a root stage
Stage-0 depends on stages: Stage-1
@@ -1375,7 +1375,7 @@ POSTHOOK: Input: default@srcbucket_pruned
#### A masked pattern was here ####
OPTIMIZED SQL: SELECT `key`, CAST('One' AS STRING) AS `value`, CAST('2008-04-08' AS STRING) AS `ds`
FROM `default`.`srcbucket_pruned`
-WHERE `value` = 'One' AND (`key` IN (1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17) AND `ds` = '2008-04-08')
+WHERE `value` = 'One' AND `key` IN (1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17) AND `ds` = '2008-04-08'
STAGE DEPENDENCIES:
Stage-1 is a root stage
Stage-0 depends on stages: Stage-1
@@ -1516,7 +1516,7 @@ POSTHOOK: Input: default@srcbucket_pruned
#### A masked pattern was here ####
OPTIMIZED SQL: SELECT CAST(1 AS INT) AS `key`, `value`, CAST('2008-04-08' AS STRING) AS `ds`
FROM `default`.`srcbucket_pruned`
-WHERE `key` = 1 AND (`ds` = '2008-04-08' AND `value` IN ('One', 'Two'))
+WHERE `key` = 1 AND `ds` = '2008-04-08' AND `value` IN ('One', 'Two')
STAGE DEPENDENCIES:
Stage-1 is a root stage
Stage-0 depends on stages: Stage-1
@@ -1587,7 +1587,7 @@ POSTHOOK: Input: default@srcbucket_pruned
#### A masked pattern was here ####
OPTIMIZED SQL: SELECT `key`, `value`, `ds`
FROM `default`.`srcbucket_pruned`
-WHERE `key` = 1 OR (`value` = 'One' OR `key` = 2 AND `value` = 'Two')
+WHERE `key` = 1 OR `value` = 'One' OR `key` = 2 AND `value` = 'Two'
STAGE DEPENDENCIES:
Stage-1 is a root stage
Stage-0 depends on stages: Stage-1
@@ -1656,7 +1656,7 @@ POSTHOOK: type: QUERY
POSTHOOK: Input: default@srcbucket_pruned
#### A masked pattern was here ####
OPTIMIZED SQL: SELECT *
-FROM (VALUES (NULL, NULL, NULL)) AS `t` (`key`, `value`, `ds`)
+FROM (SELECT NULL AS `key`, NULL AS `value`, NULL AS `ds`) AS `t`
WHERE 1 = 0
STAGE DEPENDENCIES:
Stage-0 is a root stage
diff --git a/ql/src/test/results/clientpositive/llap/bucketsortoptimize_insert_7.q.out b/ql/src/test/results/clientpositive/llap/bucketsortoptimize_insert_7.q.out
index ad89c4e60c7..431dd51ee20 100644
--- a/ql/src/test/results/clientpositive/llap/bucketsortoptimize_insert_7.q.out
+++ b/ql/src/test/results/clientpositive/llap/bucketsortoptimize_insert_7.q.out
@@ -95,10 +95,10 @@ STAGE PLANS:
Map Operator Tree:
TableScan
alias: b
- filterExpr: (key) IN (0, 5) (type: boolean)
+ filterExpr: ((key) IN (0, 5) and key is not null) (type: boolean)
Statistics: Num rows: 84 Data size: 7896 Basic stats: COMPLETE Column stats: COMPLETE
Filter Operator
- predicate: (key) IN (0, 5) (type: boolean)
+ predicate: ((key) IN (0, 5) and key is not null) (type: boolean)
Statistics: Num rows: 3 Data size: 282 Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
expressions: key (type: int), value (type: string)
@@ -108,10 +108,10 @@ STAGE PLANS:
Map Operator Tree:
TableScan
alias: a
- filterExpr: (key) IN (0, 5) (type: boolean)
+ filterExpr: ((key) IN (0, 5) and key is not null) (type: boolean)
Statistics: Num rows: 10 Data size: 930 Basic stats: COMPLETE Column stats: COMPLETE
Filter Operator
- predicate: (key) IN (0, 5) (type: boolean)
+ predicate: ((key) IN (0, 5) and key is not null) (type: boolean)
Statistics: Num rows: 3 Data size: 279 Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
expressions: key (type: int), value (type: string)
@@ -323,10 +323,10 @@ STAGE PLANS:
Map Operator Tree:
TableScan
alias: test_table2_n19
- filterExpr: (key) IN (0, 5) (type: boolean)
+ filterExpr: ((key) IN (0, 5) and key is not null) (type: boolean)
Statistics: Num rows: 84 Data size: 7896 Basic stats: COMPLETE Column stats: COMPLETE
Filter Operator
- predicate: (key) IN (0, 5) (type: boolean)
+ predicate: ((key) IN (0, 5) and key is not null) (type: boolean)
Statistics: Num rows: 3 Data size: 282 Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
expressions: key (type: int), value (type: string)
@@ -336,10 +336,10 @@ STAGE PLANS:
Map Operator Tree:
TableScan
alias: test_table1_n20
- filterExpr: (key) IN (0, 5) (type: boolean)
+ filterExpr: ((key) IN (0, 5) and key is not null) (type: boolean)
Statistics: Num rows: 10 Data size: 930 Basic stats: COMPLETE Column stats: COMPLETE
Filter Operator
- predicate: (key) IN (0, 5) (type: boolean)
+ predicate: ((key) IN (0, 5) and key is not null) (type: boolean)
Statistics: Num rows: 3 Data size: 279 Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
expressions: key (type: int), value (type: string)
@@ -557,10 +557,10 @@ STAGE PLANS:
Map Operator Tree:
TableScan
alias: test_table2_n19
- filterExpr: ((key < 8) and (key) IN (0, 5)) (type: boolean)
+ filterExpr: (key) IN (0, 5) (type: boolean)
Statistics: Num rows: 84 Data size: 7896 Basic stats: COMPLETE Column stats: COMPLETE
Filter Operator
- predicate: ((key < 8) and (key) IN (0, 5)) (type: boolean)
+ predicate: (key) IN (0, 5) (type: boolean)
Statistics: Num rows: 3 Data size: 282 Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
expressions: key (type: int), value (type: string)
@@ -570,10 +570,10 @@ STAGE PLANS:
Map Operator Tree:
TableScan
alias: test_table1_n20
- filterExpr: ((key < 8) and (key) IN (0, 5)) (type: boolean)
+ filterExpr: (key) IN (0, 5) (type: boolean)
Statistics: Num rows: 10 Data size: 930 Basic stats: COMPLETE Column stats: COMPLETE
Filter Operator
- predicate: ((key < 8) and (key) IN (0, 5)) (type: boolean)
+ predicate: (key) IN (0, 5) (type: boolean)
Statistics: Num rows: 3 Data size: 279 Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
expressions: key (type: int), value (type: string)
diff --git a/ql/src/test/results/clientpositive/llap/cardinality_preserving_join_opt.q.out b/ql/src/test/results/clientpositive/llap/cardinality_preserving_join_opt.q.out
index 69dea69e504..d04a42c5f65 100644
--- a/ql/src/test/results/clientpositive/llap/cardinality_preserving_join_opt.q.out
+++ b/ql/src/test/results/clientpositive/llap/cardinality_preserving_join_opt.q.out
@@ -88,10 +88,10 @@ POSTHOOK: Input: default@store_sales
CBO PLAN:
HiveProject(c_first_name=[$3], _o__c1=[$5], _o__c2=[*($1, $6)], c_customer_sk=[$4], ss_customer_sk=[$0])
HiveJoin(condition=[=($2, $4)], joinType=[inner], algorithm=[none], cost=[not available])
- HiveProject(ss_customer_sk=[$1], *=[*(+($0, $0), $2)], CAST=[CAST($1):BIGINT])
+ HiveProject(ss_customer_sk=[$1], EXPR$0=[*(+($0, $0), $2)], EXPR$1=[CAST($1):BIGINT])
HiveFilter(condition=[IS NOT NULL(CAST($1):BIGINT)])
HiveTableScan(table=[[default, store_sales]], table:alias=[ss])
- HiveProject(c_first_name=[$0], c_customer_sk=[$2], ||=[||(||(||(||($0, _UTF-16LE' ':VARCHAR(2147483647) CHARACTER SET "UTF-16LE"), $1), _UTF-16LE' ':VARCHAR(2147483647) CHARACTER SET "UTF-16LE"), $1)], -=[-(1, $3)])
+ HiveProject(c_first_name=[$0], c_customer_sk=[$2], _o__c1=[||(||(||(||($0, _UTF-16LE' ':VARCHAR(2147483647) CHARACTER SET "UTF-16LE"), $1), _UTF-16LE' ':VARCHAR(2147483647) CHARACTER SET "UTF-16LE"), $1)], EXPR$1=[-(1, $3)])
HiveFilter(condition=[IS NOT NULL($2)])
HiveTableScan(table=[[default, customer]], table:alias=[c])
@@ -158,11 +158,11 @@ CBO PLAN:
HiveProject(c_first_name=[$2], _o__c1=[$4], _o__c2=[*(*(+($6, $6), $8), $5)], c_customer_sk=[$3], ss_customer_sk=[$0])
HiveJoin(condition=[=($0, $7)], joinType=[inner], algorithm=[none], cost=[not available])
HiveJoin(condition=[=($1, $3)], joinType=[inner], algorithm=[none], cost=[not available])
- HiveProject(ss_customer_sk=[$0], CAST=[CAST($0):BIGINT])
+ HiveProject(ss_customer_sk=[$0], EXPR$1=[CAST($0):BIGINT])
HiveFilter(condition=[IS NOT NULL(CAST($0):BIGINT)])
HiveProject(ss_customer_sk=[$1])
HiveTableScan(table=[[default, store_sales]], table:alias=[ss])
- HiveProject(c_first_name=[$0], c_customer_sk=[$2], ||=[||(||(||(||($0, _UTF-16LE' ':VARCHAR(2147483647) CHARACTER SET "UTF-16LE"), $1), _UTF-16LE' ':VARCHAR(2147483647) CHARACTER SET "UTF-16LE"), $1)], -=[-(1, $3)])
+ HiveProject(c_first_name=[$0], c_customer_sk=[$2], _o__c1=[||(||(||(||($0, _UTF-16LE' ':VARCHAR(2147483647) CHARACTER SET "UTF-16LE"), $1), _UTF-16LE' ':VARCHAR(2147483647) CHARACTER SET "UTF-16LE"), $1)], EXPR$1=[-(1, $3)])
HiveFilter(condition=[IS NOT NULL($2)])
HiveProject(c_first_name=[$0], c_last_name=[$1], c_customer_sk=[$2], c_discount=[$3])
HiveTableScan(table=[[default, customer]], table:alias=[c])
@@ -240,7 +240,7 @@ CBO PLAN:
HiveProject(c_first_name=[$4], _o__c1=[||(||(||(||($4, _UTF-16LE' ':VARCHAR(2147483647) CHARACTER SET "UTF-16LE"), $5), _UTF-16LE' ':VARCHAR(2147483647) CHARACTER SET "UTF-16LE"), $5)], _o__c2=[*($1, -(1, $7))], c_customer_sk=[$3], ss_customer_sk=[$0])
HiveJoin(condition=[=($3, $6)], joinType=[inner], algorithm=[none], cost=[not available])
HiveJoin(condition=[=($2, $3)], joinType=[inner], algorithm=[none], cost=[not available])
- HiveProject(ss_customer_sk=[$1], *=[*(+($0, $0), $2)], CAST=[CAST($1):BIGINT])
+ HiveProject(ss_customer_sk=[$1], EXPR$0=[*(+($0, $0), $2)], EXPR$1=[CAST($1):BIGINT])
HiveFilter(condition=[IS NOT NULL(CAST($1):BIGINT)])
HiveProject(ss_quantity=[$0], ss_customer_sk=[$1], ss_list_price=[$2])
HiveTableScan(table=[[default, store_sales]], table:alias=[ss])
@@ -313,7 +313,7 @@ HiveProject(c_first_name=[$6], _o__c1=[||(||(||(||($6, _UTF-16LE' ':VARCHAR(2147
HiveJoin(condition=[=($2, $8)], joinType=[inner], algorithm=[none], cost=[not available])
HiveJoin(condition=[=($0, $4)], joinType=[inner], algorithm=[none], cost=[not available])
HiveJoin(condition=[=($1, $2)], joinType=[inner], algorithm=[none], cost=[not available])
- HiveProject(ss_customer_sk=[$0], CAST=[CAST($0):BIGINT])
+ HiveProject(ss_customer_sk=[$0], EXPR$1=[CAST($0):BIGINT])
HiveFilter(condition=[IS NOT NULL(CAST($0):BIGINT)])
HiveProject(ss_customer_sk=[$1])
HiveTableScan(table=[[default, store_sales]], table:alias=[ss])
@@ -375,7 +375,7 @@ CBO PLAN:
HiveProject(c_first_name=[$3], ss_customer_sk=[$0], c_customer_sk=[$2])
HiveJoin(condition=[=($2, $4)], joinType=[inner], algorithm=[none], cost=[not available])
HiveJoin(condition=[=($2, $1)], joinType=[inner], algorithm=[none], cost=[not available])
- HiveProject(ss_customer_sk=[$0], CAST=[CAST($0):BIGINT])
+ HiveProject(ss_customer_sk=[$0], EXPR$0=[CAST($0):BIGINT])
HiveFilter(condition=[IS NOT NULL(CAST($0):BIGINT)])
HiveProject(ss_customer_sk=[$1])
HiveTableScan(table=[[default, store_sales]], table:alias=[ss])
diff --git a/ql/src/test/results/clientpositive/llap/cbo_aggregate_reduce_functions_rule.q.out b/ql/src/test/results/clientpositive/llap/cbo_aggregate_reduce_functions_rule.q.out
index 45f839ad423..981aee931ac 100644
--- a/ql/src/test/results/clientpositive/llap/cbo_aggregate_reduce_functions_rule.q.out
+++ b/ql/src/test/results/clientpositive/llap/cbo_aggregate_reduce_functions_rule.q.out
@@ -158,7 +158,7 @@ POSTHOOK: type: QUERY
POSTHOOK: Input: default@test
#### A masked pattern was here ####
CBO PLAN:
-HiveProject(_o__c0=[COALESCE($0, 0E0:DOUBLE)], _o__c1=[COALESCE($1, 0E0:DOUBLE)], _o__c2=[COALESCE($2, 0E0:DOUBLE)], _o__c3=[COALESCE($3, 0E0:DOUBLE)], _o__c4=[COALESCE($4, 0E0:DOUBLE)], _o__c5=[COALESCE($5, 0E0:DOUBLE)], _o__c6=[/($6, $7)], _o__c7=[/($1, $8)], _o__c8=[/($9, $10)], _o__c9=[/($3, $11)], _o__c10=[/($12, $13)], _o__c11=[/($5, $14)], _o__c12=[POWER(/(-($15, /(*($6, $6), $7)), $7), 0.5:DECIMAL(2, 1))], _o__c13=[POWER(/(-($16, /(*($1, $1), $8)), $8), 0.5:DECIMAL(2, 1))], _o__c [...]
+HiveProject(_o__c0=[CAST(COALESCE($0, 0E0:DOUBLE)):DOUBLE], _o__c1=[CAST(COALESCE($1, 0E0:DOUBLE)):DOUBLE], _o__c2=[CAST(COALESCE($2, 0E0:DOUBLE)):DOUBLE], _o__c3=[CAST(COALESCE($3, 0E0:DOUBLE)):DOUBLE], _o__c4=[CAST(COALESCE($4, 0E0:DOUBLE)):DOUBLE], _o__c5=[CAST(COALESCE($5, 0E0:DOUBLE)):DOUBLE], _o__c6=[/($6, $7)], _o__c7=[/($1, $8)], _o__c8=[/($9, $10)], _o__c9=[/($3, $11)], _o__c10=[/($12, $13)], _o__c11=[/($5, $14)], _o__c12=[POWER(/(-($15, /(*($6, $6), $7)), $7), 0.5:DECIMAL(2, 1) [...]
HiveAggregate(group=[{}], agg#0=[sum($0)], agg#1=[sum($1)], agg#2=[sum($2)], agg#3=[sum($3)], agg#4=[sum($4)], agg#5=[sum($5)], agg#6=[sum($6)], agg#7=[count($6)], agg#8=[count($1)], agg#9=[sum($7)], agg#10=[count($7)], agg#11=[count($3)], agg#12=[sum($8)], agg#13=[count($8)], agg#14=[count($5)], agg#15=[sum($9)], agg#16=[sum($10)], agg#17=[sum($11)], agg#18=[sum($12)], agg#19=[sum($13)], agg#20=[sum($14)], agg#21=[count($0)], agg#22=[count($2)], agg#23=[count($4)])
HiveProject($f0=[$0], $f1=[CAST($0):DOUBLE], $f2=[$1], $f3=[CAST($1):DOUBLE], $f4=[$2], $f5=[CAST($2):DOUBLE], $f00=[CAST($0):DOUBLE], $f20=[CAST($1):DOUBLE], $f40=[CAST($2):DOUBLE], $f9=[*(CAST($0):DOUBLE, CAST($0):DOUBLE)], $f10=[*(CAST($0):DOUBLE, CAST($0):DOUBLE)], $f11=[*(CAST($1):DOUBLE, CAST($1):DOUBLE)], $f12=[*(CAST($1):DOUBLE, CAST($1):DOUBLE)], $f13=[*(CAST($2):DOUBLE, CAST($2):DOUBLE)], $f14=[*(CAST($2):DOUBLE, CAST($2):DOUBLE)])
HiveTableScan(table=[[default, test]], table:alias=[test])
diff --git a/ql/src/test/results/clientpositive/llap/cbo_join_transitive_pred_loop_1.q.out b/ql/src/test/results/clientpositive/llap/cbo_join_transitive_pred_loop_1.q.out
index dae047df095..bc7ac1199b6 100644
--- a/ql/src/test/results/clientpositive/llap/cbo_join_transitive_pred_loop_1.q.out
+++ b/ql/src/test/results/clientpositive/llap/cbo_join_transitive_pred_loop_1.q.out
@@ -67,6 +67,6 @@ HiveProject(month=[$0], con_usd=[$2])
HiveFilter(condition=[=($0, 202110)])
HiveTableScan(table=[[default, test2]], table:alias=[test2])
HiveProject(mth=[$0], con_usd=[$1])
- HiveFilter(condition=[IS NOT NULL($0)])
+ HiveFilter(condition=[AND(IS NOT NULL($0), IN($0, 202110, 202503))])
HiveTableScan(table=[[default, test3]], table:alias=[d])
diff --git a/ql/src/test/results/clientpositive/llap/cbo_join_transitive_pred_loop_3.q.out b/ql/src/test/results/clientpositive/llap/cbo_join_transitive_pred_loop_3.q.out
index 365cb86b2d9..fb331029b4e 100644
--- a/ql/src/test/results/clientpositive/llap/cbo_join_transitive_pred_loop_3.q.out
+++ b/ql/src/test/results/clientpositive/llap/cbo_join_transitive_pred_loop_3.q.out
@@ -54,7 +54,7 @@ CBO PLAN:
HiveProject(bd_id=[$0])
HiveJoin(condition=[=($0, $1)], joinType=[inner], algorithm=[none], cost=[not available])
HiveProject(bd_id=[$0])
- HiveFilter(condition=[AND(OR(=($0, 8), <>($0, 8)), OR(AND(=($0, 8), IN($1, _UTF-16LE'A':VARCHAR(2147483647) CHARACTER SET "UTF-16LE", _UTF-16LE'C':VARCHAR(2147483647) CHARACTER SET "UTF-16LE")), AND(=($0, 9), IN($1, _UTF-16LE'A':VARCHAR(2147483647) CHARACTER SET "UTF-16LE", _UTF-16LE'B':VARCHAR(2147483647) CHARACTER SET "UTF-16LE"))), IS NOT NULL($0))])
+ HiveFilter(condition=[AND(OR(AND(=($0, 8), IN($1, _UTF-16LE'A':VARCHAR(2147483647) CHARACTER SET "UTF-16LE", _UTF-16LE'C':VARCHAR(2147483647) CHARACTER SET "UTF-16LE")), AND(=($0, 9), IN($1, _UTF-16LE'A':VARCHAR(2147483647) CHARACTER SET "UTF-16LE", _UTF-16LE'B':VARCHAR(2147483647) CHARACTER SET "UTF-16LE"))), IS NOT NULL($0))])
HiveTableScan(table=[[default, tablea]], table:alias=[t])
HiveProject($f0=[$0])
HiveUnion(all=[true])
diff --git a/ql/src/test/results/clientpositive/llap/cbo_join_transitive_pred_loop_4.q.out b/ql/src/test/results/clientpositive/llap/cbo_join_transitive_pred_loop_4.q.out
index da16c285203..0319ac7ab2e 100644
--- a/ql/src/test/results/clientpositive/llap/cbo_join_transitive_pred_loop_4.q.out
+++ b/ql/src/test/results/clientpositive/llap/cbo_join_transitive_pred_loop_4.q.out
@@ -54,7 +54,7 @@ CBO PLAN:
HiveProject(bd_id=[$0])
HiveJoin(condition=[=($0, $1)], joinType=[inner], algorithm=[none], cost=[not available])
HiveProject(bd_id=[$0])
- HiveFilter(condition=[AND(OR(=($0, 8), <>($0, 8)), OR(AND(=($0, 8), IN($1, _UTF-16LE'A':VARCHAR(2147483647) CHARACTER SET "UTF-16LE", _UTF-16LE'C':VARCHAR(2147483647) CHARACTER SET "UTF-16LE")), AND(=($0, 9), IN($1, _UTF-16LE'A':VARCHAR(2147483647) CHARACTER SET "UTF-16LE", _UTF-16LE'B':VARCHAR(2147483647) CHARACTER SET "UTF-16LE"))), IS NOT NULL($0))])
+ HiveFilter(condition=[AND(OR(AND(=($0, 8), IN($1, _UTF-16LE'A':VARCHAR(2147483647) CHARACTER SET "UTF-16LE", _UTF-16LE'C':VARCHAR(2147483647) CHARACTER SET "UTF-16LE")), AND(=($0, 9), IN($1, _UTF-16LE'A':VARCHAR(2147483647) CHARACTER SET "UTF-16LE", _UTF-16LE'B':VARCHAR(2147483647) CHARACTER SET "UTF-16LE"))), IS NOT NULL($0))])
HiveTableScan(table=[[default, tablea]], table:alias=[t])
HiveProject($f0=[$0])
HiveFilter(condition=[=($1, 2)])
diff --git a/ql/src/test/results/clientpositive/llap/cbo_ppd_non_deterministic.q.out b/ql/src/test/results/clientpositive/llap/cbo_ppd_non_deterministic.q.out
index aa229ef9ff4..e1fc12222e3 100644
--- a/ql/src/test/results/clientpositive/llap/cbo_ppd_non_deterministic.q.out
+++ b/ql/src/test/results/clientpositive/llap/cbo_ppd_non_deterministic.q.out
@@ -125,7 +125,7 @@ STAGE PLANS:
expressions: rand() (type: double)
outputColumnNames: _col0
Filter Operator
- predicate: ((_col0 <= 0.5D) and (_col0 > 0.25D)) (type: boolean)
+ predicate: ((_col0 > 0.25D) and (_col0 <= 0.5D)) (type: boolean)
Limit
Number of rows: 20
Select Operator
@@ -164,7 +164,7 @@ STAGE PLANS:
expressions: rand() (type: double)
outputColumnNames: _col0
Filter Operator
- predicate: ((_col0 <= 0.5D) and (_col0 > 0.25D)) (type: boolean)
+ predicate: ((_col0 > 0.25D) and (_col0 <= 0.5D)) (type: boolean)
Limit
Number of rows: 20
Select Operator
diff --git a/ql/src/test/results/clientpositive/llap/cbo_proj_join_transpose.q.out b/ql/src/test/results/clientpositive/llap/cbo_proj_join_transpose.q.out
index 98cf90d37a6..7e32c037eda 100644
--- a/ql/src/test/results/clientpositive/llap/cbo_proj_join_transpose.q.out
+++ b/ql/src/test/results/clientpositive/llap/cbo_proj_join_transpose.q.out
@@ -47,7 +47,7 @@ HiveProject(act_nbr=[$0], row_num=[CAST(1):INTEGER])
HiveFilter(condition=[=($1, 1)])
HiveProject(act_id=[$2], row_number_window_0=[row_number() OVER (PARTITION BY $1 ORDER BY $0 DESC NULLS FIRST RANGE BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING)])
HiveJoin(condition=[=($1, $2)], joinType=[inner], algorithm=[none], cost=[not available])
- HiveProject(interest_rate=[$1], trim=[trim($0)])
+ HiveProject(interest_rate=[$1], EXPR$0=[trim($0)])
HiveFilter(condition=[IS NOT NULL(trim($0))])
HiveTableScan(table=[[default, table1]], table:alias=[t1])
HiveProject(act_id=[$0])
diff --git a/ql/src/test/results/clientpositive/llap/cbo_rp_outer_join_ppr.q.out b/ql/src/test/results/clientpositive/llap/cbo_rp_outer_join_ppr.q.out
index 3ae74e21b3d..87e068b607d 100644
--- a/ql/src/test/results/clientpositive/llap/cbo_rp_outer_join_ppr.q.out
+++ b/ql/src/test/results/clientpositive/llap/cbo_rp_outer_join_ppr.q.out
@@ -42,12 +42,12 @@ STAGE PLANS:
Map Operator Tree:
TableScan
alias: a
- filterExpr: ((UDFToDouble(key) < 20.0D) and (UDFToDouble(key) > 15.0D)) (type: boolean)
+ filterExpr: ((UDFToDouble(key) > 15.0D) and (UDFToDouble(key) < 20.0D)) (type: boolean)
Statistics: Num rows: 500 Data size: 89000 Basic stats: COMPLETE Column stats: COMPLETE
GatherStats: false
Filter Operator
isSamplingPred: false
- predicate: ((UDFToDouble(key) < 20.0D) and (UDFToDouble(key) > 15.0D)) (type: boolean)
+ predicate: ((UDFToDouble(key) > 15.0D) and (UDFToDouble(key) < 20.0D)) (type: boolean)
Statistics: Num rows: 55 Data size: 9790 Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
expressions: key (type: string), value (type: string)
@@ -341,12 +341,12 @@ STAGE PLANS:
Map Operator Tree:
TableScan
alias: a
- filterExpr: ((UDFToDouble(key) < 20.0D) and (UDFToDouble(key) > 15.0D)) (type: boolean)
+ filterExpr: ((UDFToDouble(key) > 15.0D) and (UDFToDouble(key) < 20.0D)) (type: boolean)
Statistics: Num rows: 500 Data size: 89000 Basic stats: COMPLETE Column stats: COMPLETE
GatherStats: false
Filter Operator
isSamplingPred: false
- predicate: ((UDFToDouble(key) < 20.0D) and (UDFToDouble(key) > 15.0D)) (type: boolean)
+ predicate: ((UDFToDouble(key) > 15.0D) and (UDFToDouble(key) < 20.0D)) (type: boolean)
Statistics: Num rows: 55 Data size: 9790 Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
expressions: key (type: string), value (type: string)
diff --git a/ql/src/test/results/clientpositive/llap/cbo_rp_simple_select.q.out b/ql/src/test/results/clientpositive/llap/cbo_rp_simple_select.q.out
index 2fd47f6f32a..1715e7ce43b 100644
--- a/ql/src/test/results/clientpositive/llap/cbo_rp_simple_select.q.out
+++ b/ql/src/test/results/clientpositive/llap/cbo_rp_simple_select.q.out
@@ -901,9 +901,9 @@ STAGE PLANS:
Processor Tree:
TableScan
alias: cbo_t2
- filterExpr: (c_int is not null or (c_int = 0)) (type: boolean)
+ filterExpr: c_int is not null (type: boolean)
Filter Operator
- predicate: (c_int is not null or (c_int = 0)) (type: boolean)
+ predicate: c_int is not null (type: boolean)
Select Operator
expressions: key (type: string), value (type: string), c_int (type: int), c_float (type: float), c_boolean (type: boolean), dt (type: string)
outputColumnNames: key, value, c_int, c_float, c_boolean, dt
diff --git a/ql/src/test/results/clientpositive/llap/cbo_simple_select.q.out b/ql/src/test/results/clientpositive/llap/cbo_simple_select.q.out
index 82e131100ea..f10b878240d 100644
--- a/ql/src/test/results/clientpositive/llap/cbo_simple_select.q.out
+++ b/ql/src/test/results/clientpositive/llap/cbo_simple_select.q.out
@@ -901,9 +901,9 @@ STAGE PLANS:
Processor Tree:
TableScan
alias: cbo_t2
- filterExpr: (c_int is not null or (c_int = 0)) (type: boolean)
+ filterExpr: c_int is not null (type: boolean)
Filter Operator
- predicate: (c_int is not null or (c_int = 0)) (type: boolean)
+ predicate: c_int is not null (type: boolean)
Select Operator
expressions: key (type: string), value (type: string), c_int (type: int), c_float (type: float), c_boolean (type: boolean), dt (type: string)
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5
diff --git a/ql/src/test/results/clientpositive/llap/concat_op.q.out b/ql/src/test/results/clientpositive/llap/concat_op.q.out
index b6e4ae7c75e..4f992fbb68a 100644
--- a/ql/src/test/results/clientpositive/llap/concat_op.q.out
+++ b/ql/src/test/results/clientpositive/llap/concat_op.q.out
@@ -292,7 +292,7 @@ POSTHOOK: query: explain formatted select key || value from src
POSTHOOK: type: QUERY
POSTHOOK: Input: default@src
#### A masked pattern was here ####
-{"CBOPlan":"{\n \"rels\": [\n {\n \"id\": \"0\",\n \"relOp\": \"org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveTableScan\",\n \"table\": [\n \"default\",\n \"src\"\n ],\n \"table:alias\": \"src\",\n \"inputs\": [],\n \"rowCount\": 500.0,\n \"avgRowSize\": 9.624,\n \"rowType\": [\n {\n \"type\": \"VARCHAR\",\n \"nullable\": true,\n \"precision\": 2147483647,\n \"name\": [...]
+{"CBOPlan":"{\n \"rels\": [\n {\n \"id\": \"0\",\n \"relOp\": \"org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveTableScan\",\n \"table\": [\n \"default\",\n \"src\"\n ],\n \"table:alias\": \"src\",\n \"inputs\": [],\n \"rowCount\": 500.0,\n \"avgRowSize\": 9.624,\n \"rowType\": {\n \"fields\": [\n {\n \"type\": \"VARCHAR\",\n \"nullable\": true,\n \"precision\": 21 [...]
PREHOOK: query: explain formatted select key || value || key from src
PREHOOK: type: QUERY
PREHOOK: Input: default@src
@@ -301,7 +301,7 @@ POSTHOOK: query: explain formatted select key || value || key from src
POSTHOOK: type: QUERY
POSTHOOK: Input: default@src
#### A masked pattern was here ####
-{"CBOPlan":"{\n \"rels\": [\n {\n \"id\": \"0\",\n \"relOp\": \"org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveTableScan\",\n \"table\": [\n \"default\",\n \"src\"\n ],\n \"table:alias\": \"src\",\n \"inputs\": [],\n \"rowCount\": 500.0,\n \"avgRowSize\": 9.624,\n \"rowType\": [\n {\n \"type\": \"VARCHAR\",\n \"nullable\": true,\n \"precision\": 2147483647,\n \"name\": [...]
+{"CBOPlan":"{\n \"rels\": [\n {\n \"id\": \"0\",\n \"relOp\": \"org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveTableScan\",\n \"table\": [\n \"default\",\n \"src\"\n ],\n \"table:alias\": \"src\",\n \"inputs\": [],\n \"rowCount\": 500.0,\n \"avgRowSize\": 9.624,\n \"rowType\": {\n \"fields\": [\n {\n \"type\": \"VARCHAR\",\n \"nullable\": true,\n \"precision\": 21 [...]
PREHOOK: query: explain formatted select key || value || key || value from src
PREHOOK: type: QUERY
PREHOOK: Input: default@src
@@ -310,4 +310,4 @@ POSTHOOK: query: explain formatted select key || value || key || value from src
POSTHOOK: type: QUERY
POSTHOOK: Input: default@src
#### A masked pattern was here ####
-{"CBOPlan":"{\n \"rels\": [\n {\n \"id\": \"0\",\n \"relOp\": \"org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveTableScan\",\n \"table\": [\n \"default\",\n \"src\"\n ],\n \"table:alias\": \"src\",\n \"inputs\": [],\n \"rowCount\": 500.0,\n \"avgRowSize\": 9.624,\n \"rowType\": [\n {\n \"type\": \"VARCHAR\",\n \"nullable\": true,\n \"precision\": 2147483647,\n \"name\": [...]
+{"CBOPlan":"{\n \"rels\": [\n {\n \"id\": \"0\",\n \"relOp\": \"org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveTableScan\",\n \"table\": [\n \"default\",\n \"src\"\n ],\n \"table:alias\": \"src\",\n \"inputs\": [],\n \"rowCount\": 500.0,\n \"avgRowSize\": 9.624,\n \"rowType\": {\n \"fields\": [\n {\n \"type\": \"VARCHAR\",\n \"nullable\": true,\n \"precision\": 21 [...]
diff --git a/ql/src/test/results/clientpositive/llap/constant_prop_3.q.out b/ql/src/test/results/clientpositive/llap/constant_prop_3.q.out
index 94edbb1def7..51932053eab 100644
--- a/ql/src/test/results/clientpositive/llap/constant_prop_3.q.out
+++ b/ql/src/test/results/clientpositive/llap/constant_prop_3.q.out
@@ -193,10 +193,10 @@ STAGE PLANS:
Map Operator Tree:
TableScan
alias: part_hive
- filterExpr: ((p_size) IN (22, 14, 27, 49, 21, 33, 35, 28) and (p_brand <> 'Brand#34') and p_partkey is not null and (not (p_type like 'ECONOMY BRUSHED%'))) (type: boolean)
+ filterExpr: ((not (p_type like 'ECONOMY BRUSHED%')) and (p_brand <> 'Brand#34') and (p_size) IN (14, 21, 22, 27, 28, 33, 35, 49) and p_partkey is not null) (type: boolean)
Statistics: Num rows: 1 Data size: 176 Basic stats: COMPLETE Column stats: COMPLETE
Filter Operator
- predicate: ((p_size) IN (22, 14, 27, 49, 21, 33, 35, 28) and (p_brand <> 'Brand#34') and p_partkey is not null and (not (p_type like 'ECONOMY BRUSHED%'))) (type: boolean)
+ predicate: ((not (p_type like 'ECONOMY BRUSHED%')) and (p_brand <> 'Brand#34') and (p_size) IN (14, 21, 22, 27, 28, 33, 35, 49) and p_partkey is not null) (type: boolean)
Statistics: Num rows: 1 Data size: 176 Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
expressions: p_partkey (type: int), p_brand (type: string), p_type (type: string), p_size (type: int)
diff --git a/ql/src/test/results/clientpositive/llap/constraints_explain_ddl.q.out b/ql/src/test/results/clientpositive/llap/constraints_explain_ddl.q.out
index 7d28167c8fe..968833a457d 100644
--- a/ql/src/test/results/clientpositive/llap/constraints_explain_ddl.q.out
+++ b/ql/src/test/results/clientpositive/llap/constraints_explain_ddl.q.out
@@ -116,7 +116,7 @@ EXPLAIN CBO SELECT c_custkey from customer_removal_n0 where c_nation IN ('USA',
EXPLAIN VECTORIZED SELECT c_custkey from customer_removal_n0 where c_nation IN ('USA', 'INDIA') group by c_custkey;
CBO PLAN:HiveProject(c_custkey=[$0])
- HiveFilter(condition=[IN($4, _UTF-16LE'USA':VARCHAR(2147483647) CHARACTER SET "UTF-16LE", _UTF-16LE'INDIA':VARCHAR(2147483647) CHARACTER SET "UTF-16LE")])
+ HiveFilter(condition=[IN($4, _UTF-16LE'INDIA':VARCHAR(2147483647) CHARACTER SET "UTF-16LE", _UTF-16LE'USA':VARCHAR(2147483647) CHARACTER SET "UTF-16LE")])
HiveTableScan(table=[[default, customer_removal_n0]], table:alias=[customer_removal_n0])
PLAN VECTORIZATION:
@@ -133,9 +133,9 @@ STAGE PLANS:
Processor Tree:
TableScan
alias: customer_removal_n0
- filterExpr: (c_nation) IN ('USA', 'INDIA') (type: boolean)
+ filterExpr: (c_nation) IN ('INDIA', 'USA') (type: boolean)
Filter Operator
- predicate: (c_nation) IN ('USA', 'INDIA') (type: boolean)
+ predicate: (c_nation) IN ('INDIA', 'USA') (type: boolean)
Select Operator
expressions: c_custkey (type: bigint)
outputColumnNames: _col0
@@ -187,7 +187,7 @@ EXPLAIN CBO SELECT c_custkey from customer_removal_n0 where c_nation IN ('USA',
EXPLAIN VECTORIZED SELECT c_custkey from customer_removal_n0 where c_nation IN ('USA', 'INDIA') group by c_custkey, c_nation;
CBO PLAN:HiveProject(c_custkey=[$0])
- HiveFilter(condition=[IN($4, _UTF-16LE'USA':VARCHAR(2147483647) CHARACTER SET "UTF-16LE", _UTF-16LE'INDIA':VARCHAR(2147483647) CHARACTER SET "UTF-16LE")])
+ HiveFilter(condition=[IN($4, _UTF-16LE'INDIA':VARCHAR(2147483647) CHARACTER SET "UTF-16LE", _UTF-16LE'USA':VARCHAR(2147483647) CHARACTER SET "UTF-16LE")])
HiveTableScan(table=[[default, customer_removal_n0]], table:alias=[customer_removal_n0])
PLAN VECTORIZATION:
@@ -204,9 +204,9 @@ STAGE PLANS:
Processor Tree:
TableScan
alias: customer_removal_n0
- filterExpr: (c_nation) IN ('USA', 'INDIA') (type: boolean)
+ filterExpr: (c_nation) IN ('INDIA', 'USA') (type: boolean)
Filter Operator
- predicate: (c_nation) IN ('USA', 'INDIA') (type: boolean)
+ predicate: (c_nation) IN ('INDIA', 'USA') (type: boolean)
Select Operator
expressions: c_custkey (type: bigint)
outputColumnNames: _col0
@@ -1349,7 +1349,7 @@ EXPLAIN VECTORIZED SELECT count(c_custkey) from customer_removal_n0 where c_nati
group by c_custkey, c_nation;
CBO PLAN:HiveProject(_o__c0=[$1])
HiveAggregate(group=[{0}], agg#0=[count()])
- HiveFilter(condition=[IN($4, _UTF-16LE'USA':VARCHAR(2147483647) CHARACTER SET "UTF-16LE", _UTF-16LE'INDIA':VARCHAR(2147483647) CHARACTER SET "UTF-16LE")])
+ HiveFilter(condition=[IN($4, _UTF-16LE'INDIA':VARCHAR(2147483647) CHARACTER SET "UTF-16LE", _UTF-16LE'USA':VARCHAR(2147483647) CHARACTER SET "UTF-16LE")])
HiveTableScan(table=[[default, customer_removal_n0]], table:alias=[customer_removal_n0])
PLAN VECTORIZATION:
@@ -1372,10 +1372,10 @@ STAGE PLANS:
Map Operator Tree:
TableScan
alias: customer_removal_n0
- filterExpr: (c_nation) IN ('USA', 'INDIA') (type: boolean)
+ filterExpr: (c_nation) IN ('INDIA', 'USA') (type: boolean)
Statistics: Num rows: 1 Data size: 92 Basic stats: COMPLETE Column stats: COMPLETE
Filter Operator
- predicate: (c_nation) IN ('USA', 'INDIA') (type: boolean)
+ predicate: (c_nation) IN ('INDIA', 'USA') (type: boolean)
Statistics: Num rows: 1 Data size: 92 Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
expressions: c_custkey (type: bigint)
@@ -4402,7 +4402,7 @@ CBO PLAN:HiveSortLimit(sort0=[$0], dir0=[ASC], fetch=[100])
HiveJoin(condition=[=($0, $8)], joinType=[inner], algorithm=[none], cost=[not available])
HiveProject(c_customer_sk=[$0], c_customer_id=[$1], c_first_name=[$8], c_last_name=[$9], c_preferred_cust_flag=[$10], c_birth_country=[$14], c_login=[$15], c_email_address=[$16])
HiveTableScan(table=[[default, customer]], table:alias=[customer])
- HiveProject(ss_customer_sk=[$3], /=[/(+(-(-($17, $16), $14), $15), 2:DECIMAL(10, 0))])
+ HiveProject(ss_customer_sk=[$3], $f7=[/(+(-(-($17, $16), $14), $15), 2:DECIMAL(10, 0))])
HiveFilter(condition=[IS NOT NULL($3)])
HiveTableScan(table=[[default, store_sales]], table:alias=[store_sales])
@@ -4800,7 +4800,7 @@ CBO PLAN:HiveSortLimit(sort0=[$0], dir0=[ASC], fetch=[100])
HiveJoin(condition=[=($0, $8)], joinType=[inner], algorithm=[none], cost=[not available])
HiveProject(c_customer_sk=[$0], c_customer_id=[$1], c_first_name=[$8], c_last_name=[$9], c_preferred_cust_flag=[$10], c_birth_country=[$14], c_login=[$15], c_email_address=[$16])
HiveTableScan(table=[[default, customer]], table:alias=[customer])
- HiveProject(ss_customer_sk=[$3], /=[/(+(-(-($17, $16), $14), $15), 2:DECIMAL(10, 0))])
+ HiveProject(ss_customer_sk=[$3], $f7=[/(+(-(-($17, $16), $14), $15), 2:DECIMAL(10, 0))])
HiveFilter(condition=[IS NOT NULL($3)])
HiveTableScan(table=[[default, store_sales]], table:alias=[store_sales])
@@ -5353,7 +5353,7 @@ CBO PLAN:HiveSortLimit(sort0=[$0], dir0=[ASC], fetch=[100])
HiveProject(c_customer_sk=[$0], c_customer_id=[$1], c_first_name=[$8], c_last_name=[$9], c_preferred_cust_flag=[$10], c_birth_country=[$14], c_login=[$15], c_email_address=[$16])
HiveTableScan(table=[[default, customer]], table:alias=[customer])
HiveJoin(condition=[=($0, $3)], joinType=[inner], algorithm=[none], cost=[not available])
- HiveProject(ss_sold_date_sk=[$0], ss_customer_sk=[$3], /=[/(+(-(-($17, $16), $14), $15), 2:DECIMAL(10, 0))])
+ HiveProject(ss_sold_date_sk=[$0], ss_customer_sk=[$3], $f8=[/(+(-(-($17, $16), $14), $15), 2:DECIMAL(10, 0))])
HiveFilter(condition=[AND(IS NOT NULL($3), IS NOT NULL($0))])
HiveTableScan(table=[[default, store_sales]], table:alias=[store_sales])
HiveProject(d_date_sk=[$0])
@@ -5366,7 +5366,7 @@ CBO PLAN:HiveSortLimit(sort0=[$0], dir0=[ASC], fetch=[100])
HiveProject(c_customer_sk=[$0], c_customer_id=[$1], c_first_name=[$8], c_last_name=[$9], c_preferred_cust_flag=[$10], c_birth_country=[$14], c_login=[$15], c_email_address=[$16])
HiveTableScan(table=[[default, customer]], table:alias=[customer])
HiveJoin(condition=[=($0, $3)], joinType=[inner], algorithm=[none], cost=[not available])
- HiveProject(ss_sold_date_sk=[$0], ss_customer_sk=[$3], /=[/(+(-(-($17, $16), $14), $15), 2:DECIMAL(10, 0))])
+ HiveProject(ss_sold_date_sk=[$0], ss_customer_sk=[$3], $f8=[/(+(-(-($17, $16), $14), $15), 2:DECIMAL(10, 0))])
HiveFilter(condition=[AND(IS NOT NULL($3), IS NOT NULL($0))])
HiveTableScan(table=[[default, store_sales]], table:alias=[store_sales])
HiveProject(d_date_sk=[$0])
diff --git a/ql/src/test/results/clientpositive/llap/constraints_optimization.q.out b/ql/src/test/results/clientpositive/llap/constraints_optimization.q.out
index ea4ce50633b..aacb7d63e3d 100644
--- a/ql/src/test/results/clientpositive/llap/constraints_optimization.q.out
+++ b/ql/src/test/results/clientpositive/llap/constraints_optimization.q.out
@@ -90,9 +90,9 @@ STAGE PLANS:
Processor Tree:
TableScan
alias: customer_removal_n0
- filterExpr: (c_nation) IN ('USA', 'INDIA') (type: boolean)
+ filterExpr: (c_nation) IN ('INDIA', 'USA') (type: boolean)
Filter Operator
- predicate: (c_nation) IN ('USA', 'INDIA') (type: boolean)
+ predicate: (c_nation) IN ('INDIA', 'USA') (type: boolean)
Select Operator
expressions: c_custkey (type: bigint)
outputColumnNames: _col0
@@ -118,9 +118,9 @@ STAGE PLANS:
Processor Tree:
TableScan
alias: customer_removal_n0
- filterExpr: (c_nation) IN ('USA', 'INDIA') (type: boolean)
+ filterExpr: (c_nation) IN ('INDIA', 'USA') (type: boolean)
Filter Operator
- predicate: (c_nation) IN ('USA', 'INDIA') (type: boolean)
+ predicate: (c_nation) IN ('INDIA', 'USA') (type: boolean)
Select Operator
expressions: c_custkey (type: bigint)
outputColumnNames: _col0
@@ -697,10 +697,10 @@ STAGE PLANS:
Map Operator Tree:
TableScan
alias: customer_removal_n0
- filterExpr: (c_nation) IN ('USA', 'INDIA') (type: boolean)
+ filterExpr: (c_nation) IN ('INDIA', 'USA') (type: boolean)
Statistics: Num rows: 1 Data size: 92 Basic stats: COMPLETE Column stats: COMPLETE
Filter Operator
- predicate: (c_nation) IN ('USA', 'INDIA') (type: boolean)
+ predicate: (c_nation) IN ('INDIA', 'USA') (type: boolean)
Statistics: Num rows: 1 Data size: 92 Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
expressions: c_custkey (type: bigint)
@@ -2359,7 +2359,7 @@ HiveSortLimit(sort0=[$0], dir0=[ASC], fetch=[100])
HiveJoin(condition=[=($0, $8)], joinType=[inner], algorithm=[none], cost=[not available])
HiveProject(c_customer_sk=[$0], c_customer_id=[$1], c_first_name=[$8], c_last_name=[$9], c_preferred_cust_flag=[$10], c_birth_country=[$14], c_login=[$15], c_email_address=[$16])
HiveTableScan(table=[[default, customer]], table:alias=[customer])
- HiveProject(ss_customer_sk=[$3], /=[/(+(-(-($17, $16), $14), $15), 2:DECIMAL(10, 0))])
+ HiveProject(ss_customer_sk=[$3], $f7=[/(+(-(-($17, $16), $14), $15), 2:DECIMAL(10, 0))])
HiveFilter(condition=[IS NOT NULL($3)])
HiveTableScan(table=[[default, store_sales]], table:alias=[store_sales])
@@ -2434,7 +2434,7 @@ HiveSortLimit(sort0=[$0], dir0=[ASC], fetch=[100])
HiveJoin(condition=[=($0, $8)], joinType=[inner], algorithm=[none], cost=[not available])
HiveProject(c_customer_sk=[$0], c_customer_id=[$1], c_first_name=[$8], c_last_name=[$9], c_preferred_cust_flag=[$10], c_birth_country=[$14], c_login=[$15], c_email_address=[$16])
HiveTableScan(table=[[default, customer]], table:alias=[customer])
- HiveProject(ss_customer_sk=[$3], /=[/(+(-(-($17, $16), $14), $15), 2:DECIMAL(10, 0))])
+ HiveProject(ss_customer_sk=[$3], $f7=[/(+(-(-($17, $16), $14), $15), 2:DECIMAL(10, 0))])
HiveFilter(condition=[IS NOT NULL($3)])
HiveTableScan(table=[[default, store_sales]], table:alias=[store_sales])
@@ -2596,7 +2596,7 @@ HiveSortLimit(sort0=[$0], dir0=[ASC], fetch=[100])
HiveProject(c_customer_sk=[$0], c_customer_id=[$1], c_first_name=[$8], c_last_name=[$9], c_preferred_cust_flag=[$10], c_birth_country=[$14], c_login=[$15], c_email_address=[$16])
HiveTableScan(table=[[default, customer]], table:alias=[customer])
HiveJoin(condition=[=($0, $3)], joinType=[inner], algorithm=[none], cost=[not available])
- HiveProject(ss_sold_date_sk=[$0], ss_customer_sk=[$3], /=[/(+(-(-($17, $16), $14), $15), 2:DECIMAL(10, 0))])
+ HiveProject(ss_sold_date_sk=[$0], ss_customer_sk=[$3], $f8=[/(+(-(-($17, $16), $14), $15), 2:DECIMAL(10, 0))])
HiveFilter(condition=[AND(IS NOT NULL($3), IS NOT NULL($0))])
HiveTableScan(table=[[default, store_sales]], table:alias=[store_sales])
HiveProject(d_date_sk=[$0])
@@ -2609,7 +2609,7 @@ HiveSortLimit(sort0=[$0], dir0=[ASC], fetch=[100])
HiveProject(c_customer_sk=[$0], c_customer_id=[$1], c_first_name=[$8], c_last_name=[$9], c_preferred_cust_flag=[$10], c_birth_country=[$14], c_login=[$15], c_email_address=[$16])
HiveTableScan(table=[[default, customer]], table:alias=[customer])
HiveJoin(condition=[=($0, $3)], joinType=[inner], algorithm=[none], cost=[not available])
- HiveProject(ss_sold_date_sk=[$0], ss_customer_sk=[$3], /=[/(+(-(-($17, $16), $14), $15), 2:DECIMAL(10, 0))])
+ HiveProject(ss_sold_date_sk=[$0], ss_customer_sk=[$3], $f8=[/(+(-(-($17, $16), $14), $15), 2:DECIMAL(10, 0))])
HiveFilter(condition=[AND(IS NOT NULL($3), IS NOT NULL($0))])
HiveTableScan(table=[[default, store_sales]], table:alias=[store_sales])
HiveProject(d_date_sk=[$0])
@@ -3034,7 +3034,7 @@ POSTHOOK: Input: default@store_sales
#### A masked pattern was here ####
CBO PLAN:
HiveFilter(condition=[>($3, 4)])
- HiveProject(substr=[$1], log2=[$2], d_date=[$0], $f3=[$3])
+ HiveProject($f0=[$1], $f1=[$2], d_date=[$0], $f3=[$3])
HiveAggregate(group=[{3, 5, 6}], agg#0=[count()])
HiveJoin(condition=[=($1, $4)], joinType=[inner], algorithm=[none], cost=[not available])
HiveJoin(condition=[=($0, $2)], joinType=[inner], algorithm=[none], cost=[not available])
@@ -3044,7 +3044,7 @@ HiveFilter(condition=[>($3, 4)])
HiveProject(d_date_sk=[$0], d_date=[$2])
HiveFilter(condition=[IS NOT NULL($0)])
HiveTableScan(table=[[default, date_dim]], table:alias=[date_dim])
- HiveProject(c_customer_sk=[$0], substr=[substr($8, 1, 30)], log2=[log2($0)])
+ HiveProject(c_customer_sk=[$0], $f0=[substr($8, 1, 30)], $f1=[log2($0)])
HiveTableScan(table=[[default, customer]], table:alias=[customer])
PREHOOK: query: -- group by with keys consisting of pk from multiple tables with extra columns from both side
@@ -3127,15 +3127,15 @@ POSTHOOK: Input: default@store_sales
CBO PLAN:
HiveProject(c_customer_sk=[$0])
HiveFilter(condition=[>($3, 4)])
- HiveProject(c_customer_sk=[$4], log2=[$2], d_date_sk=[$1], $f3=[$3])
+ HiveProject(c_customer_sk=[$4], $f2=[$2], d_date_sk=[$1], $f3=[$3])
HiveJoin(condition=[=($0, $4)], joinType=[inner], algorithm=[none], cost=[not available])
- HiveProject(ss_item_sk=[$0], d_date_sk=[$1], log2=[$2], $f3=[$3])
+ HiveProject(ss_item_sk=[$0], d_date_sk=[$1], $f2=[$2], $f3=[$3])
HiveAggregate(group=[{1, 2, 3}], agg#0=[count()])
HiveJoin(condition=[=($0, $2)], joinType=[inner], algorithm=[none], cost=[not available])
HiveProject(ss_sold_date_sk=[$0], ss_item_sk=[$2])
HiveFilter(condition=[IS NOT NULL($0)])
HiveTableScan(table=[[default, store_sales]], table:alias=[store_sales])
- HiveProject(d_date_sk=[$0], log2=[log2($2)])
+ HiveProject(d_date_sk=[$0], $f2=[log2($2)])
HiveFilter(condition=[IS NOT NULL($0)])
HiveTableScan(table=[[default, date_dim]], table:alias=[date_dim])
HiveProject(c_customer_sk=[$0])
@@ -3290,7 +3290,7 @@ CBO PLAN:
HiveProject(i=[$0])
HiveFilter(condition=[=($2, 2)])
HiveAggregate(group=[{0, 1}], agg#0=[count($2)])
- HiveProject($f0=[$0], $f1=[$1], $f2=[$2])
+ HiveProject(i=[$0], j=[$1], $f2=[$2])
HiveUnion(all=[true])
HiveProject(i=[$0], j=[$1], $f2=[$2])
HiveAggregate(group=[{0, 1}], agg#0=[count()])
diff --git a/ql/src/test/results/clientpositive/llap/correlationoptimizer10.q.out b/ql/src/test/results/clientpositive/llap/correlationoptimizer10.q.out
index b212b77cb42..d1719935c32 100644
--- a/ql/src/test/results/clientpositive/llap/correlationoptimizer10.q.out
+++ b/ql/src/test/results/clientpositive/llap/correlationoptimizer10.q.out
@@ -449,10 +449,10 @@ STAGE PLANS:
Map Operator Tree:
TableScan
alias: xx
- filterExpr: ((UDFToDouble(key) < 200.0D) and (UDFToDouble(key) > 20.0D)) (type: boolean)
+ filterExpr: ((UDFToDouble(key) > 20.0D) and (UDFToDouble(key) < 200.0D)) (type: boolean)
Statistics: Num rows: 25 Data size: 4375 Basic stats: COMPLETE Column stats: COMPLETE
Filter Operator
- predicate: ((UDFToDouble(key) < 200.0D) and (UDFToDouble(key) > 20.0D)) (type: boolean)
+ predicate: ((UDFToDouble(key) > 20.0D) and (UDFToDouble(key) < 200.0D)) (type: boolean)
Statistics: Num rows: 2 Data size: 350 Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
expressions: key (type: string), value (type: string)
@@ -471,10 +471,10 @@ STAGE PLANS:
Map Operator Tree:
TableScan
alias: x
- filterExpr: ((UDFToDouble(key) < 200.0D) and (UDFToDouble(key) > 20.0D)) (type: boolean)
+ filterExpr: ((UDFToDouble(key) > 20.0D) and (UDFToDouble(key) < 200.0D)) (type: boolean)
Statistics: Num rows: 500 Data size: 43500 Basic stats: COMPLETE Column stats: COMPLETE
Filter Operator
- predicate: ((UDFToDouble(key) < 200.0D) and (UDFToDouble(key) > 20.0D)) (type: boolean)
+ predicate: ((UDFToDouble(key) > 20.0D) and (UDFToDouble(key) < 200.0D)) (type: boolean)
Statistics: Num rows: 55 Data size: 4785 Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
expressions: key (type: string)
@@ -634,10 +634,10 @@ STAGE PLANS:
Map Operator Tree:
TableScan
alias: xx
- filterExpr: ((UDFToDouble(key) < 200.0D) and (UDFToDouble(key) > 20.0D)) (type: boolean)
+ filterExpr: ((UDFToDouble(key) > 20.0D) and (UDFToDouble(key) < 200.0D)) (type: boolean)
Statistics: Num rows: 25 Data size: 4375 Basic stats: COMPLETE Column stats: COMPLETE
Filter Operator
- predicate: ((UDFToDouble(key) < 200.0D) and (UDFToDouble(key) > 20.0D)) (type: boolean)
+ predicate: ((UDFToDouble(key) > 20.0D) and (UDFToDouble(key) < 200.0D)) (type: boolean)
Statistics: Num rows: 2 Data size: 350 Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
expressions: key (type: string), value (type: string)
@@ -656,10 +656,10 @@ STAGE PLANS:
Map Operator Tree:
TableScan
alias: x
- filterExpr: ((UDFToDouble(key) < 200.0D) and (UDFToDouble(key) > 20.0D)) (type: boolean)
+ filterExpr: ((UDFToDouble(key) > 20.0D) and (UDFToDouble(key) < 200.0D)) (type: boolean)
Statistics: Num rows: 500 Data size: 43500 Basic stats: COMPLETE Column stats: COMPLETE
Filter Operator
- predicate: ((UDFToDouble(key) < 200.0D) and (UDFToDouble(key) > 20.0D)) (type: boolean)
+ predicate: ((UDFToDouble(key) > 20.0D) and (UDFToDouble(key) < 200.0D)) (type: boolean)
Statistics: Num rows: 55 Data size: 4785 Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
expressions: key (type: string)
@@ -815,10 +815,10 @@ STAGE PLANS:
Map Operator Tree:
TableScan
alias: xx
- filterExpr: ((UDFToDouble(key) < 200.0D) and (UDFToDouble(key) > 180.0D)) (type: boolean)
+ filterExpr: ((UDFToDouble(key) > 180.0D) and (UDFToDouble(key) < 200.0D)) (type: boolean)
Statistics: Num rows: 500 Data size: 89000 Basic stats: COMPLETE Column stats: COMPLETE
Filter Operator
- predicate: ((UDFToDouble(key) < 200.0D) and (UDFToDouble(key) > 180.0D)) (type: boolean)
+ predicate: ((UDFToDouble(key) > 180.0D) and (UDFToDouble(key) < 200.0D)) (type: boolean)
Statistics: Num rows: 55 Data size: 9790 Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
expressions: key (type: string), value (type: string)
@@ -847,10 +847,10 @@ STAGE PLANS:
Map Operator Tree:
TableScan
alias: y
- filterExpr: ((UDFToDouble(key) < 200.0D) and (UDFToDouble(key) > 180.0D)) (type: boolean)
+ filterExpr: ((UDFToDouble(key) > 180.0D) and (UDFToDouble(key) < 200.0D)) (type: boolean)
Statistics: Num rows: 500 Data size: 43500 Basic stats: COMPLETE Column stats: COMPLETE
Filter Operator
- predicate: ((UDFToDouble(key) < 200.0D) and (UDFToDouble(key) > 180.0D)) (type: boolean)
+ predicate: ((UDFToDouble(key) > 180.0D) and (UDFToDouble(key) < 200.0D)) (type: boolean)
Statistics: Num rows: 55 Data size: 4785 Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
expressions: key (type: string)
@@ -999,10 +999,10 @@ STAGE PLANS:
Map Operator Tree:
TableScan
alias: xx
- filterExpr: ((UDFToDouble(key) < 200.0D) and (UDFToDouble(key) > 180.0D)) (type: boolean)
+ filterExpr: ((UDFToDouble(key) > 180.0D) and (UDFToDouble(key) < 200.0D)) (type: boolean)
Statistics: Num rows: 500 Data size: 89000 Basic stats: COMPLETE Column stats: COMPLETE
Filter Operator
- predicate: ((UDFToDouble(key) < 200.0D) and (UDFToDouble(key) > 180.0D)) (type: boolean)
+ predicate: ((UDFToDouble(key) > 180.0D) and (UDFToDouble(key) < 200.0D)) (type: boolean)
Statistics: Num rows: 55 Data size: 9790 Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
expressions: key (type: string), value (type: string)
@@ -1031,10 +1031,10 @@ STAGE PLANS:
Map Operator Tree:
TableScan
alias: y
- filterExpr: ((UDFToDouble(key) < 200.0D) and (UDFToDouble(key) > 180.0D)) (type: boolean)
+ filterExpr: ((UDFToDouble(key) > 180.0D) and (UDFToDouble(key) < 200.0D)) (type: boolean)
Statistics: Num rows: 500 Data size: 43500 Basic stats: COMPLETE Column stats: COMPLETE
Filter Operator
- predicate: ((UDFToDouble(key) < 200.0D) and (UDFToDouble(key) > 180.0D)) (type: boolean)
+ predicate: ((UDFToDouble(key) > 180.0D) and (UDFToDouble(key) < 200.0D)) (type: boolean)
Statistics: Num rows: 55 Data size: 4785 Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
expressions: key (type: string)
diff --git a/ql/src/test/results/clientpositive/llap/correlationoptimizer8.q.out b/ql/src/test/results/clientpositive/llap/correlationoptimizer8.q.out
index e957a86b623..19af2ee5634 100644
--- a/ql/src/test/results/clientpositive/llap/correlationoptimizer8.q.out
+++ b/ql/src/test/results/clientpositive/llap/correlationoptimizer8.q.out
@@ -82,21 +82,21 @@ STAGE PLANS:
Map Operator Tree:
TableScan
alias: x
- filterExpr: key is not null (type: boolean)
+ filterExpr: (((UDFToDouble(key) < 20.0D) or (UDFToDouble(key) > 100.0D)) and key is not null) (type: boolean)
Statistics: Num rows: 25 Data size: 4375 Basic stats: COMPLETE Column stats: COMPLETE
Filter Operator
- predicate: key is not null (type: boolean)
- Statistics: Num rows: 25 Data size: 4375 Basic stats: COMPLETE Column stats: COMPLETE
+ predicate: (((UDFToDouble(key) < 20.0D) or (UDFToDouble(key) > 100.0D)) and key is not null) (type: boolean)
+ Statistics: Num rows: 16 Data size: 2800 Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
expressions: key (type: string), value (type: string)
outputColumnNames: _col0, _col1
- Statistics: Num rows: 25 Data size: 4375 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 16 Data size: 2800 Basic stats: COMPLETE Column stats: COMPLETE
Reduce Output Operator
key expressions: _col0 (type: string)
null sort order: z
sort order: +
Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 25 Data size: 4375 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 16 Data size: 2800 Basic stats: COMPLETE Column stats: COMPLETE
value expressions: _col1 (type: string)
Execution mode: vectorized, llap
LLAP IO: all inputs
@@ -126,14 +126,14 @@ STAGE PLANS:
0 _col0 (type: string)
1 _col0 (type: string)
outputColumnNames: _col1, _col2, _col3
- Statistics: Num rows: 50 Data size: 9150 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 32 Data size: 5856 Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
expressions: _col2 (type: string), _col3 (type: string), _col1 (type: bigint)
outputColumnNames: _col0, _col1, _col2
- Statistics: Num rows: 50 Data size: 9150 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 32 Data size: 5856 Basic stats: COMPLETE Column stats: COMPLETE
File Output Operator
compressed: false
- Statistics: Num rows: 50 Data size: 9150 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 32 Data size: 5856 Basic stats: COMPLETE Column stats: COMPLETE
table:
input format: org.apache.hadoop.mapred.SequenceFileInputFormat
output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
@@ -282,21 +282,21 @@ STAGE PLANS:
Map Operator Tree:
TableScan
alias: x
- filterExpr: key is not null (type: boolean)
+ filterExpr: (((UDFToDouble(key) < 20.0D) or (UDFToDouble(key) > 100.0D)) and key is not null) (type: boolean)
Statistics: Num rows: 25 Data size: 4375 Basic stats: COMPLETE Column stats: COMPLETE
Filter Operator
- predicate: key is not null (type: boolean)
- Statistics: Num rows: 25 Data size: 4375 Basic stats: COMPLETE Column stats: COMPLETE
+ predicate: (((UDFToDouble(key) < 20.0D) or (UDFToDouble(key) > 100.0D)) and key is not null) (type: boolean)
+ Statistics: Num rows: 16 Data size: 2800 Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
expressions: key (type: string), value (type: string)
outputColumnNames: _col0, _col1
- Statistics: Num rows: 25 Data size: 4375 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 16 Data size: 2800 Basic stats: COMPLETE Column stats: COMPLETE
Reduce Output Operator
key expressions: _col0 (type: string)
null sort order: z
sort order: +
Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 25 Data size: 4375 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 16 Data size: 2800 Basic stats: COMPLETE Column stats: COMPLETE
value expressions: _col1 (type: string)
Execution mode: vectorized, llap
LLAP IO: all inputs
@@ -326,14 +326,14 @@ STAGE PLANS:
0 _col0 (type: string)
1 _col0 (type: string)
outputColumnNames: _col1, _col2, _col3
- Statistics: Num rows: 50 Data size: 9150 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 32 Data size: 5856 Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
expressions: _col2 (type: string), _col3 (type: string), _col1 (type: bigint)
outputColumnNames: _col0, _col1, _col2
- Statistics: Num rows: 50 Data size: 9150 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 32 Data size: 5856 Basic stats: COMPLETE Column stats: COMPLETE
File Output Operator
compressed: false
- Statistics: Num rows: 50 Data size: 9150 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 32 Data size: 5856 Basic stats: COMPLETE Column stats: COMPLETE
table:
input format: org.apache.hadoop.mapred.SequenceFileInputFormat
output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
@@ -922,21 +922,21 @@ STAGE PLANS:
Map Operator Tree:
TableScan
alias: x
- filterExpr: key is not null (type: boolean)
+ filterExpr: (((UDFToDouble(key) < 20.0D) or (UDFToDouble(key) > 100.0D)) and key is not null) (type: boolean)
Statistics: Num rows: 25 Data size: 4375 Basic stats: COMPLETE Column stats: COMPLETE
Filter Operator
- predicate: key is not null (type: boolean)
- Statistics: Num rows: 25 Data size: 4375 Basic stats: COMPLETE Column stats: COMPLETE
+ predicate: (((UDFToDouble(key) < 20.0D) or (UDFToDouble(key) > 100.0D)) and key is not null) (type: boolean)
+ Statistics: Num rows: 16 Data size: 2800 Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
expressions: key (type: string), value (type: string)
outputColumnNames: _col0, _col1
- Statistics: Num rows: 25 Data size: 4375 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 16 Data size: 2800 Basic stats: COMPLETE Column stats: COMPLETE
Reduce Output Operator
key expressions: _col0 (type: string)
null sort order: z
sort order: +
Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 25 Data size: 4375 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 16 Data size: 2800 Basic stats: COMPLETE Column stats: COMPLETE
value expressions: _col1 (type: string)
Execution mode: vectorized, llap
LLAP IO: all inputs
@@ -970,14 +970,14 @@ STAGE PLANS:
0 _col0 (type: string)
1 _col0 (type: string)
outputColumnNames: _col1, _col2, _col3
- Statistics: Num rows: 50 Data size: 9150 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 32 Data size: 5856 Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
expressions: _col2 (type: string), _col3 (type: string), _col1 (type: bigint)
outputColumnNames: _col0, _col1, _col2
- Statistics: Num rows: 50 Data size: 9150 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 32 Data size: 5856 Basic stats: COMPLETE Column stats: COMPLETE
File Output Operator
compressed: false
- Statistics: Num rows: 50 Data size: 9150 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 32 Data size: 5856 Basic stats: COMPLETE Column stats: COMPLETE
table:
input format: org.apache.hadoop.mapred.SequenceFileInputFormat
output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
diff --git a/ql/src/test/results/clientpositive/llap/correlationoptimizer9.q.out b/ql/src/test/results/clientpositive/llap/correlationoptimizer9.q.out
index bdd9e47f53d..1ed559f302d 100644
--- a/ql/src/test/results/clientpositive/llap/correlationoptimizer9.q.out
+++ b/ql/src/test/results/clientpositive/llap/correlationoptimizer9.q.out
@@ -56,10 +56,10 @@ STAGE PLANS:
Map Operator Tree:
TableScan
alias: x
- filterExpr: ((c1 < 120) and (c1 > 100)) (type: boolean)
+ filterExpr: ((c1 > 100) and (c1 < 120)) (type: boolean)
Statistics: Num rows: 1028 Data size: 4112 Basic stats: COMPLETE Column stats: COMPLETE
Filter Operator
- predicate: ((c1 < 120) and (c1 > 100)) (type: boolean)
+ predicate: ((c1 > 100) and (c1 < 120)) (type: boolean)
Statistics: Num rows: 198 Data size: 792 Basic stats: COMPLETE Column stats: COMPLETE
Group By Operator
aggregations: count()
@@ -208,10 +208,10 @@ STAGE PLANS:
Map Operator Tree:
TableScan
alias: x
- filterExpr: ((c1 < 120) and (c1 > 100)) (type: boolean)
+ filterExpr: ((c1 > 100) and (c1 < 120)) (type: boolean)
Statistics: Num rows: 1028 Data size: 4112 Basic stats: COMPLETE Column stats: COMPLETE
Filter Operator
- predicate: ((c1 < 120) and (c1 > 100)) (type: boolean)
+ predicate: ((c1 > 100) and (c1 < 120)) (type: boolean)
Statistics: Num rows: 198 Data size: 792 Basic stats: COMPLETE Column stats: COMPLETE
Group By Operator
aggregations: count()
diff --git a/ql/src/test/results/clientpositive/llap/ctas.q.out b/ql/src/test/results/clientpositive/llap/ctas.q.out
index 0350019155c..ac6acd49523 100644
--- a/ql/src/test/results/clientpositive/llap/ctas.q.out
+++ b/ql/src/test/results/clientpositive/llap/ctas.q.out
@@ -1183,4 +1183,4 @@ POSTHOOK: Input: default@acid_ctas_part@k=0
POSTHOOK: Input: default@acid_ctas_part@k=10
POSTHOOK: Input: default@acid_ctas_part@k=100
#### A masked pattern was here ####
-{"CBOPlan":"{\n \"rels\": [\n {\n \"id\": \"0\",\n \"relOp\": \"org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveTableScan\",\n \"table\": [\n \"default\",\n \"acid_ctas_part\"\n ],\n \"table:alias\": \"acid_ctas_part\",\n \"inputs\": [],\n \"rowCount\": 273.0,\n \"avgRowSize\": 200.0,\n \"rowType\": [\n {\n \"type\": \"VARCHAR\",\n \"nullable\": true,\n \"precision\": 2147483647, [...]
+{"CBOPlan":"{\n \"rels\": [\n {\n \"id\": \"0\",\n \"relOp\": \"org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveTableScan\",\n \"table\": [\n \"default\",\n \"acid_ctas_part\"\n ],\n \"table:alias\": \"acid_ctas_part\",\n \"inputs\": [],\n \"rowCount\": 273.0,\n \"avgRowSize\": 200.0,\n \"rowType\": {\n \"fields\": [\n {\n \"type\": \"VARCHAR\",\n \"nullable\": true,\n [...]
diff --git a/ql/src/test/results/clientpositive/llap/cte_cbo_plan_json.q.out b/ql/src/test/results/clientpositive/llap/cte_cbo_plan_json.q.out
index 953b4682ddc..e51a7a9d29d 100644
--- a/ql/src/test/results/clientpositive/llap/cte_cbo_plan_json.q.out
+++ b/ql/src/test/results/clientpositive/llap/cte_cbo_plan_json.q.out
@@ -34,4 +34,4 @@ POSTHOOK: type: QUERY
POSTHOOK: Input: default@cte_suggestion_0
POSTHOOK: Input: default@emps
#### A masked pattern was here ####
-{"CBOPlan":"{\n \"rels\": [\n {\n \"id\": \"0\",\n \"relOp\": \"org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveTableScan\",\n \"table\": [\n \"default\",\n \"emps\"\n ],\n \"table:alias\": \"e\",\n \"inputs\": [],\n \"rowCount\": 1.0,\n \"avgRowSize\": 161.0,\n \"rowType\": [\n {\n \"type\": \"INTEGER\",\n \"nullable\": true,\n \"name\": \"empid\"\n },\n {\n [...]
+{"CBOPlan":"{\n \"rels\": [\n {\n \"id\": \"0\",\n \"relOp\": \"org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveTableScan\",\n \"table\": [\n \"default\",\n \"emps\"\n ],\n \"table:alias\": \"e\",\n \"inputs\": [],\n \"rowCount\": 1.0,\n \"avgRowSize\": 161.0,\n \"rowType\": {\n \"fields\": [\n {\n \"type\": \"INTEGER\",\n \"nullable\": true,\n \"name\": \"empid\"\ [...]
diff --git a/ql/src/test/results/clientpositive/llap/dynamic_partition_pruning_2.q.out b/ql/src/test/results/clientpositive/llap/dynamic_partition_pruning_2.q.out
index f05f90e88f8..0c2e416d662 100644
--- a/ql/src/test/results/clientpositive/llap/dynamic_partition_pruning_2.q.out
+++ b/ql/src/test/results/clientpositive/llap/dynamic_partition_pruning_2.q.out
@@ -213,10 +213,10 @@ STAGE PLANS:
Map Operator Tree:
TableScan
alias: d1
- filterExpr: ((label) IN ('foo', 'bar') and id is not null) (type: boolean)
+ filterExpr: ((label) IN ('bar', 'foo') and id is not null) (type: boolean)
Statistics: Num rows: 3 Data size: 564 Basic stats: COMPLETE Column stats: NONE
Filter Operator
- predicate: ((label) IN ('foo', 'bar') and id is not null) (type: boolean)
+ predicate: ((label) IN ('bar', 'foo') and id is not null) (type: boolean)
Statistics: Num rows: 3 Data size: 564 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: id (type: int), label (type: string)
@@ -399,10 +399,10 @@ STAGE PLANS:
Map Operator Tree:
TableScan
alias: d1
- filterExpr: ((label) IN ('foo', 'bar') and id is not null) (type: boolean)
+ filterExpr: ((label) IN ('bar', 'foo') and id is not null) (type: boolean)
Statistics: Num rows: 3 Data size: 564 Basic stats: COMPLETE Column stats: NONE
Filter Operator
- predicate: ((label) IN ('foo', 'bar') and id is not null) (type: boolean)
+ predicate: ((label) IN ('bar', 'foo') and id is not null) (type: boolean)
Statistics: Num rows: 3 Data size: 564 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: id (type: int), label (type: string)
@@ -812,10 +812,10 @@ STAGE PLANS:
Map Operator Tree:
TableScan
alias: d1
- filterExpr: ((label) IN ('foo', 'bar') and id is not null) (type: boolean)
+ filterExpr: ((label) IN ('bar', 'foo') and id is not null) (type: boolean)
Statistics: Num rows: 3 Data size: 564 Basic stats: COMPLETE Column stats: NONE
Filter Operator
- predicate: ((label) IN ('foo', 'bar') and id is not null) (type: boolean)
+ predicate: ((label) IN ('bar', 'foo') and id is not null) (type: boolean)
Statistics: Num rows: 3 Data size: 564 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: id (type: int), label (type: string)
diff --git a/ql/src/test/results/clientpositive/llap/estimate_pkfk_push.q.out b/ql/src/test/results/clientpositive/llap/estimate_pkfk_push.q.out
index b1a7a4bccae..596f9a31cee 100644
--- a/ql/src/test/results/clientpositive/llap/estimate_pkfk_push.q.out
+++ b/ql/src/test/results/clientpositive/llap/estimate_pkfk_push.q.out
@@ -215,7 +215,7 @@ Stage-0
Select Operator [SEL_48] (rows=20/20 width=8)
Output:["_col0","_col1"]
Filter Operator [FIL_47] (rows=20/20 width=8)
- predicate:((ship_type_id) IN (1, 2) and id is not null)
+ predicate:((ship_type_id) IN (1, 2) and ship_type_id is not null and id is not null)
TableScan [TS_0] (rows=100/100 width=8)
default@ships,s,Tbl:COMPLETE,Col:COMPLETE,Output:["id","ship_type_id"]
<-Map 4 [SIMPLE_EDGE] vectorized, llap
@@ -224,7 +224,7 @@ Stage-0
Select Operator [SEL_51] (rows=2/2 width=4)
Output:["_col0"]
Filter Operator [FIL_50] (rows=2/2 width=4)
- predicate:(id) IN (1, 2)
+ predicate:((id) IN (1, 2) and id is not null)
TableScan [TS_3] (rows=10/10 width=4)
default@ship_types,st,Tbl:COMPLETE,Col:COMPLETE,Output:["id"]
diff --git a/ql/src/test/results/clientpositive/llap/excluded_rule_explain.q.out b/ql/src/test/results/clientpositive/llap/excluded_rule_explain.q.out
index 0ada0c3113d..537c37dc649 100644
--- a/ql/src/test/results/clientpositive/llap/excluded_rule_explain.q.out
+++ b/ql/src/test/results/clientpositive/llap/excluded_rule_explain.q.out
@@ -18,7 +18,7 @@ POSTHOOK: query: EXPLAIN FORMATTED CBO SELECT 1
POSTHOOK: type: QUERY
POSTHOOK: Input: _dummy_database@_dummy_table
#### A masked pattern was here ####
-{"CBOPlan":"{\n \"rels\": [\n {\n \"id\": \"0\",\n \"relOp\": \"org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveTableScan\",\n \"table\": [\n \"_dummy_database\",\n \"_dummy_table\"\n ],\n \"table:alias\": \"_dummy_table\",\n \"inputs\": [],\n \"rowCount\": 1.0,\n \"avgRowSize\": 129.0,\n \"rowType\": [\n {\n \"type\": \"BIGINT\",\n \"nullable\": true,\n \"name\": \"BLOCK__OFFSET [...]
+{"CBOPlan":"{\n \"rels\": [\n {\n \"id\": \"0\",\n \"relOp\": \"org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveTableScan\",\n \"table\": [\n \"_dummy_database\",\n \"_dummy_table\"\n ],\n \"table:alias\": \"_dummy_table\",\n \"inputs\": [],\n \"rowCount\": 1.0,\n \"avgRowSize\": 129.0,\n \"rowType\": {\n \"fields\": [\n {\n \"type\": \"BIGINT\",\n \"nullable\": true,\n [...]
PREHOOK: query: EXPLAIN SELECT 1
PREHOOK: type: QUERY
PREHOOK: Input: _dummy_database@_dummy_table
@@ -51,7 +51,7 @@ POSTHOOK: query: EXPLAIN FORMATTED SELECT 1
POSTHOOK: type: QUERY
POSTHOOK: Input: _dummy_database@_dummy_table
#### A masked pattern was here ####
-{"CBOPlan":"{\n \"rels\": [\n {\n \"id\": \"0\",\n \"relOp\": \"org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveTableScan\",\n \"table\": [\n \"_dummy_database\",\n \"_dummy_table\"\n ],\n \"table:alias\": \"_dummy_table\",\n \"inputs\": [],\n \"rowCount\": 1.0,\n \"avgRowSize\": 0.0,\n \"rowType\": [\n {\n \"type\": \"BIGINT\",\n \"nullable\": true,\n \"name\": \"BLOCK__OFFSET__ [...]
+{"CBOPlan":"{\n \"rels\": [\n {\n \"id\": \"0\",\n \"relOp\": \"org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveTableScan\",\n \"table\": [\n \"_dummy_database\",\n \"_dummy_table\"\n ],\n \"table:alias\": \"_dummy_table\",\n \"inputs\": [],\n \"rowCount\": 1.0,\n \"avgRowSize\": 0.0,\n \"rowType\": {\n \"fields\": [\n {\n \"type\": \"BIGINT\",\n \"nullable\": true,\n [...]
PREHOOK: query: EXPLAIN CBO SELECT 1
PREHOOK: type: QUERY
PREHOOK: Input: _dummy_database@_dummy_table
@@ -74,7 +74,7 @@ POSTHOOK: query: EXPLAIN FORMATTED CBO SELECT 1
POSTHOOK: type: QUERY
POSTHOOK: Input: _dummy_database@_dummy_table
#### A masked pattern was here ####
-{"CBOPlan":"{\n \"rels\": [\n {\n \"id\": \"0\",\n \"relOp\": \"org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveTableScan\",\n \"table\": [\n \"_dummy_database\",\n \"_dummy_table\"\n ],\n \"table:alias\": \"_dummy_table\",\n \"inputs\": [],\n \"rowCount\": 1.0,\n \"avgRowSize\": 129.0,\n \"rowType\": [\n {\n \"type\": \"BIGINT\",\n \"nullable\": true,\n \"name\": \"BLOCK__OFFSET [...]
+{"CBOPlan":"{\n \"rels\": [\n {\n \"id\": \"0\",\n \"relOp\": \"org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveTableScan\",\n \"table\": [\n \"_dummy_database\",\n \"_dummy_table\"\n ],\n \"table:alias\": \"_dummy_table\",\n \"inputs\": [],\n \"rowCount\": 1.0,\n \"avgRowSize\": 129.0,\n \"rowType\": {\n \"fields\": [\n {\n \"type\": \"BIGINT\",\n \"nullable\": true,\n [...]
PREHOOK: query: EXPLAIN SELECT 1
PREHOOK: type: QUERY
PREHOOK: Input: _dummy_database@_dummy_table
@@ -109,4 +109,4 @@ POSTHOOK: query: EXPLAIN FORMATTED SELECT 1
POSTHOOK: type: QUERY
POSTHOOK: Input: _dummy_database@_dummy_table
#### A masked pattern was here ####
-{"CBOPlan":"{\n \"rels\": [\n {\n \"id\": \"0\",\n \"relOp\": \"org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveTableScan\",\n \"table\": [\n \"_dummy_database\",\n \"_dummy_table\"\n ],\n \"table:alias\": \"_dummy_table\",\n \"inputs\": [],\n \"rowCount\": 1.0,\n \"avgRowSize\": 0.0,\n \"rowType\": [\n {\n \"type\": \"BIGINT\",\n \"nullable\": true,\n \"name\": \"BLOCK__OFFSET__ [...]
+{"CBOPlan":"{\n \"rels\": [\n {\n \"id\": \"0\",\n \"relOp\": \"org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveTableScan\",\n \"table\": [\n \"_dummy_database\",\n \"_dummy_table\"\n ],\n \"table:alias\": \"_dummy_table\",\n \"inputs\": [],\n \"rowCount\": 1.0,\n \"avgRowSize\": 0.0,\n \"rowType\": {\n \"fields\": [\n {\n \"type\": \"BIGINT\",\n \"nullable\": true,\n [...]
diff --git a/ql/src/test/results/clientpositive/llap/explainuser_1.q.out b/ql/src/test/results/clientpositive/llap/explainuser_1.q.out
index 8a94de87377..8c7d36516a3 100644
--- a/ql/src/test/results/clientpositive/llap/explainuser_1.q.out
+++ b/ql/src/test/results/clientpositive/llap/explainuser_1.q.out
@@ -2681,7 +2681,7 @@ Stage-0
Select Operator [SEL_23] (rows=21 width=223)
Output:["_col0","_col1","_col2"]
Filter Operator [FIL_22] (rows=21 width=239)
- predicate:(_col4 is null or (_col4 = 0L) or (_col7 is not null or _col0 is null or (_col5 < _col4)) is not true)
+ predicate:(_col4 is null or (_col4 = 0L) or (_col0 is null or _col7 is not null or (_col5 < _col4)) is not true)
Merge Join Operator [MERGEJOIN_50] (rows=31 width=239)
Conds:RS_19._col0, _col1=RS_20._col0, _col2(Left Outer),Output:["_col0","_col1","_col2","_col4","_col5","_col7"]
<-Reducer 2 [SIMPLE_EDGE] llap
@@ -2858,7 +2858,7 @@ Stage-0
Select Operator [SEL_33] (rows=7 width=106)
Output:["_col0","_col1"]
Filter Operator [FIL_32] (rows=7 width=114)
- predicate:(_col3 is null or (_col3 = 0L) or (_col6 is not null or _col0 is null or (_col4 < _col3)) is not true)
+ predicate:(_col3 is null or (_col3 = 0L) or (_col0 is null or _col6 is not null or (_col4 < _col3)) is not true)
Merge Join Operator [MERGEJOIN_48] (rows=7 width=114)
Conds:RS_29._col0, _col1=RS_30._col0, _col2(Left Outer),Output:["_col0","_col1","_col3","_col4","_col6"]
<-Reducer 3 [SIMPLE_EDGE] llap
diff --git a/ql/src/test/results/clientpositive/llap/explainuser_4.q.out b/ql/src/test/results/clientpositive/llap/explainuser_4.q.out
index df20fd6f29f..1e2424db1b0 100644
--- a/ql/src/test/results/clientpositive/llap/explainuser_4.q.out
+++ b/ql/src/test/results/clientpositive/llap/explainuser_4.q.out
@@ -30,28 +30,28 @@ Stage-0
Stage-1
Reducer 3 vectorized, llap
File Output Operator [FS_35]
- Select Operator [SEL_34] (rows=4626 width=528)
+ Select Operator [SEL_34] (rows=3453 width=528)
Output:["_col0","_col1","_col2","_col3","_col4","_col5","_col6","_col7","_col8","_col9","_col10","_col11","_col12","_col13","_col14","_col15","_col16","_col17","_col18","_col19","_col20","_col21","_col22","_col23"]
<-Reducer 2 [SIMPLE_EDGE] llap
SHUFFLE [RS_10]
- Merge Join Operator [MERGEJOIN_27] (rows=4626 width=528)
+ Merge Join Operator [MERGEJOIN_27] (rows=3453 width=528)
Conds:RS_30._col2=RS_33._col2(Inner),Output:["_col0","_col1","_col2","_col3","_col4","_col5","_col6","_col7","_col8","_col9","_col10","_col11","_col12","_col13","_col14","_col15","_col16","_col17","_col18","_col19","_col20","_col21","_col22","_col23"]
<-Map 1 [SIMPLE_EDGE] vectorized, llap
SHUFFLE [RS_30]
PartitionCols:_col2
- Select Operator [SEL_29] (rows=3078 width=231)
+ Select Operator [SEL_29] (rows=2298 width=231)
Output:["_col0","_col1","_col2","_col3","_col4","_col5","_col6","_col7","_col8","_col9","_col10","_col11"]
- Filter Operator [FIL_28] (rows=3078 width=231)
- predicate:cint BETWEEN 1000000 AND 3000000
+ Filter Operator [FIL_28] (rows=2298 width=231)
+ predicate:(cint BETWEEN 1000000 AND 3000000 and cint is not null)
TableScan [TS_0] (rows=12288 width=231)
default@alltypesorc,a,Tbl:COMPLETE,Col:COMPLETE,Output:["ctinyint","csmallint","cint","cbigint","cfloat","cdouble","cstring1","cstring2","ctimestamp1","ctimestamp2","cboolean1","cboolean2"]
<-Map 4 [SIMPLE_EDGE] vectorized, llap
SHUFFLE [RS_33]
PartitionCols:_col2
- Select Operator [SEL_32] (rows=2298 width=231)
+ Select Operator [SEL_32] (rows=1716 width=231)
Output:["_col0","_col1","_col2","_col3","_col4","_col5","_col6","_col7","_col8","_col9","_col10","_col11"]
- Filter Operator [FIL_31] (rows=2298 width=231)
- predicate:(cint BETWEEN 1000000 AND 3000000 and cbigint is not null)
+ Filter Operator [FIL_31] (rows=1716 width=231)
+ predicate:(cint BETWEEN 1000000 AND 3000000 and cint is not null and cbigint is not null)
TableScan [TS_3] (rows=12288 width=231)
default@alltypesorc,b,Tbl:COMPLETE,Col:COMPLETE,Output:["ctinyint","csmallint","cint","cbigint","cfloat","cdouble","cstring1","cstring2","ctimestamp1","ctimestamp2","cboolean1","cboolean2"]
@@ -119,24 +119,24 @@ Stage-0
PARTITION_ONLY_SHUFFLE [RS_11]
Group By Operator [GBY_10] (rows=1 width=8)
Output:["_col0"],aggregations:["count()"]
- Merge Join Operator [MERGEJOIN_29] (rows=4626 width=8)
+ Merge Join Operator [MERGEJOIN_29] (rows=3453 width=8)
Conds:RS_32._col0=RS_35._col0(Inner)
<-Map 1 [SIMPLE_EDGE] vectorized, llap
SHUFFLE [RS_32]
PartitionCols:_col0
- Select Operator [SEL_31] (rows=3078 width=2)
+ Select Operator [SEL_31] (rows=2298 width=2)
Output:["_col0"]
- Filter Operator [FIL_30] (rows=3078 width=2)
- predicate:cint BETWEEN 1000000 AND 3000000
+ Filter Operator [FIL_30] (rows=2298 width=2)
+ predicate:(cint BETWEEN 1000000 AND 3000000 and cint is not null)
TableScan [TS_0] (rows=12288 width=2)
default@alltypesorc,a,Tbl:COMPLETE,Col:COMPLETE,Output:["cint"]
<-Map 4 [SIMPLE_EDGE] vectorized, llap
SHUFFLE [RS_35]
PartitionCols:_col0
- Select Operator [SEL_34] (rows=2298 width=2)
+ Select Operator [SEL_34] (rows=1716 width=2)
Output:["_col0"]
- Filter Operator [FIL_33] (rows=2298 width=8)
- predicate:(cint BETWEEN 1000000 AND 3000000 and cbigint is not null)
+ Filter Operator [FIL_33] (rows=1716 width=8)
+ predicate:(cint BETWEEN 1000000 AND 3000000 and cint is not null and cbigint is not null)
TableScan [TS_3] (rows=12288 width=8)
default@alltypesorc,b,Tbl:COMPLETE,Col:COMPLETE,Output:["cint","cbigint"]
@@ -192,35 +192,35 @@ Stage-0
Stage-1
Reducer 4 vectorized, llap
File Output Operator [FS_41]
- Select Operator [SEL_40] (rows=3078 width=11)
+ Select Operator [SEL_40] (rows=2298 width=11)
Output:["_col0","_col1"]
<-Reducer 3 [SIMPLE_EDGE] vectorized, llap
SHUFFLE [RS_39]
- Group By Operator [GBY_38] (rows=3078 width=11)
+ Group By Operator [GBY_38] (rows=2298 width=11)
Output:["_col0","_col1"],aggregations:["count(VALUE._col0)"],keys:KEY._col0
<-Reducer 2 [SIMPLE_EDGE] llap
SHUFFLE [RS_11]
PartitionCols:_col0
- Group By Operator [GBY_10] (rows=3078 width=11)
+ Group By Operator [GBY_10] (rows=2298 width=11)
Output:["_col0","_col1"],aggregations:["count()"],keys:_col0
- Merge Join Operator [MERGEJOIN_31] (rows=4626 width=3)
+ Merge Join Operator [MERGEJOIN_31] (rows=3453 width=3)
Conds:RS_34._col1=RS_37._col0(Inner),Output:["_col0"]
<-Map 1 [SIMPLE_EDGE] vectorized, llap
SHUFFLE [RS_34]
PartitionCols:_col1
- Select Operator [SEL_33] (rows=3078 width=5)
+ Select Operator [SEL_33] (rows=2298 width=5)
Output:["_col0","_col1"]
- Filter Operator [FIL_32] (rows=3078 width=5)
- predicate:cint BETWEEN 1000000 AND 3000000
+ Filter Operator [FIL_32] (rows=2298 width=5)
+ predicate:(cint BETWEEN 1000000 AND 3000000 and cint is not null)
TableScan [TS_0] (rows=12288 width=5)
default@alltypesorc,a,Tbl:COMPLETE,Col:COMPLETE,Output:["csmallint","cint"]
<-Map 5 [SIMPLE_EDGE] vectorized, llap
SHUFFLE [RS_37]
PartitionCols:_col0
- Select Operator [SEL_36] (rows=2298 width=2)
+ Select Operator [SEL_36] (rows=1716 width=2)
Output:["_col0"]
- Filter Operator [FIL_35] (rows=2298 width=8)
- predicate:(cint BETWEEN 1000000 AND 3000000 and cbigint is not null)
+ Filter Operator [FIL_35] (rows=1716 width=8)
+ predicate:(cint BETWEEN 1000000 AND 3000000 and cint is not null and cbigint is not null)
TableScan [TS_3] (rows=12288 width=8)
default@alltypesorc,b,Tbl:COMPLETE,Col:COMPLETE,Output:["cint","cbigint"]
@@ -294,7 +294,7 @@ Stage-0
Select Operator [SEL_34] (rows=1365 width=240)
Output:["_col0","_col1","_col2","_col3","_col4","_col5","_col6","_col7","_col8","_col9","_col10","_col11"]
Filter Operator [FIL_32] (rows=1365 width=240)
- predicate:(cint BETWEEN 1000000 AND 3000000 and cbigint is not null)
+ predicate:(cint BETWEEN 1000000 AND 3000000 and cint is not null and cbigint is not null)
TableScan [TS_0] (rows=12288 width=240)
default@alltypesorc,a,Tbl:COMPLETE,Col:NONE,Output:["ctinyint","csmallint","cint","cbigint","cfloat","cdouble","cstring1","cstring2","ctimestamp1","ctimestamp2","cboolean1","cboolean2"]
<-Reducer 2 [CUSTOM_SIMPLE_EDGE] vectorized, llap
@@ -376,7 +376,7 @@ Stage-0
Select Operator [SEL_36] (rows=1365 width=240)
Output:["_col0"]
Filter Operator [FIL_34] (rows=1365 width=240)
- predicate:(cint BETWEEN 1000000 AND 3000000 and cbigint is not null)
+ predicate:(cint BETWEEN 1000000 AND 3000000 and cint is not null and cbigint is not null)
TableScan [TS_0] (rows=12288 width=240)
default@alltypesorc,a,Tbl:COMPLETE,Col:NONE,Output:["cint","cbigint"]
<-Reducer 2 [CUSTOM_SIMPLE_EDGE] vectorized, llap
@@ -457,7 +457,7 @@ Stage-0
Select Operator [SEL_38] (rows=1365 width=240)
Output:["_col0"]
Filter Operator [FIL_36] (rows=1365 width=240)
- predicate:(cint BETWEEN 1000000 AND 3000000 and cbigint is not null)
+ predicate:(cint BETWEEN 1000000 AND 3000000 and cint is not null and cbigint is not null)
TableScan [TS_0] (rows=12288 width=240)
default@alltypesorc,a,Tbl:COMPLETE,Col:NONE,Output:["csmallint","cint","cbigint"]
<-Reducer 2 [CUSTOM_SIMPLE_EDGE] vectorized, llap
diff --git a/ql/src/test/results/clientpositive/llap/external_jdbc_table2.q.out b/ql/src/test/results/clientpositive/llap/external_jdbc_table2.q.out
index f6b2e175825..895591440c1 100644
--- a/ql/src/test/results/clientpositive/llap/external_jdbc_table2.q.out
+++ b/ql/src/test/results/clientpositive/llap/external_jdbc_table2.q.out
@@ -470,14 +470,14 @@ FROM "EXTERNAL_JDBC_SIMPLE_DERBY2_TABLE1"
TableScan
alias: db1_ext_auth1
properties:
- hive.sql.query SELECT "ikey" * 2 AS "*"
+ hive.sql.query SELECT "ikey" * 2 AS "_o__c1"
FROM "EXTERNAL_JDBC_SIMPLE_DERBY2_TABLE1"
- hive.sql.query.fieldNames *
+ hive.sql.query.fieldNames _o__c1
hive.sql.query.fieldTypes int
hive.sql.query.split true
Statistics: Num rows: 1 Data size: 4 Basic stats: COMPLETE Column stats: NONE
Select Operator
- expressions: * (type: int)
+ expressions: _o__c1 (type: int)
outputColumnNames: _col0
Statistics: Num rows: 1 Data size: 4 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
@@ -716,7 +716,7 @@ STAGE PLANS:
hive.sql.query SELECT "ikey"
FROM (SELECT "ikey", "bkey", "dkey"
FROM "EXTERNAL_JDBC_SIMPLE_DERBY2_TABLE1") AS "t"
-WHERE ("bkey" = 10 AND "dkey" = 15.15 OR "bkey" = 20 AND "dkey" = 25.25) AND "bkey" IN (10, 20) AND ("dkey" IN (15.15, 25.25) AND "ikey" IS NOT NULL)
+WHERE ("bkey" = 10 AND "dkey" = 15.15 OR "bkey" = 20 AND "dkey" = 25.25) AND "bkey" IN (10, 20) AND "dkey" IN (15.15, 25.25) AND "ikey" IS NOT NULL
hive.sql.query.fieldNames ikey
hive.sql.query.fieldTypes int
hive.sql.query.split true
diff --git a/ql/src/test/results/clientpositive/llap/external_jdbc_table4.q.out b/ql/src/test/results/clientpositive/llap/external_jdbc_table4.q.out
index 29633e02a19..76abbd5d91a 100644
--- a/ql/src/test/results/clientpositive/llap/external_jdbc_table4.q.out
+++ b/ql/src/test/results/clientpositive/llap/external_jdbc_table4.q.out
@@ -470,14 +470,14 @@ FROM "EXTERNAL_JDBC_SIMPLE_DERBY2_TABLE1"
TableScan
alias: db1_ext_auth1
properties:
- hive.sql.query SELECT "IKEY" * 2 AS "*"
+ hive.sql.query SELECT "IKEY" * 2 AS "_o__c1"
FROM "EXTERNAL_JDBC_SIMPLE_DERBY2_TABLE1"
- hive.sql.query.fieldNames *
+ hive.sql.query.fieldNames _o__c1
hive.sql.query.fieldTypes int
hive.sql.query.split true
Statistics: Num rows: 1 Data size: 4 Basic stats: COMPLETE Column stats: NONE
Select Operator
- expressions: * (type: int)
+ expressions: _o__c1 (type: int)
outputColumnNames: _col0
Statistics: Num rows: 1 Data size: 4 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
diff --git a/ql/src/test/results/clientpositive/llap/filter_cond_pushdown.q.out b/ql/src/test/results/clientpositive/llap/filter_cond_pushdown.q.out
index 9095a83cf48..f26fe1dd794 100644
--- a/ql/src/test/results/clientpositive/llap/filter_cond_pushdown.q.out
+++ b/ql/src/test/results/clientpositive/llap/filter_cond_pushdown.q.out
@@ -196,10 +196,10 @@ STAGE PLANS:
Map Operator Tree:
TableScan
alias: f
- filterExpr: ((value) IN ('2008-04-08', '2008-04-10', '2008-04-09') and key is not null) (type: boolean)
+ filterExpr: ((value) IN ('2008-04-08', '2008-04-09', '2008-04-10') and key is not null) (type: boolean)
Statistics: Num rows: 500 Data size: 89000 Basic stats: COMPLETE Column stats: COMPLETE
Filter Operator
- predicate: ((value) IN ('2008-04-08', '2008-04-10', '2008-04-09') and key is not null) (type: boolean)
+ predicate: ((value) IN ('2008-04-08', '2008-04-09', '2008-04-10') and key is not null) (type: boolean)
Statistics: Num rows: 5 Data size: 890 Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
expressions: key (type: string), (value) IN ('2008-04-08', '2008-04-10') (type: boolean), (value = '2008-04-09') (type: boolean)
@@ -455,26 +455,26 @@ STAGE PLANS:
predicate: key is not null (type: boolean)
Statistics: Num rows: 500 Data size: 89000 Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
- expressions: key (type: string), value (type: string), (value) IN ('2008-04-08', '2008-04-10') (type: boolean), (value) IN ('2008-04-08', '2008-04-09') (type: boolean)
- outputColumnNames: _col0, _col1, _col2, _col3
- Statistics: Num rows: 500 Data size: 93000 Basic stats: COMPLETE Column stats: COMPLETE
+ expressions: key (type: string), value (type: string), (value = '2008-04-08') (type: boolean)
+ outputColumnNames: _col0, _col1, _col2
+ Statistics: Num rows: 500 Data size: 91000 Basic stats: COMPLETE Column stats: COMPLETE
Reduce Output Operator
key expressions: _col0 (type: string)
null sort order: z
sort order: +
Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 500 Data size: 93000 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions: _col1 (type: string), _col2 (type: boolean), _col3 (type: boolean)
+ Statistics: Num rows: 500 Data size: 91000 Basic stats: COMPLETE Column stats: COMPLETE
+ value expressions: _col1 (type: string), _col2 (type: boolean)
Execution mode: vectorized, llap
LLAP IO: all inputs
Map 3
Map Operator Tree:
TableScan
alias: m
- filterExpr: ((value) IN ('2008-04-10', '2008-04-08') and (value <> '') and key is not null) (type: boolean)
+ filterExpr: ((value) IN ('2008-04-08', '2008-04-10') and key is not null) (type: boolean)
Statistics: Num rows: 500 Data size: 89000 Basic stats: COMPLETE Column stats: COMPLETE
Filter Operator
- predicate: ((value) IN ('2008-04-10', '2008-04-08') and (value <> '') and key is not null) (type: boolean)
+ predicate: ((value) IN ('2008-04-08', '2008-04-10') and key is not null) (type: boolean)
Statistics: Num rows: 3 Data size: 534 Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
expressions: key (type: string), value (type: string), (value = '2008-04-10') (type: boolean), (value = '2008-04-08') (type: boolean)
@@ -498,16 +498,16 @@ STAGE PLANS:
keys:
0 _col0 (type: string)
1 _col0 (type: string)
- outputColumnNames: _col0, _col1, _col2, _col3, _col5, _col6, _col7
- residual filter predicates: {((_col2 and _col3 and _col6) or _col7)}
- Statistics: Num rows: 2 Data size: 570 Basic stats: COMPLETE Column stats: COMPLETE
+ outputColumnNames: _col0, _col1, _col2, _col4, _col5, _col6
+ residual filter predicates: {((_col2 and _col5) or _col6)}
+ Statistics: Num rows: 3 Data size: 843 Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
- expressions: _col0 (type: string), _col1 (type: string), _col5 (type: string)
+ expressions: _col0 (type: string), _col1 (type: string), _col4 (type: string)
outputColumnNames: _col0, _col1, _col2
- Statistics: Num rows: 2 Data size: 538 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 3 Data size: 807 Basic stats: COMPLETE Column stats: COMPLETE
File Output Operator
compressed: false
- Statistics: Num rows: 2 Data size: 538 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 3 Data size: 807 Basic stats: COMPLETE Column stats: COMPLETE
table:
input format: org.apache.hadoop.mapred.SequenceFileInputFormat
output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
diff --git a/ql/src/test/results/clientpositive/llap/filter_join_breaktask.q.out b/ql/src/test/results/clientpositive/llap/filter_join_breaktask.q.out
index ac75c368d38..be6918f632a 100644
--- a/ql/src/test/results/clientpositive/llap/filter_join_breaktask.q.out
+++ b/ql/src/test/results/clientpositive/llap/filter_join_breaktask.q.out
@@ -42,7 +42,7 @@ INNER JOIN ((SELECT `key`
FROM `default`.`filter_join_breaktask`
WHERE `key` IS NOT NULL AND `ds` = '2008-04-08') AS `t2` INNER JOIN (SELECT `key`, `value`
FROM `default`.`filter_join_breaktask`
-WHERE `key` IS NOT NULL AND (`value` <> '' AND `ds` = '2008-04-08')) AS `t4` ON `t2`.`key` = `t4`.`key`) ON `t0`.`value` = `t4`.`value`
+WHERE `key` IS NOT NULL AND `value` <> '' AND `ds` = '2008-04-08') AS `t4` ON `t2`.`key` = `t4`.`key`) ON `t0`.`value` = `t4`.`value`
STAGE DEPENDENCIES:
Stage-1 is a root stage
Stage-0 depends on stages: Stage-1
diff --git a/ql/src/test/results/clientpositive/llap/filter_numeric.q.out b/ql/src/test/results/clientpositive/llap/filter_numeric.q.out
index 77f680570ff..4702fd6a8cf 100644
--- a/ql/src/test/results/clientpositive/llap/filter_numeric.q.out
+++ b/ql/src/test/results/clientpositive/llap/filter_numeric.q.out
@@ -80,7 +80,7 @@ STAGE PLANS:
Processor Tree:
TableScan
alias: partint
- filterExpr: ((hr <= 12) and (hr > 11)) (type: boolean)
+ filterExpr: ((hr > 11) and (hr <= 12)) (type: boolean)
Select Operator
expressions: key (type: string), value (type: string), hr (type: int)
outputColumnNames: _col0, _col1, _col2
@@ -1658,7 +1658,7 @@ STAGE PLANS:
Processor Tree:
TableScan
alias: partint
- filterExpr: hr NOT BETWEEN 12 AND 14 (type: boolean)
+ filterExpr: ((hr < 12) or (hr > 14)) (type: boolean)
Select Operator
expressions: key (type: string), value (type: string), hr (type: int)
outputColumnNames: _col0, _col1, _col2
diff --git a/ql/src/test/results/clientpositive/llap/folder_predicate.q.out b/ql/src/test/results/clientpositive/llap/folder_predicate.q.out
index 1e67ce4271a..f8b2ef3663e 100644
--- a/ql/src/test/results/clientpositive/llap/folder_predicate.q.out
+++ b/ql/src/test/results/clientpositive/llap/folder_predicate.q.out
@@ -41,9 +41,9 @@ STAGE PLANS:
Processor Tree:
TableScan
alias: predicate_fold_tb
- filterExpr: ((value <> 3) or value is null) (type: boolean)
+ filterExpr: (value is null or (value < 3) or (value > 3)) (type: boolean)
Filter Operator
- predicate: ((value <> 3) or value is null) (type: boolean)
+ predicate: (value is null or (value < 3) or (value > 3)) (type: boolean)
Select Operator
expressions: value (type: int)
outputColumnNames: _col0
diff --git a/ql/src/test/results/clientpositive/llap/fouter_join_ppr.q.out b/ql/src/test/results/clientpositive/llap/fouter_join_ppr.q.out
index d8e0b6814d1..23c86149d91 100644
--- a/ql/src/test/results/clientpositive/llap/fouter_join_ppr.q.out
+++ b/ql/src/test/results/clientpositive/llap/fouter_join_ppr.q.out
@@ -29,10 +29,10 @@ POSTHOOK: Input: default@srcpart@ds=2008-04-08/hr=12
OPTIMIZED SQL: SELECT *
FROM (SELECT `key`, `value`
FROM `default`.`src`
-WHERE `key` < 20 AND `key` > 15) AS `t0`
+WHERE `key` > 15 AND `key` < 20 AND `key` IS NOT NULL) AS `t0`
INNER JOIN (SELECT `key`, `value`
FROM `default`.`srcpart`
-WHERE `key` > 15 AND (`key` < 20 AND `ds` = '2008-04-08')) AS `t2` ON `t0`.`key` = `t2`.`key`
+WHERE `key` > 15 AND `key` < 20 AND `ds` = '2008-04-08' AND `key` IS NOT NULL) AS `t2` ON `t0`.`key` = `t2`.`key`
STAGE DEPENDENCIES:
Stage-1 is a root stage
Stage-0 depends on stages: Stage-1
@@ -49,12 +49,12 @@ STAGE PLANS:
Map Operator Tree:
TableScan
alias: a
- filterExpr: ((UDFToDouble(key) < 20.0D) and (UDFToDouble(key) > 15.0D)) (type: boolean)
+ filterExpr: ((UDFToDouble(key) > 15.0D) and (UDFToDouble(key) < 20.0D)) (type: boolean)
Statistics: Num rows: 500 Data size: 89000 Basic stats: COMPLETE Column stats: COMPLETE
GatherStats: false
Filter Operator
isSamplingPred: false
- predicate: ((UDFToDouble(key) < 20.0D) and (UDFToDouble(key) > 15.0D)) (type: boolean)
+ predicate: ((UDFToDouble(key) > 15.0D) and (UDFToDouble(key) < 20.0D)) (type: boolean)
Statistics: Num rows: 55 Data size: 9790 Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
expressions: key (type: string), value (type: string)
@@ -331,10 +331,10 @@ POSTHOOK: Input: default@srcpart@ds=2008-04-08/hr=12
OPTIMIZED SQL: SELECT *
FROM (SELECT `key`, `value`
FROM `default`.`srcpart`
-WHERE `key` < 20 AND (`key` > 15 AND `ds` = '2008-04-08')) AS `t0`
+WHERE `key` > 15 AND `key` < 20 AND `ds` = '2008-04-08' AND `key` IS NOT NULL) AS `t0`
INNER JOIN (SELECT `key`, `value`
FROM `default`.`src`
-WHERE `key` > 15 AND `key` < 20) AS `t2` ON `t0`.`key` = `t2`.`key`
+WHERE `key` > 15 AND `key` < 20 AND `key` IS NOT NULL) AS `t2` ON `t0`.`key` = `t2`.`key`
STAGE DEPENDENCIES:
Stage-1 is a root stage
Stage-0 depends on stages: Stage-1
@@ -351,12 +351,12 @@ STAGE PLANS:
Map Operator Tree:
TableScan
alias: a
- filterExpr: ((UDFToDouble(key) < 20.0D) and (UDFToDouble(key) > 15.0D)) (type: boolean)
+ filterExpr: ((UDFToDouble(key) > 15.0D) and (UDFToDouble(key) < 20.0D)) (type: boolean)
Statistics: Num rows: 1000 Data size: 178000 Basic stats: COMPLETE Column stats: COMPLETE
GatherStats: false
Filter Operator
isSamplingPred: false
- predicate: ((UDFToDouble(key) < 20.0D) and (UDFToDouble(key) > 15.0D)) (type: boolean)
+ predicate: ((UDFToDouble(key) > 15.0D) and (UDFToDouble(key) < 20.0D)) (type: boolean)
Statistics: Num rows: 111 Data size: 19758 Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
expressions: key (type: string), value (type: string)
@@ -633,10 +633,10 @@ POSTHOOK: Input: default@srcpart@ds=2008-04-08/hr=12
OPTIMIZED SQL: SELECT *
FROM (SELECT `key`, `value`
FROM `default`.`src`
-WHERE `key` < 20 AND `key` > 15) AS `t0`
+WHERE `key` > 15 AND `key` < 20 AND `key` IS NOT NULL) AS `t0`
INNER JOIN (SELECT `key`, `value`
FROM `default`.`srcpart`
-WHERE `key` > 15 AND (`key` < 20 AND `ds` = '2008-04-08')) AS `t2` ON `t0`.`key` = `t2`.`key`
+WHERE `key` > 15 AND `key` < 20 AND `ds` = '2008-04-08' AND `key` IS NOT NULL) AS `t2` ON `t0`.`key` = `t2`.`key`
STAGE DEPENDENCIES:
Stage-1 is a root stage
Stage-0 depends on stages: Stage-1
@@ -653,12 +653,12 @@ STAGE PLANS:
Map Operator Tree:
TableScan
alias: a
- filterExpr: ((UDFToDouble(key) < 20.0D) and (UDFToDouble(key) > 15.0D)) (type: boolean)
+ filterExpr: ((UDFToDouble(key) > 15.0D) and (UDFToDouble(key) < 20.0D)) (type: boolean)
Statistics: Num rows: 500 Data size: 89000 Basic stats: COMPLETE Column stats: COMPLETE
GatherStats: false
Filter Operator
isSamplingPred: false
- predicate: ((UDFToDouble(key) < 20.0D) and (UDFToDouble(key) > 15.0D)) (type: boolean)
+ predicate: ((UDFToDouble(key) > 15.0D) and (UDFToDouble(key) < 20.0D)) (type: boolean)
Statistics: Num rows: 55 Data size: 9790 Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
expressions: key (type: string), value (type: string)
@@ -935,10 +935,10 @@ POSTHOOK: Input: default@srcpart@ds=2008-04-08/hr=12
OPTIMIZED SQL: SELECT *
FROM (SELECT `key`, `value`
FROM `default`.`srcpart`
-WHERE `key` < 20 AND (`key` > 15 AND `ds` = '2008-04-08')) AS `t0`
+WHERE `key` > 15 AND `key` < 20 AND `ds` = '2008-04-08' AND `key` IS NOT NULL) AS `t0`
INNER JOIN (SELECT `key`, `value`
FROM `default`.`src`
-WHERE `key` > 15 AND `key` < 20) AS `t2` ON `t0`.`key` = `t2`.`key`
+WHERE `key` > 15 AND `key` < 20 AND `key` IS NOT NULL) AS `t2` ON `t0`.`key` = `t2`.`key`
STAGE DEPENDENCIES:
Stage-1 is a root stage
Stage-0 depends on stages: Stage-1
@@ -955,12 +955,12 @@ STAGE PLANS:
Map Operator Tree:
TableScan
alias: a
- filterExpr: ((UDFToDouble(key) < 20.0D) and (UDFToDouble(key) > 15.0D)) (type: boolean)
+ filterExpr: ((UDFToDouble(key) > 15.0D) and (UDFToDouble(key) < 20.0D)) (type: boolean)
Statistics: Num rows: 1000 Data size: 178000 Basic stats: COMPLETE Column stats: COMPLETE
GatherStats: false
Filter Operator
isSamplingPred: false
- predicate: ((UDFToDouble(key) < 20.0D) and (UDFToDouble(key) > 15.0D)) (type: boolean)
+ predicate: ((UDFToDouble(key) > 15.0D) and (UDFToDouble(key) < 20.0D)) (type: boolean)
Statistics: Num rows: 111 Data size: 19758 Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
expressions: key (type: string), value (type: string)
diff --git a/ql/src/test/results/clientpositive/llap/groupby_position.q.out b/ql/src/test/results/clientpositive/llap/groupby_position.q.out
index 86c39ea02a4..a958be32ca4 100644
--- a/ql/src/test/results/clientpositive/llap/groupby_position.q.out
+++ b/ql/src/test/results/clientpositive/llap/groupby_position.q.out
@@ -752,10 +752,10 @@ STAGE PLANS:
Map Operator Tree:
TableScan
alias: src1
- filterExpr: ((UDFToDouble(key) < 20.0D) and (UDFToDouble(key) > 15.0D)) (type: boolean)
+ filterExpr: ((UDFToDouble(key) > 15.0D) and (UDFToDouble(key) < 20.0D)) (type: boolean)
Statistics: Num rows: 500 Data size: 89000 Basic stats: COMPLETE Column stats: COMPLETE
Filter Operator
- predicate: ((UDFToDouble(key) < 20.0D) and (UDFToDouble(key) > 15.0D)) (type: boolean)
+ predicate: ((UDFToDouble(key) > 15.0D) and (UDFToDouble(key) < 20.0D)) (type: boolean)
Statistics: Num rows: 55 Data size: 9790 Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
expressions: key (type: string), value (type: string), substr(value, 5) (type: string)
diff --git a/ql/src/test/results/clientpositive/llap/groupby_rollup_empty2.q.out b/ql/src/test/results/clientpositive/llap/groupby_rollup_empty2.q.out
index 3e1d1c010e0..ea107cc1fd8 100644
--- a/ql/src/test/results/clientpositive/llap/groupby_rollup_empty2.q.out
+++ b/ql/src/test/results/clientpositive/llap/groupby_rollup_empty2.q.out
@@ -98,10 +98,10 @@ STAGE PLANS:
Map Operator Tree:
TableScan
alias: store_s0
- filterExpr: ((s_state) IN ('SD', 'FL', 'MI', 'LA', 'MO', 'SC') and s_store_sk is not null) (type: boolean)
+ filterExpr: ((s_state) IN ('FL', 'LA', 'MI', 'MO', 'SC', 'SD') and s_store_sk is not null) (type: boolean)
Statistics: Num rows: 3 Data size: 270 Basic stats: COMPLETE Column stats: COMPLETE
Filter Operator
- predicate: ((s_state) IN ('SD', 'FL', 'MI', 'LA', 'MO', 'SC') and s_store_sk is not null) (type: boolean)
+ predicate: ((s_state) IN ('FL', 'LA', 'MI', 'MO', 'SC', 'SD') and s_store_sk is not null) (type: boolean)
Statistics: Num rows: 3 Data size: 270 Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
expressions: s_store_sk (type: int), s_state (type: string)
diff --git a/ql/src/test/results/clientpositive/llap/in_typecheck_char.q.out b/ql/src/test/results/clientpositive/llap/in_typecheck_char.q.out
index 86ff6a76f60..f963d68ba3e 100644
--- a/ql/src/test/results/clientpositive/llap/in_typecheck_char.q.out
+++ b/ql/src/test/results/clientpositive/llap/in_typecheck_char.q.out
@@ -388,7 +388,7 @@ POSTHOOK: Input: default@ax
#### A masked pattern was here ####
CBO PLAN:
HiveAggregate(group=[{}], agg#0=[count()])
- HiveFilter(condition=[OR(=($1, _UTF-16LE'a '), =($1, _UTF-16LE'bb '), =($1, _UTF-16LE'aa '), =($1, _UTF-16LE'bbb '), =($1, _UTF-16LE'ab '), =($1, _UTF-16LE'ba '), =($1, _UTF-16LE'aaa '), =($1, _UTF-16LE'abc '), =($1, _UTF-16LE'bc '), =($1, _UTF-16LE'ac '), =($1, _UTF-16LE'bca '), =($1, _UTF-16LE'cab '), =($1, _UTF-16LE'cb '), =($1, _UTF-16LE'ca '), =($1, _UTF-16LE'cbc '), =($1, _UTF-16LE [...]
+ HiveFilter(condition=[IN($1, _UTF-16LE'a', _UTF-16LE'aa', _UTF-16LE'aaa', _UTF-16LE'ab', _UTF-16LE'abc', _UTF-16LE'ac', _UTF-16LE'ba', _UTF-16LE'bb', _UTF-16LE'bbb', _UTF-16LE'bc', _UTF-16LE'bca', _UTF-16LE'ca', _UTF-16LE'cab', _UTF-16LE'cb', _UTF-16LE'cbc', _UTF-16LE'z')])
HiveTableScan(table=[[default, ax]], table:alias=[ax])
PREHOOK: query: explain select count(*) from ax where t in
@@ -417,10 +417,10 @@ STAGE PLANS:
Map Operator Tree:
TableScan
alias: ax
- filterExpr: ((t = 'a ') or (t = 'bb ') or (t = 'aa ') or (t = 'bbb ') or (t = 'ab ') or (t = 'ba ') or (t = 'aaa ') or (t = 'abc ') or (t = 'bc ') or (t = 'ac ') or (t = 'bca ') or (t = 'cab ') or (t = 'cb ') or (t = 'ca ') or (t = 'cbc ') or (t = 'z ')) (type: boolean)
+ filterExpr: (t) IN ('a ', 'aa ', 'aaa ', 'ab ', 'abc ', 'ac ', 'ba ', 'bb ', 'bbb ', 'bc ', 'bca ', 'ca ', 'cab ', 'cb ', 'cbc ', 'z ') (type: boolean)
Statistics: Num rows: 3 Data size: 255 Basic stats: COMPLETE Column stats: COMPLETE
Filter Operator
- predicate: ((t = 'a ') or (t = 'bb ') or (t = 'aa ') or (t = 'bbb ') or (t = 'ab ') or (t = 'ba ') or (t = 'aaa ') or (t = 'abc ') or (t = 'bc ') or (t = 'ac ') or (t = 'bca ') or (t = 'cab ') or (t = 'cb ') or (t = 'ca ') or (t = 'cbc ') or (t = 'z ')) (type: boolean)
+ predicate: (t) IN ('a ', 'aa ', 'aaa ', 'ab ', 'abc ', 'ac ', 'ba ', 'bb ', 'bbb ', 'bc ', 'bca ', 'ca ', 'cab ', 'cb ', 'cbc ', 'z ') (type: boolean)
Statistics: Num rows: 3 Data size: 255 Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
Statistics: Num rows: 3 Data size: 255 Basic stats: COMPLETE Column stats: COMPLETE
diff --git a/ql/src/test/results/clientpositive/llap/input4.q.out b/ql/src/test/results/clientpositive/llap/input4.q.out
index 2baa09185a9..8b0a826f6be 100644
--- a/ql/src/test/results/clientpositive/llap/input4.q.out
+++ b/ql/src/test/results/clientpositive/llap/input4.q.out
@@ -53,7 +53,7 @@ SELECT Input4Alias.VALUE, Input4Alias.KEY FROM INPUT4_n0 AS Input4Alias
POSTHOOK: type: QUERY
POSTHOOK: Input: default@input4_n0
#### A masked pattern was here ####
-{"CBOPlan":"{\n \"rels\": [\n {\n \"id\": \"0\",\n \"relOp\": \"org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveTableScan\",\n \"table\": [\n \"default\",\n \"input4_n0\"\n ],\n \"table:alias\": \"input4alias\",\n \"inputs\": [],\n \"rowCount\": 1.0,\n \"avgRowSize\": 200.0,\n \"rowType\": [\n {\n \"type\": \"VARCHAR\",\n \"nullable\": true,\n \"precision\": 2147483647,\n [...]
+{"CBOPlan":"{\n \"rels\": [\n {\n \"id\": \"0\",\n \"relOp\": \"org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveTableScan\",\n \"table\": [\n \"default\",\n \"input4_n0\"\n ],\n \"table:alias\": \"input4alias\",\n \"inputs\": [],\n \"rowCount\": 1.0,\n \"avgRowSize\": 200.0,\n \"rowType\": {\n \"fields\": [\n {\n \"type\": \"VARCHAR\",\n \"nullable\": true,\n \"pre [...]
PREHOOK: query: SELECT Input4Alias.VALUE, Input4Alias.KEY FROM INPUT4_n0 AS Input4Alias
PREHOOK: type: QUERY
PREHOOK: Input: default@input4_n0
diff --git a/ql/src/test/results/clientpositive/llap/input_part1.q.out b/ql/src/test/results/clientpositive/llap/input_part1.q.out
index 443d53286f1..3a69634c074 100644
--- a/ql/src/test/results/clientpositive/llap/input_part1.q.out
+++ b/ql/src/test/results/clientpositive/llap/input_part1.q.out
@@ -22,7 +22,7 @@ POSTHOOK: Input: default@srcpart@ds=2008-04-08/hr=12
POSTHOOK: Output: default@dest1_n45
OPTIMIZED SQL: SELECT `key`, `value`, CAST('12' AS STRING) AS `hr`, CAST('2008-04-08' AS STRING) AS `ds`
FROM `default`.`srcpart`
-WHERE `key` < 100 AND (`ds` = '2008-04-08' AND `hr` = '12')
+WHERE `key` < 100 AND `ds` = '2008-04-08' AND `hr` = '12'
STAGE DEPENDENCIES:
Stage-1 is a root stage
Stage-2 depends on stages: Stage-1
diff --git a/ql/src/test/results/clientpositive/llap/intersect_all_rj.q.out b/ql/src/test/results/clientpositive/llap/intersect_all_rj.q.out
index f26bfcb73ef..a5857239812 100644
--- a/ql/src/test/results/clientpositive/llap/intersect_all_rj.q.out
+++ b/ql/src/test/results/clientpositive/llap/intersect_all_rj.q.out
@@ -178,13 +178,13 @@ HiveProject($f0=[$1])
HiveUnion(all=[true])
HiveProject($f0=[$0], $f1=[$1])
HiveAggregate(group=[{0}], agg#0=[count()])
- HiveProject(CASE=[$0])
+ HiveProject($f0=[$0])
HiveAggregate(group=[{1}])
HiveJoin(condition=[>=($0, $2)], joinType=[inner], algorithm=[none], cost=[not available])
HiveProject(bigint_col_3=[$1])
HiveFilter(condition=[IS NOT NULL($1)])
HiveTableScan(table=[[default, table_7]], table:alias=[a3])
- HiveProject(CASE=[CASE(IS NOT NULL($2), $2, CASE($0, $3, $1))], CAST=[CAST($4):BIGINT])
+ HiveProject($f0=[CASE(IS NOT NULL($2), $2, CASE($0, $3, $1))], EXPR$1=[CAST($4):BIGINT])
HiveFilter(condition=[IS NOT NULL(CAST($4):BIGINT)])
HiveTableScan(table=[[default, table_10]], table:alias=[a4])
HiveProject($f0=[$0], $f1=[$1])
diff --git a/ql/src/test/results/clientpositive/llap/is_distinct_from.q.out b/ql/src/test/results/clientpositive/llap/is_distinct_from.q.out
index 11d674d6a0d..d5dc0dab82a 100644
--- a/ql/src/test/results/clientpositive/llap/is_distinct_from.q.out
+++ b/ql/src/test/results/clientpositive/llap/is_distinct_from.q.out
@@ -247,7 +247,7 @@ STAGE PLANS:
Filter Operator
predicate: (y IS NOT DISTINCT FROM null) (type: boolean)
Select Operator
- expressions: x (type: string), y (type: string)
+ expressions: x (type: string), null (type: string)
outputColumnNames: _col0, _col1
ListSink
@@ -349,7 +349,7 @@ STAGE PLANS:
Filter Operator
predicate: (p_size IS NOT DISTINCT FROM 2) (type: boolean)
Select Operator
- expressions: p_partkey (type: int), p_name (type: string), p_mfgr (type: string), p_brand (type: string), p_type (type: string), p_size (type: int), p_container (type: string), p_retailprice (type: double), p_comment (type: string)
+ expressions: p_partkey (type: int), p_name (type: string), p_mfgr (type: string), p_brand (type: string), p_type (type: string), 2 (type: int), p_container (type: string), p_retailprice (type: double), p_comment (type: string)
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8
ListSink
diff --git a/ql/src/test/results/clientpositive/llap/join21.q.out b/ql/src/test/results/clientpositive/llap/join21.q.out
index 1874b6a5a2b..6622e02766d 100644
--- a/ql/src/test/results/clientpositive/llap/join21.q.out
+++ b/ql/src/test/results/clientpositive/llap/join21.q.out
@@ -26,10 +26,10 @@ STAGE PLANS:
Map Operator Tree:
TableScan
alias: src1
- filterExpr: ((UDFToDouble(key) < 11.0D) and (UDFToDouble(key) > 9.0D)) (type: boolean)
+ filterExpr: ((UDFToDouble(key) > 9.0D) and (UDFToDouble(key) < 11.0D)) (type: boolean)
Statistics: Num rows: 500 Data size: 89000 Basic stats: COMPLETE Column stats: COMPLETE
Filter Operator
- predicate: ((UDFToDouble(key) < 11.0D) and (UDFToDouble(key) > 9.0D)) (type: boolean)
+ predicate: ((UDFToDouble(key) > 9.0D) and (UDFToDouble(key) < 11.0D)) (type: boolean)
Statistics: Num rows: 55 Data size: 9790 Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
expressions: key (type: string), value (type: string)
diff --git a/ql/src/test/results/clientpositive/llap/join26.q.out b/ql/src/test/results/clientpositive/llap/join26.q.out
index cfc85f09e49..19cb55a5e27 100644
--- a/ql/src/test/results/clientpositive/llap/join26.q.out
+++ b/ql/src/test/results/clientpositive/llap/join26.q.out
@@ -31,7 +31,7 @@ POSTHOOK: Output: default@dest_j1_n10
OPTIMIZED SQL: SELECT `t4`.`key`, `t0`.`value`, `t2`.`value` AS `value1`
FROM (SELECT `key`, `value`
FROM `default`.`srcpart`
-WHERE `ds` = '2008-04-08' AND (`hr` = 11 AND `key` IS NOT NULL)) AS `t0`
+WHERE `ds` = '2008-04-08' AND `hr` = 11 AND `key` IS NOT NULL) AS `t0`
INNER JOIN ((SELECT `key`, `value`
FROM `default`.`src`
WHERE `key` IS NOT NULL) AS `t2` INNER JOIN (SELECT `key`
diff --git a/ql/src/test/results/clientpositive/llap/join32.q.out b/ql/src/test/results/clientpositive/llap/join32.q.out
index f13184255ce..9a19fdd28f3 100644
--- a/ql/src/test/results/clientpositive/llap/join32.q.out
+++ b/ql/src/test/results/clientpositive/llap/join32.q.out
@@ -31,7 +31,7 @@ POSTHOOK: Output: default@dest_j1_n12
OPTIMIZED SQL: SELECT `t4`.`key`, `t0`.`value`, `t2`.`value` AS `value1`
FROM (SELECT `value`
FROM `default`.`srcpart`
-WHERE `ds` = '2008-04-08' AND (`hr` = 11 AND `value` IS NOT NULL)) AS `t0`
+WHERE `ds` = '2008-04-08' AND `hr` = 11 AND `value` IS NOT NULL) AS `t0`
INNER JOIN ((SELECT `key`, `value`
FROM `default`.`src`
WHERE `key` IS NOT NULL) AS `t2` INNER JOIN (SELECT `key`, `value`
diff --git a/ql/src/test/results/clientpositive/llap/join32_lessSize.q.out b/ql/src/test/results/clientpositive/llap/join32_lessSize.q.out
index 1ed079b1eab..cd28555ea90 100644
--- a/ql/src/test/results/clientpositive/llap/join32_lessSize.q.out
+++ b/ql/src/test/results/clientpositive/llap/join32_lessSize.q.out
@@ -39,7 +39,7 @@ POSTHOOK: Output: default@dest_j1_n21
OPTIMIZED SQL: SELECT `t4`.`key`, `t0`.`value`, `t2`.`value` AS `value1`
FROM (SELECT `value`
FROM `default`.`srcpart`
-WHERE `ds` = '2008-04-08' AND (`hr` = 11 AND `value` IS NOT NULL)) AS `t0`
+WHERE `ds` = '2008-04-08' AND `hr` = 11 AND `value` IS NOT NULL) AS `t0`
INNER JOIN ((SELECT `key`, `value`
FROM `default`.`src`
WHERE `key` IS NOT NULL) AS `t2` INNER JOIN (SELECT `key`, `value`
@@ -1109,7 +1109,7 @@ POSTHOOK: Output: default@dest_j2_n1
OPTIMIZED SQL: SELECT `t4`.`key`, `t0`.`value`, `t4`.`value` AS `value1`
FROM (SELECT `value`
FROM `default`.`srcpart`
-WHERE `ds` = '2008-04-08' AND (`hr` = 11 AND `value` IS NOT NULL)) AS `t0`
+WHERE `ds` = '2008-04-08' AND `hr` = 11 AND `value` IS NOT NULL) AS `t0`
INNER JOIN ((SELECT `key`
FROM `default`.`src`
WHERE `key` IS NOT NULL) AS `t2` INNER JOIN (SELECT `key`, `value`
@@ -1608,7 +1608,7 @@ POSTHOOK: Output: default@dest_j2_n1
OPTIMIZED SQL: SELECT `t2`.`key`, `t0`.`value`, `t2`.`value` AS `value1`
FROM (SELECT `value`
FROM `default`.`srcpart`
-WHERE `ds` = '2008-04-08' AND (`hr` = 11 AND `value` IS NOT NULL)) AS `t0`
+WHERE `ds` = '2008-04-08' AND `hr` = 11 AND `value` IS NOT NULL) AS `t0`
INNER JOIN ((SELECT `key`, `value`
FROM `default`.`src1`
WHERE `value` IS NOT NULL) AS `t2` LEFT JOIN (SELECT `key`
diff --git a/ql/src/test/results/clientpositive/llap/join33.q.out b/ql/src/test/results/clientpositive/llap/join33.q.out
... 10104 lines suppressed ...