You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "Hive QA (JIRA)" <ji...@apache.org> on 2016/09/30 10:51:20 UTC
[jira] [Commented] (HIVE-14866) Set hive.limit.optimize.enable to
true by default
[ https://issues.apache.org/jira/browse/HIVE-14866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15535677#comment-15535677 ]
Hive QA commented on HIVE-14866:
--------------------------------
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12831083/HIVE-14866.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 199 failed/errored test(s), 10650 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_mapjoin]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[allcolref_in_udf]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[authorization_view_1]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[authorization_view_disable_cbo_1]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join_reordering_values]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join_without_localtask]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucket_groupby]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cbo_SortUnionTransposeRule]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[constant_prop_1]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ctas]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[distinct_windowing]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[druid_topn]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[gby_star]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[gen_udf_example_add10]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby2_limit]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby_duplicate_key]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby_grouping_sets_limit]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[input22]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[input23]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[input2_limit]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[input4_limit]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[input_part6]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join_vc]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[lateral_view]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[lateral_view_explode2]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[lateral_view_noalias]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[lateral_view_onview]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[lateral_view_outer]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[lateral_view_ppd]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[limit0]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[limit_join_transpose]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[limit_pushdown2]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[limit_pushdown3]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[limit_pushdown]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[limit_pushdown_negative]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin1]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[nonmr_fetch]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[offset_limit]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[offset_limit_ppd_optimizer]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[orc_createas1]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[orc_predicate_pushdown]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[order]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_predicate_pushdown]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[pcr]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_udf_col]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[reduce_deduplicate_exclude_join]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[regex_col]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[select_as_omitted]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[smb_mapjoin_13]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[smb_mapjoin_15]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[smblimit]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[str_to_map]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[temp_table]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[timestamp]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_between]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_case_column_pruning]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_inline]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_isnull_isnotnull]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_size]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udtf_explode]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udtf_stack]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_cast_constant]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_char_2]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_char_simple]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_coalesce]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_data_types]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_decimal_cast]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_decimal_expressions]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_elt]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_groupby_reduce]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_join_part_col_char]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_mr_diff_schema_alias]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_non_string_partition]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_nvl]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_partitioned_date_time]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_reduce_groupby_decimal]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_string_concat]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_varchar_simple]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorization_13]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorization_7]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorization_8]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorization_decimal_date]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorization_div0]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorization_limit]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorization_offset_limit]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorization_part_project]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorization_short_regress]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorized_context]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorized_parquet]
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver[hbase_queries]
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[bucketmapjoin7]
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[explainuser_1]
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[explainuser_2]
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[limit_pushdown]
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[temp_table]
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[tez_dynpart_hashjoin_3]
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[vector_cast_constant]
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[vector_char_2]
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[vector_char_simple]
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[vector_coalesce]
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[vector_data_types]
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[vector_decimal_cast]
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[vector_decimal_expressions]
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[vector_elt]
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[vector_groupby_reduce]
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[vector_mr_diff_schema_alias]
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[vector_nvl]
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[vector_partitioned_date_time]
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[vector_reduce_groupby_decimal]
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[vector_string_concat]
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[vector_varchar_simple]
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[vectorization_13]
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[vectorization_7]
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[vectorization_8]
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[vectorization_decimal_date]
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[vectorization_part_project]
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[vectorization_short_regress]
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[vectorized_context]
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[vectorized_parquet]
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[bucketmapjoin7]
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[gen_udf_example_add10]
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_1]
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2]
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_3]
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[vector_non_string_partition]
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[vectorization_div0]
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[vectorization_limit]
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udf_assert_true2]
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udf_assert_true]
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query12]
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query15]
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query17]
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query18]
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query19]
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query20]
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query21]
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query22]
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query25]
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query26]
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query27]
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query28]
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query29]
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query3]
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query40]
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query42]
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query43]
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query45]
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query46]
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query50]
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query51]
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query52]
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query54]
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query55]
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query58]
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query65]
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query66]
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query67]
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query68]
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query70]
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query72]
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query75]
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query76]
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query79]
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query7]
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query80]
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query82]
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query84]
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query85]
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query89]
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query90]
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query93]
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query94]
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query96]
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query97]
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join_reordering_values]
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join_without_localtask]
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[bucketmapjoin7]
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[join_vc]
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[lateral_view_explode2]
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[limit_pushdown]
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[mapjoin1]
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[order]
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[pcr]
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[reduce_deduplicate_exclude_join]
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[smb_mapjoin_13]
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[smb_mapjoin_15]
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[temp_table]
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vector_cast_constant]
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vector_data_types]
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vector_elt]
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vector_string_concat]
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorization_13]
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorization_decimal_date]
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorization_div0]
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorization_part_project]
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorization_short_regress]
org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.testMetaDataCounts
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testAddJarConstructorUnCaching
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/1359/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1359/console
Test logs: http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-Build-1359/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 199 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12831083 - PreCommit-HIVE-Build
> Set hive.limit.optimize.enable to true by default
> -------------------------------------------------
>
> Key: HIVE-14866
> URL: https://issues.apache.org/jira/browse/HIVE-14866
> Project: Hive
> Issue Type: Improvement
> Affects Versions: 2.1.0
> Reporter: Jesus Camacho Rodriguez
> Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-14866.patch
>
>
> Currently, we set up the global limit for the query in two different places through two different variables: SemanticAnalyzer and through an optimization rule GlobalLimitOptimizer (the latest is off by default).
> This leads to several problems that I have observed:
> - Global limit might not be set for very simple queries, e.g., if the query does not contain a RS). GlobalLimitOptimizer would set the limit in this case, but as stated above, it is off by default.
> - Some other optimizations are not checking both variables, thus missing opportunities.
> - The variable set by SemanticAnalyzer does not take into account offset of the query, which I think might lead to incorrect results if FetchOptimizer kicks in (not verified yet). GlobalLimitOptimizer does take into account offset of query.
> This issue is to set hive.limit.optimize.enable to _true_ by default, i.e., use GlobalLimitOptimizer, and thus getting rid of the variable set by SemanticAnalyzer. Maybe there are some gaps (cases covered by SemanticAnalyzer alternative and not covered by GlobalLimitOptimizer) that we will need to work on.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)