[jira] [Commented] (HIVE-11160) Auto-gather column stats

["Hive QA (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HIVE-11160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16260912#comment-16260912 ] 

Hive QA commented on HIVE-11160:
--------------------------------



Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12799200/HIVE-11160.09.patch

{color:red}ERROR:{color} -1 due to build exiting with an error

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7942/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7942/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7942/

Messages:
{noformat}
**** This message was trimmed, see log for full details ****
error: a/itests/src/test/resources/testconfiguration.properties: No such file or directory
error: a/metastore/if/hive_metastore.thrift: No such file or directory
error: a/metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.cpp: No such file or directory
error: a/metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.h: No such file or directory
error: a/metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore_server.skeleton.cpp: No such file or directory
error: a/metastore/src/gen/thrift/gen-cpp/hive_metastore_types.cpp: No such file or directory
error: a/metastore/src/gen/thrift/gen-cpp/hive_metastore_types.h: No such file or directory
error: a/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/SetPartitionsStatsRequest.java: No such file or directory
error: a/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java: No such file or directory
error: a/metastore/src/gen/thrift/gen-php/metastore/ThriftHiveMetastore.php: No such file or directory
error: a/metastore/src/gen/thrift/gen-php/metastore/Types.php: No such file or directory
error: a/metastore/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore-remote: No such file or directory
error: a/metastore/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore.py: No such file or directory
error: a/metastore/src/gen/thrift/gen-py/hive_metastore/ttypes.py: No such file or directory
error: a/metastore/src/gen/thrift/gen-rb/hive_metastore_types.rb: No such file or directory
error: a/metastore/src/gen/thrift/gen-rb/thrift_hive_metastore.rb: No such file or directory
error: a/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java: No such file or directory
error: a/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java: No such file or directory
error: a/metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java: No such file or directory
error: a/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java: No such file or directory
error: a/metastore/src/java/org/apache/hadoop/hive/metastore/hbase/StatsCache.java: No such file or directory
error: a/metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/BinaryColumnStatsAggregator.java: No such file or directory
error: a/metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/BooleanColumnStatsAggregator.java: No such file or directory
error: a/metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/ColumnStatsAggregator.java: No such file or directory
error: a/metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/ColumnStatsAggregatorFactory.java: No such file or directory
error: a/metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/DecimalColumnStatsAggregator.java: No such file or directory
error: a/metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/DoubleColumnStatsAggregator.java: No such file or directory
error: a/metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/IExtrapolatePartStatus.java: No such file or directory
error: a/metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/LongColumnStatsAggregator.java: No such file or directory
error: a/metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/StringColumnStatsAggregator.java: No such file or directory
error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/ColumnStatsTask.java: No such file or directory
error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/FetchOperator.java: No such file or directory
error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizationContext.java: No such file or directory
error: a/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java: No such file or directory
error: a/ql/src/java/org/apache/hadoop/hive/ql/metadata/SessionHiveMetaStoreClient.java: No such file or directory
error: a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java: No such file or directory
error: a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/GenMRSkewJoinProcessor.java: No such file or directory
error: a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/PartitionPruner.java: No such file or directory
error: a/ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java: No such file or directory
error: a/ql/src/java/org/apache/hadoop/hive/ql/parse/ColumnStatsSemanticAnalyzer.java: No such file or directory
error: a/ql/src/java/org/apache/hadoop/hive/ql/parse/ParseContext.java: No such file or directory
error: a/ql/src/java/org/apache/hadoop/hive/ql/parse/QBParseInfo.java: No such file or directory
error: a/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java: No such file or directory
error: a/ql/src/java/org/apache/hadoop/hive/ql/parse/TaskCompiler.java: No such file or directory
error: a/ql/src/java/org/apache/hadoop/hive/ql/plan/ColumnStatsDesc.java: No such file or directory
error: a/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFBridge.java: No such file or directory
error: a/ql/src/test/queries/clientpositive/combine1.q: No such file or directory
error: a/ql/src/test/results/clientpositive/alter_merge_stats_orc.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/alter_numbuckets_partitioned_table2_h23.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/alter_numbuckets_partitioned_table_h23.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/alter_partition_coltype.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/alter_table_serde2.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/analyze_table_null_partition.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/annotate_stats_filter.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/annotate_stats_groupby.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/auto_join1.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/auto_join14.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/auto_join17.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/auto_join19.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/auto_join19_inclause.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/auto_join2.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/auto_join25.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/auto_join26.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/auto_join3.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/auto_join4.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/auto_join5.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/auto_join6.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/auto_join7.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/auto_join8.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/auto_join9.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/auto_join_reordering_values.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/auto_smb_mapjoin_14.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/auto_sortmerge_join_13.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/auto_sortmerge_join_6.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/auto_sortmerge_join_9.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/binary_output_format.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/bucket1.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/bucket2.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/bucket3.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/bucket_map_join_spark1.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/bucket_map_join_spark2.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/bucket_map_join_spark3.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/bucket_map_join_spark4.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/bucketmapjoin1.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/bucketmapjoin13.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/bucketmapjoin2.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/bucketmapjoin3.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/bucketmapjoin4.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/bucketmapjoin5.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/bucketmapjoin_negative.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/bucketmapjoin_negative2.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/bucketsortoptimize_insert_1.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/bucketsortoptimize_insert_2.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/bucketsortoptimize_insert_3.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/bucketsortoptimize_insert_4.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/bucketsortoptimize_insert_5.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/bucketsortoptimize_insert_6.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/bucketsortoptimize_insert_7.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/bucketsortoptimize_insert_8.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/case_sensitivity.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/cast1.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/columnStatsUpdateForStatsOptimizer_1.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/columnStatsUpdateForStatsOptimizer_2.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/columnstats_part_coltype.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/combine2.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/constprog_dp.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/constprog_type.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/correlationoptimizer5.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/cp_sel.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/dynamic_partition_skip_default.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/dynamic_rdd_cache.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/dynpart_sort_opt_vectorization.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/dynpart_sort_optimization.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/dynpart_sort_optimization2.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/explain_ddl.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/extrapolate_part_stats_full.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/extrapolate_part_stats_partial.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/extrapolate_part_stats_partial_ndv.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/filter_join_breaktask.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/groupby1.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/groupby10.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/groupby11.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/groupby12.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/groupby1_limit.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/groupby1_map.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/groupby1_map_nomap.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/groupby1_map_skew.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/groupby1_noskew.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/groupby2_map.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/groupby2_map_multi_distinct.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/groupby2_map_skew.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/groupby2_noskew.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/groupby2_noskew_multi_distinct.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/groupby3.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/groupby3_map.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/groupby3_map_multi_distinct.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/groupby3_map_skew.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/groupby3_noskew.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/groupby3_noskew_multi_distinct.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/groupby4.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/groupby4_map.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/groupby4_map_skew.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/groupby4_noskew.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/groupby5_map.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/groupby5_map_skew.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/groupby6.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/groupby6_map.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/groupby6_map_skew.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/groupby6_noskew.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/groupby_cube_multi_gby.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/groupby_map_ppr.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/groupby_map_ppr_multi_distinct.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/groupby_multi_insert_common_distinct.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/groupby_multi_single_reducer.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/groupby_multi_single_reducer2.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/groupby_multi_single_reducer3.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/groupby_ppr.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/groupby_ppr_multi_distinct.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/implicit_cast_during_insert.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/index_auto_update.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/infer_bucket_sort.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/infer_bucket_sort_convert_join.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/infer_bucket_sort_grouping_operators.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/infer_bucket_sort_list_bucket.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/infer_bucket_sort_multi_insert.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/innerjoin.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/input11.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/input11_limit.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/input12.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/input13.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/input14.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/input14_limit.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/input17.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/input18.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/input1_limit.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/input20.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/input33.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/input34.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/input35.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/input36.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/input38.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/input5.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/input6.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/input7.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/input8.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/input9.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/input_part1.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/input_part10.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/input_part2.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/input_part5.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/input_testsequencefile.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/input_testxpath.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/input_testxpath2.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/insert1.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/insert_into1.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/insert_into2.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/insert_into3.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/insert_into4.q.out: No such file or directory
error: a/ql/src/test/results/clientpositive/insert_into5.q.out: No such file or directory
'
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12799200 - PreCommit-HIVE-Build

> Auto-gather column stats
> ------------------------
>
>                 Key: HIVE-11160
>                 URL: https://issues.apache.org/jira/browse/HIVE-11160
>             Project: Hive
>          Issue Type: New Feature
>            Reporter: Pengcheng Xiong
>            Assignee: Pengcheng Xiong
>         Attachments: HIVE-11160.01.patch, HIVE-11160.02.patch, HIVE-11160.03.patch, HIVE-11160.04.patch, HIVE-11160.05.patch, HIVE-11160.06.patch, HIVE-11160.07.patch, HIVE-11160.08.patch, HIVE-11160.09.patch
>
>
>  Hive will collect table stats when set hive.stats.autogather=true during the INSERT OVERWRITE command. And then the users need to collect the column stats themselves using "Analyze" command. In this patch, the column stats will also be collected automatically. More specifically, INSERT OVERWRITE will automatically create new column stats. INSERT INTO will automatically merge new column stats with existing ones.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)