You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "Hive QA (JIRA)" <ji...@apache.org> on 2017/06/09 12:33:18 UTC
[jira] [Commented] (HIVE-16867) Extend shared scan optimizer to
reuse computation from other operators
[ https://issues.apache.org/jira/browse/HIVE-16867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16044372#comment-16044372 ]
Hive QA commented on HIVE-16867:
--------------------------------
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12872247/HIVE-16867.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 88 failed/errored test(s), 10828 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[insert_overwrite_local_directory_1] (batchId=237)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[except_distinct] (batchId=142)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[explainuser_2] (batchId=142)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[intersect_merge] (batchId=140)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_ppd_basic] (batchId=140)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[unionDistinct_1] (batchId=141)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[auto_join0] (batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[auto_join30] (batchId=149)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[auto_smb_mapjoin_14] (batchId=155)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[auto_sortmerge_join_9] (batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1] (batchId=160)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[columnstats_part_coltype] (batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[correlationoptimizer2] (batchId=153)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[correlationoptimizer3] (batchId=160)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[correlationoptimizer6] (batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dynamic_partition_pruning] (batchId=149)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[explainuser_1] (batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[limit_pushdown] (batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[mrr] (batchId=146)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[multiMapJoin2] (batchId=158)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[offset_limit_ppd_optimizer] (batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[partition_shared_scan] (batchId=146)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_in] (batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_multi] (batchId=147)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_notin] (batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_null_agg] (batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_scalar] (batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_select] (batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_views] (batchId=146)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_top_level] (batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_auto_smb_mapjoin_14] (batchId=150)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_groupby_grouping_sets4] (batchId=149)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr] (batchId=145)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_join30] (batchId=150)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorized_dynamic_partition_pruning] (batchId=150)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2] (batchId=99)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query10] (batchId=232)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14] (batchId=232)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query16] (batchId=232)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query1] (batchId=232)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23] (batchId=232)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query24] (batchId=232)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query2] (batchId=232)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query30] (batchId=232)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query31] (batchId=232)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query32] (batchId=232)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query33] (batchId=232)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query35] (batchId=232)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query38] (batchId=232)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query39] (batchId=232)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query44] (batchId=232)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query46] (batchId=232)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query47] (batchId=232)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query49] (batchId=232)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query4] (batchId=232)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query51] (batchId=232)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query54] (batchId=232)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query56] (batchId=232)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query57] (batchId=232)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query58] (batchId=232)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query59] (batchId=232)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query5] (batchId=232)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query60] (batchId=232)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query61] (batchId=232)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query64] (batchId=232)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query65] (batchId=232)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query66] (batchId=232)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query68] (batchId=232)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query69] (batchId=232)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query70] (batchId=232)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query75] (batchId=232)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query76] (batchId=232)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query77] (batchId=232)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query78] (batchId=232)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query80] (batchId=232)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query81] (batchId=232)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query83] (batchId=232)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query85] (batchId=232)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query87] (batchId=232)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query88] (batchId=232)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query90] (batchId=232)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query92] (batchId=232)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query94] (batchId=232)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query95] (batchId=232)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query97] (batchId=232)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[dynamic_rdd_cache] (batchId=123)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[script_env_var1] (batchId=102)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[script_env_var2] (batchId=125)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5601/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5601/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5601/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 88 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12872247 - PreCommit-HIVE-Build
> Extend shared scan optimizer to reuse computation from other operators
> ----------------------------------------------------------------------
>
> Key: HIVE-16867
> URL: https://issues.apache.org/jira/browse/HIVE-16867
> Project: Hive
> Issue Type: Improvement
> Components: Physical Optimizer
> Affects Versions: 3.0.0
> Reporter: Jesus Camacho Rodriguez
> Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-16867.patch
>
>
> Follow-up of the work in HIVE-16602.
> HIVE-16602 introduced an optimization that identifies scans on input tables that can be merged so the data is read only once.
> This extension to that rule allows to reuse the computation that is done in the work containing those scans. In particular, we traverse both parts of the plan upstream and reuse the operators if possible.
> Currently, the optimizer will not go beyond the output edge(s) of that work. Follow-up extensions might remove this limitation.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)