[jira] [Updated] (HIVE-948) more query plan optimization rules

["Phabricator (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HIVE-948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Phabricator updated HIVE-948:
-----------------------------

    Attachment: HIVE-948.D8463.1.patch

navis requested code review of "HIVE-948 [jira] more query plan optimization rules".

Reviewers: JIRA

DPAL-1980 Implement cleanup precessor for logical plan

Many query plans are not optimal in that they contain redundant operators. Some examples are unnecessary select operators (select followed by select, select output being the same as input etc.). Even though these operators are not very expensive, they could account for around 10% of CPU time in some simple queries. It seems they are low-hanging fruits that we should pick first.

BTW, it seems these optimization rules should be added at the last stage of the physical optimization phase since some redundant operators are added to facilitate physical plan generation.

TEST PLAN
  EMPTY

REVISION DETAIL
  https://reviews.facebook.net/D8463

AFFECTED FILES
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/CleanupProcessor.java
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java
  ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDescUtils.java
  ql/src/java/org/apache/hadoop/hive/ql/ppd/OpProcFactory.java
  ql/src/java/org/apache/hadoop/hive/ql/ppd/PredicateTransitivePropagate.java
  ql/src/test/results/clientpositive/alias_casted_column.q.out
  ql/src/test/results/clientpositive/ambiguous_col.q.out
  ql/src/test/results/clientpositive/auto_join1.q.out
  ql/src/test/results/clientpositive/auto_join12.q.out
  ql/src/test/results/clientpositive/auto_join14_hadoop20.q.out
  ql/src/test/results/clientpositive/auto_join17.q.out
  ql/src/test/results/clientpositive/auto_join19.q.out
  ql/src/test/results/clientpositive/auto_join2.q.out
  ql/src/test/results/clientpositive/auto_join20.q.out
  ql/src/test/results/clientpositive/auto_join22.q.out
  ql/src/test/results/clientpositive/auto_join26.q.out
  ql/src/test/results/clientpositive/auto_join28.q.out
  ql/src/test/results/clientpositive/auto_join29.q.out
  ql/src/test/results/clientpositive/auto_join3.q.out
  ql/src/test/results/clientpositive/auto_join4.q.out
  ql/src/test/results/clientpositive/auto_join5.q.out
  ql/src/test/results/clientpositive/auto_join6.q.out
  ql/src/test/results/clientpositive/auto_join7.q.out
  ql/src/test/results/clientpositive/auto_join8.q.out
  ql/src/test/results/clientpositive/auto_join9.q.out
  ql/src/test/results/clientpositive/binarysortable_1.q.out
  ql/src/test/results/clientpositive/bucket_groupby.q.out
  ql/src/test/results/clientpositive/bucket_map_join_1.q.out
  ql/src/test/results/clientpositive/bucket_map_join_2.q.out
  ql/src/test/results/clientpositive/bucketcontext_1.q.out
  ql/src/test/results/clientpositive/bucketcontext_2.q.out
  ql/src/test/results/clientpositive/bucketcontext_3.q.out
  ql/src/test/results/clientpositive/bucketcontext_4.q.out
  ql/src/test/results/clientpositive/bucketcontext_5.q.out
  ql/src/test/results/clientpositive/bucketcontext_6.q.out
  ql/src/test/results/clientpositive/bucketcontext_7.q.out
  ql/src/test/results/clientpositive/bucketcontext_8.q.out
  ql/src/test/results/clientpositive/bucketmapjoin1.q.out
  ql/src/test/results/clientpositive/bucketmapjoin10.q.out
  ql/src/test/results/clientpositive/bucketmapjoin11.q.out
  ql/src/test/results/clientpositive/bucketmapjoin12.q.out
  ql/src/test/results/clientpositive/bucketmapjoin13.q.out
  ql/src/test/results/clientpositive/bucketmapjoin2.q.out
  ql/src/test/results/clientpositive/bucketmapjoin3.q.out
  ql/src/test/results/clientpositive/bucketmapjoin4.q.out
  ql/src/test/results/clientpositive/bucketmapjoin5.q.out
  ql/src/test/results/clientpositive/bucketmapjoin8.q.out
  ql/src/test/results/clientpositive/bucketmapjoin9.q.out
  ql/src/test/results/clientpositive/bucketmapjoin_negative.q.out
  ql/src/test/results/clientpositive/bucketmapjoin_negative2.q.out
  ql/src/test/results/clientpositive/bucketmapjoin_negative3.q.out
  ql/src/test/results/clientpositive/column_access_stats.q.out
  ql/src/test/results/clientpositive/create_view.q.out
  ql/src/test/results/clientpositive/ppd1.q.out
  ql/src/test/results/clientpositive/ppd2.q.out
  ql/src/test/results/clientpositive/ppd_clusterby.q.out
  ql/src/test/results/clientpositive/ppd_constant_expr.q.out
  ql/src/test/results/clientpositive/ppd_gby.q.out
  ql/src/test/results/clientpositive/ppd_gby2.q.out
  ql/src/test/results/clientpositive/ppd_gby_join.q.out
  ql/src/test/results/clientpositive/ppd_join.q.out
  ql/src/test/results/clientpositive/ppd_join2.q.out
  ql/src/test/results/clientpositive/ppd_join3.q.out
  ql/src/test/results/clientpositive/ppd_multi_insert.q.out
  ql/src/test/results/clientpositive/ppd_random.q.out
  ql/src/test/results/clientpositive/ppd_repeated_alias.q.out
  ql/src/test/results/clientpositive/ppd_udf_col.q.out
  ql/src/test/results/clientpositive/ppd_union.q.out
  ql/src/test/results/clientpositive/ppd_union_view.q.out

MANAGE HERALD RULES
  https://reviews.facebook.net/herald/view/differential/

WHY DID I GET THIS EMAIL?
  https://reviews.facebook.net/herald/transcript/20637/

To: JIRA, navis

                
> more query plan optimization rules 
> -----------------------------------
>
>                 Key: HIVE-948
>                 URL: https://issues.apache.org/jira/browse/HIVE-948
>             Project: Hive
>          Issue Type: Improvement
>            Reporter: Ning Zhang
>         Attachments: HIVE-948.D8463.1.patch
>
>
> Many query plans are not optimal in that they contain redundant operators. Some examples are unnecessary select operators (select followed by select, select output being the same as input etc.). Even though these operators are not very expensive, they could account for around 10% of CPU time in some simple queries. It seems they are low-hanging fruits that we should pick first. 
> BTW, it seems these optimization rules should be added at the last stage of the physical optimization phase since some redundant operators are added to facilitate physical plan generation. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira