You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "He Yongqiang (Created) (JIRA)" <ji...@apache.org> on 2011/12/09 21:36:39 UTC
[jira] [Created] (HIVE-2642) fix Hive-2566 and make union
optimization more aggressive
fix Hive-2566 and make union optimization more aggressive
----------------------------------------------------------
Key: HIVE-2642
URL: https://issues.apache.org/jira/browse/HIVE-2642
Project: Hive
Issue Type: Improvement
Reporter: He Yongqiang
Assignee: He Yongqiang
Hive-2566 did some optimizations to union, but cause some problems. And then got reverted. This is to get it back and fix the problems we saw, and also make union optimization more aggressive.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2642) fix Hive-2566 and make union
optimization more aggressive
Posted by "Ashutosh Chauhan (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ashutosh Chauhan updated HIVE-2642:
-----------------------------------
Fix Version/s: 0.9.0
> fix Hive-2566 and make union optimization more aggressive
> ----------------------------------------------------------
>
> Key: HIVE-2642
> URL: https://issues.apache.org/jira/browse/HIVE-2642
> Project: Hive
> Issue Type: Improvement
> Reporter: He Yongqiang
> Assignee: He Yongqiang
> Fix For: 0.9.0
>
> Attachments: HIVE-2642.D735.1.patch
>
>
> Hive-2566 did some optimizations to union, but cause some problems. And then got reverted. This is to get it back and fix the problems we saw, and also make union optimization more aggressive.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2642) fix Hive-2566 and make union
optimization more aggressive
Posted by "Namit Jain (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13172854#comment-13172854 ]
Namit Jain commented on HIVE-2642:
----------------------------------
Look at union22.q.out.
map-join followed by union, an extra stage is introduced.
We dont hsve to optimize this - just wanted to make sure it is intentional.
> fix Hive-2566 and make union optimization more aggressive
> ----------------------------------------------------------
>
> Key: HIVE-2642
> URL: https://issues.apache.org/jira/browse/HIVE-2642
> Project: Hive
> Issue Type: Improvement
> Reporter: He Yongqiang
> Assignee: He Yongqiang
> Attachments: HIVE-2642.D735.1.patch
>
>
> Hive-2566 did some optimizations to union, but cause some problems. And then got reverted. This is to get it back and fix the problems we saw, and also make union optimization more aggressive.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2642) fix Hive-2566 and make union
optimization more aggressive
Posted by "Phabricator (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13172020#comment-13172020 ]
Phabricator commented on HIVE-2642:
-----------------------------------
njain has commented on the revision "HIVE-2642 [jira] fix Hive-2566 and make union optimization more aggressive".
still reviewing
INLINE COMMENTS
ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRProcContext.java:105 can you add some comments here ?
This is not really the top operators - this contains the list of intermediate tables also. This code is difficult to debug later on, so more comments would be helpful
REVISION DETAIL
https://reviews.facebook.net/D735
> fix Hive-2566 and make union optimization more aggressive
> ----------------------------------------------------------
>
> Key: HIVE-2642
> URL: https://issues.apache.org/jira/browse/HIVE-2642
> Project: Hive
> Issue Type: Improvement
> Reporter: He Yongqiang
> Assignee: He Yongqiang
> Attachments: HIVE-2642.D735.1.patch
>
>
> Hive-2566 did some optimizations to union, but cause some problems. And then got reverted. This is to get it back and fix the problems we saw, and also make union optimization more aggressive.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2642) fix Hive-2566 and make union
optimization more aggressive
Posted by "Namit Jain (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13174211#comment-13174211 ]
Namit Jain commented on HIVE-2642:
----------------------------------
1 general comment about the new test union26.q --
Reduce the test output, I mean, you dont need to load all 500 rows for this test.
It makes the test output really difficult to review.
Again, all the above 3 are not blockers - I am still reviewing, I will file a enhancement for all the follow-ups.
> fix Hive-2566 and make union optimization more aggressive
> ----------------------------------------------------------
>
> Key: HIVE-2642
> URL: https://issues.apache.org/jira/browse/HIVE-2642
> Project: Hive
> Issue Type: Improvement
> Reporter: He Yongqiang
> Assignee: He Yongqiang
> Attachments: HIVE-2642.D735.1.patch
>
>
> Hive-2566 did some optimizations to union, but cause some problems. And then got reverted. This is to get it back and fix the problems we saw, and also make union optimization more aggressive.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HIVE-2642) fix Hive-2566 and make union
optimization more aggressive
Posted by "Namit Jain (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Namit Jain resolved HIVE-2642.
------------------------------
Resolution: Fixed
Hadoop Flags: Reviewed
Committed. Thanks Yongqiang
> fix Hive-2566 and make union optimization more aggressive
> ----------------------------------------------------------
>
> Key: HIVE-2642
> URL: https://issues.apache.org/jira/browse/HIVE-2642
> Project: Hive
> Issue Type: Improvement
> Reporter: He Yongqiang
> Assignee: He Yongqiang
> Attachments: HIVE-2642.D735.1.patch
>
>
> Hive-2566 did some optimizations to union, but cause some problems. And then got reverted. This is to get it back and fix the problems we saw, and also make union optimization more aggressive.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2642) fix Hive-2566 and make union
optimization more aggressive
Posted by "Phabricator (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Phabricator updated HIVE-2642:
------------------------------
Attachment: HIVE-2642.D735.1.patch
heyongqiang requested code review of "HIVE-2642 [jira] fix Hive-2566 and make union optimization more aggressive".
Reviewers: JIRA
fix Hive-2566 and make union optimization more aggressive
Hive-2566 did some optimizations to union, but cause some problems. And then got reverted. This is to get it back and fix the problems we saw, and also make union optimization more aggressive.
TEST PLAN
EMPTY
REVISION DETAIL
https://reviews.facebook.net/D735
AFFECTED FILES
ql/src/test/results/clientpositive/merge4.q.out
ql/src/test/results/clientpositive/auto_join27.q.out
ql/src/test/results/clientpositive/union19.q.out
ql/src/test/results/clientpositive/union4.q.out
ql/src/test/results/clientpositive/union14.q.out
ql/src/test/results/clientpositive/create_view.q.out
ql/src/test/results/clientpositive/union5.q.out
ql/src/test/results/clientpositive/ql_rewrite_gbtoidx.q.out
ql/src/test/results/clientpositive/load_dyn_part14.q.out
ql/src/test/results/clientpositive/union15.q.out
ql/src/test/results/clientpositive/union24.q.out
ql/src/test/results/clientpositive/union10.q.out
ql/src/test/results/clientpositive/input25.q.out
ql/src/test/results/clientpositive/union6.q.out
ql/src/test/results/clientpositive/lineage1.q.out
ql/src/test/results/clientpositive/union25.q.out
ql/src/test/results/clientpositive/union11.q.out
ql/src/test/results/clientpositive/join35.q.out
ql/src/test/results/clientpositive/union20.q.out
ql/src/test/results/clientpositive/input26.q.out
ql/src/test/results/clientpositive/union7.q.out
ql/src/test/results/clientpositive/union17.q.out
ql/src/test/results/clientpositive/union26.q.out
ql/src/test/results/clientpositive/union12.q.out
ql/src/test/results/clientpositive/stats1.q.out
ql/src/test/results/clientpositive/union18.q.out
ql/src/test/results/clientpositive/union3.q.out
ql/src/test/results/clientpositive/union22.q.out
ql/src/test/results/clientpositive/ppd_union_view.q.out
ql/src/test/queries/clientpositive/union25.q
ql/src/test/queries/clientpositive/merge4.q
ql/src/test/queries/clientpositive/union26.q
ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRProcContext.java
ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRUnion1.java
ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRTableScan1.java
ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRRedSink3.java
ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRFileSink1.java
ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java
ql/src/java/org/apache/hadoop/hive/ql/optimizer/unionproc/UnionProcContext.java
ql/src/java/org/apache/hadoop/hive/ql/optimizer/unionproc/UnionProcFactory.java
MANAGE HERALD DIFFERENTIAL RULES
https://reviews.facebook.net/herald/view/differential/
WHY DID I GET THIS EMAIL?
https://reviews.facebook.net/herald/transcript/1581/
Tip: use the X-Herald-Rules header to filter Herald messages in your client.
> fix Hive-2566 and make union optimization more aggressive
> ----------------------------------------------------------
>
> Key: HIVE-2642
> URL: https://issues.apache.org/jira/browse/HIVE-2642
> Project: Hive
> Issue Type: Improvement
> Reporter: He Yongqiang
> Assignee: He Yongqiang
> Attachments: HIVE-2642.D735.1.patch
>
>
> Hive-2566 did some optimizations to union, but cause some problems. And then got reverted. This is to get it back and fix the problems we saw, and also make union optimization more aggressive.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2642) fix Hive-2566 and make union
optimization more aggressive
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13174519#comment-13174519 ]
Hudson commented on HIVE-2642:
------------------------------
Integrated in Hive-trunk-h0.21 #1164 (See [https://builds.apache.org/job/Hive-trunk-h0.21/1164/])
HIVE-2642 fix Hive-2566 and make union optimization more aggressive
(Yongqiang He via namit)
namit : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1221812
Files :
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRFileSink1.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRProcContext.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRRedSink3.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRTableScan1.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRUnion1.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/unionproc/UnionProcContext.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/unionproc/UnionProcFactory.java
* /hive/trunk/ql/src/test/queries/clientpositive/union26.q
* /hive/trunk/ql/src/test/results/clientpositive/auto_join27.q.out
* /hive/trunk/ql/src/test/results/clientpositive/input25.q.out
* /hive/trunk/ql/src/test/results/clientpositive/input26.q.out
* /hive/trunk/ql/src/test/results/clientpositive/join35.q.out
* /hive/trunk/ql/src/test/results/clientpositive/lineage1.q.out
* /hive/trunk/ql/src/test/results/clientpositive/load_dyn_part14.q.out
* /hive/trunk/ql/src/test/results/clientpositive/merge4.q.out
* /hive/trunk/ql/src/test/results/clientpositive/ppd_union_view.q.out
* /hive/trunk/ql/src/test/results/clientpositive/ql_rewrite_gbtoidx.q.out
* /hive/trunk/ql/src/test/results/clientpositive/stats1.q.out
* /hive/trunk/ql/src/test/results/clientpositive/union10.q.out
* /hive/trunk/ql/src/test/results/clientpositive/union11.q.out
* /hive/trunk/ql/src/test/results/clientpositive/union12.q.out
* /hive/trunk/ql/src/test/results/clientpositive/union14.q.out
* /hive/trunk/ql/src/test/results/clientpositive/union15.q.out
* /hive/trunk/ql/src/test/results/clientpositive/union17.q.out
* /hive/trunk/ql/src/test/results/clientpositive/union18.q.out
* /hive/trunk/ql/src/test/results/clientpositive/union19.q.out
* /hive/trunk/ql/src/test/results/clientpositive/union20.q.out
* /hive/trunk/ql/src/test/results/clientpositive/union22.q.out
* /hive/trunk/ql/src/test/results/clientpositive/union24.q.out
* /hive/trunk/ql/src/test/results/clientpositive/union25.q.out
* /hive/trunk/ql/src/test/results/clientpositive/union26.q.out
* /hive/trunk/ql/src/test/results/clientpositive/union3.q.out
* /hive/trunk/ql/src/test/results/clientpositive/union4.q.out
* /hive/trunk/ql/src/test/results/clientpositive/union5.q.out
* /hive/trunk/ql/src/test/results/clientpositive/union6.q.out
* /hive/trunk/ql/src/test/results/clientpositive/union7.q.out
> fix Hive-2566 and make union optimization more aggressive
> ----------------------------------------------------------
>
> Key: HIVE-2642
> URL: https://issues.apache.org/jira/browse/HIVE-2642
> Project: Hive
> Issue Type: Improvement
> Reporter: He Yongqiang
> Assignee: He Yongqiang
> Attachments: HIVE-2642.D735.1.patch
>
>
> Hive-2566 did some optimizations to union, but cause some problems. And then got reverted. This is to get it back and fix the problems we saw, and also make union optimization more aggressive.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2642) fix Hive-2566 and make union
optimization more aggressive
Posted by "He Yongqiang (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13166661#comment-13166661 ]
He Yongqiang commented on HIVE-2642:
------------------------------------
https://reviews.facebook.net/D735
> fix Hive-2566 and make union optimization more aggressive
> ----------------------------------------------------------
>
> Key: HIVE-2642
> URL: https://issues.apache.org/jira/browse/HIVE-2642
> Project: Hive
> Issue Type: Improvement
> Reporter: He Yongqiang
> Assignee: He Yongqiang
> Attachments: HIVE-2642.D735.1.patch
>
>
> Hive-2566 did some optimizations to union, but cause some problems. And then got reverted. This is to get it back and fix the problems we saw, and also make union optimization more aggressive.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2642) fix Hive-2566 and make union
optimization more aggressive
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13174329#comment-13174329 ]
Hudson commented on HIVE-2642:
------------------------------
Integrated in Hive-trunk-h0.23.0 #42 (See [https://builds.apache.org/job/Hive-trunk-h0.23.0/42/])
HIVE-2642 fix Hive-2566 and make union optimization more aggressive
(Yongqiang He via namit)
namit : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1221812
Files :
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRFileSink1.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRProcContext.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRRedSink3.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRTableScan1.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRUnion1.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/unionproc/UnionProcContext.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/unionproc/UnionProcFactory.java
* /hive/trunk/ql/src/test/queries/clientpositive/union26.q
* /hive/trunk/ql/src/test/results/clientpositive/auto_join27.q.out
* /hive/trunk/ql/src/test/results/clientpositive/input25.q.out
* /hive/trunk/ql/src/test/results/clientpositive/input26.q.out
* /hive/trunk/ql/src/test/results/clientpositive/join35.q.out
* /hive/trunk/ql/src/test/results/clientpositive/lineage1.q.out
* /hive/trunk/ql/src/test/results/clientpositive/load_dyn_part14.q.out
* /hive/trunk/ql/src/test/results/clientpositive/merge4.q.out
* /hive/trunk/ql/src/test/results/clientpositive/ppd_union_view.q.out
* /hive/trunk/ql/src/test/results/clientpositive/ql_rewrite_gbtoidx.q.out
* /hive/trunk/ql/src/test/results/clientpositive/stats1.q.out
* /hive/trunk/ql/src/test/results/clientpositive/union10.q.out
* /hive/trunk/ql/src/test/results/clientpositive/union11.q.out
* /hive/trunk/ql/src/test/results/clientpositive/union12.q.out
* /hive/trunk/ql/src/test/results/clientpositive/union14.q.out
* /hive/trunk/ql/src/test/results/clientpositive/union15.q.out
* /hive/trunk/ql/src/test/results/clientpositive/union17.q.out
* /hive/trunk/ql/src/test/results/clientpositive/union18.q.out
* /hive/trunk/ql/src/test/results/clientpositive/union19.q.out
* /hive/trunk/ql/src/test/results/clientpositive/union20.q.out
* /hive/trunk/ql/src/test/results/clientpositive/union22.q.out
* /hive/trunk/ql/src/test/results/clientpositive/union24.q.out
* /hive/trunk/ql/src/test/results/clientpositive/union25.q.out
* /hive/trunk/ql/src/test/results/clientpositive/union26.q.out
* /hive/trunk/ql/src/test/results/clientpositive/union3.q.out
* /hive/trunk/ql/src/test/results/clientpositive/union4.q.out
* /hive/trunk/ql/src/test/results/clientpositive/union5.q.out
* /hive/trunk/ql/src/test/results/clientpositive/union6.q.out
* /hive/trunk/ql/src/test/results/clientpositive/union7.q.out
> fix Hive-2566 and make union optimization more aggressive
> ----------------------------------------------------------
>
> Key: HIVE-2642
> URL: https://issues.apache.org/jira/browse/HIVE-2642
> Project: Hive
> Issue Type: Improvement
> Reporter: He Yongqiang
> Assignee: He Yongqiang
> Attachments: HIVE-2642.D735.1.patch
>
>
> Hive-2566 did some optimizations to union, but cause some problems. And then got reverted. This is to get it back and fix the problems we saw, and also make union optimization more aggressive.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2642) fix Hive-2566 and make union
optimization more aggressive
Posted by "Phabricator (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13174227#comment-13174227 ]
Phabricator commented on HIVE-2642:
-----------------------------------
njain has commented on the revision "HIVE-2642 [jira] fix Hive-2566 and make union optimization more aggressive".
INLINE COMMENTS
ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRUnion1.java:245 remove this comment
REVISION DETAIL
https://reviews.facebook.net/D735
> fix Hive-2566 and make union optimization more aggressive
> ----------------------------------------------------------
>
> Key: HIVE-2642
> URL: https://issues.apache.org/jira/browse/HIVE-2642
> Project: Hive
> Issue Type: Improvement
> Reporter: He Yongqiang
> Assignee: He Yongqiang
> Attachments: HIVE-2642.D735.1.patch
>
>
> Hive-2566 did some optimizations to union, but cause some problems. And then got reverted. This is to get it back and fix the problems we saw, and also make union optimization more aggressive.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira