You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@hive.apache.org by Navis Ryu <na...@nexr.com> on 2014/07/09 08:55:59 UTC

Review Request 23351: Support direct fetch for lateral views, sub queries, etc.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/23351/
-----------------------------------------------------------

Review request for hive.


Bugs: HIVE-5718
    https://issues.apache.org/jira/browse/HIVE-5718


Repository: hive-git


Description
-------

Extend HIVE-2925 with LV and SubQ.


Diffs
-----

  ql/src/java/org/apache/hadoop/hive/ql/exec/OperatorFactory.java 5d41fa1 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/SimpleFetchOptimizer.java 7413d2b 
  ql/src/java/org/apache/hadoop/hive/ql/parse/QB.java 908db1e 
  ql/src/java/org/apache/hadoop/hive/ql/parse/QBParseInfo.java 911ac8a 
  ql/src/java/org/apache/hadoop/hive/ql/plan/FetchWork.java 32d84ea 
  ql/src/test/queries/clientpositive/nonmr_fetch.q 2a92d17 
  ql/src/test/queries/clientpositive/nonmr_fetch_threshold.q e6343e2 
  ql/src/test/results/clientpositive/explain_logical.q.out bb26e8c 
  ql/src/test/results/clientpositive/lateral_view_noalias.q.out d51b2de 
  ql/src/test/results/clientpositive/nonmr_fetch.q.out 5a13e84 
  ql/src/test/results/clientpositive/nonmr_fetch_threshold.q.out 39cdfa6 
  ql/src/test/results/clientpositive/select_dummy_source.q.out 2742d56 
  ql/src/test/results/clientpositive/subquery_alias.q.out 37bc3a4 
  ql/src/test/results/clientpositive/udf_explode.q.out 4eeedeb 
  ql/src/test/results/clientpositive/udf_inline.q.out e065bed 
  ql/src/test/results/clientpositive/udf_reflect2.q.out 6b19277 
  ql/src/test/results/clientpositive/udf_to_unix_timestamp.q.out 447ef87 
  ql/src/test/results/clientpositive/udtf_explode.q.out ae95907 

Diff: https://reviews.apache.org/r/23351/diff/


Testing
-------


Thanks,

Navis Ryu

Re: Review Request 23351: Support direct fetch for lateral views, sub queries, etc.

Posted by Gopal V <go...@hortonworks.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/23351/#review49569
-----------------------------------------------------------

Ship it!


Ship It!

- Gopal V


On July 9, 2014, 6:55 a.m., Navis Ryu wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/23351/
> -----------------------------------------------------------
> 
> (Updated July 9, 2014, 6:55 a.m.)
> 
> 
> Review request for hive.
> 
> 
> Bugs: HIVE-5718
>     https://issues.apache.org/jira/browse/HIVE-5718
> 
> 
> Repository: hive-git
> 
> 
> Description
> -------
> 
> Extend HIVE-2925 with LV and SubQ.
> 
> 
> Diffs
> -----
> 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/OperatorFactory.java 5d41fa1 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/SimpleFetchOptimizer.java 7413d2b 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/QB.java 908db1e 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/QBParseInfo.java 911ac8a 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/FetchWork.java 32d84ea 
>   ql/src/test/queries/clientpositive/nonmr_fetch.q 2a92d17 
>   ql/src/test/queries/clientpositive/nonmr_fetch_threshold.q e6343e2 
>   ql/src/test/results/clientpositive/explain_logical.q.out bb26e8c 
>   ql/src/test/results/clientpositive/lateral_view_noalias.q.out d51b2de 
>   ql/src/test/results/clientpositive/nonmr_fetch.q.out 5a13e84 
>   ql/src/test/results/clientpositive/nonmr_fetch_threshold.q.out 39cdfa6 
>   ql/src/test/results/clientpositive/select_dummy_source.q.out 2742d56 
>   ql/src/test/results/clientpositive/subquery_alias.q.out 37bc3a4 
>   ql/src/test/results/clientpositive/udf_explode.q.out 4eeedeb 
>   ql/src/test/results/clientpositive/udf_inline.q.out e065bed 
>   ql/src/test/results/clientpositive/udf_reflect2.q.out 6b19277 
>   ql/src/test/results/clientpositive/udf_to_unix_timestamp.q.out 447ef87 
>   ql/src/test/results/clientpositive/udtf_explode.q.out ae95907 
> 
> Diff: https://reviews.apache.org/r/23351/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Navis Ryu
> 
>

Re: Review Request 23351: Support direct fetch for lateral views, sub queries, etc.

Posted by Gopal V <go...@hortonworks.com>.


> On Aug. 1, 2014, 6:21 a.m., Gopal V wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/optimizer/SimpleFetchOptimizer.java, line 161
> > <https://reviews.apache.org/r/23351/diff/1/?file=626500#file626500line161>
> >
> >     This needs some comments, because I'm lost on what this exactly does.
> 
> Navis Ryu wrote:
>     With subqueries, alias is constructed something like queryId1:queryId2:tableName. With union, it's more complicated null-subquery1:queryId2-subquery1:tableName

Can you leave that as a comment in code?


- Gopal


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/23351/#review49324
-----------------------------------------------------------


On July 9, 2014, 6:55 a.m., Navis Ryu wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/23351/
> -----------------------------------------------------------
> 
> (Updated July 9, 2014, 6:55 a.m.)
> 
> 
> Review request for hive.
> 
> 
> Bugs: HIVE-5718
>     https://issues.apache.org/jira/browse/HIVE-5718
> 
> 
> Repository: hive-git
> 
> 
> Description
> -------
> 
> Extend HIVE-2925 with LV and SubQ.
> 
> 
> Diffs
> -----
> 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/OperatorFactory.java 5d41fa1 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/SimpleFetchOptimizer.java 7413d2b 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/QB.java 908db1e 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/QBParseInfo.java 911ac8a 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/FetchWork.java 32d84ea 
>   ql/src/test/queries/clientpositive/nonmr_fetch.q 2a92d17 
>   ql/src/test/queries/clientpositive/nonmr_fetch_threshold.q e6343e2 
>   ql/src/test/results/clientpositive/explain_logical.q.out bb26e8c 
>   ql/src/test/results/clientpositive/lateral_view_noalias.q.out d51b2de 
>   ql/src/test/results/clientpositive/nonmr_fetch.q.out 5a13e84 
>   ql/src/test/results/clientpositive/nonmr_fetch_threshold.q.out 39cdfa6 
>   ql/src/test/results/clientpositive/select_dummy_source.q.out 2742d56 
>   ql/src/test/results/clientpositive/subquery_alias.q.out 37bc3a4 
>   ql/src/test/results/clientpositive/udf_explode.q.out 4eeedeb 
>   ql/src/test/results/clientpositive/udf_inline.q.out e065bed 
>   ql/src/test/results/clientpositive/udf_reflect2.q.out 6b19277 
>   ql/src/test/results/clientpositive/udf_to_unix_timestamp.q.out 447ef87 
>   ql/src/test/results/clientpositive/udtf_explode.q.out ae95907 
> 
> Diff: https://reviews.apache.org/r/23351/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Navis Ryu
> 
>

Re: Review Request 23351: Support direct fetch for lateral views, sub queries, etc.

Posted by Navis Ryu <na...@nexr.com>.


> On Aug. 1, 2014, 6:21 a.m., Gopal V wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/optimizer/SimpleFetchOptimizer.java, line 161
> > <https://reviews.apache.org/r/23351/diff/1/?file=626500#file626500line161>
> >
> >     This needs some comments, because I'm lost on what this exactly does.

With subqueries, alias is constructed something like queryId1:queryId2:tableName. With union, it's more complicated null-subquery1:queryId2-subquery1:tableName 


- Navis


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/23351/#review49324
-----------------------------------------------------------


On July 9, 2014, 6:55 a.m., Navis Ryu wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/23351/
> -----------------------------------------------------------
> 
> (Updated July 9, 2014, 6:55 a.m.)
> 
> 
> Review request for hive.
> 
> 
> Bugs: HIVE-5718
>     https://issues.apache.org/jira/browse/HIVE-5718
> 
> 
> Repository: hive-git
> 
> 
> Description
> -------
> 
> Extend HIVE-2925 with LV and SubQ.
> 
> 
> Diffs
> -----
> 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/OperatorFactory.java 5d41fa1 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/SimpleFetchOptimizer.java 7413d2b 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/QB.java 908db1e 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/QBParseInfo.java 911ac8a 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/FetchWork.java 32d84ea 
>   ql/src/test/queries/clientpositive/nonmr_fetch.q 2a92d17 
>   ql/src/test/queries/clientpositive/nonmr_fetch_threshold.q e6343e2 
>   ql/src/test/results/clientpositive/explain_logical.q.out bb26e8c 
>   ql/src/test/results/clientpositive/lateral_view_noalias.q.out d51b2de 
>   ql/src/test/results/clientpositive/nonmr_fetch.q.out 5a13e84 
>   ql/src/test/results/clientpositive/nonmr_fetch_threshold.q.out 39cdfa6 
>   ql/src/test/results/clientpositive/select_dummy_source.q.out 2742d56 
>   ql/src/test/results/clientpositive/subquery_alias.q.out 37bc3a4 
>   ql/src/test/results/clientpositive/udf_explode.q.out 4eeedeb 
>   ql/src/test/results/clientpositive/udf_inline.q.out e065bed 
>   ql/src/test/results/clientpositive/udf_reflect2.q.out 6b19277 
>   ql/src/test/results/clientpositive/udf_to_unix_timestamp.q.out 447ef87 
>   ql/src/test/results/clientpositive/udtf_explode.q.out ae95907 
> 
> Diff: https://reviews.apache.org/r/23351/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Navis Ryu
> 
>

Re: Review Request 23351: Support direct fetch for lateral views, sub queries, etc.

Posted by Gopal V <go...@hortonworks.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/23351/#review49324
-----------------------------------------------------------



ql/src/java/org/apache/hadoop/hive/ql/optimizer/SimpleFetchOptimizer.java
<https://reviews.apache.org/r/23351/#comment86301>

    This needs some comments, because I'm lost on what this exactly does.


- Gopal V


On July 9, 2014, 6:55 a.m., Navis Ryu wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/23351/
> -----------------------------------------------------------
> 
> (Updated July 9, 2014, 6:55 a.m.)
> 
> 
> Review request for hive.
> 
> 
> Bugs: HIVE-5718
>     https://issues.apache.org/jira/browse/HIVE-5718
> 
> 
> Repository: hive-git
> 
> 
> Description
> -------
> 
> Extend HIVE-2925 with LV and SubQ.
> 
> 
> Diffs
> -----
> 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/OperatorFactory.java 5d41fa1 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/SimpleFetchOptimizer.java 7413d2b 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/QB.java 908db1e 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/QBParseInfo.java 911ac8a 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/FetchWork.java 32d84ea 
>   ql/src/test/queries/clientpositive/nonmr_fetch.q 2a92d17 
>   ql/src/test/queries/clientpositive/nonmr_fetch_threshold.q e6343e2 
>   ql/src/test/results/clientpositive/explain_logical.q.out bb26e8c 
>   ql/src/test/results/clientpositive/lateral_view_noalias.q.out d51b2de 
>   ql/src/test/results/clientpositive/nonmr_fetch.q.out 5a13e84 
>   ql/src/test/results/clientpositive/nonmr_fetch_threshold.q.out 39cdfa6 
>   ql/src/test/results/clientpositive/select_dummy_source.q.out 2742d56 
>   ql/src/test/results/clientpositive/subquery_alias.q.out 37bc3a4 
>   ql/src/test/results/clientpositive/udf_explode.q.out 4eeedeb 
>   ql/src/test/results/clientpositive/udf_inline.q.out e065bed 
>   ql/src/test/results/clientpositive/udf_reflect2.q.out 6b19277 
>   ql/src/test/results/clientpositive/udf_to_unix_timestamp.q.out 447ef87 
>   ql/src/test/results/clientpositive/udtf_explode.q.out ae95907 
> 
> Diff: https://reviews.apache.org/r/23351/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Navis Ryu
> 
>

Re: Review Request 23351: Support direct fetch for lateral views, sub queries, etc.

Posted by Navis Ryu <na...@nexr.com>.


> On Nov. 3, 2014, 10:28 p.m., John Pullokkaran wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/optimizer/SimpleFetchOptimizer.java, line 162
> > <https://reviews.apache.org/r/23351/diff/1/?file=626500#file626500line162>
> >
> >     Can't we use ParseContext.topToTable to get to "Table" given a TS object instead of walking the QB tree?

Good point! Sorry for missing this comment. I'll update the patch shortly after.


- Navis


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/23351/#review59662
-----------------------------------------------------------


On July 9, 2014, 6:55 a.m., Navis Ryu wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/23351/
> -----------------------------------------------------------
> 
> (Updated July 9, 2014, 6:55 a.m.)
> 
> 
> Review request for hive.
> 
> 
> Bugs: HIVE-5718
>     https://issues.apache.org/jira/browse/HIVE-5718
> 
> 
> Repository: hive-git
> 
> 
> Description
> -------
> 
> Extend HIVE-2925 with LV and SubQ.
> 
> 
> Diffs
> -----
> 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/OperatorFactory.java 5d41fa1 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/SimpleFetchOptimizer.java 7413d2b 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/QB.java 908db1e 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/QBParseInfo.java 911ac8a 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/FetchWork.java 32d84ea 
>   ql/src/test/queries/clientpositive/nonmr_fetch.q 2a92d17 
>   ql/src/test/queries/clientpositive/nonmr_fetch_threshold.q e6343e2 
>   ql/src/test/results/clientpositive/explain_logical.q.out bb26e8c 
>   ql/src/test/results/clientpositive/lateral_view_noalias.q.out d51b2de 
>   ql/src/test/results/clientpositive/nonmr_fetch.q.out 5a13e84 
>   ql/src/test/results/clientpositive/nonmr_fetch_threshold.q.out 39cdfa6 
>   ql/src/test/results/clientpositive/select_dummy_source.q.out 2742d56 
>   ql/src/test/results/clientpositive/subquery_alias.q.out 37bc3a4 
>   ql/src/test/results/clientpositive/udf_explode.q.out 4eeedeb 
>   ql/src/test/results/clientpositive/udf_inline.q.out e065bed 
>   ql/src/test/results/clientpositive/udf_reflect2.q.out 6b19277 
>   ql/src/test/results/clientpositive/udf_to_unix_timestamp.q.out 447ef87 
>   ql/src/test/results/clientpositive/udtf_explode.q.out ae95907 
> 
> Diff: https://reviews.apache.org/r/23351/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Navis Ryu
> 
>

Re: Review Request 23351: Support direct fetch for lateral views, sub queries, etc.

Posted by John Pullokkaran <jp...@hortonworks.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/23351/#review59662
-----------------------------------------------------------



ql/src/java/org/apache/hadoop/hive/ql/optimizer/SimpleFetchOptimizer.java
<https://reviews.apache.org/r/23351/#comment100951>

    Can't we use ParseContext.topToTable to get to "Table" given a TS object instead of walking the QB tree?


- John Pullokkaran


On July 9, 2014, 6:55 a.m., Navis Ryu wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/23351/
> -----------------------------------------------------------
> 
> (Updated July 9, 2014, 6:55 a.m.)
> 
> 
> Review request for hive.
> 
> 
> Bugs: HIVE-5718
>     https://issues.apache.org/jira/browse/HIVE-5718
> 
> 
> Repository: hive-git
> 
> 
> Description
> -------
> 
> Extend HIVE-2925 with LV and SubQ.
> 
> 
> Diffs
> -----
> 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/OperatorFactory.java 5d41fa1 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/SimpleFetchOptimizer.java 7413d2b 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/QB.java 908db1e 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/QBParseInfo.java 911ac8a 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/FetchWork.java 32d84ea 
>   ql/src/test/queries/clientpositive/nonmr_fetch.q 2a92d17 
>   ql/src/test/queries/clientpositive/nonmr_fetch_threshold.q e6343e2 
>   ql/src/test/results/clientpositive/explain_logical.q.out bb26e8c 
>   ql/src/test/results/clientpositive/lateral_view_noalias.q.out d51b2de 
>   ql/src/test/results/clientpositive/nonmr_fetch.q.out 5a13e84 
>   ql/src/test/results/clientpositive/nonmr_fetch_threshold.q.out 39cdfa6 
>   ql/src/test/results/clientpositive/select_dummy_source.q.out 2742d56 
>   ql/src/test/results/clientpositive/subquery_alias.q.out 37bc3a4 
>   ql/src/test/results/clientpositive/udf_explode.q.out 4eeedeb 
>   ql/src/test/results/clientpositive/udf_inline.q.out e065bed 
>   ql/src/test/results/clientpositive/udf_reflect2.q.out 6b19277 
>   ql/src/test/results/clientpositive/udf_to_unix_timestamp.q.out 447ef87 
>   ql/src/test/results/clientpositive/udtf_explode.q.out ae95907 
> 
> Diff: https://reviews.apache.org/r/23351/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Navis Ryu
> 
>