You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by Sergey Shelukhin <se...@hortonworks.com> on 2016/05/23 17:27:24 UTC
Review Request 47735: HIVE-13617 LLAP: support non-vectorized
execution in IO
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47735/
-----------------------------------------------------------
Review request for hive and Prasanth_J.
Repository: hive-git
Description
-------
see JIRA
Diffs
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java c0843b9
llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapInputFormat.java 298f788
ql/src/java/org/apache/hadoop/hive/ql/exec/MapOperator.java afe5ee2
ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java 636f079
ql/src/java/org/apache/hadoop/hive/ql/io/BatchToRowInputFormat.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/io/BatchToRowReader.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java cfedf35
ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java 33fe3b6
ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcOiBatchToRowReader.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/plan/MapWork.java f034812
ql/src/test/queries/clientpositive/orc_llap_nonvector.q PRE-CREATION
ql/src/test/results/clientpositive/orc_llap_nonvector.q.out PRE-CREATION
Diff: https://reviews.apache.org/r/47735/diff/
Testing
-------
Thanks,
Sergey Shelukhin
Re: Review Request 47735: HIVE-13617 LLAP: support non-vectorized
execution in IO
Posted by Sergey Shelukhin <se...@hortonworks.com>.
> On May 23, 2016, 10:37 p.m., Matt McCline wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/io/BatchToRowReader.java, line 181
> > <https://reviews.apache.org/r/47735/diff/1/?file=1391686#file1391686line181>
> >
> > The VectorExtractRow class and extractRowColumn method has similar functionality. But perhaps it is not a good reuse case.
>
> Matt McCline wrote:
> Is this only being used for ORC which has already done Schema Evolution below? Or, will this be used for non-ORC tables?
It looks like schema evolution may not work with LLAP IO. Sent an email to confirm.
- Sergey
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47735/#review134467
-----------------------------------------------------------
On June 10, 2016, 10:47 p.m., Sergey Shelukhin wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47735/
> -----------------------------------------------------------
>
> (Updated June 10, 2016, 10:47 p.m.)
>
>
> Review request for hive and Prasanth_J.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> see JIRA
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 285caa3
> itests/src/test/resources/testconfiguration.properties 46350a3
> llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapInputFormat.java 298f788
> ql/src/java/org/apache/hadoop/hive/ql/exec/MapOperator.java afe5ee2
> ql/src/java/org/apache/hadoop/hive/ql/io/BatchToRowInputFormat.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/io/BatchToRowReader.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java cfedf35
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java d7a8c2f
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcOiBatchToRowReader.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 7595065
> ql/src/java/org/apache/hadoop/hive/ql/plan/MapWork.java f034812
> ql/src/test/queries/clientpositive/orc_llap_nonvector.q PRE-CREATION
> ql/src/test/results/clientpositive/llap/dynamic_partition_pruning.q.out 912cd7f
> ql/src/test/results/clientpositive/llap/hybridgrace_hashjoin_1.q.out ad3a358
> ql/src/test/results/clientpositive/llap/llap_udf.q.out e517942
> ql/src/test/results/clientpositive/llap/llapdecider.q.out 27ab92a
> ql/src/test/results/clientpositive/llap/mapjoin_decimal.q.out 2d1b818
> ql/src/test/results/clientpositive/llap/orc_llap_nonvector.q.out PRE-CREATION
> ql/src/test/results/clientpositive/llap/orc_ppd_basic.q.out 2d0984b
> ql/src/test/results/clientpositive/llap/tez_bmj_schema_evolution.q.out f99b829
> ql/src/test/results/clientpositive/llap/tez_dynpart_hashjoin_1.q.out c9417f4
> ql/src/test/results/clientpositive/llap/tez_dynpart_hashjoin_2.q.out 7f29462
> ql/src/test/results/clientpositive/llap/tez_union_group_by.q.out 11882e4
> ql/src/test/results/clientpositive/llap/tez_vector_dynpart_hashjoin_1.q.out d02e096
> ql/src/test/results/clientpositive/llap/tez_vector_dynpart_hashjoin_2.q.out c0c14e1
> ql/src/test/results/clientpositive/llap/vectorized_dynamic_partition_pruning.q.out 487d930
> ql/src/test/results/clientpositive/orc_llap_nonvector.q.out PRE-CREATION
> ql/src/test/results/clientpositive/tez/llapdecider.q.out 27ab92a
>
> Diff: https://reviews.apache.org/r/47735/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Sergey Shelukhin
>
>
Re: Review Request 47735: HIVE-13617 LLAP: support non-vectorized
execution in IO
Posted by Matt McCline <mm...@hortonworks.com>.
> On May 23, 2016, 10:37 p.m., Matt McCline wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/io/BatchToRowReader.java, line 181
> > <https://reviews.apache.org/r/47735/diff/1/?file=1391686#file1391686line181>
> >
> > The VectorExtractRow class and extractRowColumn method has similar functionality. But perhaps it is not a good reuse case.
Is this only being used for ORC which has already done Schema Evolution below? Or, will this be used for non-ORC tables?
- Matt
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47735/#review134467
-----------------------------------------------------------
On May 23, 2016, 5:27 p.m., Sergey Shelukhin wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47735/
> -----------------------------------------------------------
>
> (Updated May 23, 2016, 5:27 p.m.)
>
>
> Review request for hive and Prasanth_J.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> see JIRA
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java c0843b9
> llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapInputFormat.java 298f788
> ql/src/java/org/apache/hadoop/hive/ql/exec/MapOperator.java afe5ee2
> ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java 636f079
> ql/src/java/org/apache/hadoop/hive/ql/io/BatchToRowInputFormat.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/io/BatchToRowReader.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java cfedf35
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java 33fe3b6
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcOiBatchToRowReader.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/plan/MapWork.java f034812
> ql/src/test/queries/clientpositive/orc_llap_nonvector.q PRE-CREATION
> ql/src/test/results/clientpositive/orc_llap_nonvector.q.out PRE-CREATION
>
> Diff: https://reviews.apache.org/r/47735/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Sergey Shelukhin
>
>
Re: Review Request 47735: HIVE-13617 LLAP: support non-vectorized
execution in IO
Posted by Matt McCline <mm...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47735/#review134467
-----------------------------------------------------------
ql/src/java/org/apache/hadoop/hive/ql/io/BatchToRowReader.java (line 181)
<https://reviews.apache.org/r/47735/#comment199284>
The VectorExtractRow class and extractRowColumn method has similar functionality. But perhaps it is not a good reuse case.
- Matt McCline
On May 23, 2016, 5:27 p.m., Sergey Shelukhin wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47735/
> -----------------------------------------------------------
>
> (Updated May 23, 2016, 5:27 p.m.)
>
>
> Review request for hive and Prasanth_J.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> see JIRA
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java c0843b9
> llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapInputFormat.java 298f788
> ql/src/java/org/apache/hadoop/hive/ql/exec/MapOperator.java afe5ee2
> ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java 636f079
> ql/src/java/org/apache/hadoop/hive/ql/io/BatchToRowInputFormat.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/io/BatchToRowReader.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java cfedf35
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java 33fe3b6
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcOiBatchToRowReader.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/plan/MapWork.java f034812
> ql/src/test/queries/clientpositive/orc_llap_nonvector.q PRE-CREATION
> ql/src/test/results/clientpositive/orc_llap_nonvector.q.out PRE-CREATION
>
> Diff: https://reviews.apache.org/r/47735/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Sergey Shelukhin
>
>
Re: Review Request 47735: HIVE-13617 LLAP: support non-vectorized
execution in IO
Posted by Sergey Shelukhin <se...@hortonworks.com>.
> On June 22, 2016, 2:35 a.m., Prasanth_J wrote:
> > llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapInputFormat.java, line 418
> > <https://reviews.apache.org/r/47735/diff/2/?file=1415331#file1415331line418>
> >
> > Can this not be reused?
no, Vectorizer vectorizes entire MapWork w/o good boundaries
> On June 22, 2016, 2:35 a.m., Prasanth_J wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/io/BatchToRowReader.java, line 78
> > <https://reviews.apache.org/r/47735/diff/2/?file=1415334#file1415334line78>
> >
> > Why UnionType?
hmm?
> On June 22, 2016, 2:35 a.m., Prasanth_J wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/io/BatchToRowReader.java, line 418
> > <https://reviews.apache.org/r/47735/diff/2/?file=1415334#file1415334line418>
> >
> > Should we do enforcing of precision and scale here or is it assumed to be already enforced?
Assuming it's enforced. This code is moved from original ORC code.
- Sergey
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47735/#review139006
-----------------------------------------------------------
On June 10, 2016, 10:47 p.m., Sergey Shelukhin wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47735/
> -----------------------------------------------------------
>
> (Updated June 10, 2016, 10:47 p.m.)
>
>
> Review request for hive and Prasanth_J.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> see JIRA
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 285caa3
> itests/src/test/resources/testconfiguration.properties 46350a3
> llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapInputFormat.java 298f788
> ql/src/java/org/apache/hadoop/hive/ql/exec/MapOperator.java afe5ee2
> ql/src/java/org/apache/hadoop/hive/ql/io/BatchToRowInputFormat.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/io/BatchToRowReader.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java cfedf35
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java d7a8c2f
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcOiBatchToRowReader.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 7595065
> ql/src/java/org/apache/hadoop/hive/ql/plan/MapWork.java f034812
> ql/src/test/queries/clientpositive/orc_llap_nonvector.q PRE-CREATION
> ql/src/test/results/clientpositive/llap/dynamic_partition_pruning.q.out 912cd7f
> ql/src/test/results/clientpositive/llap/hybridgrace_hashjoin_1.q.out ad3a358
> ql/src/test/results/clientpositive/llap/llap_udf.q.out e517942
> ql/src/test/results/clientpositive/llap/llapdecider.q.out 27ab92a
> ql/src/test/results/clientpositive/llap/mapjoin_decimal.q.out 2d1b818
> ql/src/test/results/clientpositive/llap/orc_llap_nonvector.q.out PRE-CREATION
> ql/src/test/results/clientpositive/llap/orc_ppd_basic.q.out 2d0984b
> ql/src/test/results/clientpositive/llap/tez_bmj_schema_evolution.q.out f99b829
> ql/src/test/results/clientpositive/llap/tez_dynpart_hashjoin_1.q.out c9417f4
> ql/src/test/results/clientpositive/llap/tez_dynpart_hashjoin_2.q.out 7f29462
> ql/src/test/results/clientpositive/llap/tez_union_group_by.q.out 11882e4
> ql/src/test/results/clientpositive/llap/tez_vector_dynpart_hashjoin_1.q.out d02e096
> ql/src/test/results/clientpositive/llap/tez_vector_dynpart_hashjoin_2.q.out c0c14e1
> ql/src/test/results/clientpositive/llap/vectorized_dynamic_partition_pruning.q.out 487d930
> ql/src/test/results/clientpositive/orc_llap_nonvector.q.out PRE-CREATION
> ql/src/test/results/clientpositive/tez/llapdecider.q.out 27ab92a
>
> Diff: https://reviews.apache.org/r/47735/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Sergey Shelukhin
>
>
Re: Review Request 47735: HIVE-13617 LLAP: support non-vectorized
execution in IO
Posted by j....@gmail.com.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47735/#review139006
-----------------------------------------------------------
llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapInputFormat.java (line 400)
<https://reviews.apache.org/r/47735/#comment204183>
Can this not be reused?
ql/src/java/org/apache/hadoop/hive/ql/io/BatchToRowReader.java (line 78)
<https://reviews.apache.org/r/47735/#comment204189>
Why UnionType?
ql/src/java/org/apache/hadoop/hive/ql/io/BatchToRowReader.java (line 418)
<https://reviews.apache.org/r/47735/#comment204190>
Should we do enforcing of precision and scale here or is it assumed to be already enforced?
- Prasanth_J
On June 10, 2016, 10:47 p.m., Sergey Shelukhin wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47735/
> -----------------------------------------------------------
>
> (Updated June 10, 2016, 10:47 p.m.)
>
>
> Review request for hive and Prasanth_J.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> see JIRA
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 285caa3
> itests/src/test/resources/testconfiguration.properties 46350a3
> llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapInputFormat.java 298f788
> ql/src/java/org/apache/hadoop/hive/ql/exec/MapOperator.java afe5ee2
> ql/src/java/org/apache/hadoop/hive/ql/io/BatchToRowInputFormat.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/io/BatchToRowReader.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java cfedf35
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java d7a8c2f
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcOiBatchToRowReader.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 7595065
> ql/src/java/org/apache/hadoop/hive/ql/plan/MapWork.java f034812
> ql/src/test/queries/clientpositive/orc_llap_nonvector.q PRE-CREATION
> ql/src/test/results/clientpositive/llap/dynamic_partition_pruning.q.out 912cd7f
> ql/src/test/results/clientpositive/llap/hybridgrace_hashjoin_1.q.out ad3a358
> ql/src/test/results/clientpositive/llap/llap_udf.q.out e517942
> ql/src/test/results/clientpositive/llap/llapdecider.q.out 27ab92a
> ql/src/test/results/clientpositive/llap/mapjoin_decimal.q.out 2d1b818
> ql/src/test/results/clientpositive/llap/orc_llap_nonvector.q.out PRE-CREATION
> ql/src/test/results/clientpositive/llap/orc_ppd_basic.q.out 2d0984b
> ql/src/test/results/clientpositive/llap/tez_bmj_schema_evolution.q.out f99b829
> ql/src/test/results/clientpositive/llap/tez_dynpart_hashjoin_1.q.out c9417f4
> ql/src/test/results/clientpositive/llap/tez_dynpart_hashjoin_2.q.out 7f29462
> ql/src/test/results/clientpositive/llap/tez_union_group_by.q.out 11882e4
> ql/src/test/results/clientpositive/llap/tez_vector_dynpart_hashjoin_1.q.out d02e096
> ql/src/test/results/clientpositive/llap/tez_vector_dynpart_hashjoin_2.q.out c0c14e1
> ql/src/test/results/clientpositive/llap/vectorized_dynamic_partition_pruning.q.out 487d930
> ql/src/test/results/clientpositive/orc_llap_nonvector.q.out PRE-CREATION
> ql/src/test/results/clientpositive/tez/llapdecider.q.out 27ab92a
>
> Diff: https://reviews.apache.org/r/47735/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Sergey Shelukhin
>
>
Re: Review Request 47735: HIVE-13617 LLAP: support non-vectorized
execution in IO
Posted by Sergey Shelukhin <se...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47735/
-----------------------------------------------------------
(Updated June 10, 2016, 10:47 p.m.)
Review request for hive and Prasanth_J.
Repository: hive-git
Description
-------
see JIRA
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 285caa3
itests/src/test/resources/testconfiguration.properties 46350a3
llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapInputFormat.java 298f788
ql/src/java/org/apache/hadoop/hive/ql/exec/MapOperator.java afe5ee2
ql/src/java/org/apache/hadoop/hive/ql/io/BatchToRowInputFormat.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/io/BatchToRowReader.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java cfedf35
ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java d7a8c2f
ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcOiBatchToRowReader.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 7595065
ql/src/java/org/apache/hadoop/hive/ql/plan/MapWork.java f034812
ql/src/test/queries/clientpositive/orc_llap_nonvector.q PRE-CREATION
ql/src/test/results/clientpositive/llap/dynamic_partition_pruning.q.out 912cd7f
ql/src/test/results/clientpositive/llap/hybridgrace_hashjoin_1.q.out ad3a358
ql/src/test/results/clientpositive/llap/llap_udf.q.out e517942
ql/src/test/results/clientpositive/llap/llapdecider.q.out 27ab92a
ql/src/test/results/clientpositive/llap/mapjoin_decimal.q.out 2d1b818
ql/src/test/results/clientpositive/llap/orc_llap_nonvector.q.out PRE-CREATION
ql/src/test/results/clientpositive/llap/orc_ppd_basic.q.out 2d0984b
ql/src/test/results/clientpositive/llap/tez_bmj_schema_evolution.q.out f99b829
ql/src/test/results/clientpositive/llap/tez_dynpart_hashjoin_1.q.out c9417f4
ql/src/test/results/clientpositive/llap/tez_dynpart_hashjoin_2.q.out 7f29462
ql/src/test/results/clientpositive/llap/tez_union_group_by.q.out 11882e4
ql/src/test/results/clientpositive/llap/tez_vector_dynpart_hashjoin_1.q.out d02e096
ql/src/test/results/clientpositive/llap/tez_vector_dynpart_hashjoin_2.q.out c0c14e1
ql/src/test/results/clientpositive/llap/vectorized_dynamic_partition_pruning.q.out 487d930
ql/src/test/results/clientpositive/orc_llap_nonvector.q.out PRE-CREATION
ql/src/test/results/clientpositive/tez/llapdecider.q.out 27ab92a
Diff: https://reviews.apache.org/r/47735/diff/
Testing
-------
Thanks,
Sergey Shelukhin