You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by Sergey Shelukhin <se...@hortonworks.com> on 2016/11/09 02:34:41 UTC
Review Request 53593: HIVE-14089 complex type support in LLAP IO is
broken
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53593/
-----------------------------------------------------------
Review request for hive and Prasanth_J.
Repository: hive-git
Description
-------
see jira
Diffs
-----
llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapInputFormat.java a27266f
llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/OrcEncodedDataConsumer.java 2fa68a9
llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java eb8ee6c
llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcFileMetadata.java c9b0a4d
orc/src/java/org/apache/orc/impl/ConvertTreeReaderFactory.java c347181
orc/src/java/org/apache/orc/impl/TreeReaderFactory.java 245e3ce
ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java a9dbc3e
ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java ccb39da
ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReader.java 4d09dcd
ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReaderImpl.java bcb54d6
ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedTreeReaderFactory.java ebbdf8d
ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/Reader.java 4405232
ql/src/java/org/apache/hadoop/hive/ql/plan/MapWork.java 5cc3663
ql/src/test/queries/clientpositive/vector_complex_all.q 91a7368
ql/src/test/results/clientpositive/llap/vector_complex_all.q.out 08d49bc
ql/src/test/results/clientpositive/vector_complex_all.q.out a54a371
service/src/java/org/apache/hive/service/cli/operation/Operation.java 76a1286
storage-api/src/java/org/apache/hadoop/hive/common/io/encoded/EncodedColumnBatch.java 907181e
Diff: https://reviews.apache.org/r/53593/diff/
Testing
-------
Thanks,
Sergey Shelukhin
Re: Review Request 53593: HIVE-14089 complex type support in LLAP IO
is broken
Posted by Sergey Shelukhin <se...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53593/#review155539
-----------------------------------------------------------
llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java
<https://reviews.apache.org/r/53593/#comment225516>
this should not be removed
ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReaderImpl.java (line 210)
<https://reviews.apache.org/r/53593/#comment225517>
should be removed
storage-api/src/java/org/apache/hadoop/hive/common/io/encoded/EncodedColumnBatch.java (line 84)
<https://reviews.apache.org/r/53593/#comment225518>
stale comment
- Sergey Shelukhin
On Nov. 9, 2016, 2:34 a.m., Sergey Shelukhin wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/53593/
> -----------------------------------------------------------
>
> (Updated Nov. 9, 2016, 2:34 a.m.)
>
>
> Review request for hive and Prasanth_J.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> see jira
>
>
> Diffs
> -----
>
> llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapInputFormat.java a27266f
> llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/OrcEncodedDataConsumer.java 2fa68a9
> llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java eb8ee6c
> llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcFileMetadata.java c9b0a4d
> orc/src/java/org/apache/orc/impl/ConvertTreeReaderFactory.java c347181
> orc/src/java/org/apache/orc/impl/TreeReaderFactory.java 245e3ce
> ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java a9dbc3e
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java ccb39da
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReader.java 4d09dcd
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReaderImpl.java bcb54d6
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedTreeReaderFactory.java ebbdf8d
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/Reader.java 4405232
> ql/src/java/org/apache/hadoop/hive/ql/plan/MapWork.java 5cc3663
> ql/src/test/queries/clientpositive/vector_complex_all.q 91a7368
> ql/src/test/results/clientpositive/llap/vector_complex_all.q.out 08d49bc
> ql/src/test/results/clientpositive/vector_complex_all.q.out a54a371
> service/src/java/org/apache/hive/service/cli/operation/Operation.java 76a1286
> storage-api/src/java/org/apache/hadoop/hive/common/io/encoded/EncodedColumnBatch.java 907181e
>
> Diff: https://reviews.apache.org/r/53593/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Sergey Shelukhin
>
>
Re: Review Request 53593: HIVE-14089 complex type support in LLAP IO
is broken
Posted by j....@gmail.com.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53593/#review155870
-----------------------------------------------------------
Ship it!
Ship It!
- Prasanth_J
On Nov. 14, 2016, 8:45 p.m., Sergey Shelukhin wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/53593/
> -----------------------------------------------------------
>
> (Updated Nov. 14, 2016, 8:45 p.m.)
>
>
> Review request for hive and Prasanth_J.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> see jira
>
>
> Diffs
> -----
>
> llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapInputFormat.java a27266f
> llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/OrcEncodedDataConsumer.java 2fa68a9
> llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java 72f733a
> llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcFileMetadata.java c9b0a4d
> orc/src/java/org/apache/orc/impl/ConvertTreeReaderFactory.java c347181
> orc/src/java/org/apache/orc/impl/TreeReaderFactory.java 245e3ce
> ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java a9dbc3e
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java ccb39da
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReader.java 4d09dcd
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReaderImpl.java a4925b9
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedTreeReaderFactory.java ebbdf8d
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/Reader.java 4405232
> ql/src/java/org/apache/hadoop/hive/ql/plan/MapWork.java 5cc3663
> ql/src/test/queries/clientpositive/vector_complex_all.q 91a7368
> ql/src/test/results/clientpositive/llap/vector_complex_all.q.out 08d49bc
> ql/src/test/results/clientpositive/llap/vector_complex_join.q.out 97d5642
> ql/src/test/results/clientpositive/vector_complex_all.q.out a54a371
> storage-api/src/java/org/apache/hadoop/hive/common/io/encoded/EncodedColumnBatch.java 907181e
>
> Diff: https://reviews.apache.org/r/53593/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Sergey Shelukhin
>
>
Re: Review Request 53593: HIVE-14089 complex type support in LLAP IO
is broken
Posted by Sergey Shelukhin <se...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53593/
-----------------------------------------------------------
(Updated Nov. 14, 2016, 8:45 p.m.)
Review request for hive and Prasanth_J.
Repository: hive-git
Description
-------
see jira
Diffs (updated)
-----
llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapInputFormat.java a27266f
llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/OrcEncodedDataConsumer.java 2fa68a9
llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java 72f733a
llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcFileMetadata.java c9b0a4d
orc/src/java/org/apache/orc/impl/ConvertTreeReaderFactory.java c347181
orc/src/java/org/apache/orc/impl/TreeReaderFactory.java 245e3ce
ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java a9dbc3e
ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java ccb39da
ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReader.java 4d09dcd
ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReaderImpl.java a4925b9
ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedTreeReaderFactory.java ebbdf8d
ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/Reader.java 4405232
ql/src/java/org/apache/hadoop/hive/ql/plan/MapWork.java 5cc3663
ql/src/test/queries/clientpositive/vector_complex_all.q 91a7368
ql/src/test/results/clientpositive/llap/vector_complex_all.q.out 08d49bc
ql/src/test/results/clientpositive/llap/vector_complex_join.q.out 97d5642
ql/src/test/results/clientpositive/vector_complex_all.q.out a54a371
storage-api/src/java/org/apache/hadoop/hive/common/io/encoded/EncodedColumnBatch.java 907181e
Diff: https://reviews.apache.org/r/53593/diff/
Testing
-------
Thanks,
Sergey Shelukhin
Re: Review Request 53593: HIVE-14089 complex type support in LLAP IO
is broken
Posted by Sergey Shelukhin <se...@hortonworks.com>.
> On Nov. 11, 2016, 12:28 a.m., Prasanth_J wrote:
> > llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/OrcEncodedDataConsumer.java, line 329
> > <https://reviews.apache.org/r/53593/diff/1/?file=1558789#file1558789line329>
> >
> > nit: Can the batchkey be null?
no
- Sergey
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53593/#review155661
-----------------------------------------------------------
On Nov. 9, 2016, 2:34 a.m., Sergey Shelukhin wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/53593/
> -----------------------------------------------------------
>
> (Updated Nov. 9, 2016, 2:34 a.m.)
>
>
> Review request for hive and Prasanth_J.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> see jira
>
>
> Diffs
> -----
>
> llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapInputFormat.java a27266f
> llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/OrcEncodedDataConsumer.java 2fa68a9
> llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java eb8ee6c
> llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcFileMetadata.java c9b0a4d
> orc/src/java/org/apache/orc/impl/ConvertTreeReaderFactory.java c347181
> orc/src/java/org/apache/orc/impl/TreeReaderFactory.java 245e3ce
> ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java a9dbc3e
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java ccb39da
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReader.java 4d09dcd
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReaderImpl.java bcb54d6
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedTreeReaderFactory.java ebbdf8d
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/Reader.java 4405232
> ql/src/java/org/apache/hadoop/hive/ql/plan/MapWork.java 5cc3663
> ql/src/test/queries/clientpositive/vector_complex_all.q 91a7368
> ql/src/test/results/clientpositive/llap/vector_complex_all.q.out 08d49bc
> ql/src/test/results/clientpositive/vector_complex_all.q.out a54a371
> service/src/java/org/apache/hive/service/cli/operation/Operation.java 76a1286
> storage-api/src/java/org/apache/hadoop/hive/common/io/encoded/EncodedColumnBatch.java 907181e
>
> Diff: https://reviews.apache.org/r/53593/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Sergey Shelukhin
>
>
Re: Review Request 53593: HIVE-14089 complex type support in LLAP IO
is broken
Posted by j....@gmail.com.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53593/#review155661
-----------------------------------------------------------
llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapInputFormat.java (line 119)
<https://reviews.apache.org/r/53593/#comment225689>
Remove it altogether?
llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/OrcEncodedDataConsumer.java (line 259)
<https://reviews.apache.org/r/53593/#comment225734>
nit: Can the batchkey be null?
orc/src/java/org/apache/orc/impl/TreeReaderFactory.java (line 1831)
<https://reviews.apache.org/r/53593/#comment225733>
RLE reader for tags stream (tags stored in DATA stream) should be created.
ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReaderImpl.java (line 219)
<https://reviews.apache.org/r/53593/#comment225727>
All error changes seem unnecessary? Can we log this at DEBUG or TRACE instead?
ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedTreeReaderFactory.java (line 2021)
<https://reviews.apache.org/r/53593/#comment225732>
We should do seek into lengths when the stream is not empty.
ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedTreeReaderFactory.java (line 2141)
<https://reviews.apache.org/r/53593/#comment225731>
We should seek into lengths too. No sure how easy it would be to add test case for this. Since we don't do PPD on complext types.
- Prasanth_J
On Nov. 9, 2016, 2:34 a.m., Sergey Shelukhin wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/53593/
> -----------------------------------------------------------
>
> (Updated Nov. 9, 2016, 2:34 a.m.)
>
>
> Review request for hive and Prasanth_J.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> see jira
>
>
> Diffs
> -----
>
> llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapInputFormat.java a27266f
> llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/OrcEncodedDataConsumer.java 2fa68a9
> llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java eb8ee6c
> llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcFileMetadata.java c9b0a4d
> orc/src/java/org/apache/orc/impl/ConvertTreeReaderFactory.java c347181
> orc/src/java/org/apache/orc/impl/TreeReaderFactory.java 245e3ce
> ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java a9dbc3e
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java ccb39da
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReader.java 4d09dcd
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReaderImpl.java bcb54d6
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedTreeReaderFactory.java ebbdf8d
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/Reader.java 4405232
> ql/src/java/org/apache/hadoop/hive/ql/plan/MapWork.java 5cc3663
> ql/src/test/queries/clientpositive/vector_complex_all.q 91a7368
> ql/src/test/results/clientpositive/llap/vector_complex_all.q.out 08d49bc
> ql/src/test/results/clientpositive/vector_complex_all.q.out a54a371
> service/src/java/org/apache/hive/service/cli/operation/Operation.java 76a1286
> storage-api/src/java/org/apache/hadoop/hive/common/io/encoded/EncodedColumnBatch.java 907181e
>
> Diff: https://reviews.apache.org/r/53593/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Sergey Shelukhin
>
>