You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by Rohit Dholakia <ro...@simba.com> on 2015/06/23 22:39:30 UTC
Review Request 35792: HIVE-10438 - Architecture for ResultSet
Compression via external plugin
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35792/
-----------------------------------------------------------
Review request for hive, Vaibhav Gumashta and Xuefu Zhang.
Repository: hive-git
Description
-------
This patch enables ResultSet compression for Hive using external plugins. The patch proposes a plugin architecture that enables using external plugins to compress ResultSets on-the-fly.
Diffs
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 27f68df
conf/hive-site.xml dab494e
service/if/TCLIService.thrift baf583f
service/src/gen/thrift/gen-cpp/TCLIService_types.h 7bceabd
service/src/gen/thrift/gen-cpp/TCLIService_types.cpp 86eeea3
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TEnColumn.java PRE-CREATION
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TExecuteStatementReq.java 4f157ad
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TGetTablesReq.java c973fcc
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TOpenSessionReq.java c048161
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TOpenSessionResp.java 351f78b
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TProtocolVersion.java a4279d2
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TRowSet.java d16c8a4
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TStatus.java 24a746e
service/src/gen/thrift/gen-py/TCLIService/ttypes.py 068727c
service/src/gen/thrift/gen-rb/t_c_l_i_service_types.rb b482533
service/src/java/org/apache/hive/service/cli/Column.java 2e21f18
service/src/java/org/apache/hive/service/cli/ColumnBasedSet.java 47a582e
service/src/java/org/apache/hive/service/cli/ColumnCompressor.java PRE-CREATION
service/src/java/org/apache/hive/service/cli/ColumnCompressorService.java PRE-CREATION
service/src/java/org/apache/hive/service/cli/EncodedColumnBasedSet.java PRE-CREATION
service/src/java/org/apache/hive/service/cli/RowSetFactory.java e8f68ea
service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java dfb7faa
Diff: https://reviews.apache.org/r/35792/diff/
Testing
-------
Testing has been done using a docker container-based query submitter that has an integer decompressor as part of it. Using the integer compressor (also provided) and the decompressor, the end-to-end functionality can be observed.
File Attachments
----------------
Patch file
https://reviews.apache.org/media/uploaded/files/2015/06/23/16aa08f8-2393-460a-83ef-72464fc537db__HIVE-10438.patch
Thanks,
Rohit Dholakia
Re: Review Request 35792: HIVE-10438 - Architecture for ResultSet
Compression via external plugin
Posted by Xuefu Zhang <xz...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35792/#review89285
-----------------------------------------------------------
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java (line 2202)
<https://reviews.apache.org/r/35792/#comment141873>
1. Remove leading spaces or tabs.
2. The default should be "false"
conf/hive-site.xml (line 22)
<https://reviews.apache.org/r/35792/#comment141875>
We probably don't want to add anything here, as these properties have default values which are sufficient.
service/src/java/org/apache/hive/service/cli/ColumnCompressor.java (line 15)
<https://reviews.apache.org/r/35792/#comment141880>
Documentation for these public APIs?
service/src/java/org/apache/hive/service/cli/ColumnCompressorService.java (line 11)
<https://reviews.apache.org/r/35792/#comment141878>
1. Why hashtable? We should use HashMap or ConcurrentHashMap rather than HashTable.
2. <> will break people building Hive with Java 6.
service/src/java/org/apache/hive/service/cli/ColumnCompressorService.java (line 19)
<https://reviews.apache.org/r/35792/#comment141876>
Remove trailing spaces. Same for other occurrences.
First round of review comments. Most cosmatic. More will come.
- Xuefu Zhang
On June 23, 2015, 8:39 p.m., Rohit Dholakia wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/35792/
> -----------------------------------------------------------
>
> (Updated June 23, 2015, 8:39 p.m.)
>
>
> Review request for hive, Vaibhav Gumashta and Xuefu Zhang.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> This patch enables ResultSet compression for Hive using external plugins. The patch proposes a plugin architecture that enables using external plugins to compress ResultSets on-the-fly.
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 27f68df
> conf/hive-site.xml dab494e
> service/if/TCLIService.thrift baf583f
> service/src/gen/thrift/gen-cpp/TCLIService_types.h 7bceabd
> service/src/gen/thrift/gen-cpp/TCLIService_types.cpp 86eeea3
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TEnColumn.java PRE-CREATION
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TExecuteStatementReq.java 4f157ad
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TGetTablesReq.java c973fcc
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TOpenSessionReq.java c048161
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TOpenSessionResp.java 351f78b
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TProtocolVersion.java a4279d2
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TRowSet.java d16c8a4
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TStatus.java 24a746e
> service/src/gen/thrift/gen-py/TCLIService/ttypes.py 068727c
> service/src/gen/thrift/gen-rb/t_c_l_i_service_types.rb b482533
> service/src/java/org/apache/hive/service/cli/Column.java 2e21f18
> service/src/java/org/apache/hive/service/cli/ColumnBasedSet.java 47a582e
> service/src/java/org/apache/hive/service/cli/ColumnCompressor.java PRE-CREATION
> service/src/java/org/apache/hive/service/cli/ColumnCompressorService.java PRE-CREATION
> service/src/java/org/apache/hive/service/cli/EncodedColumnBasedSet.java PRE-CREATION
> service/src/java/org/apache/hive/service/cli/RowSetFactory.java e8f68ea
> service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java dfb7faa
>
> Diff: https://reviews.apache.org/r/35792/diff/
>
>
> Testing
> -------
>
> Testing has been done using a docker container-based query submitter that has an integer decompressor as part of it. Using the integer compressor (also provided) and the decompressor, the end-to-end functionality can be observed.
>
>
> File Attachments
> ----------------
>
> Patch file
> https://reviews.apache.org/media/uploaded/files/2015/06/23/16aa08f8-2393-460a-83ef-72464fc537db__HIVE-10438.patch
>
>
> Thanks,
>
> Rohit Dholakia
>
>
Re: Review Request 35792: HIVE-10438 - Architecture for ResultSet
Compression via external plugin
Posted by Xuefu Zhang <xz...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35792/#review102547
-----------------------------------------------------------
Besides the comments in the code, I have the following questions:
1. Do we have to support multiple compressors and otherwise the functionality is incomplete? Or, can we support one and later enhance if demain arises?
2. As to client telling the server its capability regarding supported compressor, what's the practice in the traditional databases? Is JSON string the typical practice. I feel that JSON is a little unwieldy.
service/if/TCLIService.thrift (line 406)
<https://reviews.apache.org/r/35792/#comment160251>
What's the use of type and size? Are these not available by looking at result set schema?
service/src/java/org/apache/hive/service/cli/compression/ColumnCompressorService.java (line 34)
<https://reviews.apache.org/r/35792/#comment160249>
It seems that calling this as a serive is a little confusing. Maybe we can just call it "factory"?
service/src/java/org/apache/hive/service/cli/compression/EncodedColumnBasedSet.java (line 92)
<https://reviews.apache.org/r/35792/#comment160246>
If hiveConf is mandatory, should we enforce it by requiring it in the constructor rather than a set method?
service/src/java/org/apache/hive/service/cli/compression/EncodedColumnBasedSet.java (line 110)
<https://reviews.apache.org/r/35792/#comment160248>
Can we do this once per user session instead of per result set?
- Xuefu Zhang
On Aug. 17, 2015, 10:37 p.m., Rohit Dholakia wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/35792/
> -----------------------------------------------------------
>
> (Updated Aug. 17, 2015, 10:37 p.m.)
>
>
> Review request for hive, Vaibhav Gumashta, Xiaojian Wang, Xiao Meng, and Xuefu Zhang.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> This patch enables ResultSet compression for Hive using external plugins. The patch proposes a plugin architecture that enables using external plugins to compress ResultSets on-the-fly.
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 730f5be
> jdbc/src/java/org/apache/hive/jdbc/HiveConnection.java bb2b695
> service/if/TCLIService.thrift baf583f
> service/src/gen/thrift/gen-cpp/TCLIService.h 29a9f4a
> service/src/gen/thrift/gen-cpp/TCLIService_types.h 4536b41
> service/src/gen/thrift/gen-cpp/TCLIService_types.cpp 742cfdc
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TEnColumn.java PRE-CREATION
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TExecuteStatementReq.java feaed34
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TGetTablesReq.java 805e69f
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TOpenSessionReq.java 657f868
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TOpenSessionResp.java 48f4b45
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TProtocolVersion.java 6e714c6
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TRowSet.java cc1a148
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TStatus.java 1cd7980
> service/src/gen/thrift/gen-py/TCLIService/ttypes.py efee8ef
> service/src/gen/thrift/gen-rb/t_c_l_i_service_types.rb bfb2b69
> service/src/java/org/apache/hive/service/cli/Column.java 2e21f18
> service/src/java/org/apache/hive/service/cli/ColumnBasedSet.java 47a582e
> service/src/java/org/apache/hive/service/cli/RowSetFactory.java e8f68ea
> service/src/java/org/apache/hive/service/cli/compression/ColumnCompressor.java PRE-CREATION
> service/src/java/org/apache/hive/service/cli/compression/ColumnCompressorService.java PRE-CREATION
> service/src/java/org/apache/hive/service/cli/compression/EncodedColumnBasedSet.java PRE-CREATION
> service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java 67bc778
> service/src/test/org/apache/hive/service/cli/compression/SnappyIntCompressor.java PRE-CREATION
> service/src/test/org/apache/hive/service/cli/compression/TestEncodedColumnBasedSet.java PRE-CREATION
> service/src/test/resources/META-INF/services/org.apache.hive.service.cli.compression.ColumnCompressor PRE-CREATION
>
> Diff: https://reviews.apache.org/r/35792/diff/
>
>
> Testing
> -------
>
> Testing has been done using a docker container-based query submitter that has an integer decompressor as part of it. Using the integer compressor (also provided) and the decompressor, the end-to-end functionality can be observed.
>
>
> File Attachments
> ----------------
>
> Patch file
> https://reviews.apache.org/media/uploaded/files/2015/06/23/16aa08f8-2393-460a-83ef-72464fc537db__HIVE-10438.patch
>
>
> Thanks,
>
> Rohit Dholakia
>
>
Re: Review Request 35792: HIVE-10438 - Architecture for ResultSet
Compression via external plugin
Posted by Rohit Dholakia <ro...@simba.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35792/
-----------------------------------------------------------
(Updated Aug. 17, 2015, 10:37 p.m.)
Review request for hive, Vaibhav Gumashta, Xiaojian Wang, Xiao Meng, and Xuefu Zhang.
Changes
-------
Reverted two .gitignore files.
Repository: hive-git
Description
-------
This patch enables ResultSet compression for Hive using external plugins. The patch proposes a plugin architecture that enables using external plugins to compress ResultSets on-the-fly.
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 730f5be
jdbc/src/java/org/apache/hive/jdbc/HiveConnection.java bb2b695
service/if/TCLIService.thrift baf583f
service/src/gen/thrift/gen-cpp/TCLIService.h 29a9f4a
service/src/gen/thrift/gen-cpp/TCLIService_types.h 4536b41
service/src/gen/thrift/gen-cpp/TCLIService_types.cpp 742cfdc
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TEnColumn.java PRE-CREATION
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TExecuteStatementReq.java feaed34
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TGetTablesReq.java 805e69f
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TOpenSessionReq.java 657f868
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TOpenSessionResp.java 48f4b45
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TProtocolVersion.java 6e714c6
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TRowSet.java cc1a148
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TStatus.java 1cd7980
service/src/gen/thrift/gen-py/TCLIService/ttypes.py efee8ef
service/src/gen/thrift/gen-rb/t_c_l_i_service_types.rb bfb2b69
service/src/java/org/apache/hive/service/cli/Column.java 2e21f18
service/src/java/org/apache/hive/service/cli/ColumnBasedSet.java 47a582e
service/src/java/org/apache/hive/service/cli/RowSetFactory.java e8f68ea
service/src/java/org/apache/hive/service/cli/compression/ColumnCompressor.java PRE-CREATION
service/src/java/org/apache/hive/service/cli/compression/ColumnCompressorService.java PRE-CREATION
service/src/java/org/apache/hive/service/cli/compression/EncodedColumnBasedSet.java PRE-CREATION
service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java 67bc778
service/src/test/org/apache/hive/service/cli/compression/SnappyIntCompressor.java PRE-CREATION
service/src/test/org/apache/hive/service/cli/compression/TestEncodedColumnBasedSet.java PRE-CREATION
service/src/test/resources/META-INF/services/org.apache.hive.service.cli.compression.ColumnCompressor PRE-CREATION
Diff: https://reviews.apache.org/r/35792/diff/
Testing
-------
Testing has been done using a docker container-based query submitter that has an integer decompressor as part of it. Using the integer compressor (also provided) and the decompressor, the end-to-end functionality can be observed.
File Attachments
----------------
Patch file
https://reviews.apache.org/media/uploaded/files/2015/06/23/16aa08f8-2393-460a-83ef-72464fc537db__HIVE-10438.patch
Thanks,
Rohit Dholakia
Re: Review Request 35792: HIVE-10438 - Architecture for ResultSet
Compression via external plugin
Posted by Rohit Dholakia <ro...@simba.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35792/
-----------------------------------------------------------
(Updated Aug. 17, 2015, 8:32 p.m.)
Review request for hive, Vaibhav Gumashta, Xiaojian Wang, Xiao Meng, and Xuefu Zhang.
Changes
-------
1. Fixed whitespace issues.
2. A bug in one of the tests.
3. Some generated files that are necessary weren't committed before. They are now.
Repository: hive-git
Description
-------
This patch enables ResultSet compression for Hive using external plugins. The patch proposes a plugin architecture that enables using external plugins to compress ResultSets on-the-fly.
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 730f5be
hcatalog/core/.gitignore 0a7a9c5
jdbc/src/java/org/apache/hive/jdbc/HiveConnection.java bb2b695
metastore/bin/.gitignore 0e4bba6
service/if/TCLIService.thrift baf583f
service/src/gen/thrift/gen-cpp/TCLIService.h 29a9f4a
service/src/gen/thrift/gen-cpp/TCLIService_types.h 4536b41
service/src/gen/thrift/gen-cpp/TCLIService_types.cpp 742cfdc
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TEnColumn.java PRE-CREATION
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TExecuteStatementReq.java feaed34
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TGetTablesReq.java 805e69f
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TOpenSessionReq.java 657f868
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TOpenSessionResp.java 48f4b45
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TProtocolVersion.java 6e714c6
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TRowSet.java cc1a148
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TStatus.java 1cd7980
service/src/gen/thrift/gen-py/TCLIService/ttypes.py efee8ef
service/src/gen/thrift/gen-rb/t_c_l_i_service_types.rb bfb2b69
service/src/java/org/apache/hive/service/cli/Column.java 2e21f18
service/src/java/org/apache/hive/service/cli/ColumnBasedSet.java 47a582e
service/src/java/org/apache/hive/service/cli/RowSetFactory.java e8f68ea
service/src/java/org/apache/hive/service/cli/compression/ColumnCompressor.java PRE-CREATION
service/src/java/org/apache/hive/service/cli/compression/ColumnCompressorService.java PRE-CREATION
service/src/java/org/apache/hive/service/cli/compression/EncodedColumnBasedSet.java PRE-CREATION
service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java 67bc778
service/src/test/org/apache/hive/service/cli/compression/SnappyIntCompressor.java PRE-CREATION
service/src/test/org/apache/hive/service/cli/compression/TestEncodedColumnBasedSet.java PRE-CREATION
service/src/test/resources/META-INF/services/org.apache.hive.service.cli.compression.ColumnCompressor PRE-CREATION
Diff: https://reviews.apache.org/r/35792/diff/
Testing
-------
Testing has been done using a docker container-based query submitter that has an integer decompressor as part of it. Using the integer compressor (also provided) and the decompressor, the end-to-end functionality can be observed.
File Attachments
----------------
Patch file
https://reviews.apache.org/media/uploaded/files/2015/06/23/16aa08f8-2393-460a-83ef-72464fc537db__HIVE-10438.patch
Thanks,
Rohit Dholakia
Re: Review Request 35792: HIVE-10438 - Architecture for ResultSet
Compression via external plugin
Posted by Rohit Dholakia <ro...@simba.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35792/
-----------------------------------------------------------
(Updated Aug. 15, 2015, 1:11 a.m.)
Review request for hive, Vaibhav Gumashta, Xiaojian Wang, Xiao Meng, and Xuefu Zhang.
Changes
-------
Removed the generated code that wasn't necessary. Reuploading the patch.
Repository: hive-git
Description
-------
This patch enables ResultSet compression for Hive using external plugins. The patch proposes a plugin architecture that enables using external plugins to compress ResultSets on-the-fly.
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 730f5be
jdbc/src/java/org/apache/hive/jdbc/HiveConnection.java bb2b695
service/if/TCLIService.thrift baf583f
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TEnColumn.java PRE-CREATION
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TProtocolVersion.java 6e714c6
service/src/gen/thrift/gen-rb/t_c_l_i_service_types.rb bfb2b69
service/src/java/org/apache/hive/service/cli/Column.java 2e21f18
service/src/java/org/apache/hive/service/cli/ColumnBasedSet.java 47a582e
service/src/java/org/apache/hive/service/cli/RowSetFactory.java e8f68ea
service/src/java/org/apache/hive/service/cli/compression/ColumnCompressor.java PRE-CREATION
service/src/java/org/apache/hive/service/cli/compression/ColumnCompressorService.java PRE-CREATION
service/src/java/org/apache/hive/service/cli/compression/EncodedColumnBasedSet.java PRE-CREATION
service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java 67bc778
service/src/test/org/apache/hive/service/cli/compression/SnappyIntCompressor.java PRE-CREATION
service/src/test/org/apache/hive/service/cli/compression/TestEncodedColumnBasedSet.java PRE-CREATION
service/src/test/resources/META-INF/services/org.apache.hive.service.cli.compression.ColumnCompressor PRE-CREATION
Diff: https://reviews.apache.org/r/35792/diff/
Testing
-------
Testing has been done using a docker container-based query submitter that has an integer decompressor as part of it. Using the integer compressor (also provided) and the decompressor, the end-to-end functionality can be observed.
File Attachments
----------------
Patch file
https://reviews.apache.org/media/uploaded/files/2015/06/23/16aa08f8-2393-460a-83ef-72464fc537db__HIVE-10438.patch
Thanks,
Rohit Dholakia
Re: Review Request 35792: HIVE-10438 - Architecture for ResultSet
Compression via external plugin
Posted by Rohit Dholakia <ro...@simba.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35792/
-----------------------------------------------------------
(Updated Aug. 15, 2015, 12:29 a.m.)
Review request for hive, Vaibhav Gumashta, Xiaojian Wang, Xiao Meng, and Xuefu Zhang.
Changes
-------
Addressed comments from Vaibhav.
1. Updated comments in EncodedColumnBasedSet to describe how client sets the compressorInfo.
2. Fixed bug w.r.t protocol version V9 and above.
3. Logged exception in ColumnCompressorService.
4. Whitespace changes.
Repository: hive-git
Description
-------
This patch enables ResultSet compression for Hive using external plugins. The patch proposes a plugin architecture that enables using external plugins to compress ResultSets on-the-fly.
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 730f5be
jdbc/src/java/org/apache/hive/jdbc/HiveConnection.java bb2b695
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/AbortTxnRequest.java 37e5bf1
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/AddDynamicPartitions.java 9c78c49
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/AddPartitionsRequest.java fcfaaf3
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/AddPartitionsResult.java 9022019
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/AggrStats.java 917cec0
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/AlreadyExistsException.java d7a317b
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/BinaryColumnStatsData.java 00b312d
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/BooleanColumnStatsData.java a0f3ab8
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/CheckLockRequest.java 82e3031
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ColumnStatistics.java 510dace
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ColumnStatisticsDesc.java cf967b6
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ColumnStatisticsObj.java 3c2b123
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/CommitTxnRequest.java 5e8cd04
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/CompactionRequest.java e9088e0
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ConfigValSecurityException.java 000670a
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/CurrentNotificationEventId.java d55d874
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Database.java 56b7281
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Date.java 0b406d6
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/DateColumnStatsData.java 7a3d4ed
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Decimal.java 9215ce9
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/DecimalColumnStatsData.java e64ca36
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/DoubleColumnStatsData.java 2509ed5
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/DropPartitionsExpr.java 5e3a2d1
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/DropPartitionsRequest.java 24536ba
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/DropPartitionsResult.java c139e65
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/EnvironmentContext.java 6accb8d
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/FieldSchema.java ba69622
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/FireEventRequest.java f3d439c
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/FireEventResponse.java d95ae06
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Function.java 50eff73
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetAllFunctionsResponse.java 0a9e27b
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetOpenTxnsInfoResponse.java 61dea16
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetOpenTxnsResponse.java bd21de5
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPrincipalsInRoleRequest.java 45e0cc1
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPrincipalsInRoleResponse.java 805b898
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetRoleGrantsForPrincipalRequest.java c2c7259
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetRoleGrantsForPrincipalResponse.java 7156665
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GrantRevokePrivilegeRequest.java 94e5a5e
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GrantRevokePrivilegeResponse.java 1949aed
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GrantRevokeRoleRequest.java ac3527a
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GrantRevokeRoleResponse.java 81ae31f
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/HeartbeatRequest.java 4a9ba7f
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/HeartbeatTxnRangeRequest.java dd6ed17
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/HeartbeatTxnRangeResponse.java 11bd82b
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/HiveObjectPrivilege.java 931b4e1
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/HiveObjectRef.java 2573cea
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Index.java 81227c3
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/IndexAlreadyExistsException.java 50517ed
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/InsertEventRequestData.java 2e14cf9
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/InvalidInputException.java 87e2f6b
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/InvalidObjectException.java 14dd7d8
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/InvalidOperationException.java 2e3f1e3
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/InvalidPartitionException.java 1d9b565
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/LockComponent.java 319f8bb
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/LockRequest.java 6894bfa
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/LockResponse.java 08acaeb
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/LongColumnStatsData.java 93d2386
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/MetaException.java 883a1d4
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/NoSuchLockException.java e214b36
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/NoSuchObjectException.java 1292a64
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/NoSuchTxnException.java d1c430d
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/NotificationEvent.java bcf4f51
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/NotificationEventRequest.java c2bc4e8
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/NotificationEventResponse.java 24f9ce4
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/OpenTxnRequest.java c5f9ccf
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/OpenTxnsResponse.java c233422
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Order.java da7bd55
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Partition.java 224d28e
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/PartitionListComposingSpec.java c50a100
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/PartitionSpec.java b5251af
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/PartitionSpecWithSharedSD.java 5574e0b
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/PartitionWithoutSD.java e1ec73e
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/PartitionsByExprRequest.java 6149c31
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/PartitionsByExprResult.java 740f7bd
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/PartitionsStatsRequest.java 5d1ee87
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/PartitionsStatsResult.java da33014
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/PrincipalPrivilegeSet.java 8f29f50
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/PrivilegeBag.java 2fd819c
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/PrivilegeGrantInfo.java c04e196
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ResourceUri.java 3b3df25
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Role.java 5c882d2
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/RolePrincipalGrant.java c4beb08
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Schema.java 8772180
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/SerDeInfo.java b30e698
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/SetPartitionsStatsRequest.java 7da298c
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ShowCompactRequest.java 7756384
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ShowCompactResponse.java dd1e857
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ShowCompactResponseElement.java cd7e79e
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ShowLocksRequest.java 122c070
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ShowLocksResponse.java 52b0bbc
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ShowLocksResponseElement.java 8be9b05
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/SkewedInfo.java bc64495
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/StorageDescriptor.java 165a879
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/StringColumnStatsData.java 9906ff3
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Table.java 51b9e38
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/TableStatsRequest.java 1edcaf9
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/TableStatsResult.java 25a1f25
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java cdbae95
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/TxnAbortedException.java a877338
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/TxnInfo.java 8b255b9
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/TxnOpenException.java 05af505
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Type.java 61e7ceb
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/UnknownDBException.java e05e79d
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/UnknownPartitionException.java c626bf6
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/UnknownTableException.java 2856121
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/UnlockRequest.java cf248e0
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Version.java cc8d5f5
metastore/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore-remote dc348ef
ql/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/ql/plan/api/Adjacency.java 5ea9b6e
ql/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/ql/plan/api/Graph.java e621cfa
ql/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/ql/plan/api/Operator.java 1b18aab
ql/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/ql/plan/api/Query.java 5c5e0f8
ql/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/ql/plan/api/QueryPlan.java d340d58
ql/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/ql/plan/api/Stage.java 7353933
ql/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/ql/plan/api/Task.java 09a2e7f
serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde/test/InnerStruct.java db325af
serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde/test/ThriftTestObj.java 1232ff9
serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde2/thrift/test/Complex.java 4d2f5bf
serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde2/thrift/test/IntString.java 23d7363
serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde2/thrift/test/MegaStruct.java 9447708
serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde2/thrift/test/MiniStruct.java d8c46f4
serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde2/thrift/test/SetIntString.java 58498b0
service/if/TCLIService.thrift baf583f
service/src/gen/thrift/gen-cpp/TCLIService.h 29a9f4a
service/src/gen/thrift/gen-cpp/TCLIService_types.h 4536b41
service/src/gen/thrift/gen-cpp/TCLIService_types.cpp 742cfdc
service/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/service/HiveClusterStatus.java d4b6972
service/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/service/HiveServerException.java 760c81e
service/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/service/ThriftHive.java df793b1
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TArrayTypeEntry.java 5625516
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TBinaryColumn.java 202399a
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TBoolColumn.java 921e9de
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TBoolValue.java 201c9fb
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TByteColumn.java cd9b6da
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TByteValue.java 42b5bd5
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TCLIService.java 6bdd53d
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TCancelDelegationTokenReq.java 1097869
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TCancelDelegationTokenResp.java 5469108
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TCancelOperationReq.java 83d191e
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TCancelOperationResp.java b8d96df
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TCloseOperationReq.java ca68866
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TCloseOperationResp.java 092e7e4
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TCloseSessionReq.java a5d910f
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TCloseSessionResp.java ae4b554
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TColumnDesc.java e31aa81
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TDoubleColumn.java 7236d90
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TDoubleValue.java 4b0811e
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TEnColumn.java PRE-CREATION
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TExecuteStatementReq.java feaed34
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TExecuteStatementResp.java daf7b5c
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TFetchResultsReq.java 47d9a0b
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TFetchResultsResp.java b9f3ef6
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TGetCatalogsReq.java 0e63b9a
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TGetCatalogsResp.java 7067ff7
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TGetColumnsReq.java a153968
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TGetColumnsResp.java 49ecb98
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TGetDelegationTokenReq.java 2c2fa0f
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TGetDelegationTokenResp.java 4222005
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TGetFunctionsReq.java 5bc0540
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TGetFunctionsResp.java a64c948
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TGetInfoReq.java 7dd6a14
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TGetInfoResp.java 3e16318
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TGetOperationStatusReq.java f4d5fae
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TGetOperationStatusResp.java 897f7f5
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TGetResultSetMetadataReq.java 405ffc6
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TGetResultSetMetadataResp.java 3735310
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TGetSchemasReq.java 33088b4
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TGetSchemasResp.java ec598a3
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TGetTableTypesReq.java 3e09a64
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TGetTableTypesResp.java cdba9b5
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TGetTablesReq.java 805e69f
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TGetTablesResp.java 0d03dd8
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TGetTypeInfoReq.java 0628e21
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TGetTypeInfoResp.java 6058826
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/THandleIdentifier.java f20c4d0
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TI16Column.java 267d109
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TI16Value.java ddaf9ef
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TI32Column.java c79df4f
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TI32Value.java d521373
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TI64Column.java 746ade2
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TI64Value.java b419b80
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TMapTypeEntry.java 91a85ef
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TOpenSessionReq.java 657f868
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TOpenSessionResp.java 48f4b45
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TOperationHandle.java db41117
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TPrimitiveTypeEntry.java f1c8d58
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TProtocolVersion.java 6e714c6
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TRenewDelegationTokenReq.java 91f8b00
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TRenewDelegationTokenResp.java c01cc3f
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TRow.java 197bab6
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TRowSet.java cc1a148
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TSessionHandle.java 264e155
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TStatus.java 1cd7980
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TStringColumn.java d996529
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TStringValue.java 9655f38
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TStructTypeEntry.java d58184c
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TTableSchema.java 796b73b
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TTypeDesc.java 95b4466
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TTypeQualifiers.java 15ac5a9
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TUnionTypeEntry.java eccf303
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TUserDefinedTypeEntry.java 21da61b
service/src/gen/thrift/gen-py/TCLIService/ttypes.py efee8ef
service/src/gen/thrift/gen-py/hive_service/ThriftHive-remote 747c2be
service/src/gen/thrift/gen-rb/t_c_l_i_service_types.rb bfb2b69
service/src/java/org/apache/hive/service/cli/Column.java 2e21f18
service/src/java/org/apache/hive/service/cli/ColumnBasedSet.java 47a582e
service/src/java/org/apache/hive/service/cli/RowSetFactory.java e8f68ea
service/src/java/org/apache/hive/service/cli/compression/ColumnCompressor.java PRE-CREATION
service/src/java/org/apache/hive/service/cli/compression/ColumnCompressorService.java PRE-CREATION
service/src/java/org/apache/hive/service/cli/compression/EncodedColumnBasedSet.java PRE-CREATION
service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java 67bc778
service/src/test/org/apache/hive/service/cli/compression/SnappyIntCompressor.java PRE-CREATION
service/src/test/org/apache/hive/service/cli/compression/TestEncodedColumnBasedSet.java PRE-CREATION
service/src/test/resources/META-INF/services/org.apache.hive.service.cli.compression.ColumnCompressor PRE-CREATION
Diff: https://reviews.apache.org/r/35792/diff/
Testing
-------
Testing has been done using a docker container-based query submitter that has an integer decompressor as part of it. Using the integer compressor (also provided) and the decompressor, the end-to-end functionality can be observed.
File Attachments
----------------
Patch file
https://reviews.apache.org/media/uploaded/files/2015/06/23/16aa08f8-2393-460a-83ef-72464fc537db__HIVE-10438.patch
Thanks,
Rohit Dholakia
Re: Review Request 35792: HIVE-10438 - Architecture for ResultSet
Compression via external plugin
Posted by Rohit Dholakia <ro...@simba.com>.
> On Aug. 11, 2015, 6:44 a.m., Vaibhav Gumashta wrote:
> > service/src/java/org/apache/hive/service/cli/compression/EncodedColumnBasedSet.java, line 46
> > <https://reviews.apache.org/r/35792/diff/2/?file=994492#file994492line46>
> >
> > How would the client send this string? Will it be a new API call from the driver code? Can you elaborate a bit more?
It won't be a new API call. It would be sent as part of TExecuteStatementReq. We updated the comments in EncodedColumnBasedSet to explain this.
- Rohit
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35792/#review94821
-----------------------------------------------------------
On Aug. 15, 2015, 12:29 a.m., Rohit Dholakia wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/35792/
> -----------------------------------------------------------
>
> (Updated Aug. 15, 2015, 12:29 a.m.)
>
>
> Review request for hive, Vaibhav Gumashta, Xiaojian Wang, Xiao Meng, and Xuefu Zhang.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> This patch enables ResultSet compression for Hive using external plugins. The patch proposes a plugin architecture that enables using external plugins to compress ResultSets on-the-fly.
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 730f5be
> jdbc/src/java/org/apache/hive/jdbc/HiveConnection.java bb2b695
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/AbortTxnRequest.java 37e5bf1
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/AddDynamicPartitions.java 9c78c49
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/AddPartitionsRequest.java fcfaaf3
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/AddPartitionsResult.java 9022019
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/AggrStats.java 917cec0
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/AlreadyExistsException.java d7a317b
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/BinaryColumnStatsData.java 00b312d
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/BooleanColumnStatsData.java a0f3ab8
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/CheckLockRequest.java 82e3031
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ColumnStatistics.java 510dace
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ColumnStatisticsDesc.java cf967b6
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ColumnStatisticsObj.java 3c2b123
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/CommitTxnRequest.java 5e8cd04
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/CompactionRequest.java e9088e0
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ConfigValSecurityException.java 000670a
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/CurrentNotificationEventId.java d55d874
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Database.java 56b7281
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Date.java 0b406d6
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/DateColumnStatsData.java 7a3d4ed
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Decimal.java 9215ce9
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/DecimalColumnStatsData.java e64ca36
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/DoubleColumnStatsData.java 2509ed5
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/DropPartitionsExpr.java 5e3a2d1
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/DropPartitionsRequest.java 24536ba
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/DropPartitionsResult.java c139e65
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/EnvironmentContext.java 6accb8d
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/FieldSchema.java ba69622
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/FireEventRequest.java f3d439c
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/FireEventResponse.java d95ae06
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Function.java 50eff73
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetAllFunctionsResponse.java 0a9e27b
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetOpenTxnsInfoResponse.java 61dea16
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetOpenTxnsResponse.java bd21de5
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPrincipalsInRoleRequest.java 45e0cc1
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPrincipalsInRoleResponse.java 805b898
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetRoleGrantsForPrincipalRequest.java c2c7259
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetRoleGrantsForPrincipalResponse.java 7156665
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GrantRevokePrivilegeRequest.java 94e5a5e
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GrantRevokePrivilegeResponse.java 1949aed
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GrantRevokeRoleRequest.java ac3527a
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GrantRevokeRoleResponse.java 81ae31f
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/HeartbeatRequest.java 4a9ba7f
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/HeartbeatTxnRangeRequest.java dd6ed17
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/HeartbeatTxnRangeResponse.java 11bd82b
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/HiveObjectPrivilege.java 931b4e1
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/HiveObjectRef.java 2573cea
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Index.java 81227c3
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/IndexAlreadyExistsException.java 50517ed
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/InsertEventRequestData.java 2e14cf9
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/InvalidInputException.java 87e2f6b
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/InvalidObjectException.java 14dd7d8
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/InvalidOperationException.java 2e3f1e3
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/InvalidPartitionException.java 1d9b565
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/LockComponent.java 319f8bb
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/LockRequest.java 6894bfa
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/LockResponse.java 08acaeb
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/LongColumnStatsData.java 93d2386
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/MetaException.java 883a1d4
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/NoSuchLockException.java e214b36
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/NoSuchObjectException.java 1292a64
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/NoSuchTxnException.java d1c430d
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/NotificationEvent.java bcf4f51
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/NotificationEventRequest.java c2bc4e8
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/NotificationEventResponse.java 24f9ce4
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/OpenTxnRequest.java c5f9ccf
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/OpenTxnsResponse.java c233422
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Order.java da7bd55
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Partition.java 224d28e
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/PartitionListComposingSpec.java c50a100
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/PartitionSpec.java b5251af
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/PartitionSpecWithSharedSD.java 5574e0b
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/PartitionWithoutSD.java e1ec73e
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/PartitionsByExprRequest.java 6149c31
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/PartitionsByExprResult.java 740f7bd
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/PartitionsStatsRequest.java 5d1ee87
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/PartitionsStatsResult.java da33014
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/PrincipalPrivilegeSet.java 8f29f50
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/PrivilegeBag.java 2fd819c
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/PrivilegeGrantInfo.java c04e196
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ResourceUri.java 3b3df25
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Role.java 5c882d2
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/RolePrincipalGrant.java c4beb08
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Schema.java 8772180
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/SerDeInfo.java b30e698
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/SetPartitionsStatsRequest.java 7da298c
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ShowCompactRequest.java 7756384
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ShowCompactResponse.java dd1e857
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ShowCompactResponseElement.java cd7e79e
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ShowLocksRequest.java 122c070
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ShowLocksResponse.java 52b0bbc
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ShowLocksResponseElement.java 8be9b05
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/SkewedInfo.java bc64495
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/StorageDescriptor.java 165a879
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/StringColumnStatsData.java 9906ff3
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Table.java 51b9e38
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/TableStatsRequest.java 1edcaf9
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/TableStatsResult.java 25a1f25
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java cdbae95
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/TxnAbortedException.java a877338
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/TxnInfo.java 8b255b9
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/TxnOpenException.java 05af505
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Type.java 61e7ceb
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/UnknownDBException.java e05e79d
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/UnknownPartitionException.java c626bf6
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/UnknownTableException.java 2856121
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/UnlockRequest.java cf248e0
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Version.java cc8d5f5
> metastore/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore-remote dc348ef
> ql/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/ql/plan/api/Adjacency.java 5ea9b6e
> ql/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/ql/plan/api/Graph.java e621cfa
> ql/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/ql/plan/api/Operator.java 1b18aab
> ql/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/ql/plan/api/Query.java 5c5e0f8
> ql/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/ql/plan/api/QueryPlan.java d340d58
> ql/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/ql/plan/api/Stage.java 7353933
> ql/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/ql/plan/api/Task.java 09a2e7f
> serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde/test/InnerStruct.java db325af
> serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde/test/ThriftTestObj.java 1232ff9
> serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde2/thrift/test/Complex.java 4d2f5bf
> serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde2/thrift/test/IntString.java 23d7363
> serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde2/thrift/test/MegaStruct.java 9447708
> serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde2/thrift/test/MiniStruct.java d8c46f4
> serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde2/thrift/test/SetIntString.java 58498b0
> service/if/TCLIService.thrift baf583f
> service/src/gen/thrift/gen-cpp/TCLIService.h 29a9f4a
> service/src/gen/thrift/gen-cpp/TCLIService_types.h 4536b41
> service/src/gen/thrift/gen-cpp/TCLIService_types.cpp 742cfdc
> service/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/service/HiveClusterStatus.java d4b6972
> service/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/service/HiveServerException.java 760c81e
> service/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/service/ThriftHive.java df793b1
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TArrayTypeEntry.java 5625516
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TBinaryColumn.java 202399a
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TBoolColumn.java 921e9de
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TBoolValue.java 201c9fb
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TByteColumn.java cd9b6da
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TByteValue.java 42b5bd5
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TCLIService.java 6bdd53d
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TCancelDelegationTokenReq.java 1097869
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TCancelDelegationTokenResp.java 5469108
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TCancelOperationReq.java 83d191e
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TCancelOperationResp.java b8d96df
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TCloseOperationReq.java ca68866
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TCloseOperationResp.java 092e7e4
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TCloseSessionReq.java a5d910f
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TCloseSessionResp.java ae4b554
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TColumnDesc.java e31aa81
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TDoubleColumn.java 7236d90
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TDoubleValue.java 4b0811e
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TEnColumn.java PRE-CREATION
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TExecuteStatementReq.java feaed34
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TExecuteStatementResp.java daf7b5c
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TFetchResultsReq.java 47d9a0b
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TFetchResultsResp.java b9f3ef6
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TGetCatalogsReq.java 0e63b9a
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TGetCatalogsResp.java 7067ff7
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TGetColumnsReq.java a153968
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TGetColumnsResp.java 49ecb98
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TGetDelegationTokenReq.java 2c2fa0f
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TGetDelegationTokenResp.java 4222005
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TGetFunctionsReq.java 5bc0540
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TGetFunctionsResp.java a64c948
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TGetInfoReq.java 7dd6a14
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TGetInfoResp.java 3e16318
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TGetOperationStatusReq.java f4d5fae
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TGetOperationStatusResp.java 897f7f5
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TGetResultSetMetadataReq.java 405ffc6
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TGetResultSetMetadataResp.java 3735310
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TGetSchemasReq.java 33088b4
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TGetSchemasResp.java ec598a3
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TGetTableTypesReq.java 3e09a64
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TGetTableTypesResp.java cdba9b5
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TGetTablesReq.java 805e69f
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TGetTablesResp.java 0d03dd8
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TGetTypeInfoReq.java 0628e21
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TGetTypeInfoResp.java 6058826
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/THandleIdentifier.java f20c4d0
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TI16Column.java 267d109
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TI16Value.java ddaf9ef
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TI32Column.java c79df4f
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TI32Value.java d521373
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TI64Column.java 746ade2
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TI64Value.java b419b80
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TMapTypeEntry.java 91a85ef
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TOpenSessionReq.java 657f868
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TOpenSessionResp.java 48f4b45
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TOperationHandle.java db41117
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TPrimitiveTypeEntry.java f1c8d58
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TProtocolVersion.java 6e714c6
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TRenewDelegationTokenReq.java 91f8b00
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TRenewDelegationTokenResp.java c01cc3f
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TRow.java 197bab6
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TRowSet.java cc1a148
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TSessionHandle.java 264e155
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TStatus.java 1cd7980
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TStringColumn.java d996529
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TStringValue.java 9655f38
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TStructTypeEntry.java d58184c
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TTableSchema.java 796b73b
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TTypeDesc.java 95b4466
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TTypeQualifiers.java 15ac5a9
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TUnionTypeEntry.java eccf303
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TUserDefinedTypeEntry.java 21da61b
> service/src/gen/thrift/gen-py/TCLIService/ttypes.py efee8ef
> service/src/gen/thrift/gen-py/hive_service/ThriftHive-remote 747c2be
> service/src/gen/thrift/gen-rb/t_c_l_i_service_types.rb bfb2b69
> service/src/java/org/apache/hive/service/cli/Column.java 2e21f18
> service/src/java/org/apache/hive/service/cli/ColumnBasedSet.java 47a582e
> service/src/java/org/apache/hive/service/cli/RowSetFactory.java e8f68ea
> service/src/java/org/apache/hive/service/cli/compression/ColumnCompressor.java PRE-CREATION
> service/src/java/org/apache/hive/service/cli/compression/ColumnCompressorService.java PRE-CREATION
> service/src/java/org/apache/hive/service/cli/compression/EncodedColumnBasedSet.java PRE-CREATION
> service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java 67bc778
> service/src/test/org/apache/hive/service/cli/compression/SnappyIntCompressor.java PRE-CREATION
> service/src/test/org/apache/hive/service/cli/compression/TestEncodedColumnBasedSet.java PRE-CREATION
> service/src/test/resources/META-INF/services/org.apache.hive.service.cli.compression.ColumnCompressor PRE-CREATION
>
> Diff: https://reviews.apache.org/r/35792/diff/
>
>
> Testing
> -------
>
> Testing has been done using a docker container-based query submitter that has an integer decompressor as part of it. Using the integer compressor (also provided) and the decompressor, the end-to-end functionality can be observed.
>
>
> File Attachments
> ----------------
>
> Patch file
> https://reviews.apache.org/media/uploaded/files/2015/06/23/16aa08f8-2393-460a-83ef-72464fc537db__HIVE-10438.patch
>
>
> Thanks,
>
> Rohit Dholakia
>
>
Re: Review Request 35792: HIVE-10438 - Architecture for ResultSet
Compression via external plugin
Posted by Vaibhav Gumashta <vg...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35792/#review94821
-----------------------------------------------------------
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java (line 2201)
<https://reviews.apache.org/r/35792/#comment149462>
For HiveServer2 specific configs, we use the format hive.server2.*
service/src/java/org/apache/hive/service/cli/RowSetFactory.java (line 31)
<https://reviews.apache.org/r/35792/#comment149459>
What happens to versions > V9?
service/src/java/org/apache/hive/service/cli/compression/ColumnCompressorService.java (line 76)
<https://reviews.apache.org/r/35792/#comment149562>
We should log the cause on catching this exception.
service/src/java/org/apache/hive/service/cli/compression/EncodedColumnBasedSet.java (line 46)
<https://reviews.apache.org/r/35792/#comment149563>
How would the client send this string? Will it be a new API call from the driver code? Can you elaborate a bit more?
service/src/java/org/apache/hive/service/cli/compression/EncodedColumnBasedSet.java (line 59)
<https://reviews.apache.org/r/35792/#comment149465>
Nit: whitespace.
service/src/java/org/apache/hive/service/cli/compression/EncodedColumnBasedSet.java (line 96)
<https://reviews.apache.org/r/35792/#comment149564>
How is CompressorInfo being set?
service/src/java/org/apache/hive/service/cli/compression/EncodedColumnBasedSet.java (line 97)
<https://reviews.apache.org/r/35792/#comment149466>
Nit: whitespace.
service/src/test/org/apache/hive/service/cli/compression/SnappyIntCompressor.java (line 64)
<https://reviews.apache.org/r/35792/#comment149533>
Nit: whitespace.
service/src/test/org/apache/hive/service/cli/compression/TestEncodedColumnBasedSet.java (line 56)
<https://reviews.apache.org/r/35792/#comment149534>
Nit: whitespace.
- Vaibhav Gumashta
On June 29, 2015, 5:41 p.m., Rohit Dholakia wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/35792/
> -----------------------------------------------------------
>
> (Updated June 29, 2015, 5:41 p.m.)
>
>
> Review request for hive, Vaibhav Gumashta, Xiaojian Wang, Xiao Meng, and Xuefu Zhang.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> This patch enables ResultSet compression for Hive using external plugins. The patch proposes a plugin architecture that enables using external plugins to compress ResultSets on-the-fly.
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 27f68df
> service/if/TCLIService.thrift baf583f
> service/src/gen/thrift/gen-cpp/TCLIService_types.h 7bceabd
> service/src/gen/thrift/gen-cpp/TCLIService_types.cpp 86eeea3
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TEnColumn.java PRE-CREATION
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TExecuteStatementReq.java 4f157ad
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TGetTablesReq.java c973fcc
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TOpenSessionReq.java c048161
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TOpenSessionResp.java 351f78b
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TProtocolVersion.java a4279d2
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TRowSet.java d16c8a4
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TStatus.java 24a746e
> service/src/gen/thrift/gen-py/TCLIService/ttypes.py 068727c
> service/src/gen/thrift/gen-rb/t_c_l_i_service_types.rb b482533
> service/src/java/org/apache/hive/service/cli/Column.java 2e21f18
> service/src/java/org/apache/hive/service/cli/ColumnBasedSet.java 47a582e
> service/src/java/org/apache/hive/service/cli/RowSetFactory.java e8f68ea
> service/src/java/org/apache/hive/service/cli/compression/ColumnCompressor.java PRE-CREATION
> service/src/java/org/apache/hive/service/cli/compression/ColumnCompressorService.java PRE-CREATION
> service/src/java/org/apache/hive/service/cli/compression/EncodedColumnBasedSet.java PRE-CREATION
> service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java dfb7faa
> service/src/main/resources/META-INF/services/org.apache.hive.service.cli.compression.ColumnCompressor PRE-CREATION
> service/src/test/org/apache/hive/service/cli/compression/SnappyIntCompressor.java PRE-CREATION
> service/src/test/org/apache/hive/service/cli/compression/TestEncodedColumnBasedSet.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/35792/diff/
>
>
> Testing
> -------
>
> Testing has been done using a docker container-based query submitter that has an integer decompressor as part of it. Using the integer compressor (also provided) and the decompressor, the end-to-end functionality can be observed.
>
>
> File Attachments
> ----------------
>
> Patch file
> https://reviews.apache.org/media/uploaded/files/2015/06/23/16aa08f8-2393-460a-83ef-72464fc537db__HIVE-10438.patch
>
>
> Thanks,
>
> Rohit Dholakia
>
>
Re: Review Request 35792: HIVE-10438 - Architecture for ResultSet
Compression via external plugin
Posted by Rohit Dholakia <ro...@simba.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35792/
-----------------------------------------------------------
(Updated June 29, 2015, 5:41 p.m.)
Review request for hive, Vaibhav Gumashta, Xiaojian Wang, Xiao Meng, and Xuefu Zhang.
Changes
-------
We have added the review comments as mentioned below:
1. Moved to a new package (cli.compression)
2. Added tests for EncodedColumnBased (using snappy as default compression)
3. Fixed whitespace.
4. Removed default values from hive-site.xml.
Repository: hive-git
Description
-------
This patch enables ResultSet compression for Hive using external plugins. The patch proposes a plugin architecture that enables using external plugins to compress ResultSets on-the-fly.
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 27f68df
service/if/TCLIService.thrift baf583f
service/src/gen/thrift/gen-cpp/TCLIService_types.h 7bceabd
service/src/gen/thrift/gen-cpp/TCLIService_types.cpp 86eeea3
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TEnColumn.java PRE-CREATION
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TExecuteStatementReq.java 4f157ad
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TGetTablesReq.java c973fcc
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TOpenSessionReq.java c048161
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TOpenSessionResp.java 351f78b
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TProtocolVersion.java a4279d2
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TRowSet.java d16c8a4
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TStatus.java 24a746e
service/src/gen/thrift/gen-py/TCLIService/ttypes.py 068727c
service/src/gen/thrift/gen-rb/t_c_l_i_service_types.rb b482533
service/src/java/org/apache/hive/service/cli/Column.java 2e21f18
service/src/java/org/apache/hive/service/cli/ColumnBasedSet.java 47a582e
service/src/java/org/apache/hive/service/cli/RowSetFactory.java e8f68ea
service/src/java/org/apache/hive/service/cli/compression/ColumnCompressor.java PRE-CREATION
service/src/java/org/apache/hive/service/cli/compression/ColumnCompressorService.java PRE-CREATION
service/src/java/org/apache/hive/service/cli/compression/EncodedColumnBasedSet.java PRE-CREATION
service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java dfb7faa
service/src/main/resources/META-INF/services/org.apache.hive.service.cli.compression.ColumnCompressor PRE-CREATION
service/src/test/org/apache/hive/service/cli/compression/SnappyIntCompressor.java PRE-CREATION
service/src/test/org/apache/hive/service/cli/compression/TestEncodedColumnBasedSet.java PRE-CREATION
Diff: https://reviews.apache.org/r/35792/diff/
Testing
-------
Testing has been done using a docker container-based query submitter that has an integer decompressor as part of it. Using the integer compressor (also provided) and the decompressor, the end-to-end functionality can be observed.
File Attachments (updated)
----------------
Patch file
https://reviews.apache.org/media/uploaded/files/2015/06/23/16aa08f8-2393-460a-83ef-72464fc537db__HIVE-10438.patch
Thanks,
Rohit Dholakia
Re: Review Request 35792: HIVE-10438 - Architecture for ResultSet
Compression via external plugin
Posted by Rohit Dholakia <ro...@simba.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35792/
-----------------------------------------------------------
(Updated June 25, 2015, 11:08 p.m.)
Review request for hive, Vaibhav Gumashta, Xiaojian Wang, Xiao Meng, and Xuefu Zhang.
Changes
-------
Fixed the issues that were raised. Added tests for EncodedColumnBasedSet. Moved the new classes to a separate package.
Repository: hive-git
Description
-------
This patch enables ResultSet compression for Hive using external plugins. The patch proposes a plugin architecture that enables using external plugins to compress ResultSets on-the-fly.
Diffs
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 27f68df
conf/hive-site.xml dab494e
service/if/TCLIService.thrift baf583f
service/src/gen/thrift/gen-cpp/TCLIService_types.h 7bceabd
service/src/gen/thrift/gen-cpp/TCLIService_types.cpp 86eeea3
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TEnColumn.java PRE-CREATION
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TExecuteStatementReq.java 4f157ad
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TGetTablesReq.java c973fcc
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TOpenSessionReq.java c048161
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TOpenSessionResp.java 351f78b
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TProtocolVersion.java a4279d2
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TRowSet.java d16c8a4
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TStatus.java 24a746e
service/src/gen/thrift/gen-py/TCLIService/ttypes.py 068727c
service/src/gen/thrift/gen-rb/t_c_l_i_service_types.rb b482533
service/src/java/org/apache/hive/service/cli/Column.java 2e21f18
service/src/java/org/apache/hive/service/cli/ColumnBasedSet.java 47a582e
service/src/java/org/apache/hive/service/cli/ColumnCompressor.java PRE-CREATION
service/src/java/org/apache/hive/service/cli/ColumnCompressorService.java PRE-CREATION
service/src/java/org/apache/hive/service/cli/EncodedColumnBasedSet.java PRE-CREATION
service/src/java/org/apache/hive/service/cli/RowSetFactory.java e8f68ea
service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java dfb7faa
Diff: https://reviews.apache.org/r/35792/diff/
Testing
-------
Testing has been done using a docker container-based query submitter that has an integer decompressor as part of it. Using the integer compressor (also provided) and the decompressor, the end-to-end functionality can be observed.
File Attachments (updated)
----------------
Patch file
https://reviews.apache.org/media/uploaded/files/2015/06/23/16aa08f8-2393-460a-83ef-72464fc537db__HIVE-10438.patch
Updated patch
https://reviews.apache.org/media/uploaded/files/2015/06/25/b7f8ab23-a900-45b8-8be6-f1ae2cb89aca__HIVE-10438-1.patch
Thanks,
Rohit Dholakia