You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Krishna Kumar (Commented) (JIRA)" <ji...@apache.org> on 2011/12/05 11:27:41 UTC
[jira] [Commented] (HIVE-2604) Add UberCompressor Serde/Codec to
contrib which allows per-column compression strategies
[ https://issues.apache.org/jira/browse/HIVE-2604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13162709#comment-13162709 ]
Krishna Kumar commented on HIVE-2604:
-------------------------------------
The current implementation works as follows:
- Adds a serde UberCompressorSerde, which is used to convert the cell values to bytes
- Adds a codec UberCompressionCodec which uses user-specific config to compress each block of column values through one of three possible mechanisms
- Config for the column: "codec:<codecname>" - Apply a CompressionCodec on the UberCompressorSerde serialized bytestream
- Config for the column: "codec:<codecname>,<serdename>" Re-serialize the bytestream through serdename and then apply codecname on it
- Config for the column: "compressor:<compressorname>" compress the cell values by sending them through the type-specific compressor
- [As a future enhancement, the config, say if is "dynamic", can let the codec decide the mechanism on the current block stats/previous seen blocks]
The idea is to maintain the ability to use a serde/codec combination (as we do now) for any columns which are not 'interesting' and use type-specific compressors only for special columns.
Type-specific compressor is also an extension point only; no implementation attached to this jira. Have attached one sample compressor to HIVE-2623, while many others are possible.
> Add UberCompressor Serde/Codec to contrib which allows per-column compression strategies
> ----------------------------------------------------------------------------------------
>
> Key: HIVE-2604
> URL: https://issues.apache.org/jira/browse/HIVE-2604
> Project: Hive
> Issue Type: Sub-task
> Components: Query Processor, Serializers/Deserializers
> Reporter: Krishna Kumar
> Assignee: Krishna Kumar
> Attachments: HIVE-2604.v0.patch
>
>
> The strategies supported are
> 1. using a specified codec on the column
> 2. using a specific codec on the column which is serialized via a specific serde
> 3. using a specific "TypeSpecificCompressor" instance
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira