[jira] [Comment Edited] (HBASE-12883) Support block encoding based on knowing set of column qualifiers up front

["Andrew Purtell (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HBASE-12883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14284148#comment-14284148 ] 

Andrew Purtell edited comment on HBASE-12883 at 1/20/15 6:22 PM:
-----------------------------------------------------------------

Block encoders get a HFileContext reference. The reader could retrieve such a dictionary from an HFile meta entry and pass it through HFileContext. Up in Store we create HFileContexts with a builder for passing down HFile level configuration. There we could look for a schema attribute that holds a supplied dictionary and pass it through. Phoenix could set up this dictionary of column qualifiers in the column schema. The HFile writer could store the dictionary into an HFile meta entry when writing out the file. If the application messes up they have an unreadable HFile, that can be documented as the contract of this option. Note that since we store the dictionary with the file, the file is always readable even if the dictionary in the schema is changed, that would only affect and be for newly written files. By "mess up" I mean supplying a corrupt dictionary, and I suppose we wouldn't even get so far as a corrupt HFile, we'd fail quickly in block encoding and the regionserver would abort as consequence as soon as there's a flush using a broken dictionary. But since this option is wizard level that could be fine.


was (Author: apurtell):
Block encoders get a HFileContext reference. The reader could retrieve such a dictionary from an HFile meta entry and pass it through HFileContext. Up in Store we create HFileContexts with a builder for passing down HFile level configuration. There we could look for a schema attribute that holds a supplied dictionary and pass it through. Phoenix could set up this dictionary of column qualifiers in the column schema. The HFile writer could store the dictionary into an HFile meta entry when writing out the file. If the application messes up they have an unreadable HFile, that can be documented as the contract of this option. 

> Support block encoding based on knowing set of column qualifiers up front
> -------------------------------------------------------------------------
>
>                 Key: HBASE-12883
>                 URL: https://issues.apache.org/jira/browse/HBASE-12883
>             Project: HBase
>          Issue Type: Bug
>            Reporter: James Taylor
>              Labels: Phoenix
>
> Phoenix knows up front the set of column qualifiers a row will have. We could likely get some good compression with little CPU based on this by having a block encoding scheme that leverages this information. It could be made non-Phoenix specific by identifying the set of column qualifiers through meta data to the block encoder.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)