[jira] [Commented] (ACCUMULO-790) RFile should compress using common prefixes of key elements

["Keith Turner (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ACCUMULO-790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13491772#comment-13491772 ] 

Keith Turner commented on ACCUMULO-790:
---------------------------------------

When RFile was created not only was the file format considered, the end to end data process from client write to client read was also considered.   The encoding that is utilized in RFile is also utilized to send data over the wire.  RFile allowed much faster read than MapFile, but that speedup would not have mattered as much if the data could not be sent over the network faster.   See Key.compress() and Key.decompress().   If these changes further speed up read rates, can we do something to speed up transfer rates?  There may be nothing to do, but its something we can experiment with.


                
> RFile should compress using common prefixes of key elements
> -----------------------------------------------------------
>
>                 Key: ACCUMULO-790
>                 URL: https://issues.apache.org/jira/browse/ACCUMULO-790
>             Project: Accumulo
>          Issue Type: Improvement
>          Components: tserver
>            Reporter: Christopher Tubbs
>            Assignee: Christopher Tubbs
>              Labels: compression, file, hackathon, optimization, rfile
>             Fix For: 1.5.0
>
>
> Relative keys have proven themselves as a great way to compress within dimensions of the key. However, we could probably do better, since we know that our data is sorted lexicographically, we can make a reasonable assumption that we will get better compression if we only store the fact that a key (or portion of a key) has a common prefix with the previous key, even if it is not an exact match.
> Currently, in RFile, unused bits from the delete flag byte are being used to store flags that show whether an element of the key is exactly the same as the previous, or if it is different. We can change the semantics of these flags to store three states per element of the key: exact match as previous key, has a common prefix as previous key, no relative key compression. If we don't want to add a byte to store 2 bits for 3 states per element, we can just take the ordinal value of the unused 7 bits of the delete flag field and map it to an enumeration of relative key flags.
> In the case of a common prefix flag enabled for a given element of the current key when reading the RFile, we can interpret the first bytes of that element as a VInt expressing the length of the common prefix relative to the previous key's same element. Because this will add at least one byte to the the length of that element, we will not want to use the common prefix compression if the common prefix is less than 2 bytes. For less than 2 bytes in common (1 or 0 bytes in common), we'd select the no compression flag for that element.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira