[jira] [Commented] (KUDU-1836) Enable compression of delta files

["Todd Lipcon (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KUDU-1836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15830686#comment-15830686 ] 

Todd Lipcon commented on KUDU-1836:
-----------------------------------

A couple numbers from a ~10GB tablet I analyzed from an internal use case which has lots of UPSERTs. In particular, this workload has a lot of cases where rows get upserted but the changed value is no different than the previous contents of the row (so multiple deltas in a row are basically dupes and highly compressible). This is obviously close to a best-case, but it's also not a contrived use case (this is a real app).

{code}
Codec       Total size of deltas
----------------------
NONE        10458MB
LZ4         413MB   (25x)
GZIP        296MB   (35x)
{code}

> Enable compression of delta files
> ---------------------------------
>
>                 Key: KUDU-1836
>                 URL: https://issues.apache.org/jira/browse/KUDU-1836
>             Project: Kudu
>          Issue Type: Improvement
>          Components: perf, tablet
>    Affects Versions: 1.3.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>
> deltafiles are currently not compressed. This can be somewhat problematic for workloads that are heavy on updates -- the deltafiles can be much larger than the underlying data in many cases, especially if the user has a dataset which is highly column-compressible. We should consider enabling LZ compression of deltafiles by default.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)