[jira] [Commented] (KUDU-463) Add checksumming to cfile and other on-disk formats

["Will Berkeley (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KUDU-463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15492256#comment-15492256 ] 

Will Berkeley commented on KUDU-463:
------------------------------------

[~tlipcon] [~adar]: this is pretty easy to do backwards-compatibly. There's some overhead on read, write which should be quantified as part of the patch, but the read overhead can be tuned by policy on when checksums are checked. There should also be a capability in the kudu tool to check checksums of a column. 

Do you think it would be better (eventually) to do checksums lower, in the Kudu filesystem layer instead of the CFile? CFile checksums could be done in the meantime, since, at a glance, adding checksums to the block layer is harder, esp re: backwards compatibility

> Add checksumming to cfile and other on-disk formats
> ---------------------------------------------------
>
>                 Key: KUDU-463
>                 URL: https://issues.apache.org/jira/browse/KUDU-463
>             Project: Kudu
>          Issue Type: Sub-task
>          Components: cfile, tablet
>    Affects Versions: Private Beta
>            Reporter: Todd Lipcon
>            Assignee: Adar Dembo
>              Labels: kudu-roadmap
>
> We should add CRC32C checksums to cfile blocks, metadata blocks, etc, to protect against silent disk corruption. We should probably do this prior to a public release, since it will likely have a negative performance impact, and we don't want to have a public regression.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)