You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@tez.apache.org by "Jonathan Turner Eagles (Jira)" <ji...@apache.org> on 2020/06/05 18:08:00 UTC

[jira] [Commented] (TEZ-1719) Allow IFile reducer merge-sort to disable crc32 checksums

    [ https://issues.apache.org/jira/browse/TEZ-1719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17126989#comment-17126989 ] 

Jonathan Turner Eagles commented on TEZ-1719:
---------------------------------------------

Just know to be careful when enabling this as modern disks write in the background and will return write success. In my case data came was read as all zeros from intermediate spill

> Allow IFile reducer merge-sort to disable crc32 checksums
> ---------------------------------------------------------
>
>                 Key: TEZ-1719
>                 URL: https://issues.apache.org/jira/browse/TEZ-1719
>             Project: Apache Tez
>          Issue Type: Bug
>    Affects Versions: 0.6.0
>            Reporter: Gopal Vijayaraghavan
>            Assignee: László Bodor
>            Priority: Major
>              Labels: Performance
>
> Next-gen filesystems like BTRFS and ZFS provide their own checksumming for disk data.
> Using PureJavaCrc32 for data written for temporary spills to such filesystems is a complete waste of CPU resources.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)