You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@avro.apache.org by "Doug Cutting (JIRA)" <ji...@apache.org> on 2011/04/07 00:10:06 UTC

[jira] [Created] (AVRO-798) add checksums to Snappy codec

add checksums to Snappy codec
-----------------------------

                 Key: AVRO-798
                 URL: https://issues.apache.org/jira/browse/AVRO-798
             Project: Avro
          Issue Type: Improvement
          Components: java, spec
            Reporter: Doug Cutting
            Assignee: Doug Cutting
             Fix For: 1.5.1


A checksum might be included with each compressed block to better detect errors.  While some filesystems (e.g. HDFS) may checksum data, not all do.  Data files may also accumulate errors when copied between filesystems.  For back-compatibility, we cannot easily add checksums to existing data files, but a new codec provides us with the opportunity to do this.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (AVRO-798) add checksums to Snappy codec

Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/AVRO-798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Doug Cutting updated AVRO-798:
------------------------------

    Attachment: AVRO-798.patch

Patch that also updates spec.

> add checksums to Snappy codec
> -----------------------------
>
>                 Key: AVRO-798
>                 URL: https://issues.apache.org/jira/browse/AVRO-798
>             Project: Avro
>          Issue Type: Improvement
>          Components: java, spec
>            Reporter: Doug Cutting
>            Assignee: Doug Cutting
>             Fix For: 1.5.1
>
>         Attachments: AVRO-798.patch, AVRO-798.patch
>
>
> A checksum might be included with each compressed block to better detect errors.  While some filesystems (e.g. HDFS) may checksum data, not all do.  Data files may also accumulate errors when copied between filesystems.  For back-compatibility, we cannot easily add checksums to existing data files, but a new codec provides us with the opportunity to do this.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (AVRO-798) add checksums to Snappy codec

Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/AVRO-798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Doug Cutting updated AVRO-798:
------------------------------

    Resolution: Fixed
        Status: Resolved  (was: Patch Available)

I committed this.

> add checksums to Snappy codec
> -----------------------------
>
>                 Key: AVRO-798
>                 URL: https://issues.apache.org/jira/browse/AVRO-798
>             Project: Avro
>          Issue Type: Improvement
>          Components: java, spec
>            Reporter: Doug Cutting
>            Assignee: Doug Cutting
>             Fix For: 1.5.1
>
>         Attachments: AVRO-798.patch, AVRO-798.patch
>
>
> A checksum might be included with each compressed block to better detect errors.  While some filesystems (e.g. HDFS) may checksum data, not all do.  Data files may also accumulate errors when copied between filesystems.  For back-compatibility, we cannot easily add checksums to existing data files, but a new codec provides us with the opportunity to do this.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (AVRO-798) add checksums to Snappy codec

Posted by "Scott Carey (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/AVRO-798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13016630#comment-13016630 ] 

Scott Carey commented on AVRO-798:
----------------------------------

Should we use PureJavaCRC32?  Its a lot faster, though less so for larger arrays.


http://svn.apache.org/viewvc/hadoop/common/trunk/src/java/org/apache/hadoop/util/PureJavaCrc32.java?revision=953881&view=markup

We can just copy that code into a class in o.a.a.util.

> add checksums to Snappy codec
> -----------------------------
>
>                 Key: AVRO-798
>                 URL: https://issues.apache.org/jira/browse/AVRO-798
>             Project: Avro
>          Issue Type: Improvement
>          Components: java, spec
>            Reporter: Doug Cutting
>            Assignee: Doug Cutting
>             Fix For: 1.5.1
>
>         Attachments: AVRO-798.patch, AVRO-798.patch
>
>
> A checksum might be included with each compressed block to better detect errors.  While some filesystems (e.g. HDFS) may checksum data, not all do.  Data files may also accumulate errors when copied between filesystems.  For back-compatibility, we cannot easily add checksums to existing data files, but a new codec provides us with the opportunity to do this.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (AVRO-798) add checksums to Snappy codec

Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/AVRO-798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Doug Cutting updated AVRO-798:
------------------------------

    Attachment: AVRO-798.patch

Here's a patch that implements this.

> add checksums to Snappy codec
> -----------------------------
>
>                 Key: AVRO-798
>                 URL: https://issues.apache.org/jira/browse/AVRO-798
>             Project: Avro
>          Issue Type: Improvement
>          Components: java, spec
>            Reporter: Doug Cutting
>            Assignee: Doug Cutting
>             Fix For: 1.5.1
>
>         Attachments: AVRO-798.patch
>
>
> A checksum might be included with each compressed block to better detect errors.  While some filesystems (e.g. HDFS) may checksum data, not all do.  Data files may also accumulate errors when copied between filesystems.  For back-compatibility, we cannot easily add checksums to existing data files, but a new codec provides us with the opportunity to do this.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (AVRO-798) add checksums to Snappy codec

Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/AVRO-798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13018627#comment-13018627 ] 

Doug Cutting commented on AVRO-798:
-----------------------------------

Any objections to committing this?

> add checksums to Snappy codec
> -----------------------------
>
>                 Key: AVRO-798
>                 URL: https://issues.apache.org/jira/browse/AVRO-798
>             Project: Avro
>          Issue Type: Improvement
>          Components: java, spec
>            Reporter: Doug Cutting
>            Assignee: Doug Cutting
>             Fix For: 1.5.1
>
>         Attachments: AVRO-798.patch, AVRO-798.patch
>
>
> A checksum might be included with each compressed block to better detect errors.  While some filesystems (e.g. HDFS) may checksum data, not all do.  Data files may also accumulate errors when copied between filesystems.  For back-compatibility, we cannot easily add checksums to existing data files, but a new codec provides us with the opportunity to do this.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (AVRO-798) add checksums to Snappy codec

Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/AVRO-798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Doug Cutting updated AVRO-798:
------------------------------

    Status: Patch Available  (was: Open)

> add checksums to Snappy codec
> -----------------------------
>
>                 Key: AVRO-798
>                 URL: https://issues.apache.org/jira/browse/AVRO-798
>             Project: Avro
>          Issue Type: Improvement
>          Components: java, spec
>            Reporter: Doug Cutting
>            Assignee: Doug Cutting
>             Fix For: 1.5.1
>
>         Attachments: AVRO-798.patch, AVRO-798.patch
>
>
> A checksum might be included with each compressed block to better detect errors.  While some filesystems (e.g. HDFS) may checksum data, not all do.  Data files may also accumulate errors when copied between filesystems.  For back-compatibility, we cannot easily add checksums to existing data files, but a new codec provides us with the opportunity to do this.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira