You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@avro.apache.org by "Doug Cutting (JIRA)" <ji...@apache.org> on 2011/04/07 00:10:06 UTC
[jira] [Created] (AVRO-798) add checksums to Snappy codec
add checksums to Snappy codec
-----------------------------
Key: AVRO-798
URL: https://issues.apache.org/jira/browse/AVRO-798
Project: Avro
Issue Type: Improvement
Components: java, spec
Reporter: Doug Cutting
Assignee: Doug Cutting
Fix For: 1.5.1
A checksum might be included with each compressed block to better detect errors. While some filesystems (e.g. HDFS) may checksum data, not all do. Data files may also accumulate errors when copied between filesystems. For back-compatibility, we cannot easily add checksums to existing data files, but a new codec provides us with the opportunity to do this.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AVRO-798) add checksums to Snappy codec
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Cutting updated AVRO-798:
------------------------------
Attachment: AVRO-798.patch
Patch that also updates spec.
> add checksums to Snappy codec
> -----------------------------
>
> Key: AVRO-798
> URL: https://issues.apache.org/jira/browse/AVRO-798
> Project: Avro
> Issue Type: Improvement
> Components: java, spec
> Reporter: Doug Cutting
> Assignee: Doug Cutting
> Fix For: 1.5.1
>
> Attachments: AVRO-798.patch, AVRO-798.patch
>
>
> A checksum might be included with each compressed block to better detect errors. While some filesystems (e.g. HDFS) may checksum data, not all do. Data files may also accumulate errors when copied between filesystems. For back-compatibility, we cannot easily add checksums to existing data files, but a new codec provides us with the opportunity to do this.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AVRO-798) add checksums to Snappy codec
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Cutting updated AVRO-798:
------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
I committed this.
> add checksums to Snappy codec
> -----------------------------
>
> Key: AVRO-798
> URL: https://issues.apache.org/jira/browse/AVRO-798
> Project: Avro
> Issue Type: Improvement
> Components: java, spec
> Reporter: Doug Cutting
> Assignee: Doug Cutting
> Fix For: 1.5.1
>
> Attachments: AVRO-798.patch, AVRO-798.patch
>
>
> A checksum might be included with each compressed block to better detect errors. While some filesystems (e.g. HDFS) may checksum data, not all do. Data files may also accumulate errors when copied between filesystems. For back-compatibility, we cannot easily add checksums to existing data files, but a new codec provides us with the opportunity to do this.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-798) add checksums to Snappy codec
Posted by "Scott Carey (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13016630#comment-13016630 ]
Scott Carey commented on AVRO-798:
----------------------------------
Should we use PureJavaCRC32? Its a lot faster, though less so for larger arrays.
http://svn.apache.org/viewvc/hadoop/common/trunk/src/java/org/apache/hadoop/util/PureJavaCrc32.java?revision=953881&view=markup
We can just copy that code into a class in o.a.a.util.
> add checksums to Snappy codec
> -----------------------------
>
> Key: AVRO-798
> URL: https://issues.apache.org/jira/browse/AVRO-798
> Project: Avro
> Issue Type: Improvement
> Components: java, spec
> Reporter: Doug Cutting
> Assignee: Doug Cutting
> Fix For: 1.5.1
>
> Attachments: AVRO-798.patch, AVRO-798.patch
>
>
> A checksum might be included with each compressed block to better detect errors. While some filesystems (e.g. HDFS) may checksum data, not all do. Data files may also accumulate errors when copied between filesystems. For back-compatibility, we cannot easily add checksums to existing data files, but a new codec provides us with the opportunity to do this.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AVRO-798) add checksums to Snappy codec
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Cutting updated AVRO-798:
------------------------------
Attachment: AVRO-798.patch
Here's a patch that implements this.
> add checksums to Snappy codec
> -----------------------------
>
> Key: AVRO-798
> URL: https://issues.apache.org/jira/browse/AVRO-798
> Project: Avro
> Issue Type: Improvement
> Components: java, spec
> Reporter: Doug Cutting
> Assignee: Doug Cutting
> Fix For: 1.5.1
>
> Attachments: AVRO-798.patch
>
>
> A checksum might be included with each compressed block to better detect errors. While some filesystems (e.g. HDFS) may checksum data, not all do. Data files may also accumulate errors when copied between filesystems. For back-compatibility, we cannot easily add checksums to existing data files, but a new codec provides us with the opportunity to do this.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-798) add checksums to Snappy codec
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13018627#comment-13018627 ]
Doug Cutting commented on AVRO-798:
-----------------------------------
Any objections to committing this?
> add checksums to Snappy codec
> -----------------------------
>
> Key: AVRO-798
> URL: https://issues.apache.org/jira/browse/AVRO-798
> Project: Avro
> Issue Type: Improvement
> Components: java, spec
> Reporter: Doug Cutting
> Assignee: Doug Cutting
> Fix For: 1.5.1
>
> Attachments: AVRO-798.patch, AVRO-798.patch
>
>
> A checksum might be included with each compressed block to better detect errors. While some filesystems (e.g. HDFS) may checksum data, not all do. Data files may also accumulate errors when copied between filesystems. For back-compatibility, we cannot easily add checksums to existing data files, but a new codec provides us with the opportunity to do this.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AVRO-798) add checksums to Snappy codec
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Cutting updated AVRO-798:
------------------------------
Status: Patch Available (was: Open)
> add checksums to Snappy codec
> -----------------------------
>
> Key: AVRO-798
> URL: https://issues.apache.org/jira/browse/AVRO-798
> Project: Avro
> Issue Type: Improvement
> Components: java, spec
> Reporter: Doug Cutting
> Assignee: Doug Cutting
> Fix For: 1.5.1
>
> Attachments: AVRO-798.patch, AVRO-798.patch
>
>
> A checksum might be included with each compressed block to better detect errors. While some filesystems (e.g. HDFS) may checksum data, not all do. Data files may also accumulate errors when copied between filesystems. For back-compatibility, we cannot easily add checksums to existing data files, but a new codec provides us with the opportunity to do this.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira