You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@avro.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2016/09/10 23:00:25 UTC
[jira] [Commented] (AVRO-1873) avro gem doesn't compatible with
other languages with snappy compression
[ https://issues.apache.org/jira/browse/AVRO-1873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15480596#comment-15480596 ]
ASF GitHub Bot commented on AVRO-1873:
--------------------------------------
GitHub user rdblue opened a pull request:
https://github.com/apache/avro/pull/121
AVRO-1873: Add CRC32 checksum to Snappy-compressed blocks.
Java and other implementations require this CRC32 checksum of the
uncompressed content in order to read the data. This implements the
checksum, with backward-compatibility for files written by old versions
of avro-ruby. If the checksum doesn't match, avro-ruby will decompress
the incoming bytes and pass them on assuming that the file is from an
old reader.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/rdblue/avro AVRO-1873-fix-snappy-checksum
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/avro/pull/121.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #121
----
commit 202db76952a1a682b7bb14a5ab7d040674bee777
Author: Ryan Blue <bl...@apache.org>
Date: 2016-09-10T22:57:30Z
AVRO-1873: Add CRC32 checksum to Snappy-compressed blocks.
Java and other implementations require this CRC32 checksum of the
uncompressed content in order to read the data. This implements the
checksum, with backward-compatibility for files written by old versions
of avro-ruby. If the checksum doesn't match, avro-ruby will decompress
the incoming bytes and pass them on assuming that the file is from an
old reader.
----
> avro gem doesn't compatible with other languages with snappy compression
> ------------------------------------------------------------------------
>
> Key: AVRO-1873
> URL: https://issues.apache.org/jira/browse/AVRO-1873
> Project: Avro
> Issue Type: Bug
> Components: ruby
> Affects Versions: 1.8.1
> Environment: CentOS 6.8 64bit, Snappy 1.1.0, Python 3.5, Ruby 2.2.3
> Reporter: Pumsuk Cho
> Priority: Blocker
> Fix For: 1.8.2
>
>
> I've tested avro gem today, then found some weird result.
> With python library like "fastavro", generated an avro file snappy compressed. This file works fine with avro-tools-1.8.1.jar.
> java -jar avro-tools-1.8.1.jar tojson testing.avro returns what I expected.
> But NOT compatible with ruby using avro gem returns "Invalid Input" message. And snappy compressed avro file made with avro gem doesn't work with avro-tools nor in python with avro-python3 and fastavro.
> my ruby codes are below:
> schema = Avro::Schema.paese(File.open('test.avsc', 'r').read)
> avrofile = File.open('test.avro', 'wb')
> writer = Avro::IO::DatumWriter.new(schema)
> datawriter = Avro::DataFile::Writer.new file, writer, schema, 'snappy'
> datawriter<< {"title" => "Avro", "author" => "Apache Foundation"}
> datawriter.close
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)