You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@avro.apache.org by "Pritchard, Charles X. -ND" <Ch...@disney.com> on 2014/10/15 02:33:01 UTC

invalid file with avro tools random and tojson

Thought I’d ask for a second set of eyes.
I’m working with a large schema, a basic translation of some tab separated data with many fields (over 300).

It appears that the output of Random can not be read back as JSON, with this schema.
Is this a bug, or me simply mis-using the method calls in Avro tools?



$ java -jar avro-tools-1.7.7.jar random --schema-file schema.avsc --count 20 - > test.avro
$ java -jar avro-tools-1.7.7.jar tojson test.avro
Exception in thread "main" org.apache.avro.AvroRuntimeException: java.io.IOException: Block size invalid or too large for this implementation: -51
	at org.apache.avro.file.DataFileStream.hasNextBlock(DataFileStream.java:275)
	at org.apache.avro.file.DataFileStream.hasNext(DataFileStream.java:197)
	at org.apache.avro.tool.DataFileReadTool.run(DataFileReadTool.java:77)
	at org.apache.avro.tool.Main.run(Main.java:84)
	at org.apache.avro.tool.Main.main(Main.java:73)
Caused by: java.io.IOException: Block size invalid or too large for this implementation: -51
	at org.apache.avro.file.DataFileStream.hasNextBlock(DataFileStream.java:266)
	... 4 more

Re: invalid file with avro tools random and tojson

Posted by Doug Cutting <cu...@apache.org>.

This is a bug.  It works if you change the command to not use standard
output, e.g.:

  random --schema-file schema.avsc --count 20 test.avro

The problem is that TestUtil.java prints something to standard output
that should go to standard error.

I filed an issue in Jira for this and will post a fix there soon.

  https://issues.apache.org/jira/browse/AVRO-1597

Doug

On Tue, Oct 14, 2014 at 5:33 PM, Pritchard, Charles X. -ND
<Ch...@disney.com> wrote:
> Thought I’d ask for a second set of eyes.
> I’m working with a large schema, a basic translation of some tab separated data with many fields (over 300).
>
> It appears that the output of Random can not be read back as JSON, with this schema.
> Is this a bug, or me simply mis-using the method calls in Avro tools?
>
>
>
> $ java -jar avro-tools-1.7.7.jar random --schema-file schema.avsc --count 20 - > test.avro
> $ java -jar avro-tools-1.7.7.jar tojson test.avro
> Exception in thread "main" org.apache.avro.AvroRuntimeException: java.io.IOException: Block size invalid or too large for this implementation: -51
>         at org.apache.avro.file.DataFileStream.hasNextBlock(DataFileStream.java:275)
>         at org.apache.avro.file.DataFileStream.hasNext(DataFileStream.java:197)
>         at org.apache.avro.tool.DataFileReadTool.run(DataFileReadTool.java:77)
>         at org.apache.avro.tool.Main.run(Main.java:84)
>         at org.apache.avro.tool.Main.main(Main.java:73)
> Caused by: java.io.IOException: Block size invalid or too large for this implementation: -51
>         at org.apache.avro.file.DataFileStream.hasNextBlock(DataFileStream.java:266)
>         ... 4 more
>