You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@avro.apache.org by "Robert Fuller (JIRA)" <ji...@apache.org> on 2012/05/14 14:10:13 UTC
[jira] [Updated] (AVRO-1081) GenericDatumWriter does not support
native ByteBuffers
[ https://issues.apache.org/jira/browse/AVRO-1081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Fuller updated AVRO-1081:
--------------------------------
Attachment: patch.diff.txt
ByteBufferTest.java
Attached one possible way of fixing this.
I am not 100% happy with the solution, but it should work for our case for now. We are writing several avro files concurrently from within a heavily multithreaded application, and cannot afford to load many of the files into memory at once at that point.
When reading the files again (in hadoop m/r job) it's ok for now to read the bytes back to memory (will read them into a byte array, but for the purpose of the test wanted to put them into a byte buffer).
Am also now wondering whether you might consider direct support for "File" (or something like FileData) in avro. This would allow a way to include into avro items which exceed the amount of available memory. But maybe it's not a requirement anybody else would have...
> GenericDatumWriter does not support native ByteBuffers
> ------------------------------------------------------
>
> Key: AVRO-1081
> URL: https://issues.apache.org/jira/browse/AVRO-1081
> Project: Avro
> Issue Type: Bug
> Affects Versions: 1.6.3
> Reporter: Robert Fuller
> Attachments: ByteBufferTest.java, ByteBufferTest.java, patch.diff.txt, patch.diff.txt
>
>
> An exception is thrown when trying to encode bytes backed by a file.
> java.lang.UnsupportedOperationException: null
> at java.nio.ByteBuffer.arrayOffset(ByteBuffer.java:968) ~[na:1.6.0_31]
> at org.apache.avro.io.BinaryEncoder.writeBytes(BinaryEncoder.java:61) ~[avro-1.6.3.jar:1.6.3]
> Note arrayOffset is an optional method, see:
> http://docs.oracle.com/javase/6/docs/api/java/nio/ByteBuffer.html#arrayOffset%28%29
> FileChannel returns native ByteBuffer not HeapedByteBuffer
> See here:
> http://mail-archives.apache.org/mod_mbox/avro-user/201202.mbox/%3CCB57F421.6BFE2%25scott@richrelevance.com%3E
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira