You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@avro.apache.org by "Flo Leibert (JIRA)" <ji...@apache.org> on 2009/09/28 09:55:16 UTC
[jira] Created: (AVRO-126) Support embedded schema
(de)serialization not only for files but for arbitrary byte arrays
Support embedded schema (de)serialization not only for files but for arbitrary byte arrays
------------------------------------------------------------------------------------------
Key: AVRO-126
URL: https://issues.apache.org/jira/browse/AVRO-126
Project: Avro
Issue Type: Improvement
Components: java
Reporter: Flo Leibert
While DataFileWriter can serialize to an arbitrary output stream, DataFileReader cannot. However, it's useful to allow for this. Thus, a SeekableByteArrayInput should be available to wrap a byte[] and allow for Deserialization from a byte input stream.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-126) Support embedded schema
(de)serialization not only for files but for arbitrary byte arrays
Posted by "Flo Leibert (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Flo Leibert updated AVRO-126:
-----------------------------
Attachment: AVRO-126.patch
SeekableByteArrayInput now available
> Support embedded schema (de)serialization not only for files but for arbitrary byte arrays
> ------------------------------------------------------------------------------------------
>
> Key: AVRO-126
> URL: https://issues.apache.org/jira/browse/AVRO-126
> Project: Avro
> Issue Type: Improvement
> Components: java
> Reporter: Flo Leibert
> Attachments: AVRO-126.patch
>
>
> While DataFileWriter can serialize to an arbitrary output stream, DataFileReader cannot. However, it's useful to allow for this. Thus, a SeekableByteArrayInput should be available to wrap a byte[] and allow for Deserialization from a byte input stream.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AVRO-126) Support embedded schema
(de)serialization not only for files but for arbitrary byte arrays
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12760714#action_12760714 ]
Doug Cutting commented on AVRO-126:
-----------------------------------
Are you using a new DataFile-formatted blob per instance? That's pretty heavy-weight.
> Support embedded schema (de)serialization not only for files but for arbitrary byte arrays
> ------------------------------------------------------------------------------------------
>
> Key: AVRO-126
> URL: https://issues.apache.org/jira/browse/AVRO-126
> Project: Avro
> Issue Type: Improvement
> Components: java
> Reporter: Flo Leibert
> Attachments: AVRO-126.patch
>
>
> While DataFileWriter can serialize to an arbitrary output stream, DataFileReader cannot. However, it's useful to allow for this. Thus, a SeekableByteArrayInput should be available to wrap a byte[] and allow for Deserialization from a byte input stream.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-126) Support embedded schema
(de)serialization not only for files but for arbitrary byte arrays
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Cutting updated AVRO-126:
------------------------------
Resolution: Duplicate
Fix Version/s: (was: 1.3.0)
Status: Resolved (was: Patch Available)
> Support embedded schema (de)serialization not only for files but for arbitrary byte arrays
> ------------------------------------------------------------------------------------------
>
> Key: AVRO-126
> URL: https://issues.apache.org/jira/browse/AVRO-126
> Project: Avro
> Issue Type: Improvement
> Components: java
> Reporter: Flo Leibert
> Attachments: AVRO-126.patch
>
>
> While DataFileWriter can serialize to an arbitrary output stream, DataFileReader cannot. However, it's useful to allow for this. Thus, a SeekableByteArrayInput should be available to wrap a byte[] and allow for Deserialization from a byte input stream.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-126) Support embedded schema
(de)serialization not only for files but for arbitrary byte arrays
Posted by "Flo Leibert (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Flo Leibert updated AVRO-126:
-----------------------------
I agree it's heavy weight - I still think there are some usecases for this - i.e. when you have a 1000s of different schemata where each schema has a big amount of data...
> Support embedded schema (de)serialization not only for files but for arbitrary byte arrays
> ------------------------------------------------------------------------------------------
>
> Key: AVRO-126
> URL: https://issues.apache.org/jira/browse/AVRO-126
> Project: Avro
> Issue Type: Improvement
> Components: java
> Reporter: Flo Leibert
> Attachments: AVRO-126.patch
>
>
> While DataFileWriter can serialize to an arbitrary output stream, DataFileReader cannot. However, it's useful to allow for this. Thus, a SeekableByteArrayInput should be available to wrap a byte[] and allow for Deserialization from a byte input stream.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-126) Support embedded schema
(de)serialization not only for files but for arbitrary byte arrays
Posted by "Flo Leibert (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Flo Leibert updated AVRO-126:
-----------------------------
Status: Patch Available (was: Open)
> Support embedded schema (de)serialization not only for files but for arbitrary byte arrays
> ------------------------------------------------------------------------------------------
>
> Key: AVRO-126
> URL: https://issues.apache.org/jira/browse/AVRO-126
> Project: Avro
> Issue Type: Improvement
> Components: java
> Reporter: Flo Leibert
> Attachments: AVRO-126.patch
>
>
> While DataFileWriter can serialize to an arbitrary output stream, DataFileReader cannot. However, it's useful to allow for this. Thus, a SeekableByteArrayInput should be available to wrap a byte[] and allow for Deserialization from a byte input stream.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-126) Support embedded schema
(de)serialization not only for files but for arbitrary byte arrays
Posted by "Flo Leibert (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Flo Leibert updated AVRO-126:
-----------------------------
The DatumReader / DatumWriter don't support the EmbeddedSchema - I propose renaming the DataFileReader/Writer to DataStreamReader/Writer and thus supporting use cases outside the FS spectrum. I am using those for storing data in Voldemort where I don't have file access but I'm passed a byte array...
> Support embedded schema (de)serialization not only for files but for arbitrary byte arrays
> ------------------------------------------------------------------------------------------
>
> Key: AVRO-126
> URL: https://issues.apache.org/jira/browse/AVRO-126
> Project: Avro
> Issue Type: Improvement
> Components: java
> Reporter: Flo Leibert
> Attachments: AVRO-126.patch
>
>
> While DataFileWriter can serialize to an arbitrary output stream, DataFileReader cannot. However, it's useful to allow for this. Thus, a SeekableByteArrayInput should be available to wrap a byte[] and allow for Deserialization from a byte input stream.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AVRO-126) Support embedded schema
(de)serialization not only for files but for arbitrary byte arrays
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12798746#action_12798746 ]
Doug Cutting commented on AVRO-126:
-----------------------------------
I think this was fixed in AVRO-160. There's a new class DataFileStream that accepts an arbitrary input stream. Does this suffice?
> Support embedded schema (de)serialization not only for files but for arbitrary byte arrays
> ------------------------------------------------------------------------------------------
>
> Key: AVRO-126
> URL: https://issues.apache.org/jira/browse/AVRO-126
> Project: Avro
> Issue Type: Improvement
> Components: java
> Reporter: Flo Leibert
> Fix For: 1.3.0
>
> Attachments: AVRO-126.patch
>
>
> While DataFileWriter can serialize to an arbitrary output stream, DataFileReader cannot. However, it's useful to allow for this. Thus, a SeekableByteArrayInput should be available to wrap a byte[] and allow for Deserialization from a byte input stream.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AVRO-126) Support embedded schema
(de)serialization not only for files but for arbitrary byte arrays
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12760672#action_12760672 ]
Doug Cutting commented on AVRO-126:
-----------------------------------
Can you further elaborate the use case? In particular, what is the non-file use case for DataFileReader? If you need to directly write and read instances from streams, then you can use DatumWriter and DatumReader directly, no? Thanks!
> Support embedded schema (de)serialization not only for files but for arbitrary byte arrays
> ------------------------------------------------------------------------------------------
>
> Key: AVRO-126
> URL: https://issues.apache.org/jira/browse/AVRO-126
> Project: Avro
> Issue Type: Improvement
> Components: java
> Reporter: Flo Leibert
> Attachments: AVRO-126.patch
>
>
> While DataFileWriter can serialize to an arbitrary output stream, DataFileReader cannot. However, it's useful to allow for this. Thus, a SeekableByteArrayInput should be available to wrap a byte[] and allow for Deserialization from a byte input stream.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-126) Support embedded schema
(de)serialization not only for files but for arbitrary byte arrays
Posted by "Jeff Hammerbacher (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jeff Hammerbacher updated AVRO-126:
-----------------------------------
Fix Version/s: 1.3.0
I think we should resolve this issue before the 1.3 release.
> Support embedded schema (de)serialization not only for files but for arbitrary byte arrays
> ------------------------------------------------------------------------------------------
>
> Key: AVRO-126
> URL: https://issues.apache.org/jira/browse/AVRO-126
> Project: Avro
> Issue Type: Improvement
> Components: java
> Reporter: Flo Leibert
> Fix For: 1.3.0
>
> Attachments: AVRO-126.patch
>
>
> While DataFileWriter can serialize to an arbitrary output stream, DataFileReader cannot. However, it's useful to allow for this. Thus, a SeekableByteArrayInput should be available to wrap a byte[] and allow for Deserialization from a byte input stream.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.