You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@avro.apache.org by "Lucas Martin-King (JIRA)" <ji...@apache.org> on 2011/09/23 06:09:26 UTC
[jira] [Created] (AVRO-893) New datum_t API does not write out spec
compatible files - header, metadata and blocks are missing.
New datum_t API does not write out spec compatible files - header, metadata and blocks are missing.
---------------------------------------------------------------------------------------------------
Key: AVRO-893
URL: https://issues.apache.org/jira/browse/AVRO-893
Project: Avro
Issue Type: Improvement
Components: c
Affects Versions: 1.5.4
Environment: N/A
Reporter: Lucas Martin-King
We have switched over to the new datum_t API for our application, as it provides more AVRO types that we need (enum, array, etc), however we have noticed that there isn't a way to write out a spec compatible AVRO file.
The old API was able to write out a header as well as the schema, the finally each data block.
I could possibly try and re-implement this myself, however I am unsure as to the design (how it would fit into the new datum API).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AVRO-893) New value_t API does not write out spec
compatible files - header, metadata and blocks are missing.
Posted by "Lucas Martin-King (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lucas Martin-King updated AVRO-893:
-----------------------------------
Remaining Estimate: (was: 24h)
Original Estimate: (was: 24h)
> New value_t API does not write out spec compatible files - header, metadata and blocks are missing.
> ---------------------------------------------------------------------------------------------------
>
> Key: AVRO-893
> URL: https://issues.apache.org/jira/browse/AVRO-893
> Project: Avro
> Issue Type: Improvement
> Components: c
> Affects Versions: 1.5.4
> Environment: N/A
> Reporter: Lucas Martin-King
> Labels: file
>
> We have switched over to the new value_t API for our application, as it provides more AVRO types that we need (enum, array, etc), however we have noticed that there isn't a way to write out a spec compatible AVRO file.
> The old API was able to write out a header as well as the schema, the finally each data block (and sync marker).
> I could possibly try and re-implement this myself, however I am unsure as to the design (how it would fit into the new datum API).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AVRO-893) New datum_t API does not write out spec
compatible files - header, metadata and blocks are missing.
Posted by "Lucas Martin-King (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lucas Martin-King updated AVRO-893:
-----------------------------------
Description:
We have switched over to the new datum_t API for our application, as it provides more AVRO types that we need (enum, array, etc), however we have noticed that there isn't a way to write out a spec compatible AVRO file.
The old API was able to write out a header as well as the schema, the finally each data block (and sync marker).
I could possibly try and re-implement this myself, however I am unsure as to the design (how it would fit into the new datum API).
was:
We have switched over to the new datum_t API for our application, as it provides more AVRO types that we need (enum, array, etc), however we have noticed that there isn't a way to write out a spec compatible AVRO file.
The old API was able to write out a header as well as the schema, the finally each data block.
I could possibly try and re-implement this myself, however I am unsure as to the design (how it would fit into the new datum API).
> New datum_t API does not write out spec compatible files - header, metadata and blocks are missing.
> ---------------------------------------------------------------------------------------------------
>
> Key: AVRO-893
> URL: https://issues.apache.org/jira/browse/AVRO-893
> Project: Avro
> Issue Type: Improvement
> Components: c
> Affects Versions: 1.5.4
> Environment: N/A
> Reporter: Lucas Martin-King
> Labels: file
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> We have switched over to the new datum_t API for our application, as it provides more AVRO types that we need (enum, array, etc), however we have noticed that there isn't a way to write out a spec compatible AVRO file.
> The old API was able to write out a header as well as the schema, the finally each data block (and sync marker).
> I could possibly try and re-implement this myself, however I am unsure as to the design (how it would fit into the new datum API).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-893) New value_t API does not write out
spec compatible files - header, metadata and blocks are missing.
Posted by "Douglas Creager (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13114363#comment-13114363 ]
Douglas Creager commented on AVRO-893:
--------------------------------------
I actually have an implementation of this in my github tracking branch, which I haven't merged into SVN yet. I assume you're using HEAD, and not any of the 1.5 releases, since those don't have the new value_t API in them. I'll try to get these features merged into SVN this week, but in the meantime, you could try building the latest on my github branch:
https://github.com/dcreager/avro/tree/dcreager-mods
You use the same {{avro_file_writer_t}} type from before, but there's a new {{avro_file_writer_append_value}} function that takes in an {{avro_value_t}} instance.
> New value_t API does not write out spec compatible files - header, metadata and blocks are missing.
> ---------------------------------------------------------------------------------------------------
>
> Key: AVRO-893
> URL: https://issues.apache.org/jira/browse/AVRO-893
> Project: Avro
> Issue Type: Improvement
> Components: c
> Affects Versions: 1.5.4
> Environment: N/A
> Reporter: Lucas Martin-King
> Labels: file
>
> We have switched over to the new value_t API for our application, as it provides more AVRO types that we need (enum, array, etc), however we have noticed that there isn't a way to write out a spec compatible AVRO file.
> The old API was able to write out a header as well as the schema, the finally each data block (and sync marker).
> I could possibly try and re-implement this myself, however I am unsure as to the design (how it would fit into the new datum API).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AVRO-893) New value_t API does not write out spec
compatible files - header, metadata and blocks are missing.
Posted by "Lucas Martin-King (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lucas Martin-King updated AVRO-893:
-----------------------------------
Description:
We have switched over to the new value_t API for our application, as it provides more AVRO types that we need (enum, array, etc), however we have noticed that there isn't a way to write out a spec compatible AVRO file.
The old API was able to write out a header as well as the schema, the finally each data block (and sync marker).
I could possibly try and re-implement this myself, however I am unsure as to the design (how it would fit into the new datum API).
was:
We have switched over to the new datum_t API for our application, as it provides more AVRO types that we need (enum, array, etc), however we have noticed that there isn't a way to write out a spec compatible AVRO file.
The old API was able to write out a header as well as the schema, the finally each data block (and sync marker).
I could possibly try and re-implement this myself, however I am unsure as to the design (how it would fit into the new datum API).
Summary: New value_t API does not write out spec compatible files - header, metadata and blocks are missing. (was: New datum_t API does not write out spec compatible files - header, metadata and blocks are missing.)
> New value_t API does not write out spec compatible files - header, metadata and blocks are missing.
> ---------------------------------------------------------------------------------------------------
>
> Key: AVRO-893
> URL: https://issues.apache.org/jira/browse/AVRO-893
> Project: Avro
> Issue Type: Improvement
> Components: c
> Affects Versions: 1.5.4
> Environment: N/A
> Reporter: Lucas Martin-King
> Labels: file
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> We have switched over to the new value_t API for our application, as it provides more AVRO types that we need (enum, array, etc), however we have noticed that there isn't a way to write out a spec compatible AVRO file.
> The old API was able to write out a header as well as the schema, the finally each data block (and sync marker).
> I could possibly try and re-implement this myself, however I am unsure as to the design (how it would fit into the new datum API).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (AVRO-893) New value_t API does not write out
spec compatible files - header, metadata and blocks are missing.
Posted by "Douglas Creager (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Douglas Creager resolved AVRO-893.
----------------------------------
Resolution: Fixed
Fix Version/s: 1.6.0
Assignee: Douglas Creager
I merged this into SVN trunk. You should be able to write avro_value_t instances into an Avro data file now.
> New value_t API does not write out spec compatible files - header, metadata and blocks are missing.
> ---------------------------------------------------------------------------------------------------
>
> Key: AVRO-893
> URL: https://issues.apache.org/jira/browse/AVRO-893
> Project: Avro
> Issue Type: Improvement
> Components: c
> Affects Versions: 1.5.4
> Environment: N/A
> Reporter: Lucas Martin-King
> Assignee: Douglas Creager
> Labels: file
> Fix For: 1.6.0
>
>
> We have switched over to the new value_t API for our application, as it provides more AVRO types that we need (enum, array, etc), however we have noticed that there isn't a way to write out a spec compatible AVRO file.
> The old API was able to write out a header as well as the schema, the finally each data block (and sync marker).
> I could possibly try and re-implement this myself, however I am unsure as to the design (how it would fit into the new datum API).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira