You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@avro.apache.org by "Scott Banachowski (JIRA)" <ji...@apache.org> on 2010/04/02 20:24:27 UTC
[jira] Created: (AVRO-498) Clarify spec for object container files
Clarify spec for object container files
---------------------------------------
Key: AVRO-498
URL: https://issues.apache.org/jira/browse/AVRO-498
Project: Avro
Issue Type: Improvement
Components: spec
Reporter: Scott Banachowski
I have some proposed changes for the spec in the Object Container Files section.
The fixes are:
1) call out explicitly that when it says long, it means avro-encoded long.
2) since the implementations are serializing the metadata as an avro map, this map should be followed by a block size of 0 (consistent with avro map encoding)
3) fix the example header schema so it passes JSONLint (it had an extra comma after the last array value).
Number 2 above is something I'm not sure about, because I don't know if it's a compatible change... I hope it does not break any implementations.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-498) Clarify spec for object container files
Posted by "Scott Banachowski (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Scott Banachowski updated AVRO-498:
-----------------------------------
Attachment: AVRO-498.patch
patch to spec.xml
> Clarify spec for object container files
> ---------------------------------------
>
> Key: AVRO-498
> URL: https://issues.apache.org/jira/browse/AVRO-498
> Project: Avro
> Issue Type: Improvement
> Components: spec
> Reporter: Scott Banachowski
> Attachments: AVRO-498.patch
>
>
> I have some proposed changes for the spec in the Object Container Files section.
> The fixes are:
> 1) call out explicitly that when it says long, it means avro-encoded long.
> 2) since the implementations are serializing the metadata as an avro map, this map should be followed by a block size of 0 (consistent with avro map encoding)
> 3) fix the example header schema so it passes JSONLint (it had an extra comma after the last array value).
> Number 2 above is something I'm not sure about, because I don't know if it's a compatible change... I hope it does not break any implementations.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-498) Clarify spec for object container files
Posted by "Scott Banachowski (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Scott Banachowski updated AVRO-498:
-----------------------------------
Status: Patch Available (was: Open)
> Clarify spec for object container files
> ---------------------------------------
>
> Key: AVRO-498
> URL: https://issues.apache.org/jira/browse/AVRO-498
> Project: Avro
> Issue Type: Improvement
> Components: spec
> Reporter: Scott Banachowski
> Attachments: AVRO-498.patch
>
>
> I have some proposed changes for the spec in the Object Container Files section.
> The fixes are:
> 1) call out explicitly that when it says long, it means avro-encoded long.
> 2) since the implementations are serializing the metadata as an avro map, this map should be followed by a block size of 0 (consistent with avro map encoding)
> 3) fix the example header schema so it passes JSONLint (it had an extra comma after the last array value).
> Number 2 above is something I'm not sure about, because I don't know if it's a compatible change... I hope it does not break any implementations.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AVRO-498) Clarify spec for object container
files
Posted by "Scott Carey (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12852926#action_12852926 ]
Scott Carey commented on AVRO-498:
----------------------------------
bq. Perhaps we should instead change the spec to say that metadata is written as if with the schema {"type":"map", "values": "bytes"}?
+1
IMO that is the clearest way to describe it.
> Clarify spec for object container files
> ---------------------------------------
>
> Key: AVRO-498
> URL: https://issues.apache.org/jira/browse/AVRO-498
> Project: Avro
> Issue Type: Improvement
> Components: spec
> Reporter: Scott Banachowski
> Attachments: AVRO-498.patch
>
>
> I have some proposed changes for the spec in the Object Container Files section.
> The fixes are:
> 1) call out explicitly that when it says long, it means avro-encoded long.
> 2) since the implementations are serializing the metadata as an avro map, this map should be followed by a block size of 0 (consistent with avro map encoding)
> 3) fix the example header schema so it passes JSONLint (it had an extra comma after the last array value).
> Number 2 above is something I'm not sure about, because I don't know if it's a compatible change... I hope it does not break any implementations.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AVRO-498) Clarify spec for object container
files
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12852907#action_12852907 ]
Doug Cutting commented on AVRO-498:
-----------------------------------
Whenever the term 'long' is used in the spec it's intended to refer to an Avro-encode long. Anything else would be called something like "eight-byte, big-endian value" or somesuch. Perhaps that's what need's clarifying?
As for the metadata, I don't think this is incompatible, since I think every impl reads it as a map, and that this is a spec bug. Perhaps we should instead change the spec to say that metadata is written as if with the schema {"type":"map", "values": "bytes"}?
> Clarify spec for object container files
> ---------------------------------------
>
> Key: AVRO-498
> URL: https://issues.apache.org/jira/browse/AVRO-498
> Project: Avro
> Issue Type: Improvement
> Components: spec
> Reporter: Scott Banachowski
> Attachments: AVRO-498.patch
>
>
> I have some proposed changes for the spec in the Object Container Files section.
> The fixes are:
> 1) call out explicitly that when it says long, it means avro-encoded long.
> 2) since the implementations are serializing the metadata as an avro map, this map should be followed by a block size of 0 (consistent with avro map encoding)
> 3) fix the example header schema so it passes JSONLint (it had an extra comma after the last array value).
> Number 2 above is something I'm not sure about, because I don't know if it's a compatible change... I hope it does not break any implementations.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AVRO-498) Clarify spec for object container
files
Posted by "Scott Banachowski (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12852951#action_12852951 ]
Scott Banachowski commented on AVRO-498:
----------------------------------------
Yes, in both cases (longs and metadata), I assumed these were the case from the context, but feel the spec could be more explicit about it.
> Clarify spec for object container files
> ---------------------------------------
>
> Key: AVRO-498
> URL: https://issues.apache.org/jira/browse/AVRO-498
> Project: Avro
> Issue Type: Improvement
> Components: spec
> Reporter: Scott Banachowski
> Attachments: AVRO-498.patch
>
>
> I have some proposed changes for the spec in the Object Container Files section.
> The fixes are:
> 1) call out explicitly that when it says long, it means avro-encoded long.
> 2) since the implementations are serializing the metadata as an avro map, this map should be followed by a block size of 0 (consistent with avro map encoding)
> 3) fix the example header schema so it passes JSONLint (it had an extra comma after the last array value).
> Number 2 above is something I'm not sure about, because I don't know if it's a compatible change... I hope it does not break any implementations.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.