You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@avro.apache.org by "David Rosenstrauch (JIRA)" <ji...@apache.org> on 2010/07/27 23:09:16 UTC
[jira] Created: (AVRO-601) Enhance schema parser to allow arbitrary
properties to be defined on a record field
Enhance schema parser to allow arbitrary properties to be defined on a record field
-----------------------------------------------------------------------------------
Key: AVRO-601
URL: https://issues.apache.org/jira/browse/AVRO-601
Project: Avro
Issue Type: Improvement
Components: java, spec
Environment: All
Reporter: David Rosenstrauch
Priority: Minor
It's currently not possible to add an arbitrary property on a record field. e.g., in the following example, although the schema parses fine, the "alias" property gets thrown away:
{
"name": "KVPair",
"type": "record",
"fields" : [
{"name": "key", "type": "int", "alias": "EventTime"},
{"name": "values", "type": "bytes"}
]
}
I had read the Avro spec and thought this was actually allowed ("Attributes not defined in this document are permitted as metadata, but must not affect the format of serialized data.") but it appears that such additional properties are only allowed at the schema level, not at the field level too.
This would be a really useful feature to have, as it would allow the user to define meta-data on a field. Currently the only other way to achieve the same behavior would be to do some kind of hack using the field's "doc" attribute, which would be rather ugly.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-601) Enhance schema parser to allow arbitrary
properties to be defined on a record field
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Cutting updated AVRO-601:
------------------------------
Attachment: AVRO-601.patch
Added a test case. I'll commit this later today unless there are objections.
> Enhance schema parser to allow arbitrary properties to be defined on a record field
> -----------------------------------------------------------------------------------
>
> Key: AVRO-601
> URL: https://issues.apache.org/jira/browse/AVRO-601
> Project: Avro
> Issue Type: Improvement
> Components: java, spec
> Environment: All
> Reporter: David Rosenstrauch
> Assignee: Doug Cutting
> Priority: Minor
> Fix For: 1.4.0
>
> Attachments: AVRO-601.patch, AVRO-601.patch, AVRO-601.patch
>
>
> It's currently not possible to add an arbitrary property on a record field. e.g., in the following example, although the schema parses fine, the "alias" property gets thrown away:
> {
> "name": "KVPair",
> "type": "record",
> "fields" : [
> {"name": "key", "type": "int", "alias": "EventTime"},
> {"name": "values", "type": "bytes"}
> ]
> }
> I had read the Avro spec and thought this was actually allowed ("Attributes not defined in this document are permitted as metadata, but must not affect the format of serialized data.") but it appears that such additional properties are only allowed at the schema level, not at the field level too.
> This would be a really useful feature to have, as it would allow the user to define meta-data on a field. Currently the only other way to achieve the same behavior would be to do some kind of hack using the field's "doc" attribute, which would be rather ugly.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-601) Enhance schema parser to allow arbitrary
properties to be defined on a record field
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Cutting updated AVRO-601:
------------------------------
Status: Patch Available (was: Open)
Fix Version/s: 1.4.0
Note also that AVRO-600 would make "aliases" a reserved field.
> Enhance schema parser to allow arbitrary properties to be defined on a record field
> -----------------------------------------------------------------------------------
>
> Key: AVRO-601
> URL: https://issues.apache.org/jira/browse/AVRO-601
> Project: Avro
> Issue Type: Improvement
> Components: java, spec
> Environment: All
> Reporter: David Rosenstrauch
> Assignee: Doug Cutting
> Priority: Minor
> Fix For: 1.4.0
>
> Attachments: AVRO-601.patch
>
>
> It's currently not possible to add an arbitrary property on a record field. e.g., in the following example, although the schema parses fine, the "alias" property gets thrown away:
> {
> "name": "KVPair",
> "type": "record",
> "fields" : [
> {"name": "key", "type": "int", "alias": "EventTime"},
> {"name": "values", "type": "bytes"}
> ]
> }
> I had read the Avro spec and thought this was actually allowed ("Attributes not defined in this document are permitted as metadata, but must not affect the format of serialized data.") but it appears that such additional properties are only allowed at the schema level, not at the field level too.
> This would be a really useful feature to have, as it would allow the user to define meta-data on a field. Currently the only other way to achieve the same behavior would be to do some kind of hack using the field's "doc" attribute, which would be rather ugly.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-601) Enhance schema parser to allow arbitrary
properties to be defined on a record field
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Cutting updated AVRO-601:
------------------------------
Status: Resolved (was: Patch Available)
Hadoop Flags: [Reviewed]
Resolution: Fixed
I just committed this.
> Enhance schema parser to allow arbitrary properties to be defined on a record field
> -----------------------------------------------------------------------------------
>
> Key: AVRO-601
> URL: https://issues.apache.org/jira/browse/AVRO-601
> Project: Avro
> Issue Type: Improvement
> Components: java, spec
> Environment: All
> Reporter: David Rosenstrauch
> Assignee: Doug Cutting
> Priority: Minor
> Fix For: 1.4.0
>
> Attachments: AVRO-601.patch, AVRO-601.patch, AVRO-601.patch, AVRO-601.patch
>
>
> It's currently not possible to add an arbitrary property on a record field. e.g., in the following example, although the schema parses fine, the "alias" property gets thrown away:
> {
> "name": "KVPair",
> "type": "record",
> "fields" : [
> {"name": "key", "type": "int", "alias": "EventTime"},
> {"name": "values", "type": "bytes"}
> ]
> }
> I had read the Avro spec and thought this was actually allowed ("Attributes not defined in this document are permitted as metadata, but must not affect the format of serialized data.") but it appears that such additional properties are only allowed at the schema level, not at the field level too.
> This would be a really useful feature to have, as it would allow the user to define meta-data on a field. Currently the only other way to achieve the same behavior would be to do some kind of hack using the field's "doc" attribute, which would be rather ugly.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AVRO-601) Enhance schema parser to allow
arbitrary properties to be defined on a record field
Posted by "David Rosenstrauch (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12892931#action_12892931 ]
David Rosenstrauch commented on AVRO-601:
-----------------------------------------
Implementation suggestion:
In other words, what I'd like to be able to do, given the schema above is like so:
Schema schema = Schema.parse(schemaText);
Schema.Field keyField = schema.getField("key");
String keyFieldAlias = keyField.getProp("alias");
> Enhance schema parser to allow arbitrary properties to be defined on a record field
> -----------------------------------------------------------------------------------
>
> Key: AVRO-601
> URL: https://issues.apache.org/jira/browse/AVRO-601
> Project: Avro
> Issue Type: Improvement
> Components: java, spec
> Environment: All
> Reporter: David Rosenstrauch
> Priority: Minor
>
> It's currently not possible to add an arbitrary property on a record field. e.g., in the following example, although the schema parses fine, the "alias" property gets thrown away:
> {
> "name": "KVPair",
> "type": "record",
> "fields" : [
> {"name": "key", "type": "int", "alias": "EventTime"},
> {"name": "values", "type": "bytes"}
> ]
> }
> I had read the Avro spec and thought this was actually allowed ("Attributes not defined in this document are permitted as metadata, but must not affect the format of serialized data.") but it appears that such additional properties are only allowed at the schema level, not at the field level too.
> This would be a really useful feature to have, as it would allow the user to define meta-data on a field. Currently the only other way to achieve the same behavior would be to do some kind of hack using the field's "doc" attribute, which would be rather ugly.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-601) Enhance schema parser to allow arbitrary
properties to be defined on a record field
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Cutting updated AVRO-601:
------------------------------
Attachment: AVRO-601.patch
Here's a patch that implements this. Does this work for you?
> Enhance schema parser to allow arbitrary properties to be defined on a record field
> -----------------------------------------------------------------------------------
>
> Key: AVRO-601
> URL: https://issues.apache.org/jira/browse/AVRO-601
> Project: Avro
> Issue Type: Improvement
> Components: java, spec
> Environment: All
> Reporter: David Rosenstrauch
> Assignee: Doug Cutting
> Priority: Minor
> Fix For: 1.4.0
>
> Attachments: AVRO-601.patch
>
>
> It's currently not possible to add an arbitrary property on a record field. e.g., in the following example, although the schema parses fine, the "alias" property gets thrown away:
> {
> "name": "KVPair",
> "type": "record",
> "fields" : [
> {"name": "key", "type": "int", "alias": "EventTime"},
> {"name": "values", "type": "bytes"}
> ]
> }
> I had read the Avro spec and thought this was actually allowed ("Attributes not defined in this document are permitted as metadata, but must not affect the format of serialized data.") but it appears that such additional properties are only allowed at the schema level, not at the field level too.
> This would be a really useful feature to have, as it would allow the user to define meta-data on a field. Currently the only other way to achieve the same behavior would be to do some kind of hack using the field's "doc" attribute, which would be rather ugly.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (AVRO-601) Enhance schema parser to allow
arbitrary properties to be defined on a record field
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Cutting reassigned AVRO-601:
---------------------------------
Assignee: Doug Cutting
> Enhance schema parser to allow arbitrary properties to be defined on a record field
> -----------------------------------------------------------------------------------
>
> Key: AVRO-601
> URL: https://issues.apache.org/jira/browse/AVRO-601
> Project: Avro
> Issue Type: Improvement
> Components: java, spec
> Environment: All
> Reporter: David Rosenstrauch
> Assignee: Doug Cutting
> Priority: Minor
>
> It's currently not possible to add an arbitrary property on a record field. e.g., in the following example, although the schema parses fine, the "alias" property gets thrown away:
> {
> "name": "KVPair",
> "type": "record",
> "fields" : [
> {"name": "key", "type": "int", "alias": "EventTime"},
> {"name": "values", "type": "bytes"}
> ]
> }
> I had read the Avro spec and thought this was actually allowed ("Attributes not defined in this document are permitted as metadata, but must not affect the format of serialized data.") but it appears that such additional properties are only allowed at the schema level, not at the field level too.
> This would be a really useful feature to have, as it would allow the user to define meta-data on a field. Currently the only other way to achieve the same behavior would be to do some kind of hack using the field's "doc" attribute, which would be rather ugly.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-601) Enhance schema parser to allow arbitrary
properties to be defined on a record field
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Cutting updated AVRO-601:
------------------------------
Attachment: AVRO-601.patch
Thiru, thanks for the review!
> In Props.add() , if the new key is already present and the new value for the key is the same as the existing one, we add the key-value pair, which is not not really needed.
Here's a new version that fixes that.
> Enhance schema parser to allow arbitrary properties to be defined on a record field
> -----------------------------------------------------------------------------------
>
> Key: AVRO-601
> URL: https://issues.apache.org/jira/browse/AVRO-601
> Project: Avro
> Issue Type: Improvement
> Components: java, spec
> Environment: All
> Reporter: David Rosenstrauch
> Assignee: Doug Cutting
> Priority: Minor
> Fix For: 1.4.0
>
> Attachments: AVRO-601.patch, AVRO-601.patch, AVRO-601.patch, AVRO-601.patch
>
>
> It's currently not possible to add an arbitrary property on a record field. e.g., in the following example, although the schema parses fine, the "alias" property gets thrown away:
> {
> "name": "KVPair",
> "type": "record",
> "fields" : [
> {"name": "key", "type": "int", "alias": "EventTime"},
> {"name": "values", "type": "bytes"}
> ]
> }
> I had read the Avro spec and thought this was actually allowed ("Attributes not defined in this document are permitted as metadata, but must not affect the format of serialized data.") but it appears that such additional properties are only allowed at the schema level, not at the field level too.
> This would be a really useful feature to have, as it would allow the user to define meta-data on a field. Currently the only other way to achieve the same behavior would be to do some kind of hack using the field's "doc" attribute, which would be rather ugly.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AVRO-601) Enhance schema parser to allow
arbitrary properties to be defined on a record field
Posted by "David Rosenstrauch (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12894668#action_12894668 ]
David Rosenstrauch commented on AVRO-601:
-----------------------------------------
My apologies - I've been swamped with work, and haven't had a chance to review/test the patch. Will try to test it if I have time, but if I can't get to it in a timely fashion, and you're convinced it works and fixes the bug, then by all means please proceed to release it. If the fix provides the functionality/API that I detailed in my first comment on this ticket (or something close to it) I'm sure I'll be happy.
Thanks much for the fix (and the quick turn-around) on this!
> Enhance schema parser to allow arbitrary properties to be defined on a record field
> -----------------------------------------------------------------------------------
>
> Key: AVRO-601
> URL: https://issues.apache.org/jira/browse/AVRO-601
> Project: Avro
> Issue Type: Improvement
> Components: java, spec
> Environment: All
> Reporter: David Rosenstrauch
> Assignee: Doug Cutting
> Priority: Minor
> Fix For: 1.4.0
>
> Attachments: AVRO-601.patch, AVRO-601.patch, AVRO-601.patch, AVRO-601.patch
>
>
> It's currently not possible to add an arbitrary property on a record field. e.g., in the following example, although the schema parses fine, the "alias" property gets thrown away:
> {
> "name": "KVPair",
> "type": "record",
> "fields" : [
> {"name": "key", "type": "int", "alias": "EventTime"},
> {"name": "values", "type": "bytes"}
> ]
> }
> I had read the Avro spec and thought this was actually allowed ("Attributes not defined in this document are permitted as metadata, but must not affect the format of serialized data.") but it appears that such additional properties are only allowed at the schema level, not at the field level too.
> This would be a really useful feature to have, as it would allow the user to define meta-data on a field. Currently the only other way to achieve the same behavior would be to do some kind of hack using the field's "doc" attribute, which would be rather ugly.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AVRO-601) Enhance schema parser to allow
arbitrary properties to be defined on a record field
Posted by "Thiruvalluvan M. G. (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12894972#action_12894972 ]
Thiruvalluvan M. G. commented on AVRO-601:
------------------------------------------
+1
Looks good to me. Thanks Doug for the change.
> Enhance schema parser to allow arbitrary properties to be defined on a record field
> -----------------------------------------------------------------------------------
>
> Key: AVRO-601
> URL: https://issues.apache.org/jira/browse/AVRO-601
> Project: Avro
> Issue Type: Improvement
> Components: java, spec
> Environment: All
> Reporter: David Rosenstrauch
> Assignee: Doug Cutting
> Priority: Minor
> Fix For: 1.4.0
>
> Attachments: AVRO-601.patch, AVRO-601.patch, AVRO-601.patch, AVRO-601.patch
>
>
> It's currently not possible to add an arbitrary property on a record field. e.g., in the following example, although the schema parses fine, the "alias" property gets thrown away:
> {
> "name": "KVPair",
> "type": "record",
> "fields" : [
> {"name": "key", "type": "int", "alias": "EventTime"},
> {"name": "values", "type": "bytes"}
> ]
> }
> I had read the Avro spec and thought this was actually allowed ("Attributes not defined in this document are permitted as metadata, but must not affect the format of serialized data.") but it appears that such additional properties are only allowed at the schema level, not at the field level too.
> This would be a really useful feature to have, as it would allow the user to define meta-data on a field. Currently the only other way to achieve the same behavior would be to do some kind of hack using the field's "doc" attribute, which would be rather ugly.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AVRO-601) Enhance schema parser to allow
arbitrary properties to be defined on a record field
Posted by "Thiruvalluvan M. G. (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12894653#action_12894653 ]
Thiruvalluvan M. G. commented on AVRO-601:
------------------------------------------
+1
Looks good to me except for a very minor non-issue. In Props.add() , if the new key is already present and the new value for the key is the same as the existing one, we add the key-value pair, which is not not really needed.
> Enhance schema parser to allow arbitrary properties to be defined on a record field
> -----------------------------------------------------------------------------------
>
> Key: AVRO-601
> URL: https://issues.apache.org/jira/browse/AVRO-601
> Project: Avro
> Issue Type: Improvement
> Components: java, spec
> Environment: All
> Reporter: David Rosenstrauch
> Assignee: Doug Cutting
> Priority: Minor
> Fix For: 1.4.0
>
> Attachments: AVRO-601.patch, AVRO-601.patch, AVRO-601.patch
>
>
> It's currently not possible to add an arbitrary property on a record field. e.g., in the following example, although the schema parses fine, the "alias" property gets thrown away:
> {
> "name": "KVPair",
> "type": "record",
> "fields" : [
> {"name": "key", "type": "int", "alias": "EventTime"},
> {"name": "values", "type": "bytes"}
> ]
> }
> I had read the Avro spec and thought this was actually allowed ("Attributes not defined in this document are permitted as metadata, but must not affect the format of serialized data.") but it appears that such additional properties are only allowed at the schema level, not at the field level too.
> This would be a really useful feature to have, as it would allow the user to define meta-data on a field. Currently the only other way to achieve the same behavior would be to do some kind of hack using the field's "doc" attribute, which would be rather ugly.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-601) Enhance schema parser to allow arbitrary
properties to be defined on a record field
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Cutting updated AVRO-601:
------------------------------
Attachment: AVRO-601.patch
Slightly improved patch that updates Field#equals() to examine props too.
> Enhance schema parser to allow arbitrary properties to be defined on a record field
> -----------------------------------------------------------------------------------
>
> Key: AVRO-601
> URL: https://issues.apache.org/jira/browse/AVRO-601
> Project: Avro
> Issue Type: Improvement
> Components: java, spec
> Environment: All
> Reporter: David Rosenstrauch
> Assignee: Doug Cutting
> Priority: Minor
> Fix For: 1.4.0
>
> Attachments: AVRO-601.patch, AVRO-601.patch
>
>
> It's currently not possible to add an arbitrary property on a record field. e.g., in the following example, although the schema parses fine, the "alias" property gets thrown away:
> {
> "name": "KVPair",
> "type": "record",
> "fields" : [
> {"name": "key", "type": "int", "alias": "EventTime"},
> {"name": "values", "type": "bytes"}
> ]
> }
> I had read the Avro spec and thought this was actually allowed ("Attributes not defined in this document are permitted as metadata, but must not affect the format of serialized data.") but it appears that such additional properties are only allowed at the schema level, not at the field level too.
> This would be a really useful feature to have, as it would allow the user to define meta-data on a field. Currently the only other way to achieve the same behavior would be to do some kind of hack using the field's "doc" attribute, which would be rather ugly.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.