You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@avro.apache.org by "Cristian Opris (JIRA)" <ji...@apache.org> on 2012/09/12 19:18:08 UTC
[jira] [Created] (AVRO-1157) Idl tool - parse properties as plain
String OR expose the property as Json
Cristian Opris created AVRO-1157:
------------------------------------
Summary: Idl tool - parse properties as plain String OR expose the property as Json
Key: AVRO-1157
URL: https://issues.apache.org/jira/browse/AVRO-1157
Project: Avro
Issue Type: Improvement
Components: java, spec
Affects Versions: 1.7.1
Reporter: Cristian Opris
Priority: Minor
The IDL Tool currently parses "annotations" as Json objects but only really supports textual values.
If one wants to have actual json annotations, they need to write json as an escaped json string.
Proposed solutions:
- expose the property as Json if it's parsed as Json
- OR parse the property as a string
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AVRO-1157) Idl tool - parse properties as plain
String OR expose the property as Json
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-1157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Cutting updated AVRO-1157:
-------------------------------
Attachment: AVRO-1157.patch
Here's a patch that adds support for arbitrary Json as property values. This is supported in both Idl and through the Schema and Protocol APIs.
> Idl tool - parse properties as plain String OR expose the property as Json
> --------------------------------------------------------------------------
>
> Key: AVRO-1157
> URL: https://issues.apache.org/jira/browse/AVRO-1157
> Project: Avro
> Issue Type: Improvement
> Components: java, spec
> Affects Versions: 1.7.1
> Reporter: Cristian Opris
> Priority: Minor
> Fix For: 1.7.3
>
> Attachments: AVRO-1157.patch, AVRO-1157.patch
>
>
> The IDL Tool currently parses "annotations" as Json objects but only really supports textual values.
> If one wants to have actual json annotations, they need to write json as an escaped json string.
> Proposed solutions:
> - expose the property as Json if it's parsed as Json
> - OR parse the property as a string
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1157) Idl tool - parse properties as plain
String OR expose the property as Json
Posted by "Cristian Opris (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-1157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13455721#comment-13455721 ]
Cristian Opris commented on AVRO-1157:
--------------------------------------
Personally I would go the whole way and add support for Json props to schema, protocol, etc. I suggested the toString() solution as a quick solution only.
What I was trying to do is to have stuctured annotations and parse those against a schema themselves (similar to how Java defines annotations as a kind of interface)
I could have a separate annotation for each piece of meta information as you suggest, but this is kind of nasty if I have lots of stuff to put in there.
The same argument would go agains @aliases, why did you decide to go with an array of values instead of multiple @alias annotations ??
My point is the current behaviour is inconsistent and rather odd...
> Idl tool - parse properties as plain String OR expose the property as Json
> --------------------------------------------------------------------------
>
> Key: AVRO-1157
> URL: https://issues.apache.org/jira/browse/AVRO-1157
> Project: Avro
> Issue Type: Improvement
> Components: java, spec
> Affects Versions: 1.7.1
> Reporter: Cristian Opris
> Priority: Minor
> Attachments: AVRO-1157.patch
>
>
> The IDL Tool currently parses "annotations" as Json objects but only really supports textual values.
> If one wants to have actual json annotations, they need to write json as an escaped json string.
> Proposed solutions:
> - expose the property as Json if it's parsed as Json
> - OR parse the property as a string
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1157) Idl tool - parse properties as plain
String OR expose the property as Json
Posted by "Cristian Opris (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-1157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13454298#comment-13454298 ]
Cristian Opris commented on AVRO-1157:
--------------------------------------
I wasn't aware of aliases, in fact I was actually wondering what the rationale behind parsing props as Json was if it returns a string in the end.
I'm looking into having structured annotations, same as in Java.
I could then use Avro as a cross-language schema def format with metadata, instead of Java interfaces.
Say I want to map an "entity" type record to cassandra, currently I need to do this:
@Entity("{\"columnFamily\":\"test\"}")
record TestEntity {
@Id("{\"column\":\"id\"}")
int id;
@Column("{\"column\":\"name\"}")
string name;
}
Note the ugly escaping
The proposed changes are what I had in mind, I can see this being generic enough to be useful for other people as well.
> Idl tool - parse properties as plain String OR expose the property as Json
> --------------------------------------------------------------------------
>
> Key: AVRO-1157
> URL: https://issues.apache.org/jira/browse/AVRO-1157
> Project: Avro
> Issue Type: Improvement
> Components: java, spec
> Affects Versions: 1.7.1
> Reporter: Cristian Opris
> Priority: Minor
>
> The IDL Tool currently parses "annotations" as Json objects but only really supports textual values.
> If one wants to have actual json annotations, they need to write json as an escaped json string.
> Proposed solutions:
> - expose the property as Json if it's parsed as Json
> - OR parse the property as a string
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1157) Idl tool - parse properties as plain
String OR expose the property as Json
Posted by "Cristian Opris (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-1157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13454907#comment-13454907 ]
Cristian Opris commented on AVRO-1157:
--------------------------------------
Yeah, I think that's a reasonable solution, is there a chance this could make it in a patch ?
Thanks vm
> Idl tool - parse properties as plain String OR expose the property as Json
> --------------------------------------------------------------------------
>
> Key: AVRO-1157
> URL: https://issues.apache.org/jira/browse/AVRO-1157
> Project: Avro
> Issue Type: Improvement
> Components: java, spec
> Affects Versions: 1.7.1
> Reporter: Cristian Opris
> Priority: Minor
>
> The IDL Tool currently parses "annotations" as Json objects but only really supports textual values.
> If one wants to have actual json annotations, they need to write json as an escaped json string.
> Proposed solutions:
> - expose the property as Json if it's parsed as Json
> - OR parse the property as a string
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1157) Idl tool - parse properties as plain
String OR expose the property as Json
Posted by "Cristian Opris (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-1157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13456962#comment-13456962 ]
Cristian Opris commented on AVRO-1157:
--------------------------------------
Thank you Doug, that would be brilliant. I could try and put something together and submit it, have to see if I'm actually allowed to first
> Idl tool - parse properties as plain String OR expose the property as Json
> --------------------------------------------------------------------------
>
> Key: AVRO-1157
> URL: https://issues.apache.org/jira/browse/AVRO-1157
> Project: Avro
> Issue Type: Improvement
> Components: java, spec
> Affects Versions: 1.7.1
> Reporter: Cristian Opris
> Priority: Minor
> Attachments: AVRO-1157.patch
>
>
> The IDL Tool currently parses "annotations" as Json objects but only really supports textual values.
> If one wants to have actual json annotations, they need to write json as an escaped json string.
> Proposed solutions:
> - expose the property as Json if it's parsed as Json
> - OR parse the property as a string
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1157) Idl tool - parse properties as plain
String OR expose the property as Json
Posted by "Cristian Opris (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-1157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13454302#comment-13454302 ]
Cristian Opris commented on AVRO-1157:
--------------------------------------
A simple hack to allow this and keep API compatibility would be to do something like:
value.toJsonString()
instead of
if (value.isTextual())
return value.getTextValue();
throw error(key+" property must be textual: "+value, token);
in Idl.getTextProp()
> Idl tool - parse properties as plain String OR expose the property as Json
> --------------------------------------------------------------------------
>
> Key: AVRO-1157
> URL: https://issues.apache.org/jira/browse/AVRO-1157
> Project: Avro
> Issue Type: Improvement
> Components: java, spec
> Affects Versions: 1.7.1
> Reporter: Cristian Opris
> Priority: Minor
>
> The IDL Tool currently parses "annotations" as Json objects but only really supports textual values.
> If one wants to have actual json annotations, they need to write json as an escaped json string.
> Proposed solutions:
> - expose the property as Json if it's parsed as Json
> - OR parse the property as a string
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1157) Idl tool - parse properties as plain
String OR expose the property as Json
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-1157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13454182#comment-13454182 ]
Doug Cutting commented on AVRO-1157:
------------------------------------
Current IDL annotations are both Json strings (for Schema properties) and Json arrays of strings (for aliases). So we cannot simply parse all annotations as strings, as that would break aliases. Also, we'd still need some sort of string escaping.
The feature you seem to want is an improved ability to specify a schema or protocol property whose value is Json. This would mostly require Schema and Protocol API changes. We might:
- change Schema's props field from Map<String,String> to Map<String,JsonNode>.
- change Schema#addProp(String,String) to create a Json TextNode.
- change Schema#getProp(String) to, if the value isTextual(), return the String otherwise return null.
- deprecate Schema#getProps() and change its implementation to return the subset with string values.
- add a new method, 'void Schema#getJsonProp(String, JsonNode)'
- add a new method, 'JsonNode Schema#getJsonProp(String)'
- add a new method, 'Map<String,JsonNode> Schema#getJsonProps()'
- make similar updates to the Protocol API.
- update the IDL parser to use these new APIs.
Can you give an example of a Json-valued Schema or Protocol property that you're using, in order to help motivate such a change?
> Idl tool - parse properties as plain String OR expose the property as Json
> --------------------------------------------------------------------------
>
> Key: AVRO-1157
> URL: https://issues.apache.org/jira/browse/AVRO-1157
> Project: Avro
> Issue Type: Improvement
> Components: java, spec
> Affects Versions: 1.7.1
> Reporter: Cristian Opris
> Priority: Minor
>
> The IDL Tool currently parses "annotations" as Json objects but only really supports textual values.
> If one wants to have actual json annotations, they need to write json as an escaped json string.
> Proposed solutions:
> - expose the property as Json if it's parsed as Json
> - OR parse the property as a string
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1157) Idl tool - parse properties as plain
String OR expose the property as Json
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-1157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13455980#comment-13455980 ]
Doug Cutting commented on AVRO-1157:
------------------------------------
> My point is the current behaviour is inconsistent and rather odd...
I think of it as consistent but limited. The syntax for annotations is consistently JSon but the implementation is limited in that it currently ignores annotations that are not strings, or, in the case of aliases, arrays of strings. It's designed so that we can add support for full Json annotations without breaking anything. If we made the quick fix discussed above then we'd have a harder time subsequently adding real support for Json annotations without breaking applications.
I agree that adding full support for Json annotations to Protocol, Schema and Field would be an improvement. I'll try to create a patch that implements this when I have a chance and if someone else doesn't do so first.
> Idl tool - parse properties as plain String OR expose the property as Json
> --------------------------------------------------------------------------
>
> Key: AVRO-1157
> URL: https://issues.apache.org/jira/browse/AVRO-1157
> Project: Avro
> Issue Type: Improvement
> Components: java, spec
> Affects Versions: 1.7.1
> Reporter: Cristian Opris
> Priority: Minor
> Attachments: AVRO-1157.patch
>
>
> The IDL Tool currently parses "annotations" as Json objects but only really supports textual values.
> If one wants to have actual json annotations, they need to write json as an escaped json string.
> Proposed solutions:
> - expose the property as Json if it's parsed as Json
> - OR parse the property as a string
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AVRO-1157) Idl tool - parse properties as plain
String OR expose the property as Json
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-1157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Cutting updated AVRO-1157:
-------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
I committed this.
> Idl tool - parse properties as plain String OR expose the property as Json
> --------------------------------------------------------------------------
>
> Key: AVRO-1157
> URL: https://issues.apache.org/jira/browse/AVRO-1157
> Project: Avro
> Issue Type: Improvement
> Components: java, spec
> Affects Versions: 1.7.1
> Reporter: Cristian Opris
> Assignee: Doug Cutting
> Priority: Minor
> Fix For: 1.7.3
>
> Attachments: AVRO-1157.patch, AVRO-1157.patch
>
>
> The IDL Tool currently parses "annotations" as Json objects but only really supports textual values.
> If one wants to have actual json annotations, they need to write json as an escaped json string.
> Proposed solutions:
> - expose the property as Json if it's parsed as Json
> - OR parse the property as a string
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1157) Idl tool - parse properties as plain
String OR expose the property as Json
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-1157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13454402#comment-13454402 ]
Doug Cutting commented on AVRO-1157:
------------------------------------
The problem with value.toString() alone is that then textual values would have quotes around them. But we might try something like:
{code}
return value.isTextual() ? value.getTextValue() : value.toString();
{code}
With that you should be able to add Json-format annotations without so many escapes, e.g.:
@Entity({"column":"id"})
The value stored in the schema would always be a string, and sometimes it would be a string that's Json and sometimes not, just like today.
> Idl tool - parse properties as plain String OR expose the property as Json
> --------------------------------------------------------------------------
>
> Key: AVRO-1157
> URL: https://issues.apache.org/jira/browse/AVRO-1157
> Project: Avro
> Issue Type: Improvement
> Components: java, spec
> Affects Versions: 1.7.1
> Reporter: Cristian Opris
> Priority: Minor
>
> The IDL Tool currently parses "annotations" as Json objects but only really supports textual values.
> If one wants to have actual json annotations, they need to write json as an escaped json string.
> Proposed solutions:
> - expose the property as Json if it's parsed as Json
> - OR parse the property as a string
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AVRO-1157) Idl tool - parse properties as plain
String OR expose the property as Json
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-1157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Cutting updated AVRO-1157:
-------------------------------
Fix Version/s: 1.7.3
Assignee: Doug Cutting
Status: Patch Available (was: Open)
These changes are all back-compatible, so we could include it in 1.7.3.
I'll commit this soon unless there are objections.
> Idl tool - parse properties as plain String OR expose the property as Json
> --------------------------------------------------------------------------
>
> Key: AVRO-1157
> URL: https://issues.apache.org/jira/browse/AVRO-1157
> Project: Avro
> Issue Type: Improvement
> Components: java, spec
> Affects Versions: 1.7.1
> Reporter: Cristian Opris
> Assignee: Doug Cutting
> Priority: Minor
> Fix For: 1.7.3
>
> Attachments: AVRO-1157.patch, AVRO-1157.patch
>
>
> The IDL Tool currently parses "annotations" as Json objects but only really supports textual values.
> If one wants to have actual json annotations, they need to write json as an escaped json string.
> Proposed solutions:
> - expose the property as Json if it's parsed as Json
> - OR parse the property as a string
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AVRO-1157) Idl tool - parse properties as plain
String OR expose the property as Json
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-1157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Cutting updated AVRO-1157:
-------------------------------
Attachment: AVRO-1157.patch
Here's a patch that implements this. However note how strange the addition to simple.avpr looks. If we really want to support complex Json annotations then we should really extend the APIs of Protocol, Schema, and Field.
What's the motivating use case? In the examples you provide above, why can't you simply use something like, @EntityColumnFamily("test"), @ColumnId("id") and @ColumnName("name")?
> Idl tool - parse properties as plain String OR expose the property as Json
> --------------------------------------------------------------------------
>
> Key: AVRO-1157
> URL: https://issues.apache.org/jira/browse/AVRO-1157
> Project: Avro
> Issue Type: Improvement
> Components: java, spec
> Affects Versions: 1.7.1
> Reporter: Cristian Opris
> Priority: Minor
> Attachments: AVRO-1157.patch
>
>
> The IDL Tool currently parses "annotations" as Json objects but only really supports textual values.
> If one wants to have actual json annotations, they need to write json as an escaped json string.
> Proposed solutions:
> - expose the property as Json if it's parsed as Json
> - OR parse the property as a string
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira