You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@avro.apache.org by "George Fletcher (Created) (JIRA)" <ji...@apache.org> on 2011/11/29 21:33:40 UTC
[jira] [Created] (AVRO-965) Enhance the IDL parser to allow
properties for protocols and messages
Enhance the IDL parser to allow properties for protocols and messages
---------------------------------------------------------------------
Key: AVRO-965
URL: https://issues.apache.org/jira/browse/AVRO-965
Project: Avro
Issue Type: Improvement
Reporter: George Fletcher
Enhance the IDL parser to support arbitrary properties for protocol and message types. This will allow for attaching metadata to a protocol or message and can be used for versioning and in some cases language annotations.
This was partly discussed as part of JIRA ticket 886
https://issues.apache.org/jira/browse/AVRO-886
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-965) Enhance the IDL parser to allow
properties for protocols and messages
Posted by "Doug Cutting (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13163820#comment-13163820 ]
Doug Cutting commented on AVRO-965:
-----------------------------------
The properties might be instead parsed in ProtocolBody. E.g. that might look something like:
{code}
"{"
<IMPORT> ... { .... }
|
( SchemaProperty(props) )*
( NamedSchemaDeclaration(props) { ... } | MessageDeclaration(props) { ... }
"}"
{code}
Does that make sense?
> Enhance the IDL parser to allow properties for protocols and messages
> ---------------------------------------------------------------------
>
> Key: AVRO-965
> URL: https://issues.apache.org/jira/browse/AVRO-965
> Project: Avro
> Issue Type: Improvement
> Reporter: George Fletcher
> Priority: Minor
>
> Enhance the IDL parser to support arbitrary properties for protocol and message types. This will allow for attaching metadata to a protocol or message and can be used for versioning and in some cases language annotations.
> This was partly discussed as part of JIRA ticket 886
> https://issues.apache.org/jira/browse/AVRO-886
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-965) Enhance the IDL parser to allow
properties for protocols and messages
Posted by "George Fletcher (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13163813#comment-13163813 ]
George Fletcher commented on AVRO-965:
--------------------------------------
You are correct. I meant Props not Property:) I'll refactor to just reference Schema.Props in Protocol.java.
I tried modifying MessageDeclaration like this
{code}
Message MessageDeclaration(Protocol p):
{
String msgDoc;
String name;
Schema request;
Schema response;
boolean oneWay = false;
List<Schema> errorSchemata = new ArrayList<Schema>();
errorSchemata.add(Protocol.SYSTEM_ERROR);
Map<String, JsonNode> props = new LinkedHashMap<String, JsonNode>();
Map<String, String> propMap = new LinkedHashMap<String, String>();
}
{
msgDoc = MessageDocumentation()
( SchemaProperty(props) )*
response = ResultType()
name = Identifier()
request = FormalParameters()
[ "oneway" {oneWay = true; } | "throws" ErrorList(errorSchemata) ]
";"
{
for (String key : props.keySet())
if (props.get(key).isTextual()) { // ignore other non-textual
propMap.put(key, getTextProp(key, props, token));
}
...
{code}
However, the issue isn't in parsing the MessageDeclaration but rather when the parser is processing the NamedSchemaDeclaration. In this case, the NamedSchemaDeclaration matches because of the @propName() construct but then expects the resulting object to be a named schema and not a message declaration. It seems like I'd need to modify NamedSchemaDeclaration but I didn't want to start down that path as that seems like a bigger change than necessary.
> Enhance the IDL parser to allow properties for protocols and messages
> ---------------------------------------------------------------------
>
> Key: AVRO-965
> URL: https://issues.apache.org/jira/browse/AVRO-965
> Project: Avro
> Issue Type: Improvement
> Reporter: George Fletcher
> Priority: Minor
>
> Enhance the IDL parser to support arbitrary properties for protocol and message types. This will allow for attaching metadata to a protocol or message and can be used for versioning and in some cases language annotations.
> This was partly discussed as part of JIRA ticket 886
> https://issues.apache.org/jira/browse/AVRO-886
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AVRO-965) Enhance the IDL parser to allow
properties for protocols and messages
Posted by "George Fletcher (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
George Fletcher updated AVRO-965:
---------------------------------
Fix Version/s: 1.6.2
Affects Version/s: 1.6.1
Status: Patch Available (was: Open)
Initial patch submission.
> Enhance the IDL parser to allow properties for protocols and messages
> ---------------------------------------------------------------------
>
> Key: AVRO-965
> URL: https://issues.apache.org/jira/browse/AVRO-965
> Project: Avro
> Issue Type: Improvement
> Affects Versions: 1.6.1
> Reporter: George Fletcher
> Priority: Minor
> Fix For: 1.6.2
>
> Attachments: avro-jira-965.patch
>
>
> Enhance the IDL parser to support arbitrary properties for protocol and message types. This will allow for attaching metadata to a protocol or message and can be used for versioning and in some cases language annotations.
> This was partly discussed as part of JIRA ticket 886
> https://issues.apache.org/jira/browse/AVRO-886
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AVRO-965) Enhance the IDL parser to allow
properties for protocols and messages
Posted by "George Fletcher (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
George Fletcher updated AVRO-965:
---------------------------------
Priority: Minor (was: Major)
> Enhance the IDL parser to allow properties for protocols and messages
> ---------------------------------------------------------------------
>
> Key: AVRO-965
> URL: https://issues.apache.org/jira/browse/AVRO-965
> Project: Avro
> Issue Type: Improvement
> Reporter: George Fletcher
> Priority: Minor
>
> Enhance the IDL parser to support arbitrary properties for protocol and message types. This will allow for attaching metadata to a protocol or message and can be used for versioning and in some cases language annotations.
> This was partly discussed as part of JIRA ticket 886
> https://issues.apache.org/jira/browse/AVRO-886
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-965) Enhance the IDL parser to allow
properties for protocols and messages
Posted by "George Fletcher (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13162844#comment-13162844 ]
George Fletcher commented on AVRO-965:
--------------------------------------
So I've been working on a patch for this enhancement and have it working for Protocols and mostly working for Messages. However, I've run into an issue with the IDL parser. Basically, the IDL parser assumes that any properties (@...) only apply to named schemas. Any help here greatly appreciated.
Here is the design I've been working from for this patch...
# Add the private Property class from Schema.java to Protocol.java
# Add getProp, addProp, and props methods to both Protocol and Message classes
# Add the hashcode of the Props set when comparing for equality
# Parse the json format
#* Protocol: find all "textual", "value" child nodes that are not reserved and add them as properties
#* Message: find all "textual, "value" child nodes that are not reserved and add them as properties. Required changing the constructor to accept a Map<String,String> of properties.
# Parse the IDL format
#* Protocol: Add all properties other than namespace to the Protocol property set
#* Message: NOT WORKING
# Update tests to work with the above changes
> Enhance the IDL parser to allow properties for protocols and messages
> ---------------------------------------------------------------------
>
> Key: AVRO-965
> URL: https://issues.apache.org/jira/browse/AVRO-965
> Project: Avro
> Issue Type: Improvement
> Reporter: George Fletcher
> Priority: Minor
>
> Enhance the IDL parser to support arbitrary properties for protocol and message types. This will allow for attaching metadata to a protocol or message and can be used for versioning and in some cases language annotations.
> This was partly discussed as part of JIRA ticket 886
> https://issues.apache.org/jira/browse/AVRO-886
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-965) Enhance the IDL parser to allow
properties for protocols and messages
Posted by "George Fletcher (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13167319#comment-13167319 ]
George Fletcher commented on AVRO-965:
--------------------------------------
Thanks so much Doug!
> Enhance the IDL parser to allow properties for protocols and messages
> ---------------------------------------------------------------------
>
> Key: AVRO-965
> URL: https://issues.apache.org/jira/browse/AVRO-965
> Project: Avro
> Issue Type: Improvement
> Affects Versions: 1.6.1
> Reporter: George Fletcher
> Assignee: George Fletcher
> Priority: Minor
> Fix For: 1.6.2
>
> Attachments: AVRO-965.patch, avro-jira-965.patch
>
>
> Enhance the IDL parser to support arbitrary properties for protocol and message types. This will allow for attaching metadata to a protocol or message and can be used for versioning and in some cases language annotations.
> This was partly discussed as part of JIRA ticket 886
> https://issues.apache.org/jira/browse/AVRO-886
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AVRO-965) Enhance the IDL parser to allow
properties for protocols and messages
Posted by "Doug Cutting (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Cutting updated AVRO-965:
------------------------------
Attachment: AVRO-965.patch
Looks great! Here's an updated patch with some minor changes, including:
- wrapped lines that exceeded 80 columns
- deprecate rather than remove old Protocol#addMessage signature so that this change is back-compatible.
- factor props.clear() in idl.jj
- update idl documentation
If there are no objections I'll commit this version soon.
> Enhance the IDL parser to allow properties for protocols and messages
> ---------------------------------------------------------------------
>
> Key: AVRO-965
> URL: https://issues.apache.org/jira/browse/AVRO-965
> Project: Avro
> Issue Type: Improvement
> Affects Versions: 1.6.1
> Reporter: George Fletcher
> Priority: Minor
> Fix For: 1.6.2
>
> Attachments: AVRO-965.patch, avro-jira-965.patch
>
>
> Enhance the IDL parser to support arbitrary properties for protocol and message types. This will allow for attaching metadata to a protocol or message and can be used for versioning and in some cases language annotations.
> This was partly discussed as part of JIRA ticket 886
> https://issues.apache.org/jira/browse/AVRO-886
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-965) Enhance the IDL parser to allow
properties for protocols and messages
Posted by "George Fletcher (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13163933#comment-13163933 ]
George Fletcher commented on AVRO-965:
--------------------------------------
Yes, let me try that and see how it goes. Thanks!
> Enhance the IDL parser to allow properties for protocols and messages
> ---------------------------------------------------------------------
>
> Key: AVRO-965
> URL: https://issues.apache.org/jira/browse/AVRO-965
> Project: Avro
> Issue Type: Improvement
> Reporter: George Fletcher
> Priority: Minor
>
> Enhance the IDL parser to support arbitrary properties for protocol and message types. This will allow for attaching metadata to a protocol or message and can be used for versioning and in some cases language annotations.
> This was partly discussed as part of JIRA ticket 886
> https://issues.apache.org/jira/browse/AVRO-886
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AVRO-965) Enhance the IDL parser to allow
properties for protocols and messages
Posted by "George Fletcher (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
George Fletcher updated AVRO-965:
---------------------------------
Attachment: avro-jira-965.patch
Initial patch submission. IDL parser changed to support properties on Protocols and Messages.
> Enhance the IDL parser to allow properties for protocols and messages
> ---------------------------------------------------------------------
>
> Key: AVRO-965
> URL: https://issues.apache.org/jira/browse/AVRO-965
> Project: Avro
> Issue Type: Improvement
> Reporter: George Fletcher
> Priority: Minor
> Attachments: avro-jira-965.patch
>
>
> Enhance the IDL parser to support arbitrary properties for protocol and message types. This will allow for attaching metadata to a protocol or message and can be used for versioning and in some cases language annotations.
> This was partly discussed as part of JIRA ticket 886
> https://issues.apache.org/jira/browse/AVRO-886
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-965) Enhance the IDL parser to allow
properties for protocols and messages
Posted by "George Fletcher (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13164510#comment-13164510 ]
George Fletcher commented on AVRO-965:
--------------------------------------
The attached file has tests for arbitrary properties on Protocols and Messages. All tests are passing.
Thanks Doug, that suggestion worked!
> Enhance the IDL parser to allow properties for protocols and messages
> ---------------------------------------------------------------------
>
> Key: AVRO-965
> URL: https://issues.apache.org/jira/browse/AVRO-965
> Project: Avro
> Issue Type: Improvement
> Affects Versions: 1.6.1
> Reporter: George Fletcher
> Priority: Minor
> Fix For: 1.6.2
>
> Attachments: avro-jira-965.patch
>
>
> Enhance the IDL parser to support arbitrary properties for protocol and message types. This will allow for attaching metadata to a protocol or message and can be used for versioning and in some cases language annotations.
> This was partly discussed as part of JIRA ticket 886
> https://issues.apache.org/jira/browse/AVRO-886
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AVRO-965) Enhance the IDL parser to allow
properties for protocols and messages
Posted by "Doug Cutting (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Cutting updated AVRO-965:
------------------------------
Resolution: Fixed
Assignee: George Fletcher
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
I committed this. Thanks, George!
> Enhance the IDL parser to allow properties for protocols and messages
> ---------------------------------------------------------------------
>
> Key: AVRO-965
> URL: https://issues.apache.org/jira/browse/AVRO-965
> Project: Avro
> Issue Type: Improvement
> Affects Versions: 1.6.1
> Reporter: George Fletcher
> Assignee: George Fletcher
> Priority: Minor
> Fix For: 1.6.2
>
> Attachments: AVRO-965.patch, avro-jira-965.patch
>
>
> Enhance the IDL parser to support arbitrary properties for protocol and message types. This will allow for attaching metadata to a protocol or message and can be used for versioning and in some cases language annotations.
> This was partly discussed as part of JIRA ticket 886
> https://issues.apache.org/jira/browse/AVRO-886
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-965) Enhance the IDL parser to allow
properties for protocols and messages
Posted by "Doug Cutting (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13163020#comment-13163020 ]
Doug Cutting commented on AVRO-965:
-----------------------------------
> Add the private Property class from Schema.java to Protocol.java
I think you mean Props, not Property, is that right? If so, you might just remove the 'private' keyword so it becomes package-private. I don't think we need to make this public in the API.
> Message: NOT WORKING
In idl.jj, can't you simply modify MessageDeclaration to collect properties the same way that FieldDeclaration does?
Other than that it sounds like you have a good plan.
> Enhance the IDL parser to allow properties for protocols and messages
> ---------------------------------------------------------------------
>
> Key: AVRO-965
> URL: https://issues.apache.org/jira/browse/AVRO-965
> Project: Avro
> Issue Type: Improvement
> Reporter: George Fletcher
> Priority: Minor
>
> Enhance the IDL parser to support arbitrary properties for protocol and message types. This will allow for attaching metadata to a protocol or message and can be used for versioning and in some cases language annotations.
> This was partly discussed as part of JIRA ticket 886
> https://issues.apache.org/jira/browse/AVRO-886
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira