You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@avro.apache.org by "Tom White (JIRA)" <ji...@apache.org> on 2011/09/26 02:53:26 UTC
[jira] [Created] (AVRO-895) JsonDecoder does not tolerate JSON
records with different field order
JsonDecoder does not tolerate JSON records with different field order
---------------------------------------------------------------------
Key: AVRO-895
URL: https://issues.apache.org/jira/browse/AVRO-895
Project: Avro
Issue Type: Bug
Components: java
Reporter: Tom White
Avro records are represented as JSON objects in the JSON encoding (according to the Avro spec http://avro.apache.org/docs/1.5.4/spec.html#schema_record). The JSON spec (http://www.json.org/) states that "A [JSON] object is an unordered set of name/value pairs". However, the Java JsonDecoder fails to read a JSON object as an Avro record if its fields are not in the same order as the schema.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-895) JsonDecoder does not tolerate JSON
records with different field order
Posted by "Tom White (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13118241#comment-13118241 ]
Tom White commented on AVRO-895:
--------------------------------
+1 looks good to me.
> JsonDecoder does not tolerate JSON records with different field order
> ---------------------------------------------------------------------
>
> Key: AVRO-895
> URL: https://issues.apache.org/jira/browse/AVRO-895
> Project: Avro
> Issue Type: Bug
> Components: java
> Affects Versions: 1.5.4
> Reporter: Tom White
> Assignee: Thiruvalluvan M. G.
> Labels: patch
> Fix For: 1.6.0
>
> Attachments: AVRO-895-rev-2.patch, AVRO-895.patch, AVRO-895.patch
>
>
> Avro records are represented as JSON objects in the JSON encoding (according to the Avro spec http://avro.apache.org/docs/1.5.4/spec.html#schema_record). The JSON spec (http://www.json.org/) states that "A [JSON] object is an unordered set of name/value pairs". However, the Java JsonDecoder fails to read a JSON object as an Avro record if its fields are not in the same order as the schema.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-895) JsonDecoder does not tolerate JSON
records with different field order
Posted by "Doug Cutting (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13114920#comment-13114920 ]
Doug Cutting commented on AVRO-895:
-----------------------------------
This is has been true since JSON input was first added in AVRO-50. Thiru mentions it there in a comment (http://s.apache.org/E2N). Thiru, is it possible to fix this in the resolver or parser? The field name is known before it is parsed.
> JsonDecoder does not tolerate JSON records with different field order
> ---------------------------------------------------------------------
>
> Key: AVRO-895
> URL: https://issues.apache.org/jira/browse/AVRO-895
> Project: Avro
> Issue Type: Bug
> Components: java
> Reporter: Tom White
> Attachments: AVRO-895.patch
>
>
> Avro records are represented as JSON objects in the JSON encoding (according to the Avro spec http://avro.apache.org/docs/1.5.4/spec.html#schema_record). The JSON spec (http://www.json.org/) states that "A [JSON] object is an unordered set of name/value pairs". However, the Java JsonDecoder fails to read a JSON object as an Avro record if its fields are not in the same order as the schema.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AVRO-895) JsonDecoder does not tolerate JSON
records with different field order
Posted by "Thiruvalluvan M. G. (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thiruvalluvan M. G. updated AVRO-895:
-------------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
Committed revision 1178066.
Thanks Tom for reporting the issue and reviewing the fix.
> JsonDecoder does not tolerate JSON records with different field order
> ---------------------------------------------------------------------
>
> Key: AVRO-895
> URL: https://issues.apache.org/jira/browse/AVRO-895
> Project: Avro
> Issue Type: Bug
> Components: java
> Affects Versions: 1.5.4
> Reporter: Tom White
> Assignee: Thiruvalluvan M. G.
> Labels: patch
> Fix For: 1.6.0
>
> Attachments: AVRO-895-rev-2.patch, AVRO-895.patch, AVRO-895.patch
>
>
> Avro records are represented as JSON objects in the JSON encoding (according to the Avro spec http://avro.apache.org/docs/1.5.4/spec.html#schema_record). The JSON spec (http://www.json.org/) states that "A [JSON] object is an unordered set of name/value pairs". However, the Java JsonDecoder fails to read a JSON object as an Avro record if its fields are not in the same order as the schema.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-895) JsonDecoder does not tolerate JSON
records with different field order
Posted by "Tom White (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13118163#comment-13118163 ]
Tom White commented on AVRO-895:
--------------------------------
Thanks Thiru - this patch fixes the problem for me.
A couple of minor naming suggestions: Change JsonParser2 to ReorderingJsonParser or some such; also no need for the underscores in the test names in TestEncoders.
> JsonDecoder does not tolerate JSON records with different field order
> ---------------------------------------------------------------------
>
> Key: AVRO-895
> URL: https://issues.apache.org/jira/browse/AVRO-895
> Project: Avro
> Issue Type: Bug
> Components: java
> Affects Versions: 1.5.4
> Reporter: Tom White
> Assignee: Thiruvalluvan M. G.
> Labels: patch
> Fix For: 1.6.0
>
> Attachments: AVRO-895.patch, AVRO-895.patch
>
>
> Avro records are represented as JSON objects in the JSON encoding (according to the Avro spec http://avro.apache.org/docs/1.5.4/spec.html#schema_record). The JSON spec (http://www.json.org/) states that "A [JSON] object is an unordered set of name/value pairs". However, the Java JsonDecoder fails to read a JSON object as an Avro record if its fields are not in the same order as the schema.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AVRO-895) JsonDecoder does not tolerate JSON
records with different field order
Posted by "Thiruvalluvan M. G. (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thiruvalluvan M. G. updated AVRO-895:
-------------------------------------
Attachment: AVRO-895-rev-2.patch
Cleaner fix. No new interface. Took care of Tom's suggestions.
> JsonDecoder does not tolerate JSON records with different field order
> ---------------------------------------------------------------------
>
> Key: AVRO-895
> URL: https://issues.apache.org/jira/browse/AVRO-895
> Project: Avro
> Issue Type: Bug
> Components: java
> Affects Versions: 1.5.4
> Reporter: Tom White
> Assignee: Thiruvalluvan M. G.
> Labels: patch
> Fix For: 1.6.0
>
> Attachments: AVRO-895-rev-2.patch, AVRO-895.patch, AVRO-895.patch
>
>
> Avro records are represented as JSON objects in the JSON encoding (according to the Avro spec http://avro.apache.org/docs/1.5.4/spec.html#schema_record). The JSON spec (http://www.json.org/) states that "A [JSON] object is an unordered set of name/value pairs". However, the Java JsonDecoder fails to read a JSON object as an Avro record if its fields are not in the same order as the schema.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AVRO-895) JsonDecoder does not tolerate JSON
records with different field order
Posted by "Thiruvalluvan M. G. (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thiruvalluvan M. G. updated AVRO-895:
-------------------------------------
Attachment: AVRO-895.patch
This patch addresses the problem. A few new tests added that tests the fixes.
The fix essentially saves the JSON fields that arrive out of order and uses the saved fields later. If the fields arrive in right order, it does not store them. So performance impact (both in terms of memory and time) due to this patch should be insignificant if the fields are in order.
> JsonDecoder does not tolerate JSON records with different field order
> ---------------------------------------------------------------------
>
> Key: AVRO-895
> URL: https://issues.apache.org/jira/browse/AVRO-895
> Project: Avro
> Issue Type: Bug
> Components: java
> Reporter: Tom White
> Assignee: Thiruvalluvan M. G.
> Attachments: AVRO-895.patch, AVRO-895.patch
>
>
> Avro records are represented as JSON objects in the JSON encoding (according to the Avro spec http://avro.apache.org/docs/1.5.4/spec.html#schema_record). The JSON spec (http://www.json.org/) states that "A [JSON] object is an unordered set of name/value pairs". However, the Java JsonDecoder fails to read a JSON object as an Avro record if its fields are not in the same order as the schema.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AVRO-895) JsonDecoder does not tolerate JSON
records with different field order
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom White updated AVRO-895:
---------------------------
Attachment: AVRO-895.patch
Here's a failing test which demonstrates the problem.
> JsonDecoder does not tolerate JSON records with different field order
> ---------------------------------------------------------------------
>
> Key: AVRO-895
> URL: https://issues.apache.org/jira/browse/AVRO-895
> Project: Avro
> Issue Type: Bug
> Components: java
> Reporter: Tom White
> Attachments: AVRO-895.patch
>
>
> Avro records are represented as JSON objects in the JSON encoding (according to the Avro spec http://avro.apache.org/docs/1.5.4/spec.html#schema_record). The JSON spec (http://www.json.org/) states that "A [JSON] object is an unordered set of name/value pairs". However, the Java JsonDecoder fails to read a JSON object as an Avro record if its fields are not in the same order as the schema.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AVRO-895) JsonDecoder does not tolerate JSON
records with different field order
Posted by "Thiruvalluvan M. G. (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thiruvalluvan M. G. updated AVRO-895:
-------------------------------------
Fix Version/s: 1.6.0
Labels: patch (was: )
Affects Version/s: 1.5.4
Status: Patch Available (was: Open)
> JsonDecoder does not tolerate JSON records with different field order
> ---------------------------------------------------------------------
>
> Key: AVRO-895
> URL: https://issues.apache.org/jira/browse/AVRO-895
> Project: Avro
> Issue Type: Bug
> Components: java
> Affects Versions: 1.5.4
> Reporter: Tom White
> Assignee: Thiruvalluvan M. G.
> Labels: patch
> Fix For: 1.6.0
>
> Attachments: AVRO-895.patch, AVRO-895.patch
>
>
> Avro records are represented as JSON objects in the JSON encoding (according to the Avro spec http://avro.apache.org/docs/1.5.4/spec.html#schema_record). The JSON spec (http://www.json.org/) states that "A [JSON] object is an unordered set of name/value pairs". However, the Java JsonDecoder fails to read a JSON object as an Avro record if its fields are not in the same order as the schema.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-895) JsonDecoder does not tolerate JSON
records with different field order
Posted by "Thiruvalluvan M. G. (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13118182#comment-13118182 ]
Thiruvalluvan M. G. commented on AVRO-895:
------------------------------------------
JsonParser2 is actually a narrower version of Jackson's JsonParser. Actually there is no need for that interface. I use it just to avoid implementing all the methods of JsonParser for the saved tokens. Now looking back, it doesn't seem to be worth it. It forces us to have an adaptor to adapt JsonParser into JsonParser2, which appears wasteful.I'll get rid of that JsonParser2 altogether.
I have no issues with removing the underscores in test names.
> JsonDecoder does not tolerate JSON records with different field order
> ---------------------------------------------------------------------
>
> Key: AVRO-895
> URL: https://issues.apache.org/jira/browse/AVRO-895
> Project: Avro
> Issue Type: Bug
> Components: java
> Affects Versions: 1.5.4
> Reporter: Tom White
> Assignee: Thiruvalluvan M. G.
> Labels: patch
> Fix For: 1.6.0
>
> Attachments: AVRO-895.patch, AVRO-895.patch
>
>
> Avro records are represented as JSON objects in the JSON encoding (according to the Avro spec http://avro.apache.org/docs/1.5.4/spec.html#schema_record). The JSON spec (http://www.json.org/) states that "A [JSON] object is an unordered set of name/value pairs". However, the Java JsonDecoder fails to read a JSON object as an Avro record if its fields are not in the same order as the schema.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (AVRO-895) JsonDecoder does not tolerate JSON
records with different field order
Posted by "Thiruvalluvan M. G. (Assigned) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thiruvalluvan M. G. reassigned AVRO-895:
----------------------------------------
Assignee: Thiruvalluvan M. G.
> JsonDecoder does not tolerate JSON records with different field order
> ---------------------------------------------------------------------
>
> Key: AVRO-895
> URL: https://issues.apache.org/jira/browse/AVRO-895
> Project: Avro
> Issue Type: Bug
> Components: java
> Reporter: Tom White
> Assignee: Thiruvalluvan M. G.
> Attachments: AVRO-895.patch
>
>
> Avro records are represented as JSON objects in the JSON encoding (according to the Avro spec http://avro.apache.org/docs/1.5.4/spec.html#schema_record). The JSON spec (http://www.json.org/) states that "A [JSON] object is an unordered set of name/value pairs". However, the Java JsonDecoder fails to read a JSON object as an Avro record if its fields are not in the same order as the schema.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira