You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@avro.apache.org by "Doug Cutting (JIRA)" <ji...@apache.org> on 2009/09/22 20:04:16 UTC
[jira] Created: (AVRO-118) JSON encoder does not easily permit
multiple reads or writes
JSON encoder does not easily permit multiple reads or writes
------------------------------------------------------------
Key: AVRO-118
URL: https://issues.apache.org/jira/browse/AVRO-118
Project: Avro
Issue Type: Bug
Components: java
Affects Versions: 1.1.0
Reporter: Doug Cutting
The JSON encoder does not permit one to easily write or read sequences of instances from a single stream.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-118) JSON encoder does not easily permit
multiple reads or writes
Posted by "Thiruvalluvan M. G. (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thiruvalluvan M. G. updated AVRO-118:
-------------------------------------
Attachment: AVRO-118.patch
The basic problem with JSON encoding is that the encoder may require to write a few symbols at the end of the object even though there won't be any call from the client (as per the current Encoder interface). For example, if the schema is a record, the last call the encoder gets is from the client is for the last field of the record. We could take two approaches:
- After every write, we check if there are any implicit action required to be completed.
- Let the client indicate that the object has ended (by calling say reset()) so that the encoder can do the clean up
The former may have performance impact and the latter means an additional call for the clients. Also, it doesn't look natural to call reset after the object() even if it's there is a single object. I found another way to achieve this:
- When a new object is written any pending implicit actions can be performed. That leaves only the trailing implicit action for very last object. I added support for that in flush(). Thus, one can write as many object as desired, but needs to call flush() after the very last object. (This requirement existed even for BlockingBinaryEncoder and so isn't new). This approach takes care of decoder part automatically. (There is equivalent of flush() in decoder which may leave some trailing symbols in the stream, which is usually not a problem. If some other data is added after Vvro stream, this could be a problem. But even without this, we have buffer management problems since JSonParser may read ahead etc.). There is one problem however (this is not new, it existed even before this patch): flush() could only be called only at object boundaries.
In summary, the new implementation simplifies the API a bit. The writer can write a sequence of objects and needs to call flush() after writing all the objects. The reader doesn't need to do anything special to read a sequence of objects.
> JSON encoder does not easily permit multiple reads or writes
> ------------------------------------------------------------
>
> Key: AVRO-118
> URL: https://issues.apache.org/jira/browse/AVRO-118
> Project: Avro
> Issue Type: Bug
> Components: java
> Affects Versions: 1.1.0
> Reporter: Doug Cutting
> Attachments: AVRO-118.patch, AVRO-118.patch
>
>
> The JSON encoder does not permit one to easily write or read sequences of instances from a single stream.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (AVRO-118) JSON encoder does not easily permit
multiple reads or writes
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Cutting resolved AVRO-118.
-------------------------------
Resolution: Fixed
Fix Version/s: 1.2.0
This is perfect! I just committed it. Thanks, Thiru!
> JSON encoder does not easily permit multiple reads or writes
> ------------------------------------------------------------
>
> Key: AVRO-118
> URL: https://issues.apache.org/jira/browse/AVRO-118
> Project: Avro
> Issue Type: Bug
> Components: java
> Affects Versions: 1.1.0
> Reporter: Doug Cutting
> Fix For: 1.2.0
>
> Attachments: AVRO-118.patch, AVRO-118.patch
>
>
> The JSON encoder does not permit one to easily write or read sequences of instances from a single stream.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (AVRO-118) JSON encoder does not easily permit
multiple reads or writes
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Cutting reassigned AVRO-118:
---------------------------------
Assignee: Thiruvalluvan M. G.
> JSON encoder does not easily permit multiple reads or writes
> ------------------------------------------------------------
>
> Key: AVRO-118
> URL: https://issues.apache.org/jira/browse/AVRO-118
> Project: Avro
> Issue Type: Bug
> Components: java
> Affects Versions: 1.1.0
> Reporter: Doug Cutting
> Assignee: Thiruvalluvan M. G.
> Fix For: 1.2.0
>
> Attachments: AVRO-118.patch, AVRO-118.patch
>
>
> The JSON encoder does not permit one to easily write or read sequences of instances from a single stream.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AVRO-118) JSON encoder does not easily permit
multiple reads or writes
Posted by "Ravi Gummadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12758564#action_12758564 ]
Ravi Gummadi commented on AVRO-118:
-----------------------------------
Schema for the JsonEncoder and JsonDecoder could be an array of objects. But that needs the arrayElementCount at the beginning, but we may not know the number of objects when we start writing.
> JSON encoder does not easily permit multiple reads or writes
> ------------------------------------------------------------
>
> Key: AVRO-118
> URL: https://issues.apache.org/jira/browse/AVRO-118
> Project: Avro
> Issue Type: Bug
> Components: java
> Affects Versions: 1.1.0
> Reporter: Doug Cutting
> Attachments: AVRO-118.patch
>
>
> The JSON encoder does not permit one to easily write or read sequences of instances from a single stream.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AVRO-118) JSON encoder does not easily permit
multiple reads or writes
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12758855#action_12758855 ]
Doug Cutting commented on AVRO-118:
-----------------------------------
Do we need to add a reset() method to JsonEncoder/Decoder that resets the parser state? Ideally it would reset automatically when a complete instance is read, so that another may be read following it. Is there danger in doing that?
> JSON encoder does not easily permit multiple reads or writes
> ------------------------------------------------------------
>
> Key: AVRO-118
> URL: https://issues.apache.org/jira/browse/AVRO-118
> Project: Avro
> Issue Type: Bug
> Components: java
> Affects Versions: 1.1.0
> Reporter: Doug Cutting
> Attachments: AVRO-118.patch
>
>
> The JSON encoder does not permit one to easily write or read sequences of instances from a single stream.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-118) JSON encoder does not easily permit
multiple reads or writes
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Cutting updated AVRO-118:
------------------------------
Attachment: AVRO-118.patch
Attaching a patch that illustrates this.
The second write fails unless the 'encoder.flush(); encoder.init(out)' is uncommented.
The second read fails unless a new decoder is constructed.
> JSON encoder does not easily permit multiple reads or writes
> ------------------------------------------------------------
>
> Key: AVRO-118
> URL: https://issues.apache.org/jira/browse/AVRO-118
> Project: Avro
> Issue Type: Bug
> Components: java
> Affects Versions: 1.1.0
> Reporter: Doug Cutting
> Attachments: AVRO-118.patch
>
>
> The JSON encoder does not permit one to easily write or read sequences of instances from a single stream.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.