You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@avro.apache.org by "Raymie Stata (JIRA)" <ji...@apache.org> on 2009/05/12 14:37:45 UTC
[jira] Created: (AVRO-29) Validation and resolution for
ValueInput/ValueOutput
Validation and resolution for ValueInput/ValueOutput
----------------------------------------------------
Key: AVRO-29
URL: https://issues.apache.org/jira/browse/AVRO-29
Project: Avro
Issue Type: Improvement
Components: java
Reporter: Raymie Stata
Assignee: Thiruvalluvan M. G.
This is a companion to AVRO-25, which introduced the classes ValueOutput and ValueInput. This patch adds two capabilities: validation of ValueInput/Output calls against a schema, and schema-resolution implemented in the context of ValueInput.
ValidatingValueInput and ValidatingValueOutput take a schema and will validate calls against a schema. For example, if the schema calls for a record consisting of two longs and a double, then ValidatingOutput will allow the call-sequence readLong, readLong, readDouble and throw an error otherwise.
ResolvingValueInput takes two schemas, the writer's and the reader's schema, and automatically performs Avro's schema-resolution logic on behalf of the reader. For example, if the writer's schema calls for a long, and the readers calls for a double, then the reader can call readDouble, and ResolvingValueInput will automatically decode the long sent by the writer and convert it into the double expected by the reader.
ResolvingValueInput is an alternative to Avro's current GenericDatumReader, which also implements Avro's resolution logic. In many use-cases, the programmer has their own data structures into which they want to store data read from an Avro stream, data structures that cannot easily be put into the GenericRecord/Array class hierarchy. With ResolvingValueInput, programmers get the benefit of this resolution logic without being forced into the GenericRecord/Array class hierarchy.
We recommend that ResolvingValueInput become the standard implementation of the resolution logic, and that GenericDatumReader be implemented in terms of ResolvingValueInput. However, we haven't implemented this change pending feedback from others.
We haven't implemented default values, but can add that feature.
Implementation note: this patch is implemented by translating Avro schemas to LL(1) parsing tables. This translation is straight forward, but tedious. If you want to understand how the code works, we recommend that you look in the file "parsing.html" (included in the patch), which explains the translation.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-29) Validation and resolution for
ValueInput/ValueOutput
Posted by "Thiruvalluvan M. G. (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-29?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thiruvalluvan M. G. updated AVRO-29:
------------------------------------
Attachment: AVRO-29.patch
This new patch adjusts the class names and other things to suit the changes made by AVRO-57. The other changes suggested by Doug (keeping the validating classes in a separate package, placement of parsing.html etc.) are not in, though.
> Validation and resolution for ValueInput/ValueOutput
> ----------------------------------------------------
>
> Key: AVRO-29
> URL: https://issues.apache.org/jira/browse/AVRO-29
> Project: Avro
> Issue Type: Improvement
> Components: java
> Reporter: Raymie Stata
> Assignee: Thiruvalluvan M. G.
> Attachments: AVRO-29.patch, AVRO-29.patch, AVRO-29.patch
>
>
> This is a companion to AVRO-25, which introduced the classes ValueOutput and ValueInput. This patch adds two capabilities: validation of ValueInput/Output calls against a schema, and schema-resolution implemented in the context of ValueInput.
> ValidatingValueInput and ValidatingValueOutput take a schema and will validate calls against a schema. For example, if the schema calls for a record consisting of two longs and a double, then ValidatingOutput will allow the call-sequence readLong, readLong, readDouble and throw an error otherwise.
> ResolvingValueInput takes two schemas, the writer's and the reader's schema, and automatically performs Avro's schema-resolution logic on behalf of the reader. For example, if the writer's schema calls for a long, and the readers calls for a double, then the reader can call readDouble, and ResolvingValueInput will automatically decode the long sent by the writer and convert it into the double expected by the reader.
> ResolvingValueInput is an alternative to Avro's current GenericDatumReader, which also implements Avro's resolution logic. In many use-cases, the programmer has their own data structures into which they want to store data read from an Avro stream, data structures that cannot easily be put into the GenericRecord/Array class hierarchy. With ResolvingValueInput, programmers get the benefit of this resolution logic without being forced into the GenericRecord/Array class hierarchy.
> We recommend that ResolvingValueInput become the standard implementation of the resolution logic, and that GenericDatumReader be implemented in terms of ResolvingValueInput. However, we haven't implemented this change pending feedback from others.
> We haven't implemented default values, but can add that feature.
> Implementation note: this patch is implemented by translating Avro schemas to LL(1) parsing tables. This translation is straight forward, but tedious. If you want to understand how the code works, we recommend that you look in the file "parsing.html" (included in the patch), which explains the translation.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AVRO-29) Validation and resolution for
ValueInput/ValueOutput
Posted by "Thiruvalluvan M. G. (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-29?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12721329#action_12721329 ]
Thiruvalluvan M. G. commented on AVRO-29:
-----------------------------------------
- The only real use of ValidatingValueReader/Writer is validation. It can be used for testing new class that directly uses ValueReader/Writer objects. Since it is designed as a filter, it can be inserted into the chain to detect any corner-case bugs even in production environments. At best it can be used for diagnostic purposes.
- There are two versions of readRecord because of the difference in behavior of ResolvingValueReader compared to ValueReader. The ValueReader returns objects in the order of their declaration in the reader's schema. For ResolvingValueReader could return in a different order depending on writer's schema. If we can achieve reordering of fields (which is possible with some more effort), then we can get rid of the second version of readRecord(). In fact if reader can expect its contents in the order of its schema and if support for default values is added, all the resolution is internal to the ResolvingValueReader. Any reader can simply read as if the data is serialized according to its schema.
- The parsing table can be considered as a binary version of schema. (There is some information loss presently, but it can be taken care of). One can define an avro schema that serializes parsing table itself. With that, an RPC can send data along with its schema which a receiver can readily use to resolve against receiver's schema. This is functionally equivalent to sending the JSON version of schema, but is more efficient. This is particularly useful for scatter/gather kind of RPCs where many receivers receive the same request. The time saved thus could be significant.
- Once we agree on the usefulness of these classes, we can move them around appropriately.
> Validation and resolution for ValueInput/ValueOutput
> ----------------------------------------------------
>
> Key: AVRO-29
> URL: https://issues.apache.org/jira/browse/AVRO-29
> Project: Avro
> Issue Type: Improvement
> Components: java
> Reporter: Raymie Stata
> Assignee: Thiruvalluvan M. G.
> Attachments: AVRO-29.patch, AVRO-29.patch
>
>
> This is a companion to AVRO-25, which introduced the classes ValueOutput and ValueInput. This patch adds two capabilities: validation of ValueInput/Output calls against a schema, and schema-resolution implemented in the context of ValueInput.
> ValidatingValueInput and ValidatingValueOutput take a schema and will validate calls against a schema. For example, if the schema calls for a record consisting of two longs and a double, then ValidatingOutput will allow the call-sequence readLong, readLong, readDouble and throw an error otherwise.
> ResolvingValueInput takes two schemas, the writer's and the reader's schema, and automatically performs Avro's schema-resolution logic on behalf of the reader. For example, if the writer's schema calls for a long, and the readers calls for a double, then the reader can call readDouble, and ResolvingValueInput will automatically decode the long sent by the writer and convert it into the double expected by the reader.
> ResolvingValueInput is an alternative to Avro's current GenericDatumReader, which also implements Avro's resolution logic. In many use-cases, the programmer has their own data structures into which they want to store data read from an Avro stream, data structures that cannot easily be put into the GenericRecord/Array class hierarchy. With ResolvingValueInput, programmers get the benefit of this resolution logic without being forced into the GenericRecord/Array class hierarchy.
> We recommend that ResolvingValueInput become the standard implementation of the resolution logic, and that GenericDatumReader be implemented in terms of ResolvingValueInput. However, we haven't implemented this change pending feedback from others.
> We haven't implemented default values, but can add that feature.
> Implementation note: this patch is implemented by translating Avro schemas to LL(1) parsing tables. This translation is straight forward, but tedious. If you want to understand how the code works, we recommend that you look in the file "parsing.html" (included in the patch), which explains the translation.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AVRO-29) Validation and resolution for
ValueInput/ValueOutput
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-29?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12724102#action_12724102 ]
Doug Cutting commented on AVRO-29:
----------------------------------
My concern is mostly a documentation one. We ought to include this code, the question is how we should frame it.
Having thought a bit more, i think the case for this is applications that do not directly represent Avro data with in-memory objects, but rather stream through data. Such applications would use the Encoder/Decoder API (nee ValueReader/Writer) directly, SAX-like rather than DOM-like, and validation would be useful. It would be good to have some examples of such applications. Can anyone think of any?
I broke this patch with AVRO-57. I can try to update it later today.
> Validation and resolution for ValueInput/ValueOutput
> ----------------------------------------------------
>
> Key: AVRO-29
> URL: https://issues.apache.org/jira/browse/AVRO-29
> Project: Avro
> Issue Type: Improvement
> Components: java
> Reporter: Raymie Stata
> Assignee: Thiruvalluvan M. G.
> Attachments: AVRO-29.patch, AVRO-29.patch
>
>
> This is a companion to AVRO-25, which introduced the classes ValueOutput and ValueInput. This patch adds two capabilities: validation of ValueInput/Output calls against a schema, and schema-resolution implemented in the context of ValueInput.
> ValidatingValueInput and ValidatingValueOutput take a schema and will validate calls against a schema. For example, if the schema calls for a record consisting of two longs and a double, then ValidatingOutput will allow the call-sequence readLong, readLong, readDouble and throw an error otherwise.
> ResolvingValueInput takes two schemas, the writer's and the reader's schema, and automatically performs Avro's schema-resolution logic on behalf of the reader. For example, if the writer's schema calls for a long, and the readers calls for a double, then the reader can call readDouble, and ResolvingValueInput will automatically decode the long sent by the writer and convert it into the double expected by the reader.
> ResolvingValueInput is an alternative to Avro's current GenericDatumReader, which also implements Avro's resolution logic. In many use-cases, the programmer has their own data structures into which they want to store data read from an Avro stream, data structures that cannot easily be put into the GenericRecord/Array class hierarchy. With ResolvingValueInput, programmers get the benefit of this resolution logic without being forced into the GenericRecord/Array class hierarchy.
> We recommend that ResolvingValueInput become the standard implementation of the resolution logic, and that GenericDatumReader be implemented in terms of ResolvingValueInput. However, we haven't implemented this change pending feedback from others.
> We haven't implemented default values, but can add that feature.
> Implementation note: this patch is implemented by translating Avro schemas to LL(1) parsing tables. This translation is straight forward, but tedious. If you want to understand how the code works, we recommend that you look in the file "parsing.html" (included in the patch), which explains the translation.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AVRO-29) Validation and resolution for
ValueInput/ValueOutput
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-29?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12720921#action_12720921 ]
Doug Cutting commented on AVRO-29:
----------------------------------
Some comments and questions:
- What's the advantage of using ResovlingRecordReader over the existing code in GenericDatumReader? This replaces existing logic with a lot more code, code that's harder to understand, IMHO. It's interesting that it can be done this way, but what value does this give applications? Above Raymie states, "With ResolvingValueInput, programmers get the benefit of this resolution logic without being forced into the GenericRecord/Array class hierarchy." But applications already are not forced into this. They simply must override addField(), getField() and removeField(), as with ResolvingRecordReader. For example, SpecificDatumReader extends GenericDatumReader to read records that do not implement GenericRecord. What am I missing?
- Similarly, what's the advantage of the ValidatingValueReader/Writer? It feels a bit like belt-and-suspenders. The existing DatumReader and DatumWriter implementations do not perform invalid operations. This is tested by unit tests, so why check it again at runtime for every record, even if it can be done relatively cheaply? We don't expect there to be thousands of DatumReader and DatumWriter implementations. All existing ones extend GenericDatumReader/Writer, which should prevent invalid operations, and I have a hard time imagining an application which cannot be reasonably implemented by extending these. Again, what am I missing?
- GenericDatumReader: ValidatingValueReader import is not used;
- GenericDatumReader: It's unfortunate that this includes two versions of readRecord.
- Might the resolving/parsing classes would better live in an io.validate package?
- What should we do with the parsing.html file? If it's for developers only, then it could live with the source code (perhaps alongside the .java files), in javadoc or on the wiki. If we think end-users might be interested, then it should go into the forrest-generated docs.
> Validation and resolution for ValueInput/ValueOutput
> ----------------------------------------------------
>
> Key: AVRO-29
> URL: https://issues.apache.org/jira/browse/AVRO-29
> Project: Avro
> Issue Type: Improvement
> Components: java
> Reporter: Raymie Stata
> Assignee: Thiruvalluvan M. G.
> Attachments: AVRO-29.patch, AVRO-29.patch
>
>
> This is a companion to AVRO-25, which introduced the classes ValueOutput and ValueInput. This patch adds two capabilities: validation of ValueInput/Output calls against a schema, and schema-resolution implemented in the context of ValueInput.
> ValidatingValueInput and ValidatingValueOutput take a schema and will validate calls against a schema. For example, if the schema calls for a record consisting of two longs and a double, then ValidatingOutput will allow the call-sequence readLong, readLong, readDouble and throw an error otherwise.
> ResolvingValueInput takes two schemas, the writer's and the reader's schema, and automatically performs Avro's schema-resolution logic on behalf of the reader. For example, if the writer's schema calls for a long, and the readers calls for a double, then the reader can call readDouble, and ResolvingValueInput will automatically decode the long sent by the writer and convert it into the double expected by the reader.
> ResolvingValueInput is an alternative to Avro's current GenericDatumReader, which also implements Avro's resolution logic. In many use-cases, the programmer has their own data structures into which they want to store data read from an Avro stream, data structures that cannot easily be put into the GenericRecord/Array class hierarchy. With ResolvingValueInput, programmers get the benefit of this resolution logic without being forced into the GenericRecord/Array class hierarchy.
> We recommend that ResolvingValueInput become the standard implementation of the resolution logic, and that GenericDatumReader be implemented in terms of ResolvingValueInput. However, we haven't implemented this change pending feedback from others.
> We haven't implemented default values, but can add that feature.
> Implementation note: this patch is implemented by translating Avro schemas to LL(1) parsing tables. This translation is straight forward, but tedious. If you want to understand how the code works, we recommend that you look in the file "parsing.html" (included in the patch), which explains the translation.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (AVRO-29) Validation and resolution for
ValueInput/ValueOutput
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-29?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Cutting resolved AVRO-29.
------------------------------
Resolution: Fixed
Fix Version/s: 1.0.0
I just committed this. Thanks Thiru & Raymie!
> Validation and resolution for ValueInput/ValueOutput
> ----------------------------------------------------
>
> Key: AVRO-29
> URL: https://issues.apache.org/jira/browse/AVRO-29
> Project: Avro
> Issue Type: Improvement
> Components: java
> Reporter: Raymie Stata
> Assignee: Thiruvalluvan M. G.
> Fix For: 1.0.0
>
> Attachments: AVRO-29.patch, AVRO-29.patch, AVRO-29.patch
>
>
> This is a companion to AVRO-25, which introduced the classes ValueOutput and ValueInput. This patch adds two capabilities: validation of ValueInput/Output calls against a schema, and schema-resolution implemented in the context of ValueInput.
> ValidatingValueInput and ValidatingValueOutput take a schema and will validate calls against a schema. For example, if the schema calls for a record consisting of two longs and a double, then ValidatingOutput will allow the call-sequence readLong, readLong, readDouble and throw an error otherwise.
> ResolvingValueInput takes two schemas, the writer's and the reader's schema, and automatically performs Avro's schema-resolution logic on behalf of the reader. For example, if the writer's schema calls for a long, and the readers calls for a double, then the reader can call readDouble, and ResolvingValueInput will automatically decode the long sent by the writer and convert it into the double expected by the reader.
> ResolvingValueInput is an alternative to Avro's current GenericDatumReader, which also implements Avro's resolution logic. In many use-cases, the programmer has their own data structures into which they want to store data read from an Avro stream, data structures that cannot easily be put into the GenericRecord/Array class hierarchy. With ResolvingValueInput, programmers get the benefit of this resolution logic without being forced into the GenericRecord/Array class hierarchy.
> We recommend that ResolvingValueInput become the standard implementation of the resolution logic, and that GenericDatumReader be implemented in terms of ResolvingValueInput. However, we haven't implemented this change pending feedback from others.
> We haven't implemented default values, but can add that feature.
> Implementation note: this patch is implemented by translating Avro schemas to LL(1) parsing tables. This translation is straight forward, but tedious. If you want to understand how the code works, we recommend that you look in the file "parsing.html" (included in the patch), which explains the translation.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-29) Validation and resolution for
ValueInput/ValueOutput
Posted by "Thiruvalluvan M. G. (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-29?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thiruvalluvan M. G. updated AVRO-29:
------------------------------------
Attachment: (was: AVRO-29.patch)
> Validation and resolution for ValueInput/ValueOutput
> ----------------------------------------------------
>
> Key: AVRO-29
> URL: https://issues.apache.org/jira/browse/AVRO-29
> Project: Avro
> Issue Type: Improvement
> Components: java
> Reporter: Raymie Stata
> Assignee: Thiruvalluvan M. G.
> Attachments: AVRO-29.patch
>
>
> This is a companion to AVRO-25, which introduced the classes ValueOutput and ValueInput. This patch adds two capabilities: validation of ValueInput/Output calls against a schema, and schema-resolution implemented in the context of ValueInput.
> ValidatingValueInput and ValidatingValueOutput take a schema and will validate calls against a schema. For example, if the schema calls for a record consisting of two longs and a double, then ValidatingOutput will allow the call-sequence readLong, readLong, readDouble and throw an error otherwise.
> ResolvingValueInput takes two schemas, the writer's and the reader's schema, and automatically performs Avro's schema-resolution logic on behalf of the reader. For example, if the writer's schema calls for a long, and the readers calls for a double, then the reader can call readDouble, and ResolvingValueInput will automatically decode the long sent by the writer and convert it into the double expected by the reader.
> ResolvingValueInput is an alternative to Avro's current GenericDatumReader, which also implements Avro's resolution logic. In many use-cases, the programmer has their own data structures into which they want to store data read from an Avro stream, data structures that cannot easily be put into the GenericRecord/Array class hierarchy. With ResolvingValueInput, programmers get the benefit of this resolution logic without being forced into the GenericRecord/Array class hierarchy.
> We recommend that ResolvingValueInput become the standard implementation of the resolution logic, and that GenericDatumReader be implemented in terms of ResolvingValueInput. However, we haven't implemented this change pending feedback from others.
> We haven't implemented default values, but can add that feature.
> Implementation note: this patch is implemented by translating Avro schemas to LL(1) parsing tables. This translation is straight forward, but tedious. If you want to understand how the code works, we recommend that you look in the file "parsing.html" (included in the patch), which explains the translation.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AVRO-29) Validation and resolution for
ValueInput/ValueOutput
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-29?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12712301#action_12712301 ]
Doug Cutting commented on AVRO-29:
----------------------------------
> ValidatingValueInput and ValidatingValueOutput take a schema and will validate calls against a schema.
That would be very useful to have for debugging. If the performance were good enough, one might even use it all the time.
> With ResolvingValueInput, programmers get the benefit of this resolution logic without being forced into the GenericRecord/Array class hierarchy.
Much of the need for GenericRecord is around handling field reordering, insertion and removal. I don't see how this changes that. What am I missing? Does each data representation need to loop over the schema itself, implementing that logic? Is there an example of how a record is read somewhere?
> Validation and resolution for ValueInput/ValueOutput
> ----------------------------------------------------
>
> Key: AVRO-29
> URL: https://issues.apache.org/jira/browse/AVRO-29
> Project: Avro
> Issue Type: Improvement
> Components: java
> Reporter: Raymie Stata
> Assignee: Thiruvalluvan M. G.
> Attachments: AVRO-29.patch
>
>
> This is a companion to AVRO-25, which introduced the classes ValueOutput and ValueInput. This patch adds two capabilities: validation of ValueInput/Output calls against a schema, and schema-resolution implemented in the context of ValueInput.
> ValidatingValueInput and ValidatingValueOutput take a schema and will validate calls against a schema. For example, if the schema calls for a record consisting of two longs and a double, then ValidatingOutput will allow the call-sequence readLong, readLong, readDouble and throw an error otherwise.
> ResolvingValueInput takes two schemas, the writer's and the reader's schema, and automatically performs Avro's schema-resolution logic on behalf of the reader. For example, if the writer's schema calls for a long, and the readers calls for a double, then the reader can call readDouble, and ResolvingValueInput will automatically decode the long sent by the writer and convert it into the double expected by the reader.
> ResolvingValueInput is an alternative to Avro's current GenericDatumReader, which also implements Avro's resolution logic. In many use-cases, the programmer has their own data structures into which they want to store data read from an Avro stream, data structures that cannot easily be put into the GenericRecord/Array class hierarchy. With ResolvingValueInput, programmers get the benefit of this resolution logic without being forced into the GenericRecord/Array class hierarchy.
> We recommend that ResolvingValueInput become the standard implementation of the resolution logic, and that GenericDatumReader be implemented in terms of ResolvingValueInput. However, we haven't implemented this change pending feedback from others.
> We haven't implemented default values, but can add that feature.
> Implementation note: this patch is implemented by translating Avro schemas to LL(1) parsing tables. This translation is straight forward, but tedious. If you want to understand how the code works, we recommend that you look in the file "parsing.html" (included in the patch), which explains the translation.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-29) Validation and resolution for
ValueInput/ValueOutput
Posted by "Thiruvalluvan M. G. (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-29?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thiruvalluvan M. G. updated AVRO-29:
------------------------------------
Attachment: AVRO-29.patch
This patch has more complete implementation of the validating and resolving reader/writers. This has been adapted to the recent changes to readers and writers in JIRA AVRO-25.
The ResolvingValueReader supports the following rules for resolution:
- long can read int.
- double can read int, long, float
- Fixed matches only size matchs
- Enums match symbols (if the writer defines 3 for symbol "x" and reader defines 5 for "x", then a value 3 in the stream will return 5 to the caller)
- If writer has a Union, it matches reader's non-union type if it is one of branches of the writer and data on the stream is of that type.
- If the writer has a non-union type and reader has a union type with the writer's type as a branch, reader sees the union with that branch.
GenericDatumReader is modified to use ValidatingValueReader.
My performance tests show that using ValidatingValueWriter is about 8% slower than not using it. Using ResolvingValueReader degrades performance from 0% to 8% depending on the kind of "resolution" used. When there is no resolution (i.e. reader and writer schemas are identical), it is functionally equivalent to ValidatingValueReader, the overhead is the maximum. [These results, of course cache the resolving table. That is the resolving table is not constructed for every object being decoded].
> Validation and resolution for ValueInput/ValueOutput
> ----------------------------------------------------
>
> Key: AVRO-29
> URL: https://issues.apache.org/jira/browse/AVRO-29
> Project: Avro
> Issue Type: Improvement
> Components: java
> Reporter: Raymie Stata
> Assignee: Thiruvalluvan M. G.
> Attachments: AVRO-29.patch, AVRO-29.patch
>
>
> This is a companion to AVRO-25, which introduced the classes ValueOutput and ValueInput. This patch adds two capabilities: validation of ValueInput/Output calls against a schema, and schema-resolution implemented in the context of ValueInput.
> ValidatingValueInput and ValidatingValueOutput take a schema and will validate calls against a schema. For example, if the schema calls for a record consisting of two longs and a double, then ValidatingOutput will allow the call-sequence readLong, readLong, readDouble and throw an error otherwise.
> ResolvingValueInput takes two schemas, the writer's and the reader's schema, and automatically performs Avro's schema-resolution logic on behalf of the reader. For example, if the writer's schema calls for a long, and the readers calls for a double, then the reader can call readDouble, and ResolvingValueInput will automatically decode the long sent by the writer and convert it into the double expected by the reader.
> ResolvingValueInput is an alternative to Avro's current GenericDatumReader, which also implements Avro's resolution logic. In many use-cases, the programmer has their own data structures into which they want to store data read from an Avro stream, data structures that cannot easily be put into the GenericRecord/Array class hierarchy. With ResolvingValueInput, programmers get the benefit of this resolution logic without being forced into the GenericRecord/Array class hierarchy.
> We recommend that ResolvingValueInput become the standard implementation of the resolution logic, and that GenericDatumReader be implemented in terms of ResolvingValueInput. However, we haven't implemented this change pending feedback from others.
> We haven't implemented default values, but can add that feature.
> Implementation note: this patch is implemented by translating Avro schemas to LL(1) parsing tables. This translation is straight forward, but tedious. If you want to understand how the code works, we recommend that you look in the file "parsing.html" (included in the patch), which explains the translation.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AVRO-29) Validation and resolution for
ValueInput/ValueOutput
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-29?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12724231#action_12724231 ]
Doug Cutting commented on AVRO-29:
----------------------------------
I just reverted the changes to GenericDatumReader, as I'm unsure of the performance impacts. Perhaps, in a new Jira issue, we should reconsider this. If the parsing table must be reused for good performance, then we might cache these in a WeakIdentityHashMap<Schema,ParsingTable>, so that naive applications do not suffer.
Also, I should have mentioned with the commit, I made a few changes to the patch:
- made references to ParsingTable and Resolving table package private so that they would not show up in end-user javadoc;
- fixed javadoc comments that still linked to ValueReader/Writer; and
- moved parsing.html into the javadoc tree linked to from theses classes javadoc.
> Validation and resolution for ValueInput/ValueOutput
> ----------------------------------------------------
>
> Key: AVRO-29
> URL: https://issues.apache.org/jira/browse/AVRO-29
> Project: Avro
> Issue Type: Improvement
> Components: java
> Reporter: Raymie Stata
> Assignee: Thiruvalluvan M. G.
> Fix For: 1.0.0
>
> Attachments: AVRO-29.patch, AVRO-29.patch, AVRO-29.patch
>
>
> This is a companion to AVRO-25, which introduced the classes ValueOutput and ValueInput. This patch adds two capabilities: validation of ValueInput/Output calls against a schema, and schema-resolution implemented in the context of ValueInput.
> ValidatingValueInput and ValidatingValueOutput take a schema and will validate calls against a schema. For example, if the schema calls for a record consisting of two longs and a double, then ValidatingOutput will allow the call-sequence readLong, readLong, readDouble and throw an error otherwise.
> ResolvingValueInput takes two schemas, the writer's and the reader's schema, and automatically performs Avro's schema-resolution logic on behalf of the reader. For example, if the writer's schema calls for a long, and the readers calls for a double, then the reader can call readDouble, and ResolvingValueInput will automatically decode the long sent by the writer and convert it into the double expected by the reader.
> ResolvingValueInput is an alternative to Avro's current GenericDatumReader, which also implements Avro's resolution logic. In many use-cases, the programmer has their own data structures into which they want to store data read from an Avro stream, data structures that cannot easily be put into the GenericRecord/Array class hierarchy. With ResolvingValueInput, programmers get the benefit of this resolution logic without being forced into the GenericRecord/Array class hierarchy.
> We recommend that ResolvingValueInput become the standard implementation of the resolution logic, and that GenericDatumReader be implemented in terms of ResolvingValueInput. However, we haven't implemented this change pending feedback from others.
> We haven't implemented default values, but can add that feature.
> Implementation note: this patch is implemented by translating Avro schemas to LL(1) parsing tables. This translation is straight forward, but tedious. If you want to understand how the code works, we recommend that you look in the file "parsing.html" (included in the patch), which explains the translation.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AVRO-29) Validation and resolution for
ValueInput/ValueOutput
Posted by "Hong Tang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-29?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12708491#action_12708491 ]
Hong Tang commented on AVRO-29:
-------------------------------
Attachment?
> Validation and resolution for ValueInput/ValueOutput
> ----------------------------------------------------
>
> Key: AVRO-29
> URL: https://issues.apache.org/jira/browse/AVRO-29
> Project: Avro
> Issue Type: Improvement
> Components: java
> Reporter: Raymie Stata
> Assignee: Thiruvalluvan M. G.
>
> This is a companion to AVRO-25, which introduced the classes ValueOutput and ValueInput. This patch adds two capabilities: validation of ValueInput/Output calls against a schema, and schema-resolution implemented in the context of ValueInput.
> ValidatingValueInput and ValidatingValueOutput take a schema and will validate calls against a schema. For example, if the schema calls for a record consisting of two longs and a double, then ValidatingOutput will allow the call-sequence readLong, readLong, readDouble and throw an error otherwise.
> ResolvingValueInput takes two schemas, the writer's and the reader's schema, and automatically performs Avro's schema-resolution logic on behalf of the reader. For example, if the writer's schema calls for a long, and the readers calls for a double, then the reader can call readDouble, and ResolvingValueInput will automatically decode the long sent by the writer and convert it into the double expected by the reader.
> ResolvingValueInput is an alternative to Avro's current GenericDatumReader, which also implements Avro's resolution logic. In many use-cases, the programmer has their own data structures into which they want to store data read from an Avro stream, data structures that cannot easily be put into the GenericRecord/Array class hierarchy. With ResolvingValueInput, programmers get the benefit of this resolution logic without being forced into the GenericRecord/Array class hierarchy.
> We recommend that ResolvingValueInput become the standard implementation of the resolution logic, and that GenericDatumReader be implemented in terms of ResolvingValueInput. However, we haven't implemented this change pending feedback from others.
> We haven't implemented default values, but can add that feature.
> Implementation note: this patch is implemented by translating Avro schemas to LL(1) parsing tables. This translation is straight forward, but tedious. If you want to understand how the code works, we recommend that you look in the file "parsing.html" (included in the patch), which explains the translation.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-29) Validation and resolution for
ValueInput/ValueOutput
Posted by "Thiruvalluvan M. G. (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-29?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thiruvalluvan M. G. updated AVRO-29:
------------------------------------
Attachment: AVRO-29.patch
> Validation and resolution for ValueInput/ValueOutput
> ----------------------------------------------------
>
> Key: AVRO-29
> URL: https://issues.apache.org/jira/browse/AVRO-29
> Project: Avro
> Issue Type: Improvement
> Components: java
> Reporter: Raymie Stata
> Assignee: Thiruvalluvan M. G.
> Attachments: AVRO-29.patch
>
>
> This is a companion to AVRO-25, which introduced the classes ValueOutput and ValueInput. This patch adds two capabilities: validation of ValueInput/Output calls against a schema, and schema-resolution implemented in the context of ValueInput.
> ValidatingValueInput and ValidatingValueOutput take a schema and will validate calls against a schema. For example, if the schema calls for a record consisting of two longs and a double, then ValidatingOutput will allow the call-sequence readLong, readLong, readDouble and throw an error otherwise.
> ResolvingValueInput takes two schemas, the writer's and the reader's schema, and automatically performs Avro's schema-resolution logic on behalf of the reader. For example, if the writer's schema calls for a long, and the readers calls for a double, then the reader can call readDouble, and ResolvingValueInput will automatically decode the long sent by the writer and convert it into the double expected by the reader.
> ResolvingValueInput is an alternative to Avro's current GenericDatumReader, which also implements Avro's resolution logic. In many use-cases, the programmer has their own data structures into which they want to store data read from an Avro stream, data structures that cannot easily be put into the GenericRecord/Array class hierarchy. With ResolvingValueInput, programmers get the benefit of this resolution logic without being forced into the GenericRecord/Array class hierarchy.
> We recommend that ResolvingValueInput become the standard implementation of the resolution logic, and that GenericDatumReader be implemented in terms of ResolvingValueInput. However, we haven't implemented this change pending feedback from others.
> We haven't implemented default values, but can add that feature.
> Implementation note: this patch is implemented by translating Avro schemas to LL(1) parsing tables. This translation is straight forward, but tedious. If you want to understand how the code works, we recommend that you look in the file "parsing.html" (included in the patch), which explains the translation.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-29) Validation and resolution for
ValueInput/ValueOutput
Posted by "Thiruvalluvan M. G. (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-29?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thiruvalluvan M. G. updated AVRO-29:
------------------------------------
Attachment: AVRO-29.patch
I missed a few file in the previous patch.
> Validation and resolution for ValueInput/ValueOutput
> ----------------------------------------------------
>
> Key: AVRO-29
> URL: https://issues.apache.org/jira/browse/AVRO-29
> Project: Avro
> Issue Type: Improvement
> Components: java
> Reporter: Raymie Stata
> Assignee: Thiruvalluvan M. G.
> Attachments: AVRO-29.patch
>
>
> This is a companion to AVRO-25, which introduced the classes ValueOutput and ValueInput. This patch adds two capabilities: validation of ValueInput/Output calls against a schema, and schema-resolution implemented in the context of ValueInput.
> ValidatingValueInput and ValidatingValueOutput take a schema and will validate calls against a schema. For example, if the schema calls for a record consisting of two longs and a double, then ValidatingOutput will allow the call-sequence readLong, readLong, readDouble and throw an error otherwise.
> ResolvingValueInput takes two schemas, the writer's and the reader's schema, and automatically performs Avro's schema-resolution logic on behalf of the reader. For example, if the writer's schema calls for a long, and the readers calls for a double, then the reader can call readDouble, and ResolvingValueInput will automatically decode the long sent by the writer and convert it into the double expected by the reader.
> ResolvingValueInput is an alternative to Avro's current GenericDatumReader, which also implements Avro's resolution logic. In many use-cases, the programmer has their own data structures into which they want to store data read from an Avro stream, data structures that cannot easily be put into the GenericRecord/Array class hierarchy. With ResolvingValueInput, programmers get the benefit of this resolution logic without being forced into the GenericRecord/Array class hierarchy.
> We recommend that ResolvingValueInput become the standard implementation of the resolution logic, and that GenericDatumReader be implemented in terms of ResolvingValueInput. However, we haven't implemented this change pending feedback from others.
> We haven't implemented default values, but can add that feature.
> Implementation note: this patch is implemented by translating Avro schemas to LL(1) parsing tables. This translation is straight forward, but tedious. If you want to understand how the code works, we recommend that you look in the file "parsing.html" (included in the patch), which explains the translation.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AVRO-29) Validation and resolution for
ValueInput/ValueOutput
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-29?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12721364#action_12721364 ]
Doug Cutting commented on AVRO-29:
----------------------------------
> The parsing table can be considered as a binary version of schema.
Would this be better than simply defining a schema for schemas, something like the following?
{code}
{"type" : "record", "name" : "Schema"
"fields" : [
{"name": "type", "type": ["Record", "Map", "Array", "Union", "String", "Bytes", "Long", ...]}
]
}
{"type" : "record", "name" : "Record"
"fields" : [
{"name": "name", "type": "string"},
{"name": "fields", "type": {"type" : "array", "items": "Field"}}
]
}
{"type" : "record", "name" : "Field"
"fields" : [
{"name": "name", "type": "string"},
{"name": "schema", "type": "Schema"}
}
{"type" : "record", "name" : "Array"
"fields" : [{"name": "items", "type": "Schema"}]
}
{"type" : "record", "name" : "Map"
"fields" : [{"name": "values", "type": "Schema"}]
}
{"type" : "record", "name" : "Union"
"fields" : [{"name": "types", "type": {"type" : "array", "items": "Schema"}}]
}
{"type" : "record", "name" : "String" "fields" : [] }
{"type" : "record", "name" : "Bytes" "fields" : [] }
{"type" : "record", "name" : "Long" "fields" : [] }
{"type" : "record", "name" : "Double" "fields" : []}
{code}
Such a schema could be included in Avro, and any schema could be efficiently serialized in binary with it. Would a parsing table be substantially more efficient?
> Validation and resolution for ValueInput/ValueOutput
> ----------------------------------------------------
>
> Key: AVRO-29
> URL: https://issues.apache.org/jira/browse/AVRO-29
> Project: Avro
> Issue Type: Improvement
> Components: java
> Reporter: Raymie Stata
> Assignee: Thiruvalluvan M. G.
> Attachments: AVRO-29.patch, AVRO-29.patch
>
>
> This is a companion to AVRO-25, which introduced the classes ValueOutput and ValueInput. This patch adds two capabilities: validation of ValueInput/Output calls against a schema, and schema-resolution implemented in the context of ValueInput.
> ValidatingValueInput and ValidatingValueOutput take a schema and will validate calls against a schema. For example, if the schema calls for a record consisting of two longs and a double, then ValidatingOutput will allow the call-sequence readLong, readLong, readDouble and throw an error otherwise.
> ResolvingValueInput takes two schemas, the writer's and the reader's schema, and automatically performs Avro's schema-resolution logic on behalf of the reader. For example, if the writer's schema calls for a long, and the readers calls for a double, then the reader can call readDouble, and ResolvingValueInput will automatically decode the long sent by the writer and convert it into the double expected by the reader.
> ResolvingValueInput is an alternative to Avro's current GenericDatumReader, which also implements Avro's resolution logic. In many use-cases, the programmer has their own data structures into which they want to store data read from an Avro stream, data structures that cannot easily be put into the GenericRecord/Array class hierarchy. With ResolvingValueInput, programmers get the benefit of this resolution logic without being forced into the GenericRecord/Array class hierarchy.
> We recommend that ResolvingValueInput become the standard implementation of the resolution logic, and that GenericDatumReader be implemented in terms of ResolvingValueInput. However, we haven't implemented this change pending feedback from others.
> We haven't implemented default values, but can add that feature.
> Implementation note: this patch is implemented by translating Avro schemas to LL(1) parsing tables. This translation is straight forward, but tedious. If you want to understand how the code works, we recommend that you look in the file "parsing.html" (included in the patch), which explains the translation.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.