You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@avro.apache.org by "Raymie Stata (Created) (JIRA)" <ji...@apache.org> on 2012/01/25 07:41:42 UTC
[jira] [Created] (AVRO-1006) Fingerprints for Avro Schemas
Fingerprints for Avro Schemas
-----------------------------
Key: AVRO-1006
URL: https://issues.apache.org/jira/browse/AVRO-1006
Project: Avro
Issue Type: New Feature
Components: java
Reporter: Raymie Stata
Assignee: Raymie Stata
Add function that returns a standardized, 64-bit fingerprint that can be used as a key in various contexts.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1006) Fingerprints for Avro Schemas
Posted by "Raymie Stata (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-1006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13219815#comment-13219815 ]
Raymie Stata commented on AVRO-1006:
------------------------------------
On the one hand, the "Parsing Canonical Form" as I've defined it has important uses and I think should be part of the Avro system.
On the other hand, it's a good point that there may be additional canonical forms that satisfy different needs in the future. The class- and method-names of the previous patch implicitly assumed a single canonical form. I've modified the patch with new names that will better support the addition of new canonical forms in the future. (I think I also fixed the Forrest problems, although I haven't been able to verify.)
> Fingerprints for Avro Schemas
> -----------------------------
>
> Key: AVRO-1006
> URL: https://issues.apache.org/jira/browse/AVRO-1006
> Project: Avro
> Issue Type: New Feature
> Components: java
> Reporter: Raymie Stata
> Assignee: Raymie Stata
> Labels: features
> Attachments: AVRO-1006-prelim.patch, AVRO-1006.patch, AVRO-1006.patch, AVRO-1006.patch, schema-fingerprinting.html, schema-fingerprinting.html, schema-fingerprinting.html
>
>
> Add function that returns a standardized, 64-bit fingerprint for schemas. Fingerprints are designed such that the chances of collisions is very, very low.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1006) Fingerprints for Avro Schemas
Posted by "Douglas Kaminsky (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-1006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13221396#comment-13221396 ]
Douglas Kaminsky commented on AVRO-1006:
----------------------------------------
Is cross language support in scope for this feature? The trickiest part is getting the output to match between implementations. That's where I see most of the value in this being incorporated into the core framework instead of implemented by individual users (as we do now)
> Fingerprints for Avro Schemas
> -----------------------------
>
> Key: AVRO-1006
> URL: https://issues.apache.org/jira/browse/AVRO-1006
> Project: Avro
> Issue Type: New Feature
> Components: java
> Reporter: Raymie Stata
> Assignee: Raymie Stata
> Labels: features
> Attachments: AVRO-1006-prelim.patch, AVRO-1006.patch, AVRO-1006.patch, AVRO-1006.patch, AVRO-1006.patch, AVRO-1006.patch, schema-fingerprinting.html, schema-fingerprinting.html, schema-fingerprinting.html
>
>
> Add function that returns a standardized, 64-bit fingerprint for schemas. Fingerprints are designed such that the chances of collisions is very, very low.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1006) Fingerprints for Avro Schemas
Posted by "Raymie Stata (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-1006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13203664#comment-13203664 ]
Raymie Stata commented on AVRO-1006:
------------------------------------
Having slept on it, I don't think that the Schema class is a good place for this code. Rather, I'll create a o.a.avro.util.SchemaFP class, which will contain a static method to canonicalize schemas (ie, the code in the current patch), and will also contain static methods for generating fingerprints of various lengths (64-bit CRC fingerprints, 128-bit MD5 fingerprints, etc). I'll try to post something over the next few days. But in the meantime, please continue commenting as if that's the plan of record.
> Fingerprints for Avro Schemas
> -----------------------------
>
> Key: AVRO-1006
> URL: https://issues.apache.org/jira/browse/AVRO-1006
> Project: Avro
> Issue Type: New Feature
> Components: java
> Reporter: Raymie Stata
> Assignee: Raymie Stata
> Labels: features
> Attachments: AVRO-1006-prelim.patch, schema-fingerprinting.html, schema-fingerprinting.html, schema-fingerprinting.html
>
>
> Add function that returns a standardized, 64-bit fingerprint for schemas. Fingerprints are designed such that the chances of collisions is very, very low.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (AVRO-1006) Fingerprints for Avro Schemas
Posted by "Doug Cutting (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-1006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Cutting resolved AVRO-1006.
--------------------------------
Resolution: Fixed
Fix Version/s: 1.7.0
I just committed this. Thanks, Raymie!
> Fingerprints for Avro Schemas
> -----------------------------
>
> Key: AVRO-1006
> URL: https://issues.apache.org/jira/browse/AVRO-1006
> Project: Avro
> Issue Type: New Feature
> Components: java
> Reporter: Raymie Stata
> Assignee: Raymie Stata
> Labels: features
> Fix For: 1.7.0
>
> Attachments: AVRO-1006-prelim.patch, AVRO-1006.patch, AVRO-1006.patch, AVRO-1006.patch, AVRO-1006.patch, AVRO-1006.patch, schema-fingerprinting.html, schema-fingerprinting.html, schema-fingerprinting.html
>
>
> Add function that returns a standardized, 64-bit fingerprint for schemas. Fingerprints are designed such that the chances of collisions is very, very low.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1006) Fingerprints for Avro Schemas
Posted by "Scott Carey (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-1006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13203256#comment-13203256 ]
Scott Carey commented on AVRO-1006:
-----------------------------------
{quote}
While representing the canonical schema as Avro data reduces it (compared to Json representation) it does not eliminate ambiguity. Non-empty arrays (and maps) can be represented in Avro in more than one way.
{quote}
Can you provide an example? I am having trouble thinking of an example that doesn't fall under the other disambiguations in the document, e.g. {"type":"int"} == "int". Can we have an avro serialized canonical form without the ambiguity? The document describes two forms of normailzation: the Avro normalization, and the JSON normalization. I don't think that something that has undergone the Avro normalization can have ambiguous array definition. If so that would break both the JSON string fingerprint case as well as the avro binary fingerprint. I am suggesting we avoid the JSON normalization and its dependency on JSON serializers that support ordering by using an avro binary representation for the input to a hash. Both cases require the Avro normalization component.
It might be useful to ALWAYS store schemas in memory in normalized form -- with attributes and doc represented and attached separately. The PRIMITIVES, FULLNAME, and MINIMIZE optimizatoins can always be applied, the STRIP optimization can be trivial by using a shared canonical schema. For example, a Schema can have a CanonicalSchema member variable, plus attributes and doc. Two Schema.Fixed that only vary on their doc or attributes would share thee same CanonicalSchema. Is it ever useful to know that two schemas differ only due to FULLNAME, MINIMIZE, or PRIMITIVE expansion?
> Fingerprints for Avro Schemas
> -----------------------------
>
> Key: AVRO-1006
> URL: https://issues.apache.org/jira/browse/AVRO-1006
> Project: Avro
> Issue Type: New Feature
> Components: java
> Reporter: Raymie Stata
> Assignee: Raymie Stata
> Labels: features
> Attachments: schema-fingerprinting.html, schema-fingerprinting.html, schema-fingerprinting.html
>
>
> Add function that returns a standardized, 64-bit fingerprint for schemas. Fingerprints are designed such that the chances of collisions is very, very low.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1006) Fingerprints for Avro Schemas
Posted by "Thiruvalluvan M. G. (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-1006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13203274#comment-13203274 ]
Thiruvalluvan M. G. commented on AVRO-1006:
-------------------------------------------
bq. Can you provide an example?
A three string array can be encoded in Avro binary as:
{{<1><string1><1><string2><1><string3><0>}} or {{<3><string1><string2><string3><0>}}
So, when you and encode the fields of a record as an array, the binary version of the schema can have more than one valid representations which are equivalent but not not equal. The fingerprints could thus be different.
> Fingerprints for Avro Schemas
> -----------------------------
>
> Key: AVRO-1006
> URL: https://issues.apache.org/jira/browse/AVRO-1006
> Project: Avro
> Issue Type: New Feature
> Components: java
> Reporter: Raymie Stata
> Assignee: Raymie Stata
> Labels: features
> Attachments: schema-fingerprinting.html, schema-fingerprinting.html, schema-fingerprinting.html
>
>
> Add function that returns a standardized, 64-bit fingerprint for schemas. Fingerprints are designed such that the chances of collisions is very, very low.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1006) Fingerprints for Avro Schemas
Posted by "Raymie Stata (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-1006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13194836#comment-13194836 ]
Raymie Stata commented on AVRO-1006:
------------------------------------
We believe that, in pub/sub and other scenarios, schema repositories will become a standard part of the Avro infrastructure. Given that, to promote an ecosystem with interoperable but alternative implementations, we believe that "common" (let's not call them "standard") ways to generate good, hash-based fingerprints will be useful. In particular, while standard, cryptographic hashes are good enough for fingerprints of size greater than 128 bits, we believe that 64-bit hashes represent a very useful point in the size/diversity tradeoff space, and there should be a common way in the Avro ecosystem for generating 64-bit hashes.
Does this make sense? If so, I think I'll re-write the explanation around this line of reasoning, which is indeed closer to our original though process.
All this being said, your point about schema numbers is a good one, I'll add a note about that technique on the section about schema repositories. It seems like the right protocol here is for the "inserter" of a schema to insert the schema and get an identifier back from the repository before using the schema. Is that how you do it?
> Fingerprints for Avro Schemas
> -----------------------------
>
> Key: AVRO-1006
> URL: https://issues.apache.org/jira/browse/AVRO-1006
> Project: Avro
> Issue Type: New Feature
> Components: java
> Reporter: Raymie Stata
> Assignee: Raymie Stata
> Labels: features
> Attachments: schema-fingerprinting.html, schema-fingerprinting.html, schema-fingerprinting.html
>
>
> Add function that returns a standardized, 64-bit fingerprint for schemas. Fingerprints are designed such that the chances of collisions is very, very low.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1006) Fingerprints for Avro Schemas
Posted by "Doug Cutting (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-1006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13214932#comment-13214932 ]
Doug Cutting commented on AVRO-1006:
------------------------------------
This looks generally good. A few nits:
- In class and method names, should we abbreviate 'fp' or spell out 'fingerprint'? FP means "floating point" to my eye.
- Might we instead put this in org.apache.avro.SchemaFingerprint, rather than in util? Right now things in the util package depend only on the JDK, not on other parts of Avro.
- Public methods and classes need javadoc comments.
- The changes to the spec are not correctly processed by Forrest 0.8 for me.
> Fingerprints for Avro Schemas
> -----------------------------
>
> Key: AVRO-1006
> URL: https://issues.apache.org/jira/browse/AVRO-1006
> Project: Avro
> Issue Type: New Feature
> Components: java
> Reporter: Raymie Stata
> Assignee: Raymie Stata
> Labels: features
> Attachments: AVRO-1006-prelim.patch, AVRO-1006.patch, schema-fingerprinting.html, schema-fingerprinting.html, schema-fingerprinting.html
>
>
> Add function that returns a standardized, 64-bit fingerprint for schemas. Fingerprints are designed such that the chances of collisions is very, very low.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1006) Fingerprints for Avro Schemas
Posted by "Eric Hauser (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-1006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13203202#comment-13203202 ]
Eric Hauser commented on AVRO-1006:
-----------------------------------
I believe LinkedIn has already written a schema service for Avro that there were interested in open sourcing at one point. Someone with contacts on the Kafka team might want to contact them and see if they are able to do so.
> Fingerprints for Avro Schemas
> -----------------------------
>
> Key: AVRO-1006
> URL: https://issues.apache.org/jira/browse/AVRO-1006
> Project: Avro
> Issue Type: New Feature
> Components: java
> Reporter: Raymie Stata
> Assignee: Raymie Stata
> Labels: features
> Attachments: schema-fingerprinting.html, schema-fingerprinting.html, schema-fingerprinting.html
>
>
> Add function that returns a standardized, 64-bit fingerprint for schemas. Fingerprints are designed such that the chances of collisions is very, very low.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AVRO-1006) Fingerprints for Avro Schemas
Posted by "Raymie Stata (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-1006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raymie Stata updated AVRO-1006:
-------------------------------
Attachment: AVRO-1006.patch
Renamed methods as suggested by Doug (convenience method was already in there).
> Fingerprints for Avro Schemas
> -----------------------------
>
> Key: AVRO-1006
> URL: https://issues.apache.org/jira/browse/AVRO-1006
> Project: Avro
> Issue Type: New Feature
> Components: java
> Reporter: Raymie Stata
> Assignee: Raymie Stata
> Labels: features
> Attachments: AVRO-1006-prelim.patch, AVRO-1006.patch, AVRO-1006.patch, AVRO-1006.patch, AVRO-1006.patch, schema-fingerprinting.html, schema-fingerprinting.html, schema-fingerprinting.html
>
>
> Add function that returns a standardized, 64-bit fingerprint for schemas. Fingerprints are designed such that the chances of collisions is very, very low.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1006) Fingerprints for Avro Schemas
Posted by "Raymie Stata (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-1006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13203426#comment-13203426 ]
Raymie Stata commented on AVRO-1006:
------------------------------------
A clarification, which addresses issues raised by Doug and Scott. The need I'm solving for is to capture that part of a writer's schema which a reader needs to read data. This is a relatively straight-forward notion of "equivalence," and a very useful one. And the good news is that this notion of equivalence allows us to ignore many aspects of schemas (e.g., attributes, aliases, default values).
I've attached code that generates the canonical string as defined in the design document. The attached code is untested, but I included it so folks have a rough idea of what it would look like. Leaving aside my APL-ish style, I don't think it's terribly long or complicated.
> Fingerprints for Avro Schemas
> -----------------------------
>
> Key: AVRO-1006
> URL: https://issues.apache.org/jira/browse/AVRO-1006
> Project: Avro
> Issue Type: New Feature
> Components: java
> Reporter: Raymie Stata
> Assignee: Raymie Stata
> Labels: features
> Attachments: AVRO-1006-prelim.patch, schema-fingerprinting.html, schema-fingerprinting.html, schema-fingerprinting.html
>
>
> Add function that returns a standardized, 64-bit fingerprint for schemas. Fingerprints are designed such that the chances of collisions is very, very low.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1006) Fingerprints for Avro Schemas
Posted by "Raymie Stata (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-1006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13193699#comment-13193699 ]
Raymie Stata commented on AVRO-1006:
------------------------------------
Sam writes:{quote}On the 0.001% collision rate, that seems high to me - would a 128-bit hash be a better choice?{quote}
Thanks for pointing this out. Turns out the 0.001% is a bug in the writeup, the actual probabilities are quite a bit lower: 3E-8 (0.000003%) for a million-item cache, 3E-10 for 100K items, and 3E-12 for 10K items (I'd love to have someone check my math). Assuming an insertion per minute into a fixed-sized table (ie, random eviction), you'd expect a collision every year with the 1M item cache, every century with 100K items, and every millennia with 10K items. This seems acceptable, especially since I expect these caches to be closer to 10K items than a million (there's a bit of a discussion on this point in the updated writeup). So are you happier now with 64 bits?
(The doc defines a canonical text for schemas, and fingerprints based that text. The patch will contain a function for returning the canonical text. This approach implicitly standardizes how one would take an MD5-or SHA-xxx fingerprint of a schema, but perhaps I can be explicit on this point.)
> Fingerprints for Avro Schemas
> -----------------------------
>
> Key: AVRO-1006
> URL: https://issues.apache.org/jira/browse/AVRO-1006
> Project: Avro
> Issue Type: New Feature
> Components: java
> Reporter: Raymie Stata
> Assignee: Raymie Stata
> Labels: features
> Attachments: schema-fingerprinting.html, schema-fingerprinting.html
>
>
> Add function that returns a standardized, 64-bit fingerprint for schemas. Fingerprints are designed such that the chances of collisions is very, very low.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1006) Fingerprints for Avro Schemas
Posted by "Raymie Stata (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-1006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13192909#comment-13192909 ]
Raymie Stata commented on AVRO-1006:
------------------------------------
An _Avro schema fingerprint_ is a hash of an Avro schema. Within a collection of even a million schemas, the probability of a collision is still around 0.001%. Thus, fingerprints can be used in place of schemas.
One motivating use-case for fingerprints is a pub/sub message bus. On a pub/sub bus, since multiple writers can publish to the same topic using different schemas, each message must be associated with its schema. Rather than include the actual schema with every message, one can instead include the fingerprint of the schema, which would be smaller. When a reader encounters a fingerprint it hasn't seen before, it can look it up and cache it. (The attached document describes possible lookup mechanisms.)
The proposed approach to fingerprinting is pretty straight forward. First, we convert Avro schemas into a _canonical form._ Two equivalent schemas always have the same canonical form. Once we have the canonical form, we simply take a 64-bit "Rabin fingerprint" (a CRC) of that text.
> Fingerprints for Avro Schemas
> -----------------------------
>
> Key: AVRO-1006
> URL: https://issues.apache.org/jira/browse/AVRO-1006
> Project: Avro
> Issue Type: New Feature
> Components: java
> Reporter: Raymie Stata
> Assignee: Raymie Stata
> Labels: features
> Attachments: schema-fingerprinting.html
>
>
> Add function that returns a standardized, 64-bit fingerprint that can be used as a key in various contexts.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1006) Fingerprints for Avro Schemas
Posted by "Raymie Stata (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-1006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13194420#comment-13194420 ]
Raymie Stata commented on AVRO-1006:
------------------------------------
I fixed the math bug in the writeup (I misapplied the "birthday probability" where I should've applied a much more favorable probability). The bottom line is 64 bits seem adequate for most use-cases. For use-cases that want longer fingerprints, the spec now explicitly recommends using MD5 or the SHA-family to the canonical text of a schema.
I'm happy to continue discussing the overall proposal, but my next step here is to file patches for the "surrounding issues" (e.g., the spec fixes).
> Fingerprints for Avro Schemas
> -----------------------------
>
> Key: AVRO-1006
> URL: https://issues.apache.org/jira/browse/AVRO-1006
> Project: Avro
> Issue Type: New Feature
> Components: java
> Reporter: Raymie Stata
> Assignee: Raymie Stata
> Labels: features
> Attachments: schema-fingerprinting.html, schema-fingerprinting.html, schema-fingerprinting.html
>
>
> Add function that returns a standardized, 64-bit fingerprint for schemas. Fingerprints are designed such that the chances of collisions is very, very low.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AVRO-1006) Fingerprints for Avro Schemas
Posted by "Raymie Stata (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-1006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raymie Stata updated AVRO-1006:
-------------------------------
Attachment: schema-fingerprinting.html
Detailed description of the proposed feature.
> Fingerprints for Avro Schemas
> -----------------------------
>
> Key: AVRO-1006
> URL: https://issues.apache.org/jira/browse/AVRO-1006
> Project: Avro
> Issue Type: New Feature
> Components: java
> Reporter: Raymie Stata
> Assignee: Raymie Stata
> Labels: features
> Attachments: schema-fingerprinting.html
>
>
> Add function that returns a standardized, 64-bit fingerprint that can be used as a key in various contexts.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1006) Fingerprints for Avro Schemas
Posted by "Doug Cutting (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-1006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13203082#comment-13203082 ]
Doug Cutting commented on AVRO-1006:
------------------------------------
Some notes:
- Primitive types may have attributes, e.g., {"type":"int", "java-class":"java.lang.Short"}, so only primitives without any attributes may be represented by their name alone.
- Attributes within JSON objects are not ordered. A correct JSON parser need not preserve ordering. Relying on order-preservation may require some implementations to write their own JSON libraries.
- With multiple Avro implementations, the chance of an inconsistent canonicalization implementation is significant. Creating an adequate test suite and validating all implementations would require significant effort.
Given the above, I'd be hesitant to build a system that depends on consistent canonical schemas for correct operation. Folks who build systems that use Avro would thus be wise to design them to gracefully handle inconsistent canonicalization. For example, Avro's RPC handshake currently uses a fingerprint-like approach without requiring canonicalization. Two implementations that represent a schema using the same string will have more efficient handshakes, but implementations that produce different strings for equivalent schemas will still interoperate correctly. So a standard, recommended canonical form could be useful, but folks should perhaps not assume that every implementation is correct.
I like the idea of a schema repository. A related idea I've had is to use something like a URL shortener. Instead of mapping url->url, it could map url->schema. One would register one's schema with the shortener, then hand out references. A shortener would, as an optimization, return the same ID for equivalent schemas. The shortener would only need to rely on only a single canonicalization implementation, its own.
> Fingerprints for Avro Schemas
> -----------------------------
>
> Key: AVRO-1006
> URL: https://issues.apache.org/jira/browse/AVRO-1006
> Project: Avro
> Issue Type: New Feature
> Components: java
> Reporter: Raymie Stata
> Assignee: Raymie Stata
> Labels: features
> Attachments: schema-fingerprinting.html, schema-fingerprinting.html, schema-fingerprinting.html
>
>
> Add function that returns a standardized, 64-bit fingerprint for schemas. Fingerprints are designed such that the chances of collisions is very, very low.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1006) Fingerprints for Avro Schemas
Posted by "Scott Carey (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-1006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13203230#comment-13203230 ]
Scott Carey commented on AVRO-1006:
-----------------------------------
{quote}
The parser assumes that names are defined before they are used, although this is not required by the specification. We recommend that the spec be changed to agree with the impl (i.e., require that names are defined before they are used).
{quote}
I think we can fix the parser. I have been thinking about how to implement Schema/Field/Protocol as immutable data structures and a requirement for that is to prevent cyclic references in the Schema objects, which requires storing name references and a name based schema registry -- the same tools needed for such a parser.
{quote}
When a schema name is redefined, the parser throws an exception, even if the two definitions of the name are the same. This is contrary to the spec, which says "A schema may only contain multiple definitions of a fullname if the definitions are equivalent." We recommend that the spec be changed to agree with the implementation (i.e., disallow multiple definitions of the same name, even if the def's are the same).
{quote}
I think this is a decent spec change, especiallyl since "if the definitions are equivalent" is insufficiently defined currently -- equivalent at what level? Including all metadata, even 'doc'? It is probably best if a single schema definition does not re-define any named schema elements.
{quote}
The parser calls validateName on the symbols of an enumeration, restricting the syntax of enumeration symbols. The spec does not call for such a restriction. We recommend that the spec be changed to conform to the implementation (i.e., restrict symbols the same way we restrict names). This helps in cannonicalization (don't have to worry about Unicode normalization).
{quote}
I wonder how various language implementations deal with this currently, it would not surprise me if more than Java already have an implicit restriction beyond the spec. We should change the spec to restrict symbols to the same restriction as field names.
{quote}
Schema.validateName uses Character.isLetter and Character.isLetterOrDigit to test characters. These accept all Unicode characters (except supplemental ones). The Avro spec says that names should be restricted to ASCII letters. We think this is an implementation bug and should be fixed. (Again, nice to avoid Unicode normalization.)
{quote}
I agree.
>From the spec:
{quote}
h3. Names
Record, enums and fixed are named types. Each has a fullname that is composed of two parts; a name and a namespace. Equality of names is defined on the fullname.
The name portion of a fullname, record field names, and enum symbols must:
* start with [A-Za-z_]
* subsequently contain only [A-Za-z0-9_]
A namespace is a dot-separated sequence of such names.
{quote}
{quote}
When the parser descends into a named schema, the default namespace in the names variable is stored into the local variable savedSpace, which is restored on exit. However, if the routine exits abruptly (with an exception), this restoration does not occur. This is probably a bug, and restoration should be in a finally. (In Parser.parse, the flag validateNames is restored in a finally clause.)
{quote}
Sounds like a bug, a patch containing a reproducible test case that fails and a fix for another ticket would be great!
> Fingerprints for Avro Schemas
> -----------------------------
>
> Key: AVRO-1006
> URL: https://issues.apache.org/jira/browse/AVRO-1006
> Project: Avro
> Issue Type: New Feature
> Components: java
> Reporter: Raymie Stata
> Assignee: Raymie Stata
> Labels: features
> Attachments: schema-fingerprinting.html, schema-fingerprinting.html, schema-fingerprinting.html
>
>
> Add function that returns a standardized, 64-bit fingerprint for schemas. Fingerprints are designed such that the chances of collisions is very, very low.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1006) Fingerprints for Avro Schemas
Posted by "Sam Pullara (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-1006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13194934#comment-13194934 ]
Sam Pullara commented on AVRO-1006:
-----------------------------------
It seems that the protocol for any system, even one using hashes, would have to follow that same protocol even if the "inserter" also acts as the repository. The only critical thing is that there is some way to generate a unique identifier and store a schema with that as the key that can be looked up by any receiver of the message. Only after that is done can you send a message using that identifier (and expect it to be understood).
One advantage that using hashes have over my sequence number suggestion is that you can easily shard, federate and merge hash-based repositories, qualities that could be useful in practice. Also, looking up a schema by its canonical text rather than determining its existence in the repository through hash lookup would likely be more efficient.
> Fingerprints for Avro Schemas
> -----------------------------
>
> Key: AVRO-1006
> URL: https://issues.apache.org/jira/browse/AVRO-1006
> Project: Avro
> Issue Type: New Feature
> Components: java
> Reporter: Raymie Stata
> Assignee: Raymie Stata
> Labels: features
> Attachments: schema-fingerprinting.html, schema-fingerprinting.html, schema-fingerprinting.html
>
>
> Add function that returns a standardized, 64-bit fingerprint for schemas. Fingerprints are designed such that the chances of collisions is very, very low.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1006) Fingerprints for Avro Schemas
Posted by "Doug Cutting (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-1006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13221342#comment-13221342 ]
Doug Cutting commented on AVRO-1006:
------------------------------------
Raymie> SchemaFingerprint.fingerprint seems unnecessarily long...
Now this becomes SchemaNormalization.fp(SchemaNormalization.toParsingForm(schema)). The 'fp' might better be spelled out as 'fingerprint'. Also a utility method like SchemaNormalization.parsingFingerprint(schema) might be useful.
Graham> pass a Normalizer instance...
With the latest API, someone can already call SchemaNormalization.fingerprint() with a differently normalized schema, so I don't see the need for this. As we add more normalizers to Avro we can add new methods, so I'm not (yet) seeing the advantage of adding a Normalization interface.
> Fingerprints for Avro Schemas
> -----------------------------
>
> Key: AVRO-1006
> URL: https://issues.apache.org/jira/browse/AVRO-1006
> Project: Avro
> Issue Type: New Feature
> Components: java
> Reporter: Raymie Stata
> Assignee: Raymie Stata
> Labels: features
> Attachments: AVRO-1006-prelim.patch, AVRO-1006.patch, AVRO-1006.patch, AVRO-1006.patch, schema-fingerprinting.html, schema-fingerprinting.html, schema-fingerprinting.html
>
>
> Add function that returns a standardized, 64-bit fingerprint for schemas. Fingerprints are designed such that the chances of collisions is very, very low.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1006) Fingerprints for Avro Schemas
Posted by "Raymie Stata (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-1006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13192915#comment-13192915 ]
Raymie Stata commented on AVRO-1006:
------------------------------------
In implementing canonical forms, we identified some bugs (both spec and impl bugs) that it would be nice to resolve:
* The parser assumes that names are defined before they are used, although this is not required by the specification. We recommend that the spec be changed to agree with the impl (i.e., require that names are defined before they are used).
* When a schema name is redefined, the parser throws an exception, even if the two definitions of the name are the same. This is contrary to the spec, which says "A schema may only contain multiple definitions of a fullname if the definitions are equivalent." We recommend that the spec be changed to agree with the implementation (i.e., disallow multiple definitions of the same name, even if the def's are the same).
* The parser calls {{validateName}} on the symbols of an enumeration, restricting the syntax of enumeration symbols. The spec does not call for such a restriction. We recommend that the spec be changed to conform to the implementation (i.e., restrict symbols the same way we restrict names). This helps in cannonicalization (don't have to worry about Unicode normalization).
* {{Schema.validateName}} uses {{Character.isLetter}} and {{Character.isLetterOrDigit}} to test characters. These accept all Unicode characters (except supplemental ones). The Avro spec says that names should be restricted to ASCII letters. We think this is an implementation bug and should be fixed. (Again, nice to avoid Unicode normalization.)
* When the parser descends into a named schema, the default namespace in the {{names}} variable is stored into the local variable {{savedSpace}}, which is restored on exit. However, if the routine exits abruptly (with an exception), this restoration does not occur. This is probably a bug, and restoration should be in a finally. (In {{Parser.parse}}, the flag {{validateNames}} is restored in a finally clause.)
Before submitting fixes, I'll file separate JIRAs, but I'd like to get any feedback here first.
> Fingerprints for Avro Schemas
> -----------------------------
>
> Key: AVRO-1006
> URL: https://issues.apache.org/jira/browse/AVRO-1006
> Project: Avro
> Issue Type: New Feature
> Components: java
> Reporter: Raymie Stata
> Assignee: Raymie Stata
> Labels: features
> Attachments: schema-fingerprinting.html
>
>
> Add function that returns a standardized, 64-bit fingerprint for schemas. Fingerprints are designed such that the chances of collisions is very, very low.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AVRO-1006) Fingerprints for Avro Schemas
Posted by "Raymie Stata (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-1006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raymie Stata updated AVRO-1006:
-------------------------------
Attachment: schema-fingerprinting.html
Major change: yet another fix to the text discussing the adequacy of 64-bit fingerprints. Also a bunch of editorial fixes.
> Fingerprints for Avro Schemas
> -----------------------------
>
> Key: AVRO-1006
> URL: https://issues.apache.org/jira/browse/AVRO-1006
> Project: Avro
> Issue Type: New Feature
> Components: java
> Reporter: Raymie Stata
> Assignee: Raymie Stata
> Labels: features
> Attachments: schema-fingerprinting.html, schema-fingerprinting.html, schema-fingerprinting.html
>
>
> Add function that returns a standardized, 64-bit fingerprint for schemas. Fingerprints are designed such that the chances of collisions is very, very low.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1006) Fingerprints for Avro Schemas
Posted by "Raymie Stata (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-1006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13221403#comment-13221403 ]
Raymie Stata commented on AVRO-1006:
------------------------------------
Yes, this is meant to be a cross-language feature, and the definition of "parsing normal form" is in the Avro spec as a result.
To aid in assuring interoperability, this patch includes file-based test data that should be used across languages. Test inputs and outputs can be found in share/test/data/schema-tests.txt
> Fingerprints for Avro Schemas
> -----------------------------
>
> Key: AVRO-1006
> URL: https://issues.apache.org/jira/browse/AVRO-1006
> Project: Avro
> Issue Type: New Feature
> Components: java
> Reporter: Raymie Stata
> Assignee: Raymie Stata
> Labels: features
> Attachments: AVRO-1006-prelim.patch, AVRO-1006.patch, AVRO-1006.patch, AVRO-1006.patch, AVRO-1006.patch, AVRO-1006.patch, schema-fingerprinting.html, schema-fingerprinting.html, schema-fingerprinting.html
>
>
> Add function that returns a standardized, 64-bit fingerprint for schemas. Fingerprints are designed such that the chances of collisions is very, very low.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1006) Fingerprints for Avro Schemas
Posted by "Scott Carey (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-1006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13203284#comment-13203284 ]
Scott Carey commented on AVRO-1006:
-----------------------------------
{quote}
A three string array can be encoded in Avro binary as:
<1><string1><1><string2><1><string3><0> or <3><string1><string2><string3><0>
{quote}
Oh, I see. You mean that the binary representation of an avro array has multiple forms. That is an issue. We could require that the single block form for arrays/maps is used. I believe this is easier than implementing and testing a conforming json stringification in every language.
> Fingerprints for Avro Schemas
> -----------------------------
>
> Key: AVRO-1006
> URL: https://issues.apache.org/jira/browse/AVRO-1006
> Project: Avro
> Issue Type: New Feature
> Components: java
> Reporter: Raymie Stata
> Assignee: Raymie Stata
> Labels: features
> Attachments: schema-fingerprinting.html, schema-fingerprinting.html, schema-fingerprinting.html
>
>
> Add function that returns a standardized, 64-bit fingerprint for schemas. Fingerprints are designed such that the chances of collisions is very, very low.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1006) Fingerprints for Avro Schemas
Posted by "Raymie Stata (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-1006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13214375#comment-13214375 ]
Raymie Stata commented on AVRO-1006:
------------------------------------
I've uploaded my patch. I've put the canonicalization and fingerprinting code in a new class, util.SchemaFP, rather than extending the already too-long Schema class.
> Fingerprints for Avro Schemas
> -----------------------------
>
> Key: AVRO-1006
> URL: https://issues.apache.org/jira/browse/AVRO-1006
> Project: Avro
> Issue Type: New Feature
> Components: java
> Reporter: Raymie Stata
> Assignee: Raymie Stata
> Labels: features
> Attachments: AVRO-1006-prelim.patch, AVRO-1006.patch, schema-fingerprinting.html, schema-fingerprinting.html, schema-fingerprinting.html
>
>
> Add function that returns a standardized, 64-bit fingerprint for schemas. Fingerprints are designed such that the chances of collisions is very, very low.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1006) Fingerprints for Avro Schemas
Posted by "graham sanderson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-1006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13219859#comment-13219859 ]
graham sanderson commented on AVRO-1006:
----------------------------------------
Raymie, thank you; I think we're in agreement. However the latest patch has class SchemaNormalization with static toParsingForm and static build methods
the other static methods reference these specific versions directly.
Perhaps it would be best (since the guts of the parsing form are in the current build method) to leave SchemaNormalization as a class of static methods, but pass a Normalizer instance to most of the static method... in this case the current build method would basically be the guts of the ParsingFormNormalizer class
Note the common Normalizer instances would likely be sensibly named singletons in the SchemaNormalizer class
> Fingerprints for Avro Schemas
> -----------------------------
>
> Key: AVRO-1006
> URL: https://issues.apache.org/jira/browse/AVRO-1006
> Project: Avro
> Issue Type: New Feature
> Components: java
> Reporter: Raymie Stata
> Assignee: Raymie Stata
> Labels: features
> Attachments: AVRO-1006-prelim.patch, AVRO-1006.patch, AVRO-1006.patch, AVRO-1006.patch, schema-fingerprinting.html, schema-fingerprinting.html, schema-fingerprinting.html
>
>
> Add function that returns a standardized, 64-bit fingerprint for schemas. Fingerprints are designed such that the chances of collisions is very, very low.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1006) Fingerprints for Avro Schemas
Posted by "Sam Pullara (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-1006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13193303#comment-13193303 ]
Sam Pullara commented on AVRO-1006:
-----------------------------------
This would be great. I ended up using the text representation + SHA256 in Avrobase for my schema keys. I would love to see a standard way that worked across languages and no matter how mangled the schema JSON was rendered. On the 0.001% collision rate, that seems high to me - would a 128-bit hash be a better choice?
> Fingerprints for Avro Schemas
> -----------------------------
>
> Key: AVRO-1006
> URL: https://issues.apache.org/jira/browse/AVRO-1006
> Project: Avro
> Issue Type: New Feature
> Components: java
> Reporter: Raymie Stata
> Assignee: Raymie Stata
> Labels: features
> Attachments: schema-fingerprinting.html
>
>
> Add function that returns a standardized, 64-bit fingerprint for schemas. Fingerprints are designed such that the chances of collisions is very, very low.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1006) Fingerprints for Avro Schemas
Posted by "Thiruvalluvan M. G. (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-1006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13203219#comment-13203219 ]
Thiruvalluvan M. G. commented on AVRO-1006:
-------------------------------------------
{quote}
Doug's point about JSON being an unordered format is important and limits using the json string as the fingerprint.
Perhaps we can complete the Avro Schema for schemas (AVRO-251) which can define field order and equivalence unambiguously and all implementations should be able to support. The output bytes from the Avro binary serialization of the schema can be used to feed a hash algorithm.
{quote}
While representing the canonical schema as Avro data reduces it (compared to Json representation) it does not eliminate ambiguity. Non-empty arrays (and maps) can be represented in Avro in more than one way.
Doug's observation implies that we cannot use a third-party Json library to generate the canonical representation. For fingerprinting to work, we need some canonical representation (which by definition is not ambiguous). Either we restrict (by removing ambiguities) an existing standard or invent a new one.
I think Raymie's canonicalization rules are simple and given that we'll have only US-ASCII characters in the canonical representation, writing a JSON generator in any language will not be hard. And it will be parsable (with no new code) and human-readable.
> Fingerprints for Avro Schemas
> -----------------------------
>
> Key: AVRO-1006
> URL: https://issues.apache.org/jira/browse/AVRO-1006
> Project: Avro
> Issue Type: New Feature
> Components: java
> Reporter: Raymie Stata
> Assignee: Raymie Stata
> Labels: features
> Attachments: schema-fingerprinting.html, schema-fingerprinting.html, schema-fingerprinting.html
>
>
> Add function that returns a standardized, 64-bit fingerprint for schemas. Fingerprints are designed such that the chances of collisions is very, very low.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1006) Fingerprints for Avro Schemas
Posted by "graham sanderson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-1006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13216630#comment-13216630 ]
graham sanderson commented on AVRO-1006:
----------------------------------------
Note, more succinctly, given that I and other people in this issue have been discussing or inferring schema repositories, I think maybe it makes sense in general or at least as a separate issue to deal with a digest of the schema in its entirety, even if that results in a number of hash values mapping to equivalent schemas. Generally I think the use cases don't involve insane quantities of schemas in the first place (although I won't say that for certain, because the ability to define an anonymous schema on the fly for certain applications is rather appealing)
> Fingerprints for Avro Schemas
> -----------------------------
>
> Key: AVRO-1006
> URL: https://issues.apache.org/jira/browse/AVRO-1006
> Project: Avro
> Issue Type: New Feature
> Components: java
> Reporter: Raymie Stata
> Assignee: Raymie Stata
> Labels: features
> Attachments: AVRO-1006-prelim.patch, AVRO-1006.patch, AVRO-1006.patch, schema-fingerprinting.html, schema-fingerprinting.html, schema-fingerprinting.html
>
>
> Add function that returns a standardized, 64-bit fingerprint for schemas. Fingerprints are designed such that the chances of collisions is very, very low.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1006) Fingerprints for Avro Schemas
Posted by "Raymie Stata (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-1006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13221388#comment-13221388 ]
Raymie Stata commented on AVRO-1006:
------------------------------------
The string passed to the fingerprint method is in turn passed through to MessageDigest.getInstance. Using an enum here would prevent folks from accessing digest algo's in the underlying library. I suggest leaving that as-is.
Otherwise I'll fix the rest of these issues.
> Fingerprints for Avro Schemas
> -----------------------------
>
> Key: AVRO-1006
> URL: https://issues.apache.org/jira/browse/AVRO-1006
> Project: Avro
> Issue Type: New Feature
> Components: java
> Reporter: Raymie Stata
> Assignee: Raymie Stata
> Labels: features
> Attachments: AVRO-1006-prelim.patch, AVRO-1006.patch, AVRO-1006.patch, AVRO-1006.patch, AVRO-1006.patch, schema-fingerprinting.html, schema-fingerprinting.html, schema-fingerprinting.html
>
>
> Add function that returns a standardized, 64-bit fingerprint for schemas. Fingerprints are designed such that the chances of collisions is very, very low.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AVRO-1006) Fingerprints for Avro Schemas
Posted by "Raymie Stata (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-1006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raymie Stata updated AVRO-1006:
-------------------------------
Attachment: AVRO-1006.patch
> Fingerprints for Avro Schemas
> -----------------------------
>
> Key: AVRO-1006
> URL: https://issues.apache.org/jira/browse/AVRO-1006
> Project: Avro
> Issue Type: New Feature
> Components: java
> Reporter: Raymie Stata
> Assignee: Raymie Stata
> Labels: features
> Attachments: AVRO-1006-prelim.patch, AVRO-1006.patch, AVRO-1006.patch, AVRO-1006.patch, AVRO-1006.patch, AVRO-1006.patch, schema-fingerprinting.html, schema-fingerprinting.html, schema-fingerprinting.html
>
>
> Add function that returns a standardized, 64-bit fingerprint for schemas. Fingerprints are designed such that the chances of collisions is very, very low.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1006) Fingerprints for Avro Schemas
Posted by "Doug Cutting (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-1006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13221386#comment-13221386 ]
Doug Cutting commented on AVRO-1006:
------------------------------------
A few minor nits:
- the 'href' in one of the documentation links is missing its 'h'
- the WHITESPACE comment should perhaps read, "Eliminate all whitespace in JSON outside of string literals."
- we might define a nested FingerprintAlgorithm Enum for the implemented fingerprint algorithm names.
- SchemaNormalization should probably have a private constructor, e.g., 'private SchemaNormalization() {}'
- the #fingerprint link in the class documentation is broken.
Otherwise I'm +1 and look forward to committing this early next week unless there are objections.
> Fingerprints for Avro Schemas
> -----------------------------
>
> Key: AVRO-1006
> URL: https://issues.apache.org/jira/browse/AVRO-1006
> Project: Avro
> Issue Type: New Feature
> Components: java
> Reporter: Raymie Stata
> Assignee: Raymie Stata
> Labels: features
> Attachments: AVRO-1006-prelim.patch, AVRO-1006.patch, AVRO-1006.patch, AVRO-1006.patch, AVRO-1006.patch, schema-fingerprinting.html, schema-fingerprinting.html, schema-fingerprinting.html
>
>
> Add function that returns a standardized, 64-bit fingerprint for schemas. Fingerprints are designed such that the chances of collisions is very, very low.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1006) Fingerprints for Avro Schemas
Posted by "Doug Cutting (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-1006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13223428#comment-13223428 ]
Doug Cutting commented on AVRO-1006:
------------------------------------
Patch looks good. +1 I'll commit this unless there are objections.
> Fingerprints for Avro Schemas
> -----------------------------
>
> Key: AVRO-1006
> URL: https://issues.apache.org/jira/browse/AVRO-1006
> Project: Avro
> Issue Type: New Feature
> Components: java
> Reporter: Raymie Stata
> Assignee: Raymie Stata
> Labels: features
> Attachments: AVRO-1006-prelim.patch, AVRO-1006.patch, AVRO-1006.patch, AVRO-1006.patch, AVRO-1006.patch, AVRO-1006.patch, schema-fingerprinting.html, schema-fingerprinting.html, schema-fingerprinting.html
>
>
> Add function that returns a standardized, 64-bit fingerprint for schemas. Fingerprints are designed such that the chances of collisions is very, very low.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1006) Fingerprints for Avro Schemas
Posted by "Thiruvalluvan M. G. (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-1006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13223464#comment-13223464 ]
Thiruvalluvan M. G. commented on AVRO-1006:
-------------------------------------------
+1 Looks good to me.
> Fingerprints for Avro Schemas
> -----------------------------
>
> Key: AVRO-1006
> URL: https://issues.apache.org/jira/browse/AVRO-1006
> Project: Avro
> Issue Type: New Feature
> Components: java
> Reporter: Raymie Stata
> Assignee: Raymie Stata
> Labels: features
> Attachments: AVRO-1006-prelim.patch, AVRO-1006.patch, AVRO-1006.patch, AVRO-1006.patch, AVRO-1006.patch, AVRO-1006.patch, schema-fingerprinting.html, schema-fingerprinting.html, schema-fingerprinting.html
>
>
> Add function that returns a standardized, 64-bit fingerprint for schemas. Fingerprints are designed such that the chances of collisions is very, very low.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1006) Fingerprints for Avro Schemas
Posted by "Scott Carey (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-1006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13203237#comment-13203237 ]
Scott Carey commented on AVRO-1006:
-----------------------------------
{quote}
I think we can fix the parser. I have been thinking about how to implement Schema/Field/Protocol as immutable data structures and a requirement for that is to prevent cyclic references in the Schema objects, which requires storing name references and a name based schema registry – the same tools needed for such a parser.
{quote}
Although we could make the parser work with use-before-define, it is probably a good spec simplification to avoid it that will be easier to build and test across languages.
> Fingerprints for Avro Schemas
> -----------------------------
>
> Key: AVRO-1006
> URL: https://issues.apache.org/jira/browse/AVRO-1006
> Project: Avro
> Issue Type: New Feature
> Components: java
> Reporter: Raymie Stata
> Assignee: Raymie Stata
> Labels: features
> Attachments: schema-fingerprinting.html, schema-fingerprinting.html, schema-fingerprinting.html
>
>
> Add function that returns a standardized, 64-bit fingerprint for schemas. Fingerprints are designed such that the chances of collisions is very, very low.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1006) Fingerprints for Avro Schemas
Posted by "Raymie Stata (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-1006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13215480#comment-13215480 ]
Raymie Stata commented on AVRO-1006:
------------------------------------
> ['fp' or 'fingerprint']
I renamed classes to "fingerprint," but left methods "fp." SchemaFingerprint.fingerprint seems unnecessarily long...
> [put in o.a.avro]
Good idea -- I moved it.
> [missing javadoc]
Added
> [breaks Forrest]
I can't get Forrest running on my machine -- can you send me the errors? (Or maybe just fix them?)
> Fingerprints for Avro Schemas
> -----------------------------
>
> Key: AVRO-1006
> URL: https://issues.apache.org/jira/browse/AVRO-1006
> Project: Avro
> Issue Type: New Feature
> Components: java
> Reporter: Raymie Stata
> Assignee: Raymie Stata
> Labels: features
> Attachments: AVRO-1006-prelim.patch, AVRO-1006.patch, AVRO-1006.patch, schema-fingerprinting.html, schema-fingerprinting.html, schema-fingerprinting.html
>
>
> Add function that returns a standardized, 64-bit fingerprint for schemas. Fingerprints are designed such that the chances of collisions is very, very low.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1006) Fingerprints for Avro Schemas
Posted by "graham sanderson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-1006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13216629#comment-13216629 ]
graham sanderson commented on AVRO-1006:
----------------------------------------
"A clarification, which addresses issues raised by Doug and Scott. The need I'm solving for is to capture that part of a writer's schema which a reader needs to read data. This is a relatively straight-forward notion of "equivalence," and a very useful one. And the good news is that this notion of equivalence allows us to ignore many aspects of schemas (e.g., attributes, aliases, default values)."
Perhaps this should be made clearer (when naming the class/method), I came across this feature because of a desire to hash/fingerprint avro schemas for messaging, and was seeing if there was already a util to do it. In my case I potentially might use custom properties on fields in the schema to indicate they are being transmitted using a certain named dictionary and thus in my case they affect the ability to interpret the message, so I'd rather stick with something that I can reliably use on the producer end to encode the entire state of the schema, rather than a particular well defined sub-set of the schema.
Note that (thanks to someone making Props a LinkedHashMap since the code base I'm using) and the particular implementation of Jackson, schema.toString() in the Java impl appears like it will be fine for my purposes, and if another language implementation happens to produce a different hash value I'm cool with that, as long as it is relatively stable; for example:
SchemaInstance1 -toJson-> string x
string x -fromJson-> SchemInstance2 -> toJson string y
string x and string y being equal seems a reasonable enough guarantee for me
> Fingerprints for Avro Schemas
> -----------------------------
>
> Key: AVRO-1006
> URL: https://issues.apache.org/jira/browse/AVRO-1006
> Project: Avro
> Issue Type: New Feature
> Components: java
> Reporter: Raymie Stata
> Assignee: Raymie Stata
> Labels: features
> Attachments: AVRO-1006-prelim.patch, AVRO-1006.patch, AVRO-1006.patch, schema-fingerprinting.html, schema-fingerprinting.html, schema-fingerprinting.html
>
>
> Add function that returns a standardized, 64-bit fingerprint for schemas. Fingerprints are designed such that the chances of collisions is very, very low.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AVRO-1006) Fingerprints for Avro Schemas
Posted by "Raymie Stata (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-1006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raymie Stata updated AVRO-1006:
-------------------------------
Attachment: schema-fingerprinting.html
Updated design doc based on feedback:
* Fixed typos and clarified some text.
* Added suggestion that the spec be explicit about case-sensitivity.
* Most significantly, fixed error in stated probability of collisions, and added a bunch more text around that issue.
> Fingerprints for Avro Schemas
> -----------------------------
>
> Key: AVRO-1006
> URL: https://issues.apache.org/jira/browse/AVRO-1006
> Project: Avro
> Issue Type: New Feature
> Components: java
> Reporter: Raymie Stata
> Assignee: Raymie Stata
> Labels: features
> Attachments: schema-fingerprinting.html, schema-fingerprinting.html
>
>
> Add function that returns a standardized, 64-bit fingerprint for schemas. Fingerprints are designed such that the chances of collisions is very, very low.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1006) Fingerprints for Avro Schemas
Posted by "Raymie Stata (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-1006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13193941#comment-13193941 ]
Raymie Stata commented on AVRO-1006:
------------------------------------
Sigh. My math is wrong. I think things are better than stated, but I need to think about this more.
> Fingerprints for Avro Schemas
> -----------------------------
>
> Key: AVRO-1006
> URL: https://issues.apache.org/jira/browse/AVRO-1006
> Project: Avro
> Issue Type: New Feature
> Components: java
> Reporter: Raymie Stata
> Assignee: Raymie Stata
> Labels: features
> Attachments: schema-fingerprinting.html, schema-fingerprinting.html
>
>
> Add function that returns a standardized, 64-bit fingerprint for schemas. Fingerprints are designed such that the chances of collisions is very, very low.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AVRO-1006) Fingerprints for Avro Schemas
Posted by "Raymie Stata (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-1006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raymie Stata updated AVRO-1006:
-------------------------------
Attachment: AVRO-1006.patch
> Fingerprints for Avro Schemas
> -----------------------------
>
> Key: AVRO-1006
> URL: https://issues.apache.org/jira/browse/AVRO-1006
> Project: Avro
> Issue Type: New Feature
> Components: java
> Reporter: Raymie Stata
> Assignee: Raymie Stata
> Labels: features
> Attachments: AVRO-1006-prelim.patch, AVRO-1006.patch, schema-fingerprinting.html, schema-fingerprinting.html, schema-fingerprinting.html
>
>
> Add function that returns a standardized, 64-bit fingerprint for schemas. Fingerprints are designed such that the chances of collisions is very, very low.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AVRO-1006) Fingerprints for Avro Schemas
Posted by "Raymie Stata (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-1006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raymie Stata updated AVRO-1006:
-------------------------------
Attachment: AVRO-1006-prelim.patch
> Fingerprints for Avro Schemas
> -----------------------------
>
> Key: AVRO-1006
> URL: https://issues.apache.org/jira/browse/AVRO-1006
> Project: Avro
> Issue Type: New Feature
> Components: java
> Reporter: Raymie Stata
> Assignee: Raymie Stata
> Labels: features
> Attachments: AVRO-1006-prelim.patch, schema-fingerprinting.html, schema-fingerprinting.html, schema-fingerprinting.html
>
>
> Add function that returns a standardized, 64-bit fingerprint for schemas. Fingerprints are designed such that the chances of collisions is very, very low.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1006) Fingerprints for Avro Schemas
Posted by "Sam Pullara (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-1006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13194460#comment-13194460 ]
Sam Pullara commented on AVRO-1006:
-----------------------------------
I think the specification should really separate the canonicalization and the hashing stuff. Enumerating various ways of hashing the result of the canonical form doesn't really seem necessary since in any of the cases you have listed you could just as easily use a monotonically increasing number for the schema -- as long as your central repository or other schema provider lets you look up the number with the canonical schema text. This was the design I ultimately went with for Avrobase on top of MySQL since it is easy to lookup a row by a value of a column.
> Fingerprints for Avro Schemas
> -----------------------------
>
> Key: AVRO-1006
> URL: https://issues.apache.org/jira/browse/AVRO-1006
> Project: Avro
> Issue Type: New Feature
> Components: java
> Reporter: Raymie Stata
> Assignee: Raymie Stata
> Labels: features
> Attachments: schema-fingerprinting.html, schema-fingerprinting.html, schema-fingerprinting.html
>
>
> Add function that returns a standardized, 64-bit fingerprint for schemas. Fingerprints are designed such that the chances of collisions is very, very low.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1006) Fingerprints for Avro Schemas
Posted by "Scott Carey (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-1006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13203181#comment-13203181 ]
Scott Carey commented on AVRO-1006:
-----------------------------------
More notes:
* Schema equivalence has a few variations
** Serialization equivalent -- attribute metadata is irrelevant, {"type":"int", "java-class":"java.lang.Short"} is equal to {"int"}. Defaults and doc are also irrelevant for this case.
** Serialization and metadata equivalence, where the above two are not equivalent.
** Reversible transformation equivalence, e.g. ["int", "string"] equals ["string", "int], or records with pure field reordering.
* Other schema relationships that are related to equivalence but cannot satisfy associativity and transitivity
** Alias equivalence is not transitive, but is associative.
** Schema resolution and transformation is often neither transitive or associative.
All three equivalence variations above may be useful for different purposes, especially the first two. Serialization equivalence is important for long term storage. Full equivalence with metadata is often needed for internal state. But we may want to let users specify which optional components are included (attributes, defaults, doc). Doug's point about JSON being an unordered format is important and limits using the json string as the fingerprint.
Perhaps we can complete the Avro Schema for schemas (AVRO-251) which can define field order and equivalence unambiguously and all implementations should be able to support. The output bytes from the Avro binary serialization of the schema can be used to feed a hash algorithm.
> Fingerprints for Avro Schemas
> -----------------------------
>
> Key: AVRO-1006
> URL: https://issues.apache.org/jira/browse/AVRO-1006
> Project: Avro
> Issue Type: New Feature
> Components: java
> Reporter: Raymie Stata
> Assignee: Raymie Stata
> Labels: features
> Attachments: schema-fingerprinting.html, schema-fingerprinting.html, schema-fingerprinting.html
>
>
> Add function that returns a standardized, 64-bit fingerprint for schemas. Fingerprints are designed such that the chances of collisions is very, very low.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AVRO-1006) Fingerprints for Avro Schemas
Posted by "Raymie Stata (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-1006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raymie Stata updated AVRO-1006:
-------------------------------
Attachment: AVRO-1006.patch
> Fingerprints for Avro Schemas
> -----------------------------
>
> Key: AVRO-1006
> URL: https://issues.apache.org/jira/browse/AVRO-1006
> Project: Avro
> Issue Type: New Feature
> Components: java
> Reporter: Raymie Stata
> Assignee: Raymie Stata
> Labels: features
> Attachments: AVRO-1006-prelim.patch, AVRO-1006.patch, AVRO-1006.patch, schema-fingerprinting.html, schema-fingerprinting.html, schema-fingerprinting.html
>
>
> Add function that returns a standardized, 64-bit fingerprint for schemas. Fingerprints are designed such that the chances of collisions is very, very low.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AVRO-1006) Fingerprints for Avro Schemas
Posted by "Raymie Stata (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-1006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raymie Stata updated AVRO-1006:
-------------------------------
Attachment: AVRO-1006.patch
> Fingerprints for Avro Schemas
> -----------------------------
>
> Key: AVRO-1006
> URL: https://issues.apache.org/jira/browse/AVRO-1006
> Project: Avro
> Issue Type: New Feature
> Components: java
> Reporter: Raymie Stata
> Assignee: Raymie Stata
> Labels: features
> Attachments: AVRO-1006-prelim.patch, AVRO-1006.patch, AVRO-1006.patch, AVRO-1006.patch, schema-fingerprinting.html, schema-fingerprinting.html, schema-fingerprinting.html
>
>
> Add function that returns a standardized, 64-bit fingerprint for schemas. Fingerprints are designed such that the chances of collisions is very, very low.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AVRO-1006) Fingerprints for Avro Schemas
Posted by "Raymie Stata (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-1006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raymie Stata updated AVRO-1006:
-------------------------------
Description: Add function that returns a standardized, 64-bit fingerprint for schemas. Fingerprints are designed such that the chances of collisions is very, very low. (was: Add function that returns a standardized, 64-bit fingerprint that can be used as a key in various contexts.)
> Fingerprints for Avro Schemas
> -----------------------------
>
> Key: AVRO-1006
> URL: https://issues.apache.org/jira/browse/AVRO-1006
> Project: Avro
> Issue Type: New Feature
> Components: java
> Reporter: Raymie Stata
> Assignee: Raymie Stata
> Labels: features
> Attachments: schema-fingerprinting.html
>
>
> Add function that returns a standardized, 64-bit fingerprint for schemas. Fingerprints are designed such that the chances of collisions is very, very low.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira