You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@avro.apache.org by "opwvhk (via GitHub)" <gi...@apache.org> on 2023/08/17 08:20:37 UTC
[GitHub] [avro] opwvhk opened a new pull request, #2448: AVRO-3833: Clarify uniqueness of names and aliases
opwvhk opened a new pull request, #2448:
URL: https://github.com/apache/avro/pull/2448
## What is the purpose of the change
Update the specification to explicitly state that names must be unique (within their scope), and that aliases are names too.
## Verifying this change
Please read and verify that the change matches reasonable expectations for implementations.
## Documentation
- Does this pull request introduce a new feature? (~yes~ / **no**)
- ~If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [avro] kojiromike commented on a diff in pull request #2448: AVRO-3833: [Spec] Clarify usage of names and aliases
Posted by "kojiromike (via GitHub)" <gi...@apache.org>.
kojiromike commented on code in PR #2448:
URL: https://github.com/apache/avro/pull/2448#discussion_r1297907053
##########
doc/content/en/docs/++version++/Specification/_index.md:
##########
@@ -180,9 +180,9 @@ For example, 16-byte quantity may be declared with:
```
### Names {#names}
-Record, enums and fixed are named types. Each has a fullname that is composed of two parts; a name and a namespace, separated by a dot. Equality of names is defined on the fullname.
+Record, enums and fixed are named types. Each has a fullname that is composed of two parts; a name and a namespace, separated by a dot. Equality of names is defined on the fullname: it is an error to specify two different types with the same name.
-Record fields and enum symbols have names as well (but no namespace). Equality of fields and enum symbols is defined on the name of the field/symbol within its scope (the record/enum that defines it). Fields and enum symbols across scopes are never equal.
+Record fields and enum symbols have names as well (but no namespace). Equality of field names and enum symbols is defined within their scope (the record/enum that defines them): it is an error to define multiple fields or enum symbols with the same name in a single type. Fields and enum symbols across scopes are never equal, so field names and enum symbols can be reused in a different type.
Review Comment:
```suggestion
Record fields and enum symbols have names as well (but no namespace). Equality of field names and enum symbols is defined within their scope (the record/enum that defines them). It is an error to define multiple fields or enum symbols with the same name in a single type. Fields and enum symbols across scopes are never equal, so field names and enum symbols can be reused in a different type.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [avro] KalleOlaviNiemitalo commented on a diff in pull request #2448: AVRO-3833: [Spec] Clarify usage of names and aliases
Posted by "KalleOlaviNiemitalo (via GitHub)" <gi...@apache.org>.
KalleOlaviNiemitalo commented on code in PR #2448:
URL: https://github.com/apache/avro/pull/2448#discussion_r1298037348
##########
doc/content/en/docs/++version++/Specification/_index.md:
##########
@@ -259,13 +259,29 @@ Complex types (`record`, `enum`, `array`, `map`, `fixed`) have no namespace, but
A schema or protocol may not contain multiple definitions of a fullname. Further, a name must be defined before it is used ("before" in the depth-first, left-to-right traversal of the JSON parse tree, where the types attribute of a protocol is always deemed to come "before" the messages attribute.)
-### Aliases
+### Aliases {#aliases}
Named types and fields may have aliases. An implementation may optionally use aliases to map a writer's schema to the reader's. This facilitates both schema evolution as well as processing disparate datasets.
Aliases function by re-writing the writer's schema using aliases from the reader's schema. For example, if the writer's schema was named "Foo" and the reader's schema is named "Bar" and has an alias of "Foo", then the implementation would act as though "Foo" were named "Bar" when reading. Similarly, if data was written as a record with a field named "x" and is read as a record with a field named "y" with alias "x", then the implementation would act as though "x" were named "y" when reading.
A type alias may be specified either as a fully namespace-qualified, or relative to the namespace of the name it is an alias for. For example, if a type named "a.b" has aliases of "c" and "x.y", then the fully qualified names of its aliases are "a.c" and "x.y".
+Aliases are alternative names, and thus subject to the same uniqueness constraints as names. Aliases should be valid names, but this is not required: any string is accepted as an alias. This allows schema evolution to correct illegal names in old schemata.
Review Comment:
My understanding from <https://avro.apache.org/docs/1.11.1/specification/#aliases> is that aliases cannot be referenced in a schema; they are only used for reading data that was written using an old schema that used those strings as names. But this conversion requires parsing the old schema, so if the names in that are not valid, then it can only be done with a library that tolerates such invalid names (in addition to tolerating those strings as aliases in the new schema).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [avro] martin-g commented on a diff in pull request #2448: AVRO-3833: [Spec] Clarify usage of names and aliases
Posted by "martin-g (via GitHub)" <gi...@apache.org>.
martin-g commented on code in PR #2448:
URL: https://github.com/apache/avro/pull/2448#discussion_r1298107195
##########
doc/content/en/docs/++version++/Specification/_index.md:
##########
@@ -259,13 +259,29 @@ Complex types (`record`, `enum`, `array`, `map`, `fixed`) have no namespace, but
A schema or protocol may not contain multiple definitions of a fullname. Further, a name must be defined before it is used ("before" in the depth-first, left-to-right traversal of the JSON parse tree, where the types attribute of a protocol is always deemed to come "before" the messages attribute.)
-### Aliases
+### Aliases {#aliases}
Review Comment:
What is this ?
Is it something for the url `#fragment` ? Because if it is then IMO there is no need of it - it already works: https://avro.apache.org/docs/1.11.1/specification/#aliases
##########
doc/content/en/docs/++version++/Specification/_index.md:
##########
@@ -180,9 +180,9 @@ For example, 16-byte quantity may be declared with:
```
### Names {#names}
-Record, enums and fixed are named types. Each has a fullname that is composed of two parts; a name and a namespace, separated by a dot. Equality of names is defined on the fullname.
+Record, enums and fixed are named types. Each has a fullname that is composed of two parts; a name and a namespace, separated by a dot. Equality of names is defined on the fullname: it is an error to specify two different types with the same name.
Review Comment:
I'd add spaces before and after the dash in @kojiromike's suggestion: `... fullname - it ...`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [avro] opwvhk merged pull request #2448: AVRO-3833: [Spec] Clarify usage of names and aliases
Posted by "opwvhk (via GitHub)" <gi...@apache.org>.
opwvhk merged PR #2448:
URL: https://github.com/apache/avro/pull/2448
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [avro] opwvhk commented on a diff in pull request #2448: AVRO-3833: [Spec] Clarify usage of names and aliases
Posted by "opwvhk (via GitHub)" <gi...@apache.org>.
opwvhk commented on code in PR #2448:
URL: https://github.com/apache/avro/pull/2448#discussion_r1314998731
##########
doc/content/en/docs/++version++/Specification/_index.md:
##########
@@ -259,13 +259,29 @@ Complex types (`record`, `enum`, `array`, `map`, `fixed`) have no namespace, but
A schema or protocol may not contain multiple definitions of a fullname. Further, a name must be defined before it is used ("before" in the depth-first, left-to-right traversal of the JSON parse tree, where the types attribute of a protocol is always deemed to come "before" the messages attribute.)
-### Aliases
+### Aliases {#aliases}
Named types and fields may have aliases. An implementation may optionally use aliases to map a writer's schema to the reader's. This facilitates both schema evolution as well as processing disparate datasets.
Aliases function by re-writing the writer's schema using aliases from the reader's schema. For example, if the writer's schema was named "Foo" and the reader's schema is named "Bar" and has an alias of "Foo", then the implementation would act as though "Foo" were named "Bar" when reading. Similarly, if data was written as a record with a field named "x" and is read as a record with a field named "y" with alias "x", then the implementation would act as though "x" were named "y" when reading.
A type alias may be specified either as a fully namespace-qualified, or relative to the namespace of the name it is an alias for. For example, if a type named "a.b" has aliases of "c" and "x.y", then the fully qualified names of its aliases are "a.c" and "x.y".
+Aliases are alternative names, and thus subject to the same uniqueness constraints as names. Aliases should be valid names, but this is not required: any string is accepted as an alias. This allows schema evolution to correct illegal names in old schemata.
Review Comment:
Correct. Code reading data using (potentially) old schemas should disable validations. If this does not happen, it's a bug.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [avro] kojiromike commented on a diff in pull request #2448: AVRO-3833: [Spec] Clarify usage of names and aliases
Posted by "kojiromike (via GitHub)" <gi...@apache.org>.
kojiromike commented on code in PR #2448:
URL: https://github.com/apache/avro/pull/2448#discussion_r1297906498
##########
doc/content/en/docs/++version++/Specification/_index.md:
##########
@@ -180,9 +180,9 @@ For example, 16-byte quantity may be declared with:
```
### Names {#names}
-Record, enums and fixed are named types. Each has a fullname that is composed of two parts; a name and a namespace, separated by a dot. Equality of names is defined on the fullname.
+Record, enums and fixed are named types. Each has a fullname that is composed of two parts; a name and a namespace, separated by a dot. Equality of names is defined on the fullname: it is an error to specify two different types with the same name.
Review Comment:
```suggestion
Record, enums and fixed are named types. Each has a fullname that is composed of two parts: a name and a namespace, separated by a dot. Equality of names is defined on the fullname–it is an error to specify two different types with the same name.
```
##########
doc/content/en/docs/++version++/Specification/_index.md:
##########
@@ -180,9 +180,9 @@ For example, 16-byte quantity may be declared with:
```
### Names {#names}
-Record, enums and fixed are named types. Each has a fullname that is composed of two parts; a name and a namespace, separated by a dot. Equality of names is defined on the fullname.
+Record, enums and fixed are named types. Each has a fullname that is composed of two parts; a name and a namespace, separated by a dot. Equality of names is defined on the fullname: it is an error to specify two different types with the same name.
Review Comment:
```suggestion
Record, enums and fixed are named types. Each has a fullname that is composed of two parts: a name and a namespace, separated by a dot. Equality of names is defined on the fullname–it is an error to specify two different types with the same name.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [avro] kojiromike commented on a diff in pull request #2448: AVRO-3833: [Spec] Clarify usage of names and aliases
Posted by "kojiromike (via GitHub)" <gi...@apache.org>.
kojiromike commented on code in PR #2448:
URL: https://github.com/apache/avro/pull/2448#discussion_r1297923222
##########
doc/content/en/docs/++version++/Specification/_index.md:
##########
@@ -259,13 +259,29 @@ Complex types (`record`, `enum`, `array`, `map`, `fixed`) have no namespace, but
A schema or protocol may not contain multiple definitions of a fullname. Further, a name must be defined before it is used ("before" in the depth-first, left-to-right traversal of the JSON parse tree, where the types attribute of a protocol is always deemed to come "before" the messages attribute.)
-### Aliases
+### Aliases {#aliases}
Named types and fields may have aliases. An implementation may optionally use aliases to map a writer's schema to the reader's. This facilitates both schema evolution as well as processing disparate datasets.
Aliases function by re-writing the writer's schema using aliases from the reader's schema. For example, if the writer's schema was named "Foo" and the reader's schema is named "Bar" and has an alias of "Foo", then the implementation would act as though "Foo" were named "Bar" when reading. Similarly, if data was written as a record with a field named "x" and is read as a record with a field named "y" with alias "x", then the implementation would act as though "x" were named "y" when reading.
A type alias may be specified either as a fully namespace-qualified, or relative to the namespace of the name it is an alias for. For example, if a type named "a.b" has aliases of "c" and "x.y", then the fully qualified names of its aliases are "a.c" and "x.y".
+Aliases are alternative names, and thus subject to the same uniqueness constraints as names. Aliases should be valid names, but this is not required: any string is accepted as an alias. This allows schema evolution to correct illegal names in old schemata.
Review Comment:
I sent [a note](https://lists.apache.org/thread/v5g14yt2p7sv08129xlwqtsx23p1qonc) to the dev list to understand this better.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [avro] opwvhk commented on a diff in pull request #2448: AVRO-3833: [Spec] Clarify usage of names and aliases
Posted by "opwvhk (via GitHub)" <gi...@apache.org>.
opwvhk commented on code in PR #2448:
URL: https://github.com/apache/avro/pull/2448#discussion_r1314983283
##########
doc/content/en/docs/++version++/Specification/_index.md:
##########
@@ -259,13 +259,29 @@ Complex types (`record`, `enum`, `array`, `map`, `fixed`) have no namespace, but
A schema or protocol may not contain multiple definitions of a fullname. Further, a name must be defined before it is used ("before" in the depth-first, left-to-right traversal of the JSON parse tree, where the types attribute of a protocol is always deemed to come "before" the messages attribute.)
-### Aliases
+### Aliases {#aliases}
Review Comment:
Good point. I copied this from other places where I saw a link being used. But I guess it's not needed in all cases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [avro] opwvhk commented on a diff in pull request #2448: AVRO-3833: [Spec] Clarify usage of names and aliases
Posted by "opwvhk (via GitHub)" <gi...@apache.org>.
opwvhk commented on code in PR #2448:
URL: https://github.com/apache/avro/pull/2448#discussion_r1314999611
##########
doc/content/en/docs/++version++/Specification/_index.md:
##########
@@ -259,13 +259,29 @@ Complex types (`record`, `enum`, `array`, `map`, `fixed`) have no namespace, but
A schema or protocol may not contain multiple definitions of a fullname. Further, a name must be defined before it is used ("before" in the depth-first, left-to-right traversal of the JSON parse tree, where the types attribute of a protocol is always deemed to come "before" the messages attribute.)
-### Aliases
+### Aliases {#aliases}
Review Comment:
I'll remove it, and also the explicit anchor `{#names}`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [avro] clesaec commented on a diff in pull request #2448: AVRO-3833: [Spec] Clarify usage of names and aliases
Posted by "clesaec (via GitHub)" <gi...@apache.org>.
clesaec commented on code in PR #2448:
URL: https://github.com/apache/avro/pull/2448#discussion_r1297081672
##########
doc/content/en/docs/++version++/Specification/_index.md:
##########
@@ -259,13 +259,29 @@ Complex types (`record`, `enum`, `array`, `map`, `fixed`) have no namespace, but
A schema or protocol may not contain multiple definitions of a fullname. Further, a name must be defined before it is used ("before" in the depth-first, left-to-right traversal of the JSON parse tree, where the types attribute of a protocol is always deemed to come "before" the messages attribute.)
-### Aliases
+### Aliases {#aliases}
Named types and fields may have aliases. An implementation may optionally use aliases to map a writer's schema to the reader's. This facilitates both schema evolution as well as processing disparate datasets.
Aliases function by re-writing the writer's schema using aliases from the reader's schema. For example, if the writer's schema was named "Foo" and the reader's schema is named "Bar" and has an alias of "Foo", then the implementation would act as though "Foo" were named "Bar" when reading. Similarly, if data was written as a record with a field named "x" and is read as a record with a field named "y" with alias "x", then the implementation would act as though "x" were named "y" when reading.
A type alias may be specified either as a fully namespace-qualified, or relative to the namespace of the name it is an alias for. For example, if a type named "a.b" has aliases of "c" and "x.y", then the fully qualified names of its aliases are "a.c" and "x.y".
+Aliases are alternative names, and thus subject to the same uniqueness constraints as names. Aliases should be valid names, but this is not required: any string is accepted as an alias. This allows schema evolution to correct illegal names in old schemata.
+
+## Fixing an invalid, but previously accepted, schema
+Over time, rules and validations on schemas have changed. It is therefore possible that a schema used to work with an older version of Avro, but now fails to parse.
+
+This can have several reasons, as listed below. Each reason also describes a fix, which can be applied using [schema resolution]({{< ref "#schema-resolution" >}}): you fix the problems in the schema in a way that is compatible, and then you can use the new schema to read the old data.
+
+### Invalid names
+Invalid names of types and fields can be corrected by renaming (using an [alias]({{< ref "#aliases" >}})). This works for simple names, namespaces and fullnames.
+
+Ths fix is twofold: first, you add the invalid name as an alias to the type/field. Then, you change the name to any valid name.
Review Comment:
**This** fix ... ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org