You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@avro.apache.org by Jonathan Slusher <jo...@gmail.com> on 2023/11/13 17:43:09 UTC
Hyphens in Avro schema namespaces and the rust deserializer
Hi,
I opened an issue in the AVRO project in Jira <https://issues.apache.org/jira/browse/AVRO-3900> and I’ve been asked to submit a topic for discussion to this email group.
See this issue in the rust schema_registry_converter repo <https://github.com/gklijs/schema_registry_converter/issues/100> for details specific to the crate in rust that we’re having trouble with:
A couple of things to point out here:
1. I understand that at this time the Avro spec does not allow hyphens in its namespaces, but somehow our registry is allowing them to be created from our Debezium connectors. We have been using the confluent_python <https://github.com/confluentinc/confluent-kafka-python> module and since version 1.9.2, its deserializer seems to handle these hyphens without error. We also have several JDBC sink connectors with consumer groups that are able to use these topics.
2. We recently attempted to implement a consumer written in rust and the crate <https://github.com/gklijs/schema_registry_converter> above, which is used for deserialization, throws an exception when attempting to connect to these topics.
```
thread 'main' panicked at /app/src/utils/kafka.rs:67:35:
Error decoding value: Error: Supplied raw value "{\"type\":\"record\",\"name\":\"Envelope\",\"namespace\":\"debezium.abc-123-efg-20231005.table.u_table_dbz\",\"fields\":[{\"name\":\"before\",\"type\":[\"null\",{\"type\":\"record\",\"name\":\"Value\",\"fields\":[{\"name\":\"id\",\"type\":\"int\"},{\"name\":\"uid\",\"type\":{\"type\":\"long\",\"connect.default\":0},\"default\":0},{\"name\":\"release_id\",\"type\":{\"type\":\"long\",\"connect.default\":0},\"default\":0},{\"name\":\"notes\",\"type\":\"string\"},{\"name\":\"notes_public\",\"type\":{\"type\":\"string\",\"connect.version\":1,\"connect.parameters\":{\"allowed\":\"Y,N\"},\"connect.default\":\"N\",\"connect.name\":\"io.debezium.data.Enum\"},\"default\":\"N\"},{\"name\":\"added_ts\",\"type\":{\"type\":\"long\",\"connect.version\":1,\"connect.name\":\"io.debezium.time.Timestamp\"}}],\"connect.name\":\"debezium.abc-123-efg-20231005.table.u_table_dbz.Value\"}],\"default\":null},{\"name\":\"after\",\"type\":[\"null\",\"Value\"],\"default\":null},{\"name\":\"source\",\"type\":{\"type\":\"record\",\"name\":\"Source\",\"namespace\":\"io.debezium.connector.mysql\",\"fields\":[{\"name\":\"version\",\"type\":\"string\"},{\"name\":\"connector\",\"type\":\"string\"},{\"name\":\"name\",\"type\":\"string\"},{\"name\":\"ts_ms\",\"type\":\"long\"},{\"name\":\"snapshot\",\"type\":[{\"type\":\"string\",\"connect.version\":1,\"connect.parameters\":{\"allowed\":\"true,last,false,incremental\"},\"connect.default\":\"false\",\"connect.name\":\"io.debezium.data.Enum\"},\"null\"],\"default\":\"false\"},{\"name\":\"db\",\"type\":\"string\"},{\"name\":\"sequence\",\"type\":[\"null\",\"string\"],\"default\":null},{\"name\":\"table\",\"type\":[\"null\",\"string\"],\"default\":null},{\"name\":\"server_id\",\"type\":\"long\"},{\"name\":\"gtid\",\"type\":[\"null\",\"string\"],\"default\":null},{\"name\":\"file\",\"type\":\"string\"},{\"name\":\"pos\",\"type\":\"long\"},{\"name\":\"row\",\"type\":\"int\"},{\"name\":\"thread\",\"type\":[\"null\",\"long\"],\"default\":null},{\"name\":\"query\",\"type\":[\"null\",\"string\"],\"default\":null}],\"connect.name\":\"io.debezium.connector.mysql.Source\"}},{\"name\":\"op\",\"type\":\"string\"},{\"name\":\"ts_ms\",\"type\":[\"null\",\"long\"],\"default\":null},{\"name\":\"transaction\",\"type\":[\"null\",{\"type\":\"record\",\"name\":\"block\",\"namespace\":\"event\",\"fields\":[{\"name\":\"id\",\"type\":\"string\"},{\"name\":\"total_order\",\"type\":\"long\"},{\"name\":\"data_collection_order\",\"type\":\"long\"}],\"connect.version\":1,\"connect.name\":\"event.block\"}],\"default\":null}],\"connect.version\":1,\"connect.name\":\"debezium.abc-123-efg-20231005.table.u_table_dbz.Envelope\"}" cant be turned into a Schema, was cause by Invalid namespace debezium.abc-123-efg-20231005.table.u_table_dbz. It must match the regex '^([A-Za-z_][A-Za-z0-9_]*(\.[A-Za-z_][A-Za-z0-9_]*)*)?$', it's retriable: false, it's cached: false
```
Ideally, the Avro spec would just accept hyphens since they’re a pretty common character and unavoidable in certain circumstances. If this is easier said than done, I think at the least any library used for Avro deserialization should account for them, including the rust library. If this works in Java and Python, shouldn’t it also work in rust?
Here’s a generic example of a schema created by a Debezium connector:
```
{
"type": "record",
"name": "Envelope",
"namespace": "abc-123-efg-20231005.table.u_table_dbz",
"fields": [
{
"name": "before",
"type": [
"null",
{
...
```
Please let me know if you need any more details, and thank you!
Jon Slusher
Re: Hyphens in Avro schema namespaces and the rust deserializer
Posted by Martin Grigorov <mg...@apache.org>.
Thanks for the comment, Oscar!
On Tue, Nov 21, 2023 at 5:06 PM Oscar Westra van Holthe - Kind <
oscar@westravanholthe.nl> wrote:
> Hi all,
>
> On the topic of allowed characters, we've settled in the (recent) past on
> alphanumeric characters only for simple names. Reasoning was that this is
> most portable.
>
> Remembering that, allowing the use of hyphens (or underscores) will not
> unduly complicate things. I would, however, draw a line at characters that
> are normally not found in words, like !@#€$%^&*(){}[],<>/?|"';:...
>
> Specifically, I think we should only add to the spec to allow words of
> alphanumeric characters, optionally in snake or kebab case. In a regular
> expression, this would mean: [a-zA-Z][0-9a-zA-Z_-]*[0-9a-zA-Z]?
>
+1
I like the new regex!
Martin
>
> Kind regards,
> Oscar
>
>
> On Mon, 13 Nov 2023 at 21:06, Martin Grigorov <mg...@apache.org>
> wrote:
>
> > Hi Jon,
> >
> > Thank you for this email!
> >
> > On Mon, Nov 13, 2023 at 7:56 PM Jonathan Slusher <jo...@gmail.com>
> > wrote:
> >
> > > Hi,
> > >
> > > I opened an issue in the AVRO project in Jira <
> > > https://issues.apache.org/jira/browse/AVRO-3900> and I’ve been asked
> to
> > > submit a topic for discussion to this email group.
> > >
> > > See this issue in the rust schema_registry_converter repo <
> > > https://github.com/gklijs/schema_registry_converter/issues/100> for
> > > details specific to the crate in rust that we’re having trouble with:
> > >
> > > A couple of things to point out here:
> > >
> > > 1. I understand that at this time the Avro spec does not allow hyphens
> in
> > > its namespaces, but somehow our registry is allowing them to be created
> > > from our Debezium connectors. We have been using the confluent_python <
> > > https://github.com/confluentinc/confluent-kafka-python> module and
> since
> > > version 1.9.2, its deserializer seems to handle these hyphens without
> > > error. We also have several JDBC sink connectors with consumer groups
> > that
> > > are able to use these topics.
> > >
> > > 2. We recently attempted to implement a consumer written in rust and
> the
> > > crate <https://github.com/gklijs/schema_registry_converter> above,
> which
> > > is used for deserialization, throws an exception when attempting to
> > connect
> > > to these topics.
> > >
> > > ```
> > > thread 'main' panicked at /app/src/utils/kafka.rs:67:35:
> > > Error decoding value: Error: Supplied raw value
> > >
> >
> "{\"type\":\"record\",\"name\":\"Envelope\",\"namespace\":\"debezium.abc-123-efg-20231005.table.u_table_dbz\",\"fields\":[{\"name\":\"before\",\"type\":[\"null\",{\"type\":\"record\",\"name\":\"Value\",\"fields\":[{\"name\":\"id\",\"type\":\"int\"},{\"name\":\"uid\",\"type\":{\"type\":\"long\",\"connect.default\":0},\"default\":0},{\"name\":\"release_id\",\"type\":{\"type\":\"long\",\"connect.default\":0},\"default\":0},{\"name\":\"notes\",\"type\":\"string\"},{\"name\":\"notes_public\",\"type\":{\"type\":\"string\",\"connect.version\":1,\"connect.parameters\":{\"allowed\":\"Y,N\"},\"connect.default\":\"N\",\"
> > > connect.name
> > >
> >
> \":\"io.debezium.data.Enum\"},\"default\":\"N\"},{\"name\":\"added_ts\",\"type\":{\"type\":\"long\",\"connect.version\":1,\"
> > > connect.name\":\"io.debezium.time.Timestamp\"}}],\"connect.name
> > >
> >
> \":\"debezium.abc-123-efg-20231005.table.u_table_dbz.Value\"}],\"default\":null},{\"name\":\"after\",\"type\":[\"null\",\"Value\"],\"default\":null},{\"name\":\"source\",\"type\":{\"type\":\"record\",\"name\":\"Source\",\"namespace\":\"io.debezium.connector.mysql\",\"fields\":[{\"name\":\"version\",\"type\":\"string\"},{\"name\":\"connector\",\"type\":\"string\"},{\"name\":\"name\",\"type\":\"string\"},{\"name\":\"ts_ms\",\"type\":\"long\"},{\"name\":\"snapshot\",\"type\":[{\"type\":\"string\",\"connect.version\":1,\"connect.parameters\":{\"allowed\":\"true,last,false,incremental\"},\"connect.default\":\"false\",\"
> > > connect.name
> > >
> >
> \":\"io.debezium.data.Enum\"},\"null\"],\"default\":\"false\"},{\"name\":\"db\",\"type\":\"string\"},{\"name\":\"sequence\",\"type\":[\"null\",\"string\"],\"default\":null},{\"name\":\"table\",\"type\":[\"null\",\"string\"],\"default\":null},{\"name\":\"server_id\",\"type\":\"long\"},{\"name\":\"gtid\",\"type\":[\"null\",\"string\"],\"default\":null},{\"name\":\"file\",\"type\":\"string\"},{\"name\":\"pos\",\"type\":\"long\"},{\"name\":\"row\",\"type\":\"int\"},{\"name\":\"thread\",\"type\":[\"null\",\"long\"],\"default\":null},{\"name\":\"query\",\"type\":[\"null\",\"string\"],\"default\":null}],\"
> > > connect.name
> > >
> >
> \":\"io.debezium.connector.mysql.Source\"}},{\"name\":\"op\",\"type\":\"string\"},{\"name\":\"ts_ms\",\"type\":[\"null\",\"long\"],\"default\":null},{\"name\":\"transaction\",\"type\":[\"null\",{\"type\":\"record\",\"name\":\"block\",\"namespace\":\"event\",\"fields\":[{\"name\":\"id\",\"type\":\"string\"},{\"name\":\"total_order\",\"type\":\"long\"},{\"name\":\"data_collection_order\",\"type\":\"long\"}],\"connect.version\":1,\"
> > > connect.name
> > > \":\"event.block\"}],\"default\":null}],\"connect.version\":1,\"
> > > connect.name
> > \":\"debezium.abc-123-efg-20231005.table.u_table_dbz.Envelope\"}"
> > > cant be turned into a Schema, was cause by Invalid namespace
> > > debezium.abc-123-efg-20231005.table.u_table_dbz. It must match the
> regex
> > > '^([A-Za-z_][A-Za-z0-9_]*(\.[A-Za-z_][A-Za-z0-9_]*)*)?$', it's
> retriable:
> > > false, it's cached: false
> > > ```
> > >
> > > Ideally, the Avro spec would just accept hyphens since they’re a pretty
> > > common character and unavoidable in certain circumstances. If this is
> > > easier said than done, I think at the least any library used for Avro
> > > deserialization should account for them, including the rust library. If
> > > this works in Java and Python, shouldn’t it also work in rust?
> > >
> >
> > This is exactly the reason I asked you to raise this question here in the
> > mailing list!
> > I also agree that if most/all of the SDKs will allow hyphens in the
> > name[space] then it is a better idea to add it to the list of allowed
> > characters in the specification instead of adding logic to disable the
> > validation.
> >
> > @Avro devs: What is your opinion ?
> >
> > Martin
> >
> >
> > >
> > > Here’s a generic example of a schema created by a Debezium connector:
> > >
> > > ```
> > > {
> > > "type": "record",
> > > "name": "Envelope",
> > > "namespace": "abc-123-efg-20231005.table.u_table_dbz",
> > > "fields": [
> > > {
> > > "name": "before",
> > > "type": [
> > > "null",
> > > {
> > > ...
> > > ```
> > >
> > > Please let me know if you need any more details, and thank you!
> > >
> > > Jon Slusher
> > >
> > >
> > >
> > >
> >
>
>
> --
>
> ✉️ Oscar Westra van Holthe - Kind <os...@westravanholthe.nl>
>
Re: Hyphens in Avro schema namespaces and the rust deserializer
Posted by Oscar Westra van Holthe - Kind <os...@westravanholthe.nl>.
Hi all,
On the topic of allowed characters, we've settled in the (recent) past on
alphanumeric characters only for simple names. Reasoning was that this is
most portable.
Remembering that, allowing the use of hyphens (or underscores) will not
unduly complicate things. I would, however, draw a line at characters that
are normally not found in words, like !@#€$%^&*(){}[],<>/?|"';:...
Specifically, I think we should only add to the spec to allow words of
alphanumeric characters, optionally in snake or kebab case. In a regular
expression, this would mean: [a-zA-Z][0-9a-zA-Z_-]*[0-9a-zA-Z]?
Kind regards,
Oscar
On Mon, 13 Nov 2023 at 21:06, Martin Grigorov <mg...@apache.org> wrote:
> Hi Jon,
>
> Thank you for this email!
>
> On Mon, Nov 13, 2023 at 7:56 PM Jonathan Slusher <jo...@gmail.com>
> wrote:
>
> > Hi,
> >
> > I opened an issue in the AVRO project in Jira <
> > https://issues.apache.org/jira/browse/AVRO-3900> and I’ve been asked to
> > submit a topic for discussion to this email group.
> >
> > See this issue in the rust schema_registry_converter repo <
> > https://github.com/gklijs/schema_registry_converter/issues/100> for
> > details specific to the crate in rust that we’re having trouble with:
> >
> > A couple of things to point out here:
> >
> > 1. I understand that at this time the Avro spec does not allow hyphens in
> > its namespaces, but somehow our registry is allowing them to be created
> > from our Debezium connectors. We have been using the confluent_python <
> > https://github.com/confluentinc/confluent-kafka-python> module and since
> > version 1.9.2, its deserializer seems to handle these hyphens without
> > error. We also have several JDBC sink connectors with consumer groups
> that
> > are able to use these topics.
> >
> > 2. We recently attempted to implement a consumer written in rust and the
> > crate <https://github.com/gklijs/schema_registry_converter> above, which
> > is used for deserialization, throws an exception when attempting to
> connect
> > to these topics.
> >
> > ```
> > thread 'main' panicked at /app/src/utils/kafka.rs:67:35:
> > Error decoding value: Error: Supplied raw value
> >
> "{\"type\":\"record\",\"name\":\"Envelope\",\"namespace\":\"debezium.abc-123-efg-20231005.table.u_table_dbz\",\"fields\":[{\"name\":\"before\",\"type\":[\"null\",{\"type\":\"record\",\"name\":\"Value\",\"fields\":[{\"name\":\"id\",\"type\":\"int\"},{\"name\":\"uid\",\"type\":{\"type\":\"long\",\"connect.default\":0},\"default\":0},{\"name\":\"release_id\",\"type\":{\"type\":\"long\",\"connect.default\":0},\"default\":0},{\"name\":\"notes\",\"type\":\"string\"},{\"name\":\"notes_public\",\"type\":{\"type\":\"string\",\"connect.version\":1,\"connect.parameters\":{\"allowed\":\"Y,N\"},\"connect.default\":\"N\",\"
> > connect.name
> >
> \":\"io.debezium.data.Enum\"},\"default\":\"N\"},{\"name\":\"added_ts\",\"type\":{\"type\":\"long\",\"connect.version\":1,\"
> > connect.name\":\"io.debezium.time.Timestamp\"}}],\"connect.name
> >
> \":\"debezium.abc-123-efg-20231005.table.u_table_dbz.Value\"}],\"default\":null},{\"name\":\"after\",\"type\":[\"null\",\"Value\"],\"default\":null},{\"name\":\"source\",\"type\":{\"type\":\"record\",\"name\":\"Source\",\"namespace\":\"io.debezium.connector.mysql\",\"fields\":[{\"name\":\"version\",\"type\":\"string\"},{\"name\":\"connector\",\"type\":\"string\"},{\"name\":\"name\",\"type\":\"string\"},{\"name\":\"ts_ms\",\"type\":\"long\"},{\"name\":\"snapshot\",\"type\":[{\"type\":\"string\",\"connect.version\":1,\"connect.parameters\":{\"allowed\":\"true,last,false,incremental\"},\"connect.default\":\"false\",\"
> > connect.name
> >
> \":\"io.debezium.data.Enum\"},\"null\"],\"default\":\"false\"},{\"name\":\"db\",\"type\":\"string\"},{\"name\":\"sequence\",\"type\":[\"null\",\"string\"],\"default\":null},{\"name\":\"table\",\"type\":[\"null\",\"string\"],\"default\":null},{\"name\":\"server_id\",\"type\":\"long\"},{\"name\":\"gtid\",\"type\":[\"null\",\"string\"],\"default\":null},{\"name\":\"file\",\"type\":\"string\"},{\"name\":\"pos\",\"type\":\"long\"},{\"name\":\"row\",\"type\":\"int\"},{\"name\":\"thread\",\"type\":[\"null\",\"long\"],\"default\":null},{\"name\":\"query\",\"type\":[\"null\",\"string\"],\"default\":null}],\"
> > connect.name
> >
> \":\"io.debezium.connector.mysql.Source\"}},{\"name\":\"op\",\"type\":\"string\"},{\"name\":\"ts_ms\",\"type\":[\"null\",\"long\"],\"default\":null},{\"name\":\"transaction\",\"type\":[\"null\",{\"type\":\"record\",\"name\":\"block\",\"namespace\":\"event\",\"fields\":[{\"name\":\"id\",\"type\":\"string\"},{\"name\":\"total_order\",\"type\":\"long\"},{\"name\":\"data_collection_order\",\"type\":\"long\"}],\"connect.version\":1,\"
> > connect.name
> > \":\"event.block\"}],\"default\":null}],\"connect.version\":1,\"
> > connect.name
> \":\"debezium.abc-123-efg-20231005.table.u_table_dbz.Envelope\"}"
> > cant be turned into a Schema, was cause by Invalid namespace
> > debezium.abc-123-efg-20231005.table.u_table_dbz. It must match the regex
> > '^([A-Za-z_][A-Za-z0-9_]*(\.[A-Za-z_][A-Za-z0-9_]*)*)?$', it's retriable:
> > false, it's cached: false
> > ```
> >
> > Ideally, the Avro spec would just accept hyphens since they’re a pretty
> > common character and unavoidable in certain circumstances. If this is
> > easier said than done, I think at the least any library used for Avro
> > deserialization should account for them, including the rust library. If
> > this works in Java and Python, shouldn’t it also work in rust?
> >
>
> This is exactly the reason I asked you to raise this question here in the
> mailing list!
> I also agree that if most/all of the SDKs will allow hyphens in the
> name[space] then it is a better idea to add it to the list of allowed
> characters in the specification instead of adding logic to disable the
> validation.
>
> @Avro devs: What is your opinion ?
>
> Martin
>
>
> >
> > Here’s a generic example of a schema created by a Debezium connector:
> >
> > ```
> > {
> > "type": "record",
> > "name": "Envelope",
> > "namespace": "abc-123-efg-20231005.table.u_table_dbz",
> > "fields": [
> > {
> > "name": "before",
> > "type": [
> > "null",
> > {
> > ...
> > ```
> >
> > Please let me know if you need any more details, and thank you!
> >
> > Jon Slusher
> >
> >
> >
> >
>
--
✉️ Oscar Westra van Holthe - Kind <os...@westravanholthe.nl>
Re: Hyphens in Avro schema namespaces and the rust deserializer
Posted by Martin Grigorov <mg...@apache.org>.
Hi Jon,
Thank you for this email!
On Mon, Nov 13, 2023 at 7:56 PM Jonathan Slusher <jo...@gmail.com>
wrote:
> Hi,
>
> I opened an issue in the AVRO project in Jira <
> https://issues.apache.org/jira/browse/AVRO-3900> and I’ve been asked to
> submit a topic for discussion to this email group.
>
> See this issue in the rust schema_registry_converter repo <
> https://github.com/gklijs/schema_registry_converter/issues/100> for
> details specific to the crate in rust that we’re having trouble with:
>
> A couple of things to point out here:
>
> 1. I understand that at this time the Avro spec does not allow hyphens in
> its namespaces, but somehow our registry is allowing them to be created
> from our Debezium connectors. We have been using the confluent_python <
> https://github.com/confluentinc/confluent-kafka-python> module and since
> version 1.9.2, its deserializer seems to handle these hyphens without
> error. We also have several JDBC sink connectors with consumer groups that
> are able to use these topics.
>
> 2. We recently attempted to implement a consumer written in rust and the
> crate <https://github.com/gklijs/schema_registry_converter> above, which
> is used for deserialization, throws an exception when attempting to connect
> to these topics.
>
> ```
> thread 'main' panicked at /app/src/utils/kafka.rs:67:35:
> Error decoding value: Error: Supplied raw value
> "{\"type\":\"record\",\"name\":\"Envelope\",\"namespace\":\"debezium.abc-123-efg-20231005.table.u_table_dbz\",\"fields\":[{\"name\":\"before\",\"type\":[\"null\",{\"type\":\"record\",\"name\":\"Value\",\"fields\":[{\"name\":\"id\",\"type\":\"int\"},{\"name\":\"uid\",\"type\":{\"type\":\"long\",\"connect.default\":0},\"default\":0},{\"name\":\"release_id\",\"type\":{\"type\":\"long\",\"connect.default\":0},\"default\":0},{\"name\":\"notes\",\"type\":\"string\"},{\"name\":\"notes_public\",\"type\":{\"type\":\"string\",\"connect.version\":1,\"connect.parameters\":{\"allowed\":\"Y,N\"},\"connect.default\":\"N\",\"
> connect.name
> \":\"io.debezium.data.Enum\"},\"default\":\"N\"},{\"name\":\"added_ts\",\"type\":{\"type\":\"long\",\"connect.version\":1,\"
> connect.name\":\"io.debezium.time.Timestamp\"}}],\"connect.name
> \":\"debezium.abc-123-efg-20231005.table.u_table_dbz.Value\"}],\"default\":null},{\"name\":\"after\",\"type\":[\"null\",\"Value\"],\"default\":null},{\"name\":\"source\",\"type\":{\"type\":\"record\",\"name\":\"Source\",\"namespace\":\"io.debezium.connector.mysql\",\"fields\":[{\"name\":\"version\",\"type\":\"string\"},{\"name\":\"connector\",\"type\":\"string\"},{\"name\":\"name\",\"type\":\"string\"},{\"name\":\"ts_ms\",\"type\":\"long\"},{\"name\":\"snapshot\",\"type\":[{\"type\":\"string\",\"connect.version\":1,\"connect.parameters\":{\"allowed\":\"true,last,false,incremental\"},\"connect.default\":\"false\",\"
> connect.name
> \":\"io.debezium.data.Enum\"},\"null\"],\"default\":\"false\"},{\"name\":\"db\",\"type\":\"string\"},{\"name\":\"sequence\",\"type\":[\"null\",\"string\"],\"default\":null},{\"name\":\"table\",\"type\":[\"null\",\"string\"],\"default\":null},{\"name\":\"server_id\",\"type\":\"long\"},{\"name\":\"gtid\",\"type\":[\"null\",\"string\"],\"default\":null},{\"name\":\"file\",\"type\":\"string\"},{\"name\":\"pos\",\"type\":\"long\"},{\"name\":\"row\",\"type\":\"int\"},{\"name\":\"thread\",\"type\":[\"null\",\"long\"],\"default\":null},{\"name\":\"query\",\"type\":[\"null\",\"string\"],\"default\":null}],\"
> connect.name
> \":\"io.debezium.connector.mysql.Source\"}},{\"name\":\"op\",\"type\":\"string\"},{\"name\":\"ts_ms\",\"type\":[\"null\",\"long\"],\"default\":null},{\"name\":\"transaction\",\"type\":[\"null\",{\"type\":\"record\",\"name\":\"block\",\"namespace\":\"event\",\"fields\":[{\"name\":\"id\",\"type\":\"string\"},{\"name\":\"total_order\",\"type\":\"long\"},{\"name\":\"data_collection_order\",\"type\":\"long\"}],\"connect.version\":1,\"
> connect.name
> \":\"event.block\"}],\"default\":null}],\"connect.version\":1,\"
> connect.name\":\"debezium.abc-123-efg-20231005.table.u_table_dbz.Envelope\"}"
> cant be turned into a Schema, was cause by Invalid namespace
> debezium.abc-123-efg-20231005.table.u_table_dbz. It must match the regex
> '^([A-Za-z_][A-Za-z0-9_]*(\.[A-Za-z_][A-Za-z0-9_]*)*)?$', it's retriable:
> false, it's cached: false
> ```
>
> Ideally, the Avro spec would just accept hyphens since they’re a pretty
> common character and unavoidable in certain circumstances. If this is
> easier said than done, I think at the least any library used for Avro
> deserialization should account for them, including the rust library. If
> this works in Java and Python, shouldn’t it also work in rust?
>
This is exactly the reason I asked you to raise this question here in the
mailing list!
I also agree that if most/all of the SDKs will allow hyphens in the
name[space] then it is a better idea to add it to the list of allowed
characters in the specification instead of adding logic to disable the
validation.
@Avro devs: What is your opinion ?
Martin
>
> Here’s a generic example of a schema created by a Debezium connector:
>
> ```
> {
> "type": "record",
> "name": "Envelope",
> "namespace": "abc-123-efg-20231005.table.u_table_dbz",
> "fields": [
> {
> "name": "before",
> "type": [
> "null",
> {
> ...
> ```
>
> Please let me know if you need any more details, and thank you!
>
> Jon Slusher
>
>
>
>