You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by "Weiss, Christian" <Ch...@sva.de> on 2021/10/26 13:46:01 UTC
Unable to evolve Schema with HWXRegistry
Hey guys,
we're facing some issues while writing avro with a connected hortonworks schema registry.
We would like to achieve the follwoing:
- registered a schema and evolve it over time
- write a avro as "confluent avro" with magic byte zero followed by the schema id
Therefor we've configured an AvroRecordSetWriter and a HortonworksSchemaRegistry as conroller service, so that it should retrieve the latest version.
[cid:image001.png@01D7CA80.92DE8440]
This works, unless we increase the version of a schema in the registry. We've investigate the requests, responses and the avro file itself and figured out the following.
Actual behaviour:
- The avro file contains the id which is present in the schemaMetadataId
Expected behaviour:
- The avro file contains the id which is present in the id field of the response
Example call:
curl -X GET "https://<host>:<port>/schema-registry/api/v1/schemaregistry/schemas/<schema_name>/versions/latest?branch=MASTER" -H "accept: application/json"
Example response:
{
"id": 10,
"schemaMetadataId": 7,
"name": "schema_name",
"description": "test",
"version": 1,
"schemaText": "..",
"timestamp": 1635252068173,
"stateId": 5,
"mergeInfo": null
}
This makes it impossible to increase the version of an existing schema, because then the id and schemaMetadataId will evolve differently.
Does anyone have similar issues?
Thanks and cheers,
Christian
----------------------------
Das SVA Mail-System ist mit einem Mailverschlusselungs-Gateway ausgestattet. Wenn Sie moechten, dass an Sie gerichtete E-Mails verschluesselt werden, senden Sie einfach eine S/MIME-signierte E-Mail oder Ihren PGP Public Key an christian.weiss@sva.de.
The SVA mail system is equipped with an email encryption gateway. If you want email sent to you to be encrypted please send a S/MIME signed email or your PGP public key to christian.weiss@sva.de.
Re: Unable to evolve Schema with HWXRegistry
Posted by Pierre Villard <pi...@gmail.com>.
Sounds good, thanks for the feedback!
Le jeu. 28 oct. 2021 à 09:14, Weiss, Christian <Ch...@sva.de> a
écrit :
> Hi Pierre,
>
> we managed to fix the issue using NiFi version 1.14.
>
> As we already planned to upgrade in near future, we are fine with that.
>
>
>
> Thanks,
>
> Christian
>
>
>
> *Von:* Pierre Villard <pi...@gmail.com>
> *Gesendet:* Mittwoch, 27. Oktober 2021 20:29
> *An:* users@nifi.apache.org
> *Betreff:* Re: Unable to evolve Schema with HWXRegistry
>
>
>
> Thanks Christian.
>
> Understood, I'll try to have a look into the code and get back to you.
>
>
>
> Le mar. 26 oct. 2021 à 19:16, Weiss, Christian <Ch...@sva.de> a
> écrit :
>
> Hi Pierre,
>
>
>
> the Avro-Files, which we write to a Kafka Topic in the next step, needs to
> be compatible with the Confluent Avro-Format, because we use Confluent
> compatible deserializers on the consuming end. Therefore we need to use the
> “Confluent Schema” Write-Strategy. The Hortonworks registry provides
> Confluent compatible Endpoints to achieving this.
>
>
>
> We are actually using NiFi 11.4.
>
>
>
> Hope this helps.
>
>
>
> Thanks,
>
> Christian
>
>
>
>
>
> Christian Weiß
>
>
>
> Data Architect
>
> Big Data/Analytics & IOT
>
>
>
> SVA System Vertrieb Alexander GmbH
>
> Freiberger Str. 39
>
> 01067 Dresden
>
> Mobile: +49 151 18027982
>
>
>
> E-Mail: Christian.Weiss@sva.de
>
> http://www.sva.de
>
>
>
> Geschäftsführung: Philipp Alexander, Sven Eichelbaum
>
> Sitz der Gesellschaft: Wiesbaden
>
> Registergericht: Amtsgericht Wiesbaden, HRB 10315
> ------------------------------
>
> *Von:* Pierre Villard <pi...@gmail.com>
> *Gesendet:* Tuesday, October 26, 2021 7:03:36 PM
> *An:* users@nifi.apache.org <us...@nifi.apache.org>
> *Betreff:* Re: Unable to evolve Schema with HWXRegistry
>
>
>
> Hi Christian,
>
>
>
> Not sure to understand the overall objective. You're writing the schema
> reference using Confluent encoding but you have the schema in an
> Hortonworks Schema Registry instance?
>
>
>
> Also, what version of NiFi are you using?
>
>
>
> Thanks,
>
> Pierre
>
>
>
> Le mar. 26 oct. 2021 à 15:46, Weiss, Christian <Ch...@sva.de> a
> écrit :
>
> Hey guys,
>
>
>
> we're facing some issues while writing avro with a connected hortonworks
> schema registry.
>
>
>
> We would like to achieve the follwoing:
>
> - registered a schema and evolve it over time
>
> - write a avro as "confluent avro" with magic byte zero followed by the
> schema id
>
>
>
> Therefor we've configured an AvroRecordSetWriter and a
> HortonworksSchemaRegistry as conroller service, so that it should retrieve
> the latest version.
>
>
>
>
>
> This works, unless we increase the version of a schema in the registry.
> We've investigate the requests, responses and the avro file itself and
> figured out the following.
>
>
>
> Actual behaviour:
>
> - The avro file contains the id which is present in the schemaMetadataId
>
> Expected behaviour:
>
> - The avro file contains the id which is present in the id field of the
> response
>
>
>
> Example call:
>
> curl -X GET "
> https://<host>:<port>/schema-registry/api/v1/schemaregistry/schemas/<schema_name>/versions/latest?branch=MASTER"
> -H "accept: application/json"
>
> Example response:
>
> {
>
> "id": 10,
>
> "schemaMetadataId": 7,
>
> "name": "schema_name",
>
> "description": "test",
>
> "version": 1,
>
> "schemaText": "..",
>
> "timestamp": 1635252068173,
>
> "stateId": 5,
>
> "mergeInfo": null
>
> }
>
>
>
> This makes it impossible to increase the version of an existing schema,
> because then the id and schemaMetadataId will evolve differently.
>
>
>
> Does anyone have similar issues?
>
>
>
> Thanks and cheers,
>
> Christian
>
>
>
>
> Das SVA Mail-System ist mit einem Mailverschluesselungs-Gateway
> ausgestattet. Wenn Sie moechten, dass an Sie gerichtete E-Mails
> verschluesselt werden, senden Sie einfach eine S/MIME-signierte E-Mail oder
> Ihren PGP Public Key an christian.weiss@sva.de.
>
> The SVA mail system is equipped with an email encryption gateway. If you
> want email sent to you to be encrypted please send a S/MIME signed email or
> your PGP public key to christian.weiss@sva.de.
>
>
AW: Unable to evolve Schema with HWXRegistry
Posted by "Weiss, Christian" <Ch...@sva.de>.
Hi Pierre,
we managed to fix the issue using NiFi version 1.14.
As we already planned to upgrade in near future, we are fine with that.
Thanks,
Christian
Von: Pierre Villard <pi...@gmail.com>
Gesendet: Mittwoch, 27. Oktober 2021 20:29
An: users@nifi.apache.org
Betreff: Re: Unable to evolve Schema with HWXRegistry
Thanks Christian.
Understood, I'll try to have a look into the code and get back to you.
Le mar. 26 oct. 2021 à 19:16, Weiss, Christian <Ch...@sva.de>> a écrit :
Hi Pierre,
the Avro-Files, which we write to a Kafka Topic in the next step, needs to be compatible with the Confluent Avro-Format, because we use Confluent compatible deserializers on the consuming end. Therefore we need to use the “Confluent Schema” Write-Strategy. The Hortonworks registry provides Confluent compatible Endpoints to achieving this.
We are actually using NiFi 11.4.
Hope this helps.
Thanks,
Christian
Christian Weiß
Data Architect
Big Data/Analytics & IOT
SVA System Vertrieb Alexander GmbH
Freiberger Str. 39
01067 Dresden
Mobile: +49 151 18027982
E-Mail: Christian.Weiss@sva.de<ma...@sva.de>
http://www.sva.de
Geschäftsführung: Philipp Alexander, Sven Eichelbaum
Sitz der Gesellschaft: Wiesbaden
Registergericht: Amtsgericht Wiesbaden, HRB 10315
________________________________
Von: Pierre Villard <pi...@gmail.com>>
Gesendet: Tuesday, October 26, 2021 7:03:36 PM
An: users@nifi.apache.org<ma...@nifi.apache.org> <us...@nifi.apache.org>>
Betreff: Re: Unable to evolve Schema with HWXRegistry
Hi Christian,
Not sure to understand the overall objective. You're writing the schema reference using Confluent encoding but you have the schema in an Hortonworks Schema Registry instance?
Also, what version of NiFi are you using?
Thanks,
Pierre
Le mar. 26 oct. 2021 à 15:46, Weiss, Christian <Ch...@sva.de>> a écrit :
Hey guys,
we're facing some issues while writing avro with a connected hortonworks schema registry.
We would like to achieve the follwoing:
- registered a schema and evolve it over time
- write a avro as "confluent avro" with magic byte zero followed by the schema id
Therefor we've configured an AvroRecordSetWriter and a HortonworksSchemaRegistry as conroller service, so that it should retrieve the latest version.
[cid:image001.png@01D7CBDC.3E8BEC40]
This works, unless we increase the version of a schema in the registry. We've investigate the requests, responses and the avro file itself and figured out the following.
Actual behaviour:
- The avro file contains the id which is present in the schemaMetadataId
Expected behaviour:
- The avro file contains the id which is present in the id field of the response
Example call:
curl -X GET "https://<host>:<port>/schema-registry/api/v1/schemaregistry/schemas/<schema_name>/versions/latest?branch=MASTER<https://%3chost%3e:%3cport%3e/schema-registry/api/v1/schemaregistry/schemas/%3cschema_name%3e/versions/latest?branch=MASTER>" -H "accept: application/json"
Example response:
{
"id": 10,
"schemaMetadataId": 7,
"name": "schema_name",
"description": "test",
"version": 1,
"schemaText": "..",
"timestamp": 1635252068173,
"stateId": 5,
"mergeInfo": null
}
This makes it impossible to increase the version of an existing schema, because then the id and schemaMetadataId will evolve differently.
Does anyone have similar issues?
Thanks and cheers,
Christian
Das SVA Mail-System ist mit einem Mailverschluesselungs-Gateway ausgestattet. Wenn Sie moechten, dass an Sie gerichtete E-Mails verschluesselt werden, senden Sie einfach eine S/MIME-signierte E-Mail oder Ihren PGP Public Key an christian.weiss@sva.de<ma...@sva.de>.
The SVA mail system is equipped with an email encryption gateway. If you want email sent to you to be encrypted please send a S/MIME signed email or your PGP public key to christian.weiss@sva.de<ma...@sva.de>.
Re: Unable to evolve Schema with HWXRegistry
Posted by Pierre Villard <pi...@gmail.com>.
Thanks Christian.
Understood, I'll try to have a look into the code and get back to you.
Le mar. 26 oct. 2021 à 19:16, Weiss, Christian <Ch...@sva.de> a
écrit :
> Hi Pierre,
>
> the Avro-Files, which we write to a Kafka Topic in the next step, needs to
> be compatible with the Confluent Avro-Format, because we use Confluent
> compatible deserializers on the consuming end. Therefore we need to use the
> “Confluent Schema” Write-Strategy. The Hortonworks registry provides
> Confluent compatible Endpoints to achieving this.
>
> We are actually using NiFi 11.4.
>
> Hope this helps.
>
> Thanks,
> Christian
>
>
> Christian Weiß
>
> Data Architect
> Big Data/Analytics & IOT
>
> SVA System Vertrieb Alexander GmbH
> Freiberger Str. 39
> 01067 Dresden
> Mobile: +49 151 18027982
>
> E-Mail: Christian.Weiss@sva.de
> http://www.sva.de
>
> Geschäftsführung: Philipp Alexander, Sven Eichelbaum
> Sitz der Gesellschaft: Wiesbaden
> Registergericht: Amtsgericht Wiesbaden, HRB 10315
> ------------------------------
> *Von:* Pierre Villard <pi...@gmail.com>
> *Gesendet:* Tuesday, October 26, 2021 7:03:36 PM
> *An:* users@nifi.apache.org <us...@nifi.apache.org>
> *Betreff:* Re: Unable to evolve Schema with HWXRegistry
>
> Hi Christian,
>
> Not sure to understand the overall objective. You're writing the schema
> reference using Confluent encoding but you have the schema in an
> Hortonworks Schema Registry instance?
>
> Also, what version of NiFi are you using?
>
> Thanks,
> Pierre
>
> Le mar. 26 oct. 2021 à 15:46, Weiss, Christian <Ch...@sva.de> a
> écrit :
>
> Hey guys,
>
>
>
> we're facing some issues while writing avro with a connected hortonworks
> schema registry.
>
>
>
> We would like to achieve the follwoing:
>
> - registered a schema and evolve it over time
>
> - write a avro as "confluent avro" with magic byte zero followed by the
> schema id
>
>
>
> Therefor we've configured an AvroRecordSetWriter and a
> HortonworksSchemaRegistry as conroller service, so that it should retrieve
> the latest version.
>
>
>
>
>
> This works, unless we increase the version of a schema in the registry.
> We've investigate the requests, responses and the avro file itself and
> figured out the following.
>
>
>
> Actual behaviour:
>
> - The avro file contains the id which is present in the schemaMetadataId
>
> Expected behaviour:
>
> - The avro file contains the id which is present in the id field of the
> response
>
>
>
> Example call:
>
> curl -X GET "https://<host>:<port>/schema-registry/api/v1/schemaregistry/schemas/<schema_name>/versions/latest?branch=MASTER"
> -H "accept: application/json"
>
> Example response:
>
> {
>
> "id": 10,
>
> "schemaMetadataId": 7,
>
> "name": "schema_name",
>
> "description": "test",
>
> "version": 1,
>
> "schemaText": "..",
>
> "timestamp": 1635252068173,
>
> "stateId": 5,
>
> "mergeInfo": null
>
> }
>
>
>
> This makes it impossible to increase the version of an existing schema,
> because then the id and schemaMetadataId will evolve differently.
>
>
>
> Does anyone have similar issues?
>
>
>
> Thanks and cheers,
>
> Christian
>
>
>
> Das SVA Mail-System ist mit einem Mailverschluesselungs-Gateway
> ausgestattet. Wenn Sie moechten, dass an Sie gerichtete E-Mails
> verschluesselt werden, senden Sie einfach eine S/MIME-signierte E-Mail oder
> Ihren PGP Public Key an christian.weiss@sva.de.
>
> The SVA mail system is equipped with an email encryption gateway. If you
> want email sent to you to be encrypted please send a S/MIME signed email or
> your PGP public key to christian.weiss@sva.de.
>
>
Re: Unable to evolve Schema with HWXRegistry
Posted by "Weiss, Christian" <Ch...@sva.de>.
Hi Pierre,
the Avro-Files, which we write to a Kafka Topic in the next step, needs to be compatible with the Confluent Avro-Format, because we use Confluent compatible deserializers on the consuming end. Therefore we need to use the “Confluent Schema” Write-Strategy. The Hortonworks registry provides Confluent compatible Endpoints to achieving this.
We are actually using NiFi 11.4.
Hope this helps.
Thanks,
Christian
Christian Weiß
Data Architect
Big Data/Analytics & IOT
SVA System Vertrieb Alexander GmbH
Freiberger Str. 39
01067 Dresden
Mobile: +49 151 18027982
E-Mail: Christian.Weiss@sva.de
http://www.sva.de
Geschäftsführung: Philipp Alexander, Sven Eichelbaum
Sitz der Gesellschaft: Wiesbaden
Registergericht: Amtsgericht Wiesbaden, HRB 10315
________________________________
Von: Pierre Villard <pi...@gmail.com>
Gesendet: Tuesday, October 26, 2021 7:03:36 PM
An: users@nifi.apache.org <us...@nifi.apache.org>
Betreff: Re: Unable to evolve Schema with HWXRegistry
Hi Christian,
Not sure to understand the overall objective. You're writing the schema reference using Confluent encoding but you have the schema in an Hortonworks Schema Registry instance?
Also, what version of NiFi are you using?
Thanks,
Pierre
Le mar. 26 oct. 2021 à 15:46, Weiss, Christian <Ch...@sva.de>> a écrit :
Hey guys,
we're facing some issues while writing avro with a connected hortonworks schema registry.
We would like to achieve the follwoing:
- registered a schema and evolve it over time
- write a avro as "confluent avro" with magic byte zero followed by the schema id
Therefor we've configured an AvroRecordSetWriter and a HortonworksSchemaRegistry as conroller service, so that it should retrieve the latest version.
[cid:17cbd8d74454cff311]
This works, unless we increase the version of a schema in the registry. We've investigate the requests, responses and the avro file itself and figured out the following.
Actual behaviour:
- The avro file contains the id which is present in the schemaMetadataId
Expected behaviour:
- The avro file contains the id which is present in the id field of the response
Example call:
curl -X GET "https://<host>:<port>/schema-registry/api/v1/schemaregistry/schemas/<schema_name>/versions/latest?branch=MASTER" -H "accept: application/json"
Example response:
{
"id": 10,
"schemaMetadataId": 7,
"name": "schema_name",
"description": "test",
"version": 1,
"schemaText": "..",
"timestamp": 1635252068173,
"stateId": 5,
"mergeInfo": null
}
This makes it impossible to increase the version of an existing schema, because then the id and schemaMetadataId will evolve differently.
Does anyone have similar issues?
Thanks and cheers,
Christian
Das SVA Mail-System ist mit einem Mailverschluesselungs-Gateway ausgestattet. Wenn Sie moechten, dass an Sie gerichtete E-Mails verschluesselt werden, senden Sie einfach eine S/MIME-signierte E-Mail oder Ihren PGP Public Key an christian.weiss@sva.de<ma...@sva.de>.
The SVA mail system is equipped with an email encryption gateway. If you want email sent to you to be encrypted please send a S/MIME signed email or your PGP public key to christian.weiss@sva.de<ma...@sva.de>.
Re: Unable to evolve Schema with HWXRegistry
Posted by Pierre Villard <pi...@gmail.com>.
Hi Christian,
Not sure to understand the overall objective. You're writing the schema
reference using Confluent encoding but you have the schema in an
Hortonworks Schema Registry instance?
Also, what version of NiFi are you using?
Thanks,
Pierre
Le mar. 26 oct. 2021 à 15:46, Weiss, Christian <Ch...@sva.de> a
écrit :
> Hey guys,
>
>
>
> we're facing some issues while writing avro with a connected hortonworks
> schema registry.
>
>
>
> We would like to achieve the follwoing:
>
> - registered a schema and evolve it over time
>
> - write a avro as "confluent avro" with magic byte zero followed by the
> schema id
>
>
>
> Therefor we've configured an AvroRecordSetWriter and a
> HortonworksSchemaRegistry as conroller service, so that it should retrieve
> the latest version.
>
>
>
>
>
> This works, unless we increase the version of a schema in the registry.
> We've investigate the requests, responses and the avro file itself and
> figured out the following.
>
>
>
> Actual behaviour:
>
> - The avro file contains the id which is present in the schemaMetadataId
>
> Expected behaviour:
>
> - The avro file contains the id which is present in the id field of the
> response
>
>
>
> Example call:
>
> curl -X GET "https://<host>:<port>/schema-registry/api/v1/schemaregistry/schemas/<schema_name>/versions/latest?branch=MASTER"
> -H "accept: application/json"
>
> Example response:
>
> {
>
> "id": 10,
>
> "schemaMetadataId": 7,
>
> "name": "schema_name",
>
> "description": "test",
>
> "version": 1,
>
> "schemaText": "..",
>
> "timestamp": 1635252068173,
>
> "stateId": 5,
>
> "mergeInfo": null
>
> }
>
>
>
> This makes it impossible to increase the version of an existing schema,
> because then the id and schemaMetadataId will evolve differently.
>
>
>
> Does anyone have similar issues?
>
>
>
> Thanks and cheers,
>
> Christian
>
>
>
> Das SVA Mail-System ist mit einem Mailverschluesselungs-Gateway
> ausgestattet. Wenn Sie moechten, dass an Sie gerichtete E-Mails
> verschluesselt werden, senden Sie einfach eine S/MIME-signierte E-Mail oder
> Ihren PGP Public Key an christian.weiss@sva.de.
>
> The SVA mail system is equipped with an email encryption gateway. If you
> want email sent to you to be encrypted please send a S/MIME signed email or
> your PGP public key to christian.weiss@sva.de.
>