You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by na...@bt.com on 2020/04/20 09:59:13 UTC
Avro Single Object Binary Encoding
Hi There,
Apologies if this has been asked before!
One of our Inputs into our NIFI cluster is looking to start using Avro Single Object Encoding on their messages. This was added V1.8.2 of the Avro Schema, but from what I can see NIFI is using V1.8.1 (looking at the pom.xml on GitHub, https://github.com/apache/nifi/blob/rel/nifi-1.11.4/nifi-nar-bundles/nifi-standard-services/nifi-record-serialization-services-bundle/nifi-record-serialization-services/pom.xml) of the Avro Library. Further information on the Single Object Encoding can be found here https://avro.apache.org/docs/1.8.2/spec.html#single_object_encoding_spec.
Firstly, are there any plans to upgrade the Avro library in NIFI? Secondly with an upgrade how and will NIFI support the Avro Single Object Encoding? Has anybody implemented support for this in their NIFI clusters?
Once again apologies if this has been asked before.
Kind Regards,
Nathan
RE: Avro Single Object Binary Encoding
Posted by na...@bt.com.
Hi Andy,
Thanks for getting back to me.
I have raised NIFI-7376 on Jira as an improvement. Hopefully that’s enough detail, if not please let me know as I can add more.
I do agree with the versioning of Avro, it wasn’t something I was expecting to find in a patch release.
Nathan
From: Andy LoPresto [mailto:alopresto@apache.org]
Sent: 20 April 2020 21:07
To: users@nifi.apache.org
Subject: Re: Avro Single Object Binary Encoding
Hi Nathan,
I don’t believe this has been addressed yet. Please file a Jira ticket [1] requesting this as a new feature/improvement and document as many of the specific requirements you’re encountering as possible to provide context around the development effort. It’s interesting a new feature was added in a x.y.z release, which indicates they may not be using semantic versioning and we would have to perform serious evaluation before upgrading the library to ensure no other behavior was changed.
[1] https://issues.apache.org/jira/projects/NIFI/issues
Andy LoPresto
alopresto@apache.org<ma...@apache.org>
alopresto.apache@gmail.com<ma...@gmail.com>
He/Him
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
On Apr 20, 2020, at 2:59 AM, nathan.english@bt.com<ma...@bt.com> wrote:
Hi There,
Apologies if this has been asked before!
One of our Inputs into our NIFI cluster is looking to start using Avro Single Object Encoding on their messages. This was added V1.8.2 of the Avro Schema, but from what I can see NIFI is using V1.8.1 (looking at the pom.xml on GitHub, https://github.com/apache/nifi/blob/rel/nifi-1.11.4/nifi-nar-bundles/nifi-standard-services/nifi-record-serialization-services-bundle/nifi-record-serialization-services/pom.xml) of the Avro Library. Further information on the Single Object Encoding can be found here https://avro.apache.org/docs/1.8.2/spec.html#single_object_encoding_spec.
Firstly, are there any plans to upgrade the Avro library in NIFI? Secondly with an upgrade how and will NIFI support the Avro Single Object Encoding? Has anybody implemented support for this in their NIFI clusters?
Once again apologies if this has been asked before.
Kind Regards,
Nathan
Re: Avro Single Object Binary Encoding
Posted by Andy LoPresto <al...@apache.org>.
Hi Nathan,
I don’t believe this has been addressed yet. Please file a Jira ticket [1] requesting this as a new feature/improvement and document as many of the specific requirements you’re encountering as possible to provide context around the development effort. It’s interesting a new feature was added in a x.y.z release, which indicates they may not be using semantic versioning and we would have to perform serious evaluation before upgrading the library to ensure no other behavior was changed.
[1] https://issues.apache.org/jira/projects/NIFI/issues <https://issues.apache.org/jira/projects/NIFI/issues>
Andy LoPresto
alopresto@apache.org
alopresto.apache@gmail.com
He/Him
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> On Apr 20, 2020, at 2:59 AM, nathan.english@bt.com wrote:
>
> Hi There,
>
> Apologies if this has been asked before!
>
> One of our Inputs into our NIFI cluster is looking to start using Avro Single Object Encoding on their messages. This was added V1.8.2 of the Avro Schema, but from what I can see NIFI is using V1.8.1 (looking at the pom.xml on GitHub, https://github.com/apache/nifi/blob/rel/nifi-1.11.4/nifi-nar-bundles/nifi-standard-services/nifi-record-serialization-services-bundle/nifi-record-serialization-services/pom.xml <https://github.com/apache/nifi/blob/rel/nifi-1.11.4/nifi-nar-bundles/nifi-standard-services/nifi-record-serialization-services-bundle/nifi-record-serialization-services/pom.xml>) of the Avro Library. Further information on the Single Object Encoding can be found here https://avro.apache.org/docs/1.8.2/spec.html#single_object_encoding_spec <https://avro.apache.org/docs/1.8.2/spec.html#single_object_encoding_spec>.
>
> Firstly, are there any plans to upgrade the Avro library in NIFI? Secondly with an upgrade how and will NIFI support the Avro Single Object Encoding? Has anybody implemented support for this in their NIFI clusters?
>
> Once again apologies if this has been asked before.
>
> Kind Regards,
>
> Nathan