[jira] [Created] (NIFI-7376) Avro Single-object encoding Support

["Nathan English (Jira)" <ji...@apache.org>]

Nathan English created NIFI-7376:
------------------------------------

             Summary: Avro Single-object encoding Support
                 Key: NIFI-7376
                 URL: https://issues.apache.org/jira/browse/NIFI-7376
             Project: Apache NiFi
          Issue Type: Improvement
          Components: Core Framework
    Affects Versions: 1.9.2
            Reporter: Nathan English


For my Flows I consume Avro Binary Encoded Messages from Kafka which is currently working great! However going forward one of our inputs is looking to provide [Avro Single-object Encoding|[https://avro.apache.org/docs/1.8.2/spec.html#single_object_encoding]]. 

Avro Single Object Encoding provides us a way to where we don't have to have the overhead and message size increase of embedding the schema in the message. Single Object Encoding achieves this with a couple of extra fields to firstly confirm the message is an Avro message and secondly a fingerprint of the schema used to encode the message.

This is a massive benefit for us, because we have multiple of the same device producing messages into one Kafka Topic. This is fine until we start upgrading these devices where schema changes may occur, this is when schema fingerprinting comes into its element.

From all the information I have found, it looks to me as if Avro Single Object Encoding was added in version 1.8.2 of the Avro Specification. NiFi is currently using version 1.8.1 of the Avro Specification based on the [pom|[https://github.com/apache/nifi/blob/rel/nifi-1.11.4/nifi-nar-bundles/nifi-standard-services/nifi-record-serialization-services-bundle/nifi-record-serialization-services/pom.xml]] in NiFi Record Serialization Services section of the github project.

I'm sure there are a million ways to tackle this issue and I'm personally haven't done enough research on NiFi or Avro on it to suggest a way to resolve this, but I can tell it's not as simple as just upgrading the Avro version used.

My thoughts were to upgrade the Library version, then modify the avro schema registry to either take in a fingerprint value or have it calculated on enablement of the registry. I'm sure it's probably not as simple as I have just made it!.

More than happy to help where I can.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)