You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by "Alex Leathwood (Jira)" <ji...@apache.org> on 2025/03/04 09:37:00 UTC

[jira] [Created] (NIFI-14314) Improve Kafka3 client performance when reading Avro

Alex Leathwood created NIFI-14314:
-------------------------------------

             Summary: Improve Kafka3 client performance when reading Avro
                 Key: NIFI-14314
                 URL: https://issues.apache.org/jira/browse/NIFI-14314
             Project: Apache NiFi
          Issue Type: Improvement
          Components: Extensions
    Affects Versions: 2.2.0
            Reporter: Alex Leathwood


In NiFi-5213 "Allow AvroReader to process files w embedded schema even when the access strategy is explicit schema" did so at the expense of [throwing and catching an IOException|https://github.com/apache/nifi/blob/cd25603157556d28a4bda961ae5e52d18ac71849/nifi-extension-bundles/nifi-standard-services/nifi-record-serialization-services-bundle/nifi-record-serialization-services/src/main/java/org/apache/nifi/avro/AvroReaderWithExplicitSchema.java#L53] for each and every Kafka record without an embedded schema, even when the processor is correctly configured. 
In high-performance use cases we have observed up to 2/3 of the cpu time ot the respective worker thread being used for this check.
The exception originates for the kafka/avro client libraries and cannot easily be replaced with a lightweight implementation of Exception .


We propose to either drop this earlier improvement of user experience (PR available) )or to reimplement it so that the correct configuration is the "happy path" and only the incorrect configuration case be penalized with exceptions and ideally a warning.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)