[jira] [Commented] (NIFI-14882) ConsumeKafka improvement for Avro+SchemaRegistry input

["Alex Goos (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/NIFI-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18021066#comment-18021066 ] 

Alex Goos commented on NIFI-14882:
----------------------------------

[~joewitt] :
The rate is not particularly low, given that we deliberatly just ran one Thread for the Consumer ofr measuring the impact. In production we have many more threads configured for this, but would like not to over-spend either.
Please have a look at the attached screenshot from the Profiler. We interpret it as such as the processor spends 72% of the CPU time only in the "createRecordReader" method. 
Overall 96% is spent in "AbstractRecordStreamKafkaMessageConverter.toFlowFiles".

In the very common case of using Confluence SchemaRegistry to encode AvroSchemas for one AvroBinary Message per KafkaMessage and then continuing to pass the messages in Avro format inside the NiFi Cluster, this is pure oerhead. 

@[~exceptionfactory] :
Using the demarcator strategy would mean we'd have to inspect the bytestream and re-shuffle per Schema identifier at a later custom processor stage.

And since the internal structure of the ConsumeKafka processor is neatly divided in one class per ProcessingStrategy, we'd like this low-invasive code with the community.
I don't see how this would fit into a ControllerService. It's just a matter of batching the results of the KafkaClient ControllerService differently. 
!Screenshot from 2025-08-27 13-52-01.png!

> ConsumeKafka improvement for Avro+SchemaRegistry input
> ------------------------------------------------------
>
>                 Key: NIFI-14882
>                 URL: https://issues.apache.org/jira/browse/NIFI-14882
>             Project: Apache NiFi
>          Issue Type: Improvement
>          Components: Extensions
>    Affects Versions: 2.5.0
>         Environment: NiFi 2.4+, Kafka 3 
>            Reporter: Alex Goos
>            Priority: Major
>         Attachments: Screenshot from 2025-08-27 13-52-01.png
>
>          Time Spent: 40m
>  Remaining Estimate: 0h
>
> When Kafka receives Avro records marked with a schema identifier in the Confluent SchemaRegistry - and no immediate transformation is needed - the costly conversion into NiFiRecords and back to Avro is not needed. The performance can be substantially improved by simply batching records from the same source and with the same schema into a Avro Datafile. 
> A new ProcessingStrategy "Avro Datafile" can be added to the existing ones.
> In our setup, a VM with a Intel(R) Xeon(R) CPU E5-2695 v4 @ 2.10GHz CPU, and a single executor thread for ConsumeKafka this bumps throughput from 500MB/5Min (Precessing Strategy RECORD) to 17GB/5min
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)