You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by "Mark Payne (Jira)" <ji...@apache.org> on 2021/05/17 21:37:00 UTC
[jira] [Updated] (NIFI-8609) Improve efficiency of converting
Record object to Avro GenericRecord object
[ https://issues.apache.org/jira/browse/NIFI-8609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mark Payne updated NIFI-8609:
-----------------------------
Fix Version/s: 1.14.0
Status: Patch Available (was: Open)
> Improve efficiency of converting Record object to Avro GenericRecord object
> ---------------------------------------------------------------------------
>
> Key: NIFI-8609
> URL: https://issues.apache.org/jira/browse/NIFI-8609
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
> Reporter: Mark Payne
> Assignee: Mark Payne
> Priority: Major
> Fix For: 1.14.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> During some performance tests and profiling, I found that for a given flow, pushing Avro records to Kafka, one of the most expensive parts of the flow was converting our Record (MapRecord) object into a GenericRecord object for the Avro Writer.
> I created a simple unit test to determine a baseline for performance numbers before making any changes. The unit test creates a Record with 100 null String fields, half of which have a {{null}} value assigned to them. I then converted the record into an Avro GenericRecord via {{AvroTypeUtil.createAvroRecord(record, avroSchema);}} in a loop of 1,000,000 iterations and output how long it took; this was then repeated 1,000 times in order to allow the JVM to warm up.
> Numbers on my Macbook Pro showed after the first few iterations that the amount of time needed to convert 1 million records was on the order of 4.5 seconds.
> After updating the code, performance numbers are just under 2 seconds. So somewhere on the order of 2x better performance.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)