You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by "Mark Payne (Jira)" <ji...@apache.org> on 2021/05/17 21:37:00 UTC

[jira] [Updated] (NIFI-8609) Improve efficiency of converting Record object to Avro GenericRecord object

     [ https://issues.apache.org/jira/browse/NIFI-8609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Mark Payne updated NIFI-8609:
-----------------------------
    Fix Version/s: 1.14.0
           Status: Patch Available  (was: Open)

> Improve efficiency of converting Record object to Avro GenericRecord object
> ---------------------------------------------------------------------------
>
>                 Key: NIFI-8609
>                 URL: https://issues.apache.org/jira/browse/NIFI-8609
>             Project: Apache NiFi
>          Issue Type: Improvement
>          Components: Extensions
>            Reporter: Mark Payne
>            Assignee: Mark Payne
>            Priority: Major
>             Fix For: 1.14.0
>
>          Time Spent: 10m
>  Remaining Estimate: 0h
>
> During some performance tests and profiling, I found that for a given flow, pushing Avro records to Kafka, one of the most expensive parts of the flow was converting our Record (MapRecord) object into a GenericRecord object for the Avro Writer.
> I created a simple unit test to determine a baseline for performance numbers before making any changes. The unit test creates a Record with 100 null String fields, half of which have a {{null}} value assigned to them. I then converted the record into an Avro GenericRecord via {{AvroTypeUtil.createAvroRecord(record, avroSchema);}} in a loop of 1,000,000 iterations and output how long it took; this was then repeated 1,000 times in order to allow the JVM to warm up.
> Numbers on my Macbook Pro showed after the first few iterations that the amount of time needed to convert 1 million records was on the order of 4.5 seconds.
> After updating the code, performance numbers are just under 2 seconds. So somewhere on the order of 2x better performance.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)