You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by "Pierre Villard (Jira)" <ji...@apache.org> on 2026/02/24 15:32:00 UTC

[jira] [Updated] (NIFI-15526) Add Fast Reader Enabled property to Avro Reader

     [ https://issues.apache.org/jira/browse/NIFI-15526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Pierre Villard updated NIFI-15526:
----------------------------------
    Fix Version/s: 2.8.0
       Resolution: Fixed
           Status: Resolved  (was: Patch Available)

> Add Fast Reader Enabled property to Avro Reader
> -----------------------------------------------
>
>                 Key: NIFI-15526
>                 URL: https://issues.apache.org/jira/browse/NIFI-15526
>             Project: Apache NiFi
>          Issue Type: Improvement
>          Components: Core Framework
>    Affects Versions: 2.7.1, 2.7.2
>            Reporter: Josef Zahner
>            Assignee: Pierre Villard
>            Priority: Critical
>             Fix For: 2.8.0
>
>         Attachments: image-2026-01-29-16-14-08-609.png, image-2026-01-29-16-14-29-698.png, image-2026-01-29-16-16-01-232.png
>
>          Time Spent: 2.5h
>  Remaining Estimate: 0h
>
> We found an issue causes OOM (used java heap full) for NiFi > v2.5.0 if we use AVRO Readers/Writers. It seems that the AVRO Library used in the newest NiFi versions has AVRO {{{}org.apache.avro.fastread{}}}=true per default. This causes massive OOM problems on our side. We suspect it is due to the inheritance of the reader/writer schemas.
> If we disable fastread in {{{}/conf/bootstrap.conf{}}}, the memory leak is gone:
> {code:java}
> java.arg.28=-Dorg.apache.avro.fastread=false{code}
> Here real world outputs based on prometheus metrics from used heap for NiFi. The screenshot 1 (blue line) shows a max 32GB heap, which never goes down again, it just grows (until it would reach 32GB and crashes). Thats taken with NiFI 2.7.2 with nothing changed regarding AVRO fast read (default, seems to be enabled).
> !image-2026-01-29-16-14-08-609.png!
> Screenshot 2 (orange line) shows another cluster with more or less the same use case, but with fastread=false. No memory leak anymore. So it's clearly related to the AVRO fastread feature.
> !image-2026-01-29-16-14-29-698.png!
> Yourkit Profiler example of the FastReaderBuilder object and memory usage. 1 Object consumes about 10GB memory (fastread is on default, so enabled) .
> !image-2026-01-29-16-16-01-232.png!
>  
> Would be great if someone of the devs, which is more familiar with all this topics, could investigate it.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)