NiFi 1.12.1 content repository clean up issues

[Bruno Gante <br...@gmail.com>]

Hi all,

I have a cluster running NiFi 1.12.1 that processes a significant amount of small flowfiles (around 3K/sec).

The system is performing quite good after some fine tuning but i am facing an issue regarding content repository which does not seem to clean up properly. For example, it reached 100% usage of all 3 nodes cluster, each one with 500GB content disk in less than 3 days.Today, I´ve also tried to turn content repo in memory and regarding performance is even better, however in terms of content repository cleanup it reached 12GB memory in just 6 hours.I do not have any old queued flow file that may block claim deletion. Each flow file has a lineage duration of 1-2 min average.

Any advise or guidance would be really appreciated.

Below my content repository config:


nifi.content.claim.max.appendable.size=10 MB
nifi.content.claim.max.flow.files=100
nifi.content.repository.directory.default=../content_repository
nifi.content.repository.archive.max.retention.period=1 hours
nifi.content.repository.archive.max.usage.percentage=80%
nifi.content.repository.archive.enabled=false
nifi.content.repository.always.sync=false

On top of that, does anyone know if there´s some way to get an estimation of required content repository size depending on the number of Read/Write volume and content config?

Thanks