Inspecting content claims

[Isha Lamboo <is...@virtualsciences.nl>]

Hi all,

I’m investigating an issue where the content disk of one of our nifi 1.12 clusters is filling up while the active usage is very low (10-12 GB active data per node while the cluster view shows 98% full of 600 GB per node).
My first idea was that archiving wasn’t being run often enough or cleaning up enough data, but from what I read, I should see global backpressure being applied then.

For reference, these are the settings:
# Content Repository
nifi.content.claim.max.appendable.size=1 MB
nifi.content.repository.archive.max.retention.period=12 hours
nifi.content.repository.archive.max.usage.percentage=80%
nifi.content.repository.archive.enabled=true
nifi.content.repository.always.sync=false

Now my next suspect is a queue full of old flowfiles (50+ days in queue) in a disabled flow that keep content claims active amounting to much more diskspace than needed.
I know I can look up the Content Claim identifier in the content repository directory, but while I can see the size of the total file, I have no idea whether those are all active flowfiles or mostly archived.

Is there a more systematic way of inspecting active/inactive content in content claims?

Regards,

Isha

Re: Inspecting content claims

[Matthew Clarke <ma...@gmail.com>]

A content claim that has not been moved to an archive subdirectory within
the content_repository directory is considered still active.   Since a
content claim can contain the content for 1 too many FlowFiles and a
content claim can't be moved to archive until ALL FlowFiles referencing
that claim have reach there point of termination in your dataflows,  even a
small FlowFile left queen can hold up a large claim.  Not sure what version
of NiFi you are running, but if suggesting being in the most current and
making sure the spendable claim size set in your nifi.properties file is no
larger then 50kb.

On Tue, Apr 26, 2022, 8:20 AM Isha Lamboo <is...@virtualsciences.nl>
wrote:

> Hi all,
>
>
>
> I’m investigating an issue where the content disk of one of our nifi 1.12
> clusters is filling up while the active usage is very low (10-12 GB active
> data per node while the cluster view shows 98% full of 600 GB per node).
>
> My first idea was that archiving wasn’t being run often enough or cleaning
> up enough data, but from what I read, I should see global backpressure
> being applied then.
>
>
>
> For reference, these are the settings:
>
> # Content Repository
>
> nifi.content.claim.max.appendable.size=1 MB
>
> nifi.content.repository.archive.max.retention.period=12 hours
>
> nifi.content.repository.archive.max.usage.percentage=80%
>
> nifi.content.repository.archive.enabled=true
>
> nifi.content.repository.always.sync=false
>
>
>
> Now my next suspect is a queue full of old flowfiles (50+ days in queue)
> in a disabled flow that keep content claims active amounting to much more
> diskspace than needed.
>
> I know I can look up the Content Claim identifier in the content
> repository directory, but while I can see the size of the total file, I
> have no idea whether those are all active flowfiles or mostly archived.
>
>
>
> Is there a more systematic way of inspecting active/inactive content in
> content claims?
>
>
>
> Regards,
>
>
>
> Isha
>