You are viewing a plain text version of this content. The canonical link for it is here.

Posted to users@nifi.apache.org by Valentina Ivanova <va...@ri.se> on 2020/06/25 12:49:57 UTC

NiFi 1.11 - High repository storage usage

Hello!

I see (screenshot attached) quite high (86%) storage usage for all three Flow File, Content & Provenance Repositories in the System Diagnostics. When I check the size of the respective folders on the disk they don't even add up to 12,24 GB shown on the screenshot.
I also have the following settings for the content repository in nifi.properties:
nifi.content.repository.archive.max.retention.period=12 hours
nifi.content.repository.archive.max.usage.percentage=50%
nifi.content.repository.archive.enabled=true
nifi.content.repository.always.sync=false


So I am wondering where these numbers are coming from and how to reduce the storage utilization (I assume such high utilization will impact the performance)? Why is the content repository usage 86% even though the max.usage.percentage is set at 50%.

Many thanks in advance & all the best

Valentina

Re: NiFi 1.11 - High repository storage usage

Posted by "Wesley C. Dias de Oliveira" <wc...@gmail.com>.

Hi, Valentina.

I've experienced the same issue on an old installation.

In my case, it was related to memory usage.

The system begins to dump things to the disk when there's no available
memory.

Have you checked the memory( params?

# JVM memory settings
java.arg.2=-Xms512m
java.arg.3=-Xmx512m






Em qui., 25 de jun. de 2020 às 09:50, Valentina Ivanova <
valentina.ivanova@ri.se> escreveu:

> Hello!
>
> I see (screenshot attached) quite high (86%) storage usage for all three
> Flow File, Content & Provenance Repositories in the System Diagnostics.
> When I check the size of the respective folders on the disk they don't even
> add up to 12,24 GB shown on the screenshot.
> I also have the following settings for the content repository in
> nifi.properties:
> nifi.content.repository.archive.max.retention.period=12 hours
> nifi.content.repository.archive.max.usage.percentage=50%
> nifi.content.repository.archive.enabled=true
> nifi.content.repository.always.sync=false
>
>
> So I am wondering where these numbers are coming from and how to reduce
> the storage utilization (I assume such high utilization will impact the
> performance)? Why is the content repository usage 86% even though the
> max.usage.percentage is set at 50%.
>
> Many thanks in advance & all the best
>
> Valentina
>
>
>
>

-- 
Grato,
Wesley C. Dias de Oliveira.

Linux User nº 576838.

Re: NiFi 1.11 - High repository storage usage

Posted by Valentina Ivanova <va...@ri.se>.

Hi Wesley & Harald,

Thanks for the quick replies!

@Wesley C. Dias de Oliveira<ma...@gmail.com> I have recently increased these settings due to other reasons, so that might be the primary cause.

@Harald You are right, they are on the same drive. I assumed that usage is relative to the nifi usage/settings not the OS. I will now change so each repository gets its own partition.

Thanks!

Valentina
________________________________
From: Dobbernack, Harald (Key-Work) <ha...@key-work.de>
Sent: Thursday, 25 June 2020 15:08
To: users@nifi.apache.org <us...@nifi.apache.org>
Subject: AW: NiFi 1.11 - High repository storage usage


  *   When I check the size of the respective folders on the disk they don't even add up to 12,24 GB shown on the screenshot.



I’m guessing you have other stuff on the drive or partition as well and not only the repositories.



Von: Dobbernack, Harald (Key-Work)
Gesendet: Donnerstag, 25. Juni 2020 14:58
An: users@nifi.apache.org
Betreff: AW: NiFi 1.11 - High repository storage usage



Hi,



presumably all three respositories are on the same partition or drive? I believe the screenshot view you posted shows what the OS reports as the space usage of the whole partition/drive on which the repositories are sitting. Best practice would be to place each repository on its own partition.



Greetings

Harald



Von: Valentina Ivanova <va...@ri.se>>
Gesendet: Donnerstag, 25. Juni 2020 14:50
An: users@nifi.apache.org<ma...@nifi.apache.org>
Betreff: NiFi 1.11 - High repository storage usage



Hello!



I see (screenshot attached) quite high (86%) storage usage for all three Flow File, Content & Provenance Repositories in the System Diagnostics. When I check the size of the respective folders on the disk they don't even add up to 12,24 GB shown on the screenshot.

I also have the following settings for the content repository in nifi.properties:

nifi.content.repository.archive.max.retention.period=12 hours

nifi.content.repository.archive.max.usage.percentage=50%

nifi.content.repository.archive.enabled=true

nifi.content.repository.always.sync=false





So I am wondering where these numbers are coming from and how to reduce the storage utilization (I assume such high utilization will impact the performance)? Why is the content repository usage 86% even though the max.usage.percentage is set at 50%.



Many thanks in advance & all the best



Valentina









Harald Dobbernack

Key-Work Consulting GmbH | Kriegsstr. 100 | 76133 | Karlsruhe | Germany | www.key-work.de<https://www.key-work.de> | Datenschutz<https://www.key-work.de/de/footer/datenschutz.html>
Fon: +49-721-78203-264 | E-Mail: harald.dobbernack@key-work.de | Fax: +49-721-78203-10

Key-Work Consulting GmbH, Karlsruhe, HRB 108695, HRG Mannheim
Geschäftsführer: Andreas Stappert, Tobin Wotring

AW: NiFi 1.11 - High repository storage usage

Posted by "Dobbernack, Harald (Key-Work)" <ha...@key-work.de>.

* When I check the size of the respective folders on the disk they don't even add up to 12,24 GB shown on the screenshot.

I'm guessing you have other stuff on the drive or partition as well and not only the repositories.

Von: Dobbernack, Harald (Key-Work)
Gesendet: Donnerstag, 25. Juni 2020 14:58
An: users@nifi.apache.org
Betreff: AW: NiFi 1.11 - High repository storage usage

Hi,

presumably all three respositories are on the same partition or drive? I believe the screenshot view you posted shows what the OS reports as the space usage of the whole partition/drive on which the repositories are sitting. Best practice would be to place each repository on its own partition.

Greetings
Harald

Von: Valentina Ivanova <va...@ri.se>>
Gesendet: Donnerstag, 25. Juni 2020 14:50
An: users@nifi.apache.org<ma...@nifi.apache.org>
Betreff: NiFi 1.11 - High repository storage usage

Hello!

I see (screenshot attached) quite high (86%) storage usage for all three Flow File, Content & Provenance Repositories in the System Diagnostics. When I check the size of the respective folders on the disk they don't even add up to 12,24 GB shown on the screenshot.
I also have the following settings for the content repository in nifi.properties:
nifi.content.repository.archive.max.retention.period=12 hours
nifi.content.repository.archive.max.usage.percentage=50%
nifi.content.repository.archive.enabled=true
nifi.content.repository.always.sync=false

So I am wondering where these numbers are coming from and how to reduce the storage utilization (I assume such high utilization will impact the performance)? Why is the content repository usage 86% even though the max.usage.percentage is set at 50%.

Many thanks in advance & all the best

Valentina

Harald Dobbernack

Key-Work Consulting GmbH, Karlsruhe, HRB 108695, HRG Mannheim
Gesch?ftsf?hrer: Andreas Stappert, Tobin Wotring