Re: Content repository filling up

[Joe Witt <jo...@gmail.com>]

Olav

Your understanding sounds spot on to me and your settings seem quite fine
for what you're trying to do.  What version of NiFi are you working with?

Thanks
Joe

On Mon, Apr 24, 2017 at 9:26 PM, Olav Jordens <
Olav.Jordens@2degreesmobile.co.nz> wrote:

> Hi Users,
>
>
>
> I have had this problem intermittently for some time now – the content
> repository disk fills up even though there appear to be very few flow files
> in the system.
>
> I have read the very good explanation of content claims here:
> https://community.hortonworks.com/articles/82308/understanding-how-nifis-
> content-repository-archivi.html
>
>
>
> My data flows includes a mix of very large and very small files, and so I
> suspect that the small files within a claim are locking the large ones. I
> have followed the suggestion in the above link:
>
>
>
> *If you are working with data that ranges greatly from very small to very
> large, you may want to decrease the max appendable size and/or max flow
> file settings. By doing so you decrease the number of FlowFiles that make
> it into a single claim. This in turns reduces the likelihood of a single
> piece of data keeping large amounts of data still active in your content
> repository.*
>
>
>
> I have tried the most radical approach – one content claim per file which
> I believe should imply that as soon as a large file leaves the flow, it is
> available for removal as I have set archiving to false.
>
> My issue is that even with these settings, the nifi content repository
> fills up, and when I look inside the content repository, I see multiple
> flowfile contents contained within a single claim file, which is unexpected
> as I have set nifi.content.claim.max.flow.files=1.
>
>
>
>
>
> These are my content repository settings in nifi.properties:
>
>
>
> # Content Repository
>
> nifi.content.repository.implementation=org.apache.
> nifi.controller.repository.FileSystemRepository
>
> # Exceptionally important to get this right when having a mix of large and
> small files
>
> # We don't want a large file to be in the same claim as a small file which
> remains queued:
>
> # The claim can never be released until the small file is no longer
> enqueued and has been released
>
> # Large files, first into a claim, will take up an entire claim anyway.
>
> # So setting max.flow.files=1, there is no need to configure
> max.appendable.size
>
> nifi.content.claim.max.appendable.size=10 MB
>
> #nifi.content.claim.max.flow.files=100
>
> nifi.content.claim.max.flow.files=1
>
>
>
> #OPT
>
> #nifi.content.repository.directory.default=./content_repository
>
> nifi.content.repository.directory.default=/app/nifi/
> common/content_repository
>
>
>
> # Archiving of content is disabled - no need to keep data hanging around
> once the flow is complete.
>
> nifi.content.repository.archive.max.retention.period=12 hours
>
> nifi.content.repository.archive.max.usage.percentage=50%
>
> #nifi.content.repository.archive.enabled=true
>
> nifi.content.repository.archive.enabled=false
>
> nifi.content.repository.always.sync=false
>
> nifi.content.viewer.url=/nifi-content-viewer/
>
>
>
> Am I looking at this incorrectly?
>
>
>
> Thanks,
>
> Olav
>
>
>
>
>
>
>
> Olav Jordens
> Senior ETL Developer
> +64 226 202 429
> +64 9 919 7000 <+64%209-919%207000>
> 2degreesmobile.co.nz <http://www.2degreesmobile.co.nz>
> Two Degrees Mobile Limited | 47-49 George Street | Newmarket | Auckland |
> New Zealand
> PO Box 8355 | Symonds Street | Auckland 1150 | New Zealand | Fax +64 9
> 919 7001 <+64%209-919%207001>
>
> ------------------------------
>
> Disclaimer
> The e-mail and any files transmitted with it are confidential and may
> contain privileged or copyright information. If you are not the intended
> recipient you must not copy, distribute, or use this e-mail or the
> information contained in it for any purpose other than to notify us of the
> error. If you have received this message in error, please notify the sender
> immediately, by email or phone (+64 9 919 7000 <+64%209-919%207000>) and
> delete this email from your system. Any views expressed in this message are
> those of the individual sender, except where the sender specifically states
> them to be the views of Two Degrees Mobile Limited. We do not guarantee
> that this material is free from viruses or any other defects although due
> care has been taken to minimize the risk
>
>
>
>