Clarification regarding swap of flow file to storage

[sanjeet rath <ra...@gmail.com>]

Hi Team,

As per the nifi document, once the number of  flow file in a connection
queue will reach the threshold value, then it removed from hash map(Java
heap space) and moved to disc.
could u please confirm the deault which repository it moves to , it's
content repository or to flowfile repository.

could i configure the swap directory using nifi.swap.storage.directory
parameter ?

Thanks,
sanjeet

Re: Clarification regarding swap of flow file to storage

[sanjeet rath <ra...@gmail.com>]

Thanks Brayn,

Your clarification is much appreciated.
Yours suggestions are always helpfull for me.
We have set up our projects with allmost all the default parameter present
in nifi.properties & bootstrap config file.

I am trying to improve performance, so trying to understand each
parameters.I have also gone through the nifi dip dive document.

Could you please suggest any attributes in those config files which i can
play arround to improve the performance.

suggestion of any tools to identify the performance bottleneck will be very
helpfull.

currently for onprem cluster log monitering we r using kibana & dynatrace
and for AWS cloud cluster DataDog is used.

once again thanks a lot for your support,

Sanjeet

On Fri, 6 Dec, 2019, 7:13 PM Bryan Bende, <bb...@gmail.com> wrote:

> Hello,
>
> Swap files are written under the flow file repo, in a sub directory
> called swap. I don't think it is configurable anymore.
>
> When a queue has flow files in it, the content of those flow file is
> not in memory in the queue, its just java objects representing the
> pointers to the flow files.
>
> So swapping is to help with memory when you have too many java
> objects, it writes out the information about what was in the queue to
> a swap file, but again its not the content of those flow files which
> is why it is under flow file repo.
>
> Thanks,
>
> Bryan
>
> On Fri, Dec 6, 2019 at 5:10 AM sanjeet rath <ra...@gmail.com>
> wrote:
> >
> > Hi Team,
> >
> > Any help on my trailed mail query will be very helpful.One more thing if
> i add a new directory for swap storage will it improve any  performance in
> processing.
> >
> > Thanks & Regards,
> > Sanjeet
> >
> > On Thu, 5 Dec, 2019, 12:06 PM sanjeet rath, <ra...@gmail.com>
> wrote:
> >>
> >> Hi Team,
> >>
> >> As per the nifi document, once the number of  flow file in a connection
> queue will reach the threshold value, then it removed from hash map(Java
> heap space) and moved to disc.
> >> could u please confirm the deault which repository it moves to , it's
> content repository or to flowfile repository.
> >>
> >> could i configure the swap directory using nifi.swap.storage.directory
> parameter ?
> >>
> >> Thanks,
> >> sanjeet
>

Re: Clarification regarding swap of flow file to storage

[Bryan Bende <bb...@gmail.com>]

Hello,

Swap files are written under the flow file repo, in a sub directory
called swap. I don't think it is configurable anymore.

When a queue has flow files in it, the content of those flow file is
not in memory in the queue, its just java objects representing the
pointers to the flow files.

So swapping is to help with memory when you have too many java
objects, it writes out the information about what was in the queue to
a swap file, but again its not the content of those flow files which
is why it is under flow file repo.

Thanks,

Bryan

On Fri, Dec 6, 2019 at 5:10 AM sanjeet rath <ra...@gmail.com> wrote:
>
> Hi Team,
>
> Any help on my trailed mail query will be very helpful.One more thing if i add a new directory for swap storage will it improve any  performance in processing.
>
> Thanks & Regards,
> Sanjeet
>
> On Thu, 5 Dec, 2019, 12:06 PM sanjeet rath, <ra...@gmail.com> wrote:
>>
>> Hi Team,
>>
>> As per the nifi document, once the number of  flow file in a connection queue will reach the threshold value, then it removed from hash map(Java heap space) and moved to disc.
>> could u please confirm the deault which repository it moves to , it's content repository or to flowfile repository.
>>
>> could i configure the swap directory using nifi.swap.storage.directory parameter ?
>>
>> Thanks,
>> sanjeet

Re: Clarification regarding swap of flow file to storage

[sanjeet rath <ra...@gmail.com>]

Hi Team,

Any help on my trailed mail query will be very helpful.One more thing if i
add a new directory for swap storage will it improve any  performance in
processing.

Thanks & Regards,
Sanjeet

On Thu, 5 Dec, 2019, 12:06 PM sanjeet rath, <ra...@gmail.com> wrote:

> Hi Team,
>
> As per the nifi document, once the number of  flow file in a connection
> queue will reach the threshold value, then it removed from hash map(Java
> heap space) and moved to disc.
> could u please confirm the deault which repository it moves to , it's
> content repository or to flowfile repository.
>
> could i configure the swap directory using nifi.swap.storage.directory
> parameter ?
>
> Thanks,
> sanjeet
>