Re: NiFi ram usage

[Adam Lamar <ad...@gmail.com>]

Hi All,

Thanks for the messages, especially thanks for those doc links Andy! I
think it would definitely be an improvement to add some memory requirements
to the docs, even just a ballpark minimum figure. NiFi on my mac uses about
1GB at startup too (with no flows).

Mike, Jeff,

On a fresh Linux instance with 1GB of memory, `free -m` reports about 650MB
completely free, and 800 available. There are no databases or processes
running since the instance is completely new at that point. So while there
is some OS overhead, there is a decent amount available to work with too.

Although the jvm heap size is set to 512MB, the jvm still uses about 1.1GB
of OS memory at startup. In addition to the heap memory, the jvm needs
memory to load classes amongst other things. In reality a java process uses
more than the heap size specification only, and all of NiFi is just a lot
of code :)

The good news: I did a little experiment by removing all the nars in the
lib/ directory except for these:

- nifi-framework
- nifi-jetty-bundle
- nifi-provenance-repository
- nifi-standard-nar
- nifi-standard-services
- nifi-aws-nar (since I was using aws processors)

I also changed the bootstrap to use `java.arg.2=-Xms100m` to avoid NiFi
gobbling up too much ram initially.

Not only did NiFi start, but after moving 100GB of content, NiFi was only
using about 575MB total OS memory! About half of what I had seen
previously, and had no trouble operating on the 1GB Linux instance.

Hopefully this is useful to someone else who wants to run NiFi in a more
constrained environment (and save a few dollars on cloud resources too).

Thanks all for the discussion.

Adam


On Thu, Aug 31, 2017 at 10:20 AM, Andy LoPresto <al...@apache.org>
wrote:

> Adam,
>
> Hopefully the team is able to get your specific issue resolved here, but
> to answer a question you asked that I think may have been missed, we do
> have “System Requirements” [1] and “Configuration Best Practices” [2]
> (which is really “Additional Requirements” now) both documented. Neither
> give an explicit value for memory, so that’s an opportunity for us to
> improve the documentation.
>
> [1] https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#
> system-requirements
> [2] https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#
> configuration-best-practices
>
> Andy LoPresto
> alopresto@apache.org
> *alopresto.apache@gmail.com <al...@gmail.com>*
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> On Aug 31, 2017, at 5:38 AM, Jeff <jt...@gmail.com> wrote:
>
> Adam,
>
> Mike brings up a good point...  When your VM has started, and you haven't
> started NiFi yet, how much memory is free in the system?  An instance of
> NiFi with an empty flow should have no trouble running in 512mb of heap
> space.  I have a flow with a few processors on it and the heap usage
> averages around 250mb for me, a default bootstrap.conf.
>
> On Thu, Aug 31, 2017 at 7:59 AM Pierre Villard <
> pierre.villard.fr@gmail.com> wrote:
>
>> As Jeff, I'm a bit surprised by what you are experiencing. I've never
>> changed the default values of 512MB when working with NiFi on my laptop and
>> never hit OOM errors. Are you sure that 1GB is available on the VM before
>> starting NiFi?
>>
>> Pierre
>>
>> 2017-08-31 13:52 GMT+02:00 Mike Thomsen <mi...@gmail.com>:
>>
>>> Adam,
>>>
>>> I cannot say exactly why the default settings won't work for you on a
>>> clean installation, but it likely has to do with how small the VM is. The
>>> OS overhead alone is probably a few hundred MB of RAM. If you have anything
>>> else running, even just MySQL or MongoDB it's entirely possible that you
>>> actually don't have enough memory to give even 512MB to NiFi.
>>>
>>> My recommendation would be 4GB of RAM for the VM with Xms1G and Xmx2G
>>> for the heap sizes. That's very reasonable for experimenting with something
>>> like NiFi. The ram usage is very difficult to calculate in advance because
>>> it's based entirely on what you're doing with NiFi.
>>>
>>> Mike
>>>
>>> On Wed, Aug 30, 2017 at 11:45 PM, Adam Lamar <ad...@gmail.com>
>>> wrote:
>>>
>>>> Jeff,
>>>>
>>>> This was a new installation so I actually hadn't set up any flows yet.
>>>> NiFi wouldn't start immediately after installation (before I could
>>>> configure any flows) because the system had too little ram. The 1.1GB
>>>> figure is private (RSS) memory usage, which exceeded the 1GB instance limit
>>>> (and the instance had no swap configured).
>>>>
>>>> Is there any system requirements documentation? I couldn't find any
>>>> docs on minimum system specs, so I guess I'm wondering if the ram usage is
>>>> known and expected, and if there are any ways to get the ram usage down.
>>>>
>>>> Thanks in advance,
>>>> Adam
>>>>
>>>> ​
>>>>
>>>
>>>
>>
>