You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by jgunvaldson <jg...@cox.net> on 2020/12/03 18:03:31 UTC
NIFI and Out of Memory Error
Just looking for an opinion
Knowing (for one example) that ReplaceText Processor can be very memory intensive with large files - we are finding it more and more common to wake up to an Out of Memory error like the following
2020-12-03 15:07:21,748ZUTC ERROR [Timer-Driven Process Thread-31] o.a.nifi.processors.standard.ReplaceText ReplaceText[id=352afe80-4195-3f56-8798-aaf8be160581] ReplaceText[id=352afe80-4195-3f56-8798-aaf8be160581] failed to process session due to java.lang.OutOfMemoryError: Java heap space; Processor Administratively Yielded for 1 sec: java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
at org.apache.nifi.processors.standard.ReplaceText.onTrigger(ReplaceText.java:255)
at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1162)
My question is this. Knowing that "When an OOME occurs in a JVM this can cause the JVM to skip instructions. Skipping instructions can compromise the integrity of the JVM memory without displaying errors. You can't always tell from the outside if a JVM has compromised memory, the only safe thing to do is restart the JVM.”
And in this case “Restart NIFI”
Is that “our collective” understanding also, that a Restart of NIFI is mandatory - or optional?
Thanks
John
Re: NIFI and Out of Memory Error
Posted by jgunvaldson <jg...@cox.net>.
I passed this along to the primary developers - Thanks Mike!
Best Regards
John
> On Dec 3, 2020, at 1:11 PM, Mike Thomsen <mi...@gmail.com> wrote:
>
> One of my colleagues ran into a similar situation, and all that was
> required to fix it was to make ReplaceText work line by line. When you
> do that, you shouldn't run into any issues.
>
Re: NIFI and Out of Memory Error
Posted by Mike Thomsen <mi...@gmail.com>.
One of my colleagues ran into a similar situation, and all that was
required to fix it was to make ReplaceText work line by line. When you
do that, you shouldn't run into any issues.
On Thu, Dec 3, 2020 at 1:04 PM jgunvaldson <jg...@cox.net> wrote:
>
> Just looking for an opinion
>
> Knowing (for one example) that ReplaceText Processor can be very memory intensive with large files - we are finding it more and more common to wake up to an Out of Memory error like the following
>
> 2020-12-03 15:07:21,748ZUTC ERROR [Timer-Driven Process Thread-31] o.a.nifi.processors.standard.ReplaceText ReplaceText[id=352afe80-4195-3f56-8798-aaf8be160581] ReplaceText[id=352afe80-4195-3f56-8798-aaf8be160581] failed to process session due to java.lang.OutOfMemoryError: Java heap space; Processor Administratively Yielded for 1 sec: java.lang.OutOfMemoryError: Java heap space
> java.lang.OutOfMemoryError: Java heap space
> at org.apache.nifi.processors.standard.ReplaceText.onTrigger(ReplaceText.java:255)
> at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1162)
>
>
> My question is this. Knowing that "When an OOME occurs in a JVM this can cause the JVM to skip instructions. Skipping instructions can compromise the integrity of the JVM memory without displaying errors. You can't always tell from the outside if a JVM has compromised memory, the only safe thing to do is restart the JVM.”
>
> And in this case “Restart NIFI”
>
> Is that “our collective” understanding also, that a Restart of NIFI is mandatory - or optional?
>
> Thanks
>
> John
>
Re: NIFI and Out of Memory Error
Posted by Joe Witt <jo...@gmail.com>.
I am honestly not sure if it is required - but it is probably a good idea.
Please let us know what you find using those.
Also, definitely we should change that flow to avoid large memory
consumption. Want to share more details on the input data and config that
results in this?
On Thu, Dec 3, 2020 at 11:21 AM jgunvaldson <jg...@cox.net> wrote:
> Thanks Joe,
>
> I am getting the general opinion that on OOM restart is not optional, must
> be done. In that case I am going to also look at some of the following
>
> -XX:+ExitOnOutOfMemoryError
> -XX:+CrashOnOutOfMemoryError
>
> *ExitOnOutOfMemoryError*
> When you enable this option, the JVM exits on the first occurrence of an
> out-of-memory error. It can be used if you prefer restarting an instance of
> the JVM rather than handling out of memory errors.
>
> *CrashOnOutOfMemoryError*
> If this option is enabled, when an out-of-memory error occurs, the JVM
> crashes and produces text and binary crash files.
>
> Best Regards
> John
>
>
>
> On Dec 3, 2020, at 10:09 AM, Joe Witt <jo...@gmail.com> wrote:
>
> John,
>
> First, as a general rule it is usually very doable to build flows which
> are very stream oriented rather than entire file oriented. That processor
> by its nature isn't friendly in this way if configured to work with large
> memory chunks. Alternatives often exist.
>
> Second, I do think it is wise to restart the JVM in the event of an OOME.
> There are ways to configure your JVM to do this automatically. Googling
> 'restart JVM on oome' for instance could be helpful there.
>
> Thanks
>
>>
>>
>
Re: NIFI and Out of Memory Error
Posted by jgunvaldson <jg...@cox.net>.
Thanks Joe,
I am getting the general opinion that on OOM restart is not optional, must be done. In that case I am going to also look at some of the following
-XX:+ExitOnOutOfMemoryError
-XX:+CrashOnOutOfMemoryError
ExitOnOutOfMemoryError
When you enable this option, the JVM exits on the first occurrence of an out-of-memory error. It can be used if you prefer restarting an instance of the JVM rather than handling out of memory errors.
CrashOnOutOfMemoryError
If this option is enabled, when an out-of-memory error occurs, the JVM crashes and produces text and binary crash files.
Best Regards
John
> On Dec 3, 2020, at 10:09 AM, Joe Witt <jo...@gmail.com> wrote:
>
> John,
>
> First, as a general rule it is usually very doable to build flows which are very stream oriented rather than entire file oriented. That processor by its nature isn't friendly in this way if configured to work with large memory chunks. Alternatives often exist.
>
> Second, I do think it is wise to restart the JVM in the event of an OOME. There are ways to configure your JVM to do this automatically. Googling 'restart JVM on oome' for instance could be helpful there.
>
> Thanks
>
Re: NIFI and Out of Memory Error
Posted by Joe Witt <jo...@gmail.com>.
John,
First, as a general rule it is usually very doable to build flows which are
very stream oriented rather than entire file oriented. That processor by
its nature isn't friendly in this way if configured to work with large
memory chunks. Alternatives often exist.
Second, I do think it is wise to restart the JVM in the event of an OOME.
There are ways to configure your JVM to do this automatically. Googling
'restart JVM on oome' for instance could be helpful there.
Thanks
On Thu, Dec 3, 2020 at 11:04 AM jgunvaldson <jg...@cox.net> wrote:
> Just looking for an opinion
>
> Knowing (for one example) that ReplaceText Processor can be very memory
> intensive with large files - we are finding it more and more common to wake
> up to an Out of Memory error like the following
>
> 2020-12-03 15:07:21,748ZUTC ERROR [Timer-Driven Process Thread-31]
> o.a.nifi.processors.standard.ReplaceText
> ReplaceText[id=352afe80-4195-3f56-8798-aaf8be160581]
> ReplaceText[id=352afe80-4195-3f56-8798-aaf8be160581] failed to process
> session due to java.lang.OutOfMemoryError: Java heap space; Processor
> Administratively Yielded for 1 sec: java.lang.OutOfMemoryError: Java heap
> space
> java.lang.OutOfMemoryError: Java heap space
> at
> org.apache.nifi.processors.standard.ReplaceText.onTrigger(ReplaceText.java:255)
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1162)
>
>
> My question is this. Knowing that "When an OOME occurs in a JVM this can
> cause the JVM to skip instructions. Skipping instructions can compromise
> the integrity of the JVM memory without displaying errors. You can't always
> tell from the outside if a JVM has compromised memory, the only safe thing
> to do is restart the JVM.”
>
> And in this case “Restart NIFI”
>
> Is that “our collective” understanding also, that a Restart of NIFI is
> mandatory - or optional?
>
> Thanks
>
> John
>
>