You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by "wanglei2@geekplus.com.cn" <wa...@geekplus.com.cn> on 2019/07/27 13:09:54 UTC
OutOfMemoryError if the remote process group down
A processor send message to a remote process group。
If the remote process group crash or blocked， after some time the local nifi instance will throw OutOfMemoryError and hanged to die.
How can i avoid the Dying of the local Nifi instance?
wanglei2@geekplus.com.cn
Re: Re: OutOfMemoryError if the remote process group down
Posted by "wanglei2@geekplus.com.cn" <wa...@geekplus.com.cn>.
Thanks Purushotham Pushpavanth.
I have another question.
If don't send data through site to site and just run CaptureChangeMySQL in a singe cluster, seems there's no this error.
i stopped the downstream processor deliberately. As the picture shows, although it's blocking, the CaptureChangeMySQL seems don't process again, and no memory error.
I am using nifi-1.9.2
Thanks,
Lei
wanglei2@geekplus.com.cn
From: Purushotham Pushpavanthar
Date: 2019-07-30 04:01
To: users
Subject: Re: Re: OutOfMemoryError if the remote process group down
Hi Lei,
From your description, I'm assuming that the MiNiFi instance is going down due to OOM Error.
There is a bug in CaptureChangeMySQL processor, which has unbounded Blocking Queue as buffer for storing bin log events before translating to actual JSONs.
This Queue grows uncontrollably when the polling thread is slower than the thread offering to the buffer queue.
I faced a similar issue and mail thread related to that is here.
Workaround to avoid this is to make Retrieve All Records property to false and increasing your JVM heap size accordingly.
This will make sure your processor reads Near Real Time changes which will be very less and doesn't overload the heap.
I'm working on PR to address this issue.
Regards,
Purushotham Pushpavanth
On Sun, 28 Jul 2019 at 16:49, wanglei2@geekplus.com.cn <wa...@geekplus.com.cn> wrote:
Hi Mark,
Let me describe my scenario with details.
My sending site is a MiNiFi instance, an CaptureChangeMySQL local to a remote process group.
The jvm option: java.arg.2=-Xms512m java.arg.3=-Xmx512m
version: minifi-0.5.0
The remote instance is a cluster with two nodes. The remote process group is configured with HTTP protocol. The cluster side:
I stop the JoltTransfromJSON processor deliberately, after a while, the MiNiFi log shows:
o.a.nifi.remote.client.http.HttpClient Penalizing a peer Peer[url=http://10.44.51.33:8080/nifi-api] due to java.io.IOException: Unexpected response code: 503 errCode:Port's Destination is Full errMessage:Received port identifier 18c0bff2-016c-1000-ffff-ffffe3eb065c, but its destination is full
A few minutes later：
2019-07-28 19:03:22,273 WARN [Timer-Driven Process Thread-7] o.apache.nifi.controller.FlowController Unable to update Remote Process Groups due to java.lang.OutOfMemoryError: GC overhead limit exceeded
2019-07-28 19:03:28,508 ERROR [Remote Process Group 9b03993a-f870-3258-0000-000000000000 Thread-1] org.apache.nifi.engine.FlowEngine A flow controller task execution stopped abnormally
java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: GC overhead limit exceeded
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at org.apache.nifi.engine.FlowEngine.afterExecute(FlowEngine.java:100)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1157)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
Thanks，
Lei
wanglei2@geekplus.com.cn
From: Mark Payne
Date: 2019-07-27 21:24
To: users@nifi.apache.org
Subject: Re: OutOfMemoryError if the remote process group down
Hello,
This is not something that should be occurring, so it's difficult to say how to avoid the problem. I have a few questions to help understand the scenario better.
What version of NiFi is the sending side running?
Is the remote instance a single NiFi instance or a cluster?
When you start seeing OutOfMemoryError, how many FlowFiles are queued up waiting to be sent?
Do the FlowFiles to be sent have lots of attributes or any really large attributes?
Is the Remote Process Group configured to send data via "Raw Socket" or HTTP protocol? (this can be determined by right clicking on the Remote Process Group and going to Configure)
How large is the NiFi heap? (set in conf/bootstrap.conf, as the -Xms and -Xmx parameters)
Thanks
-Mark
On Jul 27, 2019, at 9:09 AM, wanglei2@geekplus.com.cn wrote:
A processor send message to a remote process group。
If the remote process group crash or blocked， after some time the local nifi instance will throw OutOfMemoryError and hanged to die.
How can i avoid the Dying of the local Nifi instance?
wanglei2@geekplus.com.cn
Re: Re: OutOfMemoryError if the remote process group down
Posted by Purushotham Pushpavanthar <pu...@gmail.com>.
Hi Lei,
From your description, I'm assuming that the MiNiFi instance is going down
due to OOM Error.
There is a bug <https://issues.apache.org/jira/browse/NIFI-6501> in
CaptureChangeMySQL processor, which has unbounded Blocking Queue as buffer
for storing bin log events before translating to actual JSONs.
This Queue grows uncontrollably when the polling thread is slower than the
thread offering to the buffer queue.
I faced a similar issue and mail thread related to that is here
<http://apache-nifi-developer-list.39713.n7.nabble.com/NiFi-Cluster-crashes-while-running-CaptureChangeMySQL-for-CDC-td20895.html>
.
Workaround to avoid this is to make *Retrieve All Records *property to *false
*and increasing your JVM heap size accordingly.
This will make sure your processor reads Near Real Time changes which will
be very less and doesn't overload the heap.
I'm working on PR to address this issue.
Regards,
Purushotham Pushpavanth
On Sun, 28 Jul 2019 at 16:49, wanglei2@geekplus.com.cn <
wanglei2@geekplus.com.cn> wrote:
> Hi Mark,
>
> Let me describe my scenario with details.
>
> - My sending site is a MiNiFi instance, an CaptureChangeMySQL
> local to a remote process group.
>
> The jvm option: java.arg.2=-Xms512m
> java.arg.3=-Xmx512m
> version: minifi-0.5.0
>
>
> - The remote instance is a cluster with two nodes. The remote process
> group is configured with HTTP protocol. The cluster side:
>
>
>
> I stop the JoltTransfromJSON processor deliberately, after a while,
> the MiNiFi log shows:
> o.a.nifi.remote.client.http.HttpClient Penalizing a peer Peer[url=
> http://10.44.51.33:8080/nifi-api] due to java.io.IOException: Unexpected
> response code: 503 errCode:Port's Destination is Full errMessage:Received
> port identifier 18c0bff2-016c-1000-ffff-ffffe3eb065c, but its destination
> is full
>
> A few minutes later：
> 2019-07-28 19:03:22,273 WARN [Timer-Driven Process Thread-7]
> o.apache.nifi.controller.FlowController Unable to update Remote Process
> Groups due to java.lang.OutOfMemoryError: GC overhead limit exceeded
> 2019-07-28 19:03:28,508 ERROR [Remote Process Group
> 9b03993a-f870-3258-0000-000000000000 Thread-1]
> org.apache.nifi.engine.FlowEngine A flow controller task execution stopped
> abnormally
> java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: GC
> overhead limit exceeded
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at
> org.apache.nifi.engine.FlowEngine.afterExecute(FlowEngine.java:100)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1157)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
>
> Thanks，
> Lei
>
> ------------------------------
> wanglei2@geekplus.com.cn
>
>
> *From:* Mark Payne <ma...@hotmail.com>
> *Date:* 2019-07-27 21:24
> *To:* users@nifi.apache.org
> *Subject:* Re: OutOfMemoryError if the remote process group down
> Hello,
>
> This is not something that should be occurring, so it's difficult to say
> how to avoid the problem. I have a few questions to help understand the
> scenario better.
>
> What version of NiFi is the sending side running?
> Is the remote instance a single NiFi instance or a cluster?
> When you start seeing OutOfMemoryError, how many FlowFiles are queued up
> waiting to be sent?
> Do the FlowFiles to be sent have lots of attributes or any really large
> attributes?
> Is the Remote Process Group configured to send data via "Raw Socket" or
> HTTP protocol? (this can be determined by right clicking on the Remote
> Process Group and going to Configure)
> How large is the NiFi heap? (set in conf/bootstrap.conf, as the -Xms and
> -Xmx parameters)
>
> Thanks
> -Mark
>
>
> On Jul 27, 2019, at 9:09 AM, wanglei2@geekplus.com.cn wrote:
>
>
> A processor send message to a remote process group。
> If the remote process group crash or blocked， after some time the local
> nifi instance will throw OutOfMemoryError and hanged to die.
>
> How can i avoid the Dying of the local Nifi instance?
>
>
> ------------------------------
> wanglei2@geekplus.com.cn
>
>
>
Re: OutOfMemoryError if the remote process group down
Posted by Mark Payne <ma...@hotmail.com>.
Lei,
I don't have MiNiFi setup or a MySQL instance to test against either. I attempted to replicate by creating a 2-node nifi cluster
and using a GenerateFlowFile to generate the data and send over HTTP-based site-to-site. Stopped the processor after the
input port to see if i could duplicate the behavior. Unfortunately, though, I wasn't able to cause any sort of problems.
In order to understand what's happening, you'll likely have to grab a heap dump and analyze that.
On Jul 28, 2019, at 7:18 AM, wanglei2@geekplus.com.cn<ma...@geekplus.com.cn> wrote:
Hi Mark,
Let me describe my scenario with details.
* My sending site is a MiNiFi instance, an CaptureChangeMySQL local to a remote process group.
The jvm option: java.arg.2=-Xms512m java.arg.3=-Xmx512m
version: minifi-0.5.0
<Catch.jpg>
* The remote instance is a cluster with two nodes. The remote process group is configured with HTTP protocol. The cluster side:
<Catch8314.jpg>
I stop the JoltTransfromJSON processor deliberately, after a while, the MiNiFi log shows:
o.a.nifi.remote.client.http.HttpClient Penalizing a peer Peer[url=http://10.44.51.33:8080/nifi-api] due to java.io.IOException: Unexpected response code: 503 errCode:Port's Destination is Full errMessage:Received port identifier 18c0bff2-016c-1000-ffff-ffffe3eb065c, but its destination is full
A few minutes later：
2019-07-28 19:03:22,273 WARN [Timer-Driven Process Thread-7] o.apache.nifi.controller.FlowController Unable to update Remote Process Groups due to java.lang.OutOfMemoryError: GC overhead limit exceeded
2019-07-28 19:03:28,508 ERROR [Remote Process Group 9b03993a-f870-3258-0000-000000000000 Thread-1] org.apache.nifi.engine.FlowEngine A flow controller task execution stopped abnormally
java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: GC overhead limit exceeded
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at org.apache.nifi.engine.FlowEngine.afterExecute(FlowEngine.java:100)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1157)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
Thanks，
Lei
________________________________
wanglei2@geekplus.com.cn<ma...@geekplus.com.cn>
From: Mark Payne<ma...@hotmail.com>
Date: 2019-07-27 21:24
To: users@nifi.apache.org<ma...@nifi.apache.org>
Subject: Re: OutOfMemoryError if the remote process group down
Hello,
This is not something that should be occurring, so it's difficult to say how to avoid the problem. I have a few questions to help understand the scenario better.
What version of NiFi is the sending side running?
Is the remote instance a single NiFi instance or a cluster?
When you start seeing OutOfMemoryError, how many FlowFiles are queued up waiting to be sent?
Do the FlowFiles to be sent have lots of attributes or any really large attributes?
Is the Remote Process Group configured to send data via "Raw Socket" or HTTP protocol? (this can be determined by right clicking on the Remote Process Group and going to Configure)
How large is the NiFi heap? (set in conf/bootstrap.conf, as the -Xms and -Xmx parameters)
Thanks
-Mark
On Jul 27, 2019, at 9:09 AM, wanglei2@geekplus.com.cn<ma...@geekplus.com.cn> wrote:
A processor send message to a remote process group。
If the remote process group crash or blocked， after some time the local nifi instance will throw OutOfMemoryError and hanged to die.
How can i avoid the Dying of the local Nifi instance?
________________________________
wanglei2@geekplus.com.cn<ma...@geekplus.com.cn>
Re: Re: OutOfMemoryError if the remote process group down
Posted by "wanglei2@geekplus.com.cn" <wa...@geekplus.com.cn>.
Hi Mark,
Let me describe my scenario with details.
My sending site is a MiNiFi instance, an CaptureChangeMySQL local to a remote process group.
The jvm option: java.arg.2=-Xms512m java.arg.3=-Xmx512m
version: minifi-0.5.0
The remote instance is a cluster with two nodes. The remote process group is configured with HTTP protocol. The cluster side:
I stop the JoltTransfromJSON processor deliberately, after a while, the MiNiFi log shows:
o.a.nifi.remote.client.http.HttpClient Penalizing a peer Peer[url=http://10.44.51.33:8080/nifi-api] due to java.io.IOException: Unexpected response code: 503 errCode:Port's Destination is Full errMessage:Received port identifier 18c0bff2-016c-1000-ffff-ffffe3eb065c, but its destination is full
A few minutes later：
2019-07-28 19:03:22,273 WARN [Timer-Driven Process Thread-7] o.apache.nifi.controller.FlowController Unable to update Remote Process Groups due to java.lang.OutOfMemoryError: GC overhead limit exceeded
2019-07-28 19:03:28,508 ERROR [Remote Process Group 9b03993a-f870-3258-0000-000000000000 Thread-1] org.apache.nifi.engine.FlowEngine A flow controller task execution stopped abnormally
java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: GC overhead limit exceeded
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at org.apache.nifi.engine.FlowEngine.afterExecute(FlowEngine.java:100)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1157)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
Thanks，
Lei
wanglei2@geekplus.com.cn
From: Mark Payne
Date: 2019-07-27 21:24
To: users@nifi.apache.org
Subject: Re: OutOfMemoryError if the remote process group down
Hello,
This is not something that should be occurring, so it's difficult to say how to avoid the problem. I have a few questions to help understand the scenario better.
What version of NiFi is the sending side running?
Is the remote instance a single NiFi instance or a cluster?
When you start seeing OutOfMemoryError, how many FlowFiles are queued up waiting to be sent?
Do the FlowFiles to be sent have lots of attributes or any really large attributes?
Is the Remote Process Group configured to send data via "Raw Socket" or HTTP protocol? (this can be determined by right clicking on the Remote Process Group and going to Configure)
How large is the NiFi heap? (set in conf/bootstrap.conf, as the -Xms and -Xmx parameters)
Thanks
-Mark
On Jul 27, 2019, at 9:09 AM, wanglei2@geekplus.com.cn wrote:
A processor send message to a remote process group。
If the remote process group crash or blocked， after some time the local nifi instance will throw OutOfMemoryError and hanged to die.
How can i avoid the Dying of the local Nifi instance?
wanglei2@geekplus.com.cn
Re: OutOfMemoryError if the remote process group down
Posted by Mark Payne <ma...@hotmail.com>.
Hello,
This is not something that should be occurring, so it's difficult to say how to avoid the problem. I have a few questions to help understand the scenario better.
What version of NiFi is the sending side running?
Is the remote instance a single NiFi instance or a cluster?
When you start seeing OutOfMemoryError, how many FlowFiles are queued up waiting to be sent?
Do the FlowFiles to be sent have lots of attributes or any really large attributes?
Is the Remote Process Group configured to send data via "Raw Socket" or HTTP protocol? (this can be determined by right clicking on the Remote Process Group and going to Configure)
How large is the NiFi heap? (set in conf/bootstrap.conf, as the -Xms and -Xmx parameters)
Thanks
-Mark
On Jul 27, 2019, at 9:09 AM, wanglei2@geekplus.com.cn<ma...@geekplus.com.cn> wrote:
A processor send message to a remote process group。
If the remote process group crash or blocked， after some time the local nifi instance will throw OutOfMemoryError and hanged to die.
How can i avoid the Dying of the local Nifi instance?
________________________________
wanglei2@geekplus.com.cn<ma...@geekplus.com.cn>