You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Asanka Sanjaya <an...@gmail.com> on 2018/12/20 08:07:55 UTC
Memory Leak in NiFi
Hi All,
NiFi version: 1.4.0, The same problem in v1.6.0 as well.
I have a nifi flow with multiple process groups and I have deployed nifi in
kubernetes using a docker container. I could see a very strange
behavior of memory consumption in nifi docker container. The only process
which runs in the docker container is nifi. See the attached screenshot of
the memory usage graph.
[image: image.png]
I tried stopping and starting the process groups to replicate the issue.
And I found out that the issue could be replicated by enabling a process
group which as a lot of ReplaceText processors. When I enable the process
group and parse several flow files through it, I can see a sudden spike in
the graph (more than 1 GB sometimes). See the screenshot below.
[image: image.png]
I reduced the Maximum buffer size of each processor to 100KB from 1MB while
the evaluation mode is set to Entire text mode. Then I ran the flow several
times and I couldn't reproduce the problem. So I assumed that the problem
was resolved.
Today I noticed the strange behavior again. See this graph.
[image: image.png]
But this hike was not due to this process group for sure, because no flow
file was going through this group at the time (This is a daily running job).
When I looked at the nifi log I could see this error message several times.
This error is the only error I can find in the log and this error has
logged exactly when the memory hike has happened.
018-12-20 06:08:20,939 ERROR [Timer-Driven Process Thread-9]
o.a.nifi.processors.standard.InvokeHTTP
InvokeHTTP[id=11791003-1f8c-1365-6043-eb6322e072f1] Routing to Failure due
to exception: java.net.SocketTimeoutException: connect timed out:
java.net.SocketTimeoutException: connect timed out
java.net.SocketTimeoutException: connect timed out
at java.net.PlainSocketImpl.socketConnect(Native Method)
at
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at okhttp3.internal.platform.Platform.connectSocket(Platform.java:124)
at
okhttp3.internal.connection.RealConnection.connectSocket(RealConnection.java:220)
at
okhttp3.internal.connection.RealConnection.connect(RealConnection.java:146)
at
okhttp3.internal.connection.StreamAllocation.findConnection(StreamAllocation.java:186)
at
okhttp3.internal.connection.StreamAllocation.findHealthyConnection(StreamAllocation.java:121)
at
okhttp3.internal.connection.StreamAllocation.newStream(StreamAllocation.java:100)
at
okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:42)
at
okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
at
okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
at
okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93)
at
okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
at
okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
at
okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
at
okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
at
okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:120)
at
okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
at
okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:179)
at okhttp3.RealCall.execute(RealCall.java:63)
at
org.apache.nifi.processors.standard.InvokeHTTP.onTrigger(InvokeHTTP.java:692)
at
org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
at
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1119)
at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
at
org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:128)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Above error message is due to a different processor which tries to connect
to a different server. At the time of the incident, the external server was
down so that the InvokeHTTP processor has done several retries and because
of that a sudden spike in memory. The memory is never getting released.
Can somebody assist with me regarding this problem, please?
--
*Thanks,*
Asanka Sanjaya Herath
Senior Software Engineer | Zone24x7
Re: Memory Leak in NiFi
Posted by Asanka Sanjaya <an...@gmail.com>.
Thank you, Joe. I will update the nifi to 1.7.
On Mon, Jan 28, 2019 at 9:20 AM Joe Percivall <jp...@apache.org> wrote:
> Hello Asanka,
>
> Apologies for the late reply, but this is due to a known issue[1] with the
> InvokeHttp processor where it did not close the response in all cases. This
> affected versions 1.4 through 1.6 and was fixed in 1.7.
>
> [1] https://issues.apache.org/jira/browse/NIFI-5085
>
> Joe
>
> On Thu, Dec 20, 2018 at 3:08 AM Asanka Sanjaya <an...@gmail.com> wrote:
>
>> Hi All,
>>
>> NiFi version: 1.4.0, The same problem in v1.6.0 as well.
>>
>> I have a nifi flow with multiple process groups and I have deployed nifi
>> in kubernetes using a docker container. I could see a very strange
>> behavior of memory consumption in nifi docker container. The only process
>> which runs in the docker container is nifi. See the attached screenshot of
>> the memory usage graph.
>>
>> [image: image.png]
>>
>>
>> I tried stopping and starting the process groups to replicate the issue.
>> And I found out that the issue could be replicated by enabling a process
>> group which as a lot of ReplaceText processors. When I enable the process
>> group and parse several flow files through it, I can see a sudden spike in
>> the graph (more than 1 GB sometimes). See the screenshot below.
>>
>> [image: image.png]
>>
>>
>> I reduced the Maximum buffer size of each processor to 100KB from 1MB
>> while the evaluation mode is set to Entire text mode. Then I ran the flow
>> several times and I couldn't reproduce the problem. So I assumed that the
>> problem was resolved.
>>
>> Today I noticed the strange behavior again. See this graph.
>> [image: image.png]
>>
>> But this hike was not due to this process group for sure, because no
>> flow file was going through this group at the time (This is a daily running
>> job).
>>
>> When I looked at the nifi log I could see this error message several
>> times. This error is the only error I can find in the log and this error
>> has logged exactly when the memory hike has happened.
>>
>> 018-12-20 06:08:20,939 ERROR [Timer-Driven Process Thread-9]
>> o.a.nifi.processors.standard.InvokeHTTP
>> InvokeHTTP[id=11791003-1f8c-1365-6043-eb6322e072f1] Routing to Failure due
>> to exception: java.net.SocketTimeoutException: connect timed out:
>> java.net.SocketTimeoutException: connect timed out
>> java.net.SocketTimeoutException: connect timed out
>> at java.net.PlainSocketImpl.socketConnect(Native Method)
>> at
>> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
>> at
>> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
>> at
>> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
>> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
>> at java.net.Socket.connect(Socket.java:589)
>> at okhttp3.internal.platform.Platform.connectSocket(Platform.java:124)
>> at
>> okhttp3.internal.connection.RealConnection.connectSocket(RealConnection.java:220)
>> at
>> okhttp3.internal.connection.RealConnection.connect(RealConnection.java:146)
>> at
>> okhttp3.internal.connection.StreamAllocation.findConnection(StreamAllocation.java:186)
>> at
>> okhttp3.internal.connection.StreamAllocation.findHealthyConnection(StreamAllocation.java:121)
>> at
>> okhttp3.internal.connection.StreamAllocation.newStream(StreamAllocation.java:100)
>> at
>> okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:42)
>> at
>> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
>> at
>> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
>> at
>> okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93)
>> at
>> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
>> at
>> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
>> at
>> okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
>> at
>> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
>> at
>> okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:120)
>> at
>> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
>> at
>> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
>> at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:179)
>> at okhttp3.RealCall.execute(RealCall.java:63)
>> at
>> org.apache.nifi.processors.standard.InvokeHTTP.onTrigger(InvokeHTTP.java:692)
>> at
>> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
>> at
>> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1119)
>> at
>> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
>> at
>> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
>> at
>> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:128)
>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:748)
>>
>>
>> Above error message is due to a different processor which tries to
>> connect to a different server. At the time of the incident, the external
>> server was down so that the InvokeHTTP processor has done several retries
>> and because of that a sudden spike in memory. The memory is never getting
>> released.
>>
>>
>> Can somebody assist with me regarding this problem, please?
>>
>>
>>
>> --
>>
>> *Thanks,*
>>
>> Asanka Sanjaya Herath
>>
>> Senior Software Engineer | Zone24x7
>>
>
>
> --
> *Joe Percivall*
> linkedin.com/in/Percivall
> e: jpercivall@apache.com
>
--
*Thanks,*
Asanka Sanjaya Herath
Senior Software Engineer | Zone24x7
Re: Memory Leak in NiFi
Posted by Joe Percivall <jp...@apache.org>.
Hello Asanka,
Apologies for the late reply, but this is due to a known issue[1] with the
InvokeHttp processor where it did not close the response in all cases. This
affected versions 1.4 through 1.6 and was fixed in 1.7.
[1] https://issues.apache.org/jira/browse/NIFI-5085
Joe
On Thu, Dec 20, 2018 at 3:08 AM Asanka Sanjaya <an...@gmail.com> wrote:
> Hi All,
>
> NiFi version: 1.4.0, The same problem in v1.6.0 as well.
>
> I have a nifi flow with multiple process groups and I have deployed nifi
> in kubernetes using a docker container. I could see a very strange
> behavior of memory consumption in nifi docker container. The only process
> which runs in the docker container is nifi. See the attached screenshot of
> the memory usage graph.
>
> [image: image.png]
>
>
> I tried stopping and starting the process groups to replicate the issue.
> And I found out that the issue could be replicated by enabling a process
> group which as a lot of ReplaceText processors. When I enable the process
> group and parse several flow files through it, I can see a sudden spike in
> the graph (more than 1 GB sometimes). See the screenshot below.
>
> [image: image.png]
>
>
> I reduced the Maximum buffer size of each processor to 100KB from 1MB
> while the evaluation mode is set to Entire text mode. Then I ran the flow
> several times and I couldn't reproduce the problem. So I assumed that the
> problem was resolved.
>
> Today I noticed the strange behavior again. See this graph.
> [image: image.png]
>
> But this hike was not due to this process group for sure, because no flow
> file was going through this group at the time (This is a daily running job).
>
> When I looked at the nifi log I could see this error message several
> times. This error is the only error I can find in the log and this error
> has logged exactly when the memory hike has happened.
>
> 018-12-20 06:08:20,939 ERROR [Timer-Driven Process Thread-9]
> o.a.nifi.processors.standard.InvokeHTTP
> InvokeHTTP[id=11791003-1f8c-1365-6043-eb6322e072f1] Routing to Failure due
> to exception: java.net.SocketTimeoutException: connect timed out:
> java.net.SocketTimeoutException: connect timed out
> java.net.SocketTimeoutException: connect timed out
> at java.net.PlainSocketImpl.socketConnect(Native Method)
> at
> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
> at
> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
> at
> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
> at java.net.Socket.connect(Socket.java:589)
> at okhttp3.internal.platform.Platform.connectSocket(Platform.java:124)
> at
> okhttp3.internal.connection.RealConnection.connectSocket(RealConnection.java:220)
> at
> okhttp3.internal.connection.RealConnection.connect(RealConnection.java:146)
> at
> okhttp3.internal.connection.StreamAllocation.findConnection(StreamAllocation.java:186)
> at
> okhttp3.internal.connection.StreamAllocation.findHealthyConnection(StreamAllocation.java:121)
> at
> okhttp3.internal.connection.StreamAllocation.newStream(StreamAllocation.java:100)
> at
> okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:42)
> at
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
> at
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
> at
> okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93)
> at
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
> at
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
> at
> okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
> at
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
> at
> okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:120)
> at
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
> at
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
> at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:179)
> at okhttp3.RealCall.execute(RealCall.java:63)
> at
> org.apache.nifi.processors.standard.InvokeHTTP.onTrigger(InvokeHTTP.java:692)
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1119)
> at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
> at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:128)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:748)
>
>
> Above error message is due to a different processor which tries to connect
> to a different server. At the time of the incident, the external server was
> down so that the InvokeHTTP processor has done several retries and because
> of that a sudden spike in memory. The memory is never getting released.
>
>
> Can somebody assist with me regarding this problem, please?
>
>
>
> --
>
> *Thanks,*
>
> Asanka Sanjaya Herath
>
> Senior Software Engineer | Zone24x7
>
--
*Joe Percivall*
linkedin.com/in/Percivall
e: jpercivall@apache.com