You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Vijay Chhipa <vc...@apple.com> on 2021/01/25 04:59:45 UTC
InvokeHTTP hangs after several successful calls
Hi All,
I am seeing a case where InvokeHTTP processor hangs after several successful calls to an endpoint. (Processor is paginating over millions of rows and pulling a few thousand rows at a time)
If I have more than one thread scheduled, all threads hang.
When this happens InvokeHTTP does not release the current flowfile at all. My only recourse is to try to stop the processor.
This "stop" does not complete, at which point I have to terminate the processor.
After termination of this processor, starting of the processor does not process any files. So I have to remove the processor and all its connections, delete the processor and replace it with another InvokeHTTP with same properties and connections.
I have tried to set the "Connection: close" header but it didn't help.
Has anyone seen this behaviors, how did you resolve it?
Thanks
Re: Monitoring NiFi Root Level Controller Services
Posted by jgunvaldson <jg...@cox.net>.
Hi Daniel,
And yes I am “John” (Joe was correct) - I have some experience with Nipyapi and yes, will probably be using it - I was wondering about a 3rd party (not DataDog) monitoring service (designed) for working with NiFi as recommendations from the group, but certainly - writing our own monitors to use API is probably what we will do
Best Regards
John
> On Feb 1, 2021, at 10:48 PM, Daniel Chaffelson <ch...@gmail.com> wrote:
>
> Hi John, the python client nipyapi already has convenience methods in canvas.py for this, you are looking for a component status of Running.
>
> On Tue, 2 Feb 2021, 01:18 jgunvaldson, <jgunvaldson@cox.net <ma...@cox.net>> wrote:
> Wiring up a rest client to make a rest request to the NIFI API, seems the following three are possible candidates:
>
> GET /controller-services/{id} Gets a controller service
>
> GET /controller-services/{id}/references Gets a controller service
>
> GET /controller-services/{id}/state Gets the state for a controller service
>
> I’ll do some initial tests with Postman to see what is returned in response json payload (and if 200 means up and running, or just a successful API request)
>
> Sounds good
> John
>
>
>
>> On Feb 1, 2021, at 4:46 PM, Joe Witt <joe.witt@gmail.com <ma...@gmail.com>> wrote:
>>
>> John,
>>
>> You're using a vendor distribution of NiFi. You should contact the vendor.
>>
>> You can certainly monitor the state of a controller service via the REST API. They should either be enabled or not enabled.
>>
>> Thanks
>>
>> On Mon, Feb 1, 2021 at 5:39 PM jgunvaldson <jgunvaldson@cox.net <ma...@cox.net>> wrote:
>> Hi All,
>>
>> Root level (canvas) Controller Services - We tend to setup several Root level Controller services for developers that are typically DBCPConnectionPools and DistributedMapCacheClientService and maybe a few other. MOST importantly, these controller services cannot be down and cannot be disabled - must remain Enabled at all times.
>>
>> We have now had a few outages where upon examination a Controller Service has encountered “something” that caused it to be Disabled or Down.
>>
>> Is there a standard practice we can use to “Monitor” the controller services and ensure we get alerted if any one of them goes into a Disable state?
>>
>> What do folks generally think is a good monitoring practice?
>>
>> We are on
>>
>> HDF Version 3.4.1.1.
>>
>> Powered by Apache NiFi Version 1.9.0
>> 1.9.0.3.4.1.1-4 built 05/01/2019 02:15:30 UTC
>> Tagged nifi-1.9.0-RC2
>> From 7410fa4 on branch UNKNOWN
>>
>>
>
Re: Monitoring NiFi Root Level Controller Services
Posted by Daniel Chaffelson <ch...@gmail.com>.
Hi John, the python client nipyapi already has convenience methods in
canvas.py for this, you are looking for a component status of Running.
On Tue, 2 Feb 2021, 01:18 jgunvaldson, <jg...@cox.net> wrote:
> Wiring up a rest client to make a rest request to the NIFI API, seems the
> following three are possible candidates:
>
>
> - GET /controller-services/{id} Gets a
> controller service
> -
> - GET /controller-services/{id}/references Gets a controller
> service
> -
> - GET /controller-services/{id}/state Gets the state
> for a controller service
>
>
> I’ll do some initial tests with Postman to see what is returned in
> response json payload (and if 200 means up and running, or just a
> successful API request)
>
> Sounds good
> John
>
>
>
> On Feb 1, 2021, at 4:46 PM, Joe Witt <jo...@gmail.com> wrote:
>
> John,
>
> You're using a vendor distribution of NiFi. You should contact the vendor.
>
> You can certainly monitor the state of a controller service via the REST
> API. They should either be enabled or not enabled.
>
> Thanks
>
> On Mon, Feb 1, 2021 at 5:39 PM jgunvaldson <jg...@cox.net> wrote:
>
>> Hi All,
>>
>> Root level (canvas) Controller Services - We tend to setup several Root
>> level Controller services for developers that are typically
>> DBCPConnectionPools and DistributedMapCacheClientService and maybe a few
>> other. MOST importantly, these controller services cannot be down and
>> cannot be disabled - must remain Enabled at all times.
>>
>> We have now had a few outages where upon examination a Controller Service
>> has encountered “something” that caused it to be Disabled or Down.
>>
>> Is there a standard practice we can use to “Monitor” the controller
>> services and ensure we get alerted if any one of them goes into a Disable
>> state?
>>
>> What do folks generally think is a good monitoring practice?
>>
>> We are on
>>
>> HDF Version 3.4.1.1.
>>
>> Powered by Apache NiFi Version 1.9.0
>> 1.9.0.3.4.1.1-4 built 05/01/2019 02:15:30 UTC
>> Tagged nifi-1.9.0-RC2
>> From 7410fa4 on branch UNKNOWN
>>
>>
>>
>
Re: Monitoring NiFi Root Level Controller Services
Posted by jgunvaldson <jg...@cox.net>.
Wiring up a rest client to make a rest request to the NIFI API, seems the following three are possible candidates:
GET /controller-services/{id} Gets a controller service
GET /controller-services/{id}/references Gets a controller service
GET /controller-services/{id}/state Gets the state for a controller service
I’ll do some initial tests with Postman to see what is returned in response json payload (and if 200 means up and running, or just a successful API request)
Sounds good
John
> On Feb 1, 2021, at 4:46 PM, Joe Witt <jo...@gmail.com> wrote:
>
> John,
>
> You're using a vendor distribution of NiFi. You should contact the vendor.
>
> You can certainly monitor the state of a controller service via the REST API. They should either be enabled or not enabled.
>
> Thanks
>
> On Mon, Feb 1, 2021 at 5:39 PM jgunvaldson <jgunvaldson@cox.net <ma...@cox.net>> wrote:
> Hi All,
>
> Root level (canvas) Controller Services - We tend to setup several Root level Controller services for developers that are typically DBCPConnectionPools and DistributedMapCacheClientService and maybe a few other. MOST importantly, these controller services cannot be down and cannot be disabled - must remain Enabled at all times.
>
> We have now had a few outages where upon examination a Controller Service has encountered “something” that caused it to be Disabled or Down.
>
> Is there a standard practice we can use to “Monitor” the controller services and ensure we get alerted if any one of them goes into a Disable state?
>
> What do folks generally think is a good monitoring practice?
>
> We are on
>
> HDF Version 3.4.1.1.
>
> Powered by Apache NiFi Version 1.9.0
> 1.9.0.3.4.1.1-4 built 05/01/2019 02:15:30 UTC
> Tagged nifi-1.9.0-RC2
> From 7410fa4 on branch UNKNOWN
>
>
Re: Monitoring NiFi Root Level Controller Services
Posted by Joe Witt <jo...@gmail.com>.
also sorry for referencing you as 'John' - not sure why I just assumed that
:)
On Mon, Feb 1, 2021 at 5:46 PM Joe Witt <jo...@gmail.com> wrote:
> John,
>
> You're using a vendor distribution of NiFi. You should contact the vendor.
>
> You can certainly monitor the state of a controller service via the REST
> API. They should either be enabled or not enabled.
>
> Thanks
>
> On Mon, Feb 1, 2021 at 5:39 PM jgunvaldson <jg...@cox.net> wrote:
>
>> Hi All,
>>
>> Root level (canvas) Controller Services - We tend to setup several Root
>> level Controller services for developers that are typically
>> DBCPConnectionPools and DistributedMapCacheClientService and maybe a few
>> other. MOST importantly, these controller services cannot be down and
>> cannot be disabled - must remain Enabled at all times.
>>
>> We have now had a few outages where upon examination a Controller Service
>> has encountered “something” that caused it to be Disabled or Down.
>>
>> Is there a standard practice we can use to “Monitor” the controller
>> services and ensure we get alerted if any one of them goes into a Disable
>> state?
>>
>> What do folks generally think is a good monitoring practice?
>>
>> We are on
>>
>> HDF Version 3.4.1.1.
>>
>> Powered by Apache NiFi Version 1.9.0
>> 1.9.0.3.4.1.1-4 built 05/01/2019 02:15:30 UTC
>> Tagged nifi-1.9.0-RC2
>> From 7410fa4 on branch UNKNOWN
>>
>>
>>
Re: Monitoring NiFi Root Level Controller Services
Posted by Joe Witt <jo...@gmail.com>.
John,
You're using a vendor distribution of NiFi. You should contact the vendor.
You can certainly monitor the state of a controller service via the REST
API. They should either be enabled or not enabled.
Thanks
On Mon, Feb 1, 2021 at 5:39 PM jgunvaldson <jg...@cox.net> wrote:
> Hi All,
>
> Root level (canvas) Controller Services - We tend to setup several Root
> level Controller services for developers that are typically
> DBCPConnectionPools and DistributedMapCacheClientService and maybe a few
> other. MOST importantly, these controller services cannot be down and
> cannot be disabled - must remain Enabled at all times.
>
> We have now had a few outages where upon examination a Controller Service
> has encountered “something” that caused it to be Disabled or Down.
>
> Is there a standard practice we can use to “Monitor” the controller
> services and ensure we get alerted if any one of them goes into a Disable
> state?
>
> What do folks generally think is a good monitoring practice?
>
> We are on
>
> HDF Version 3.4.1.1.
>
> Powered by Apache NiFi Version 1.9.0
> 1.9.0.3.4.1.1-4 built 05/01/2019 02:15:30 UTC
> Tagged nifi-1.9.0-RC2
> From 7410fa4 on branch UNKNOWN
>
>
>
Monitoring NiFi Root Level Controller Services
Posted by jgunvaldson <jg...@cox.net>.
Hi All,
Root level (canvas) Controller Services - We tend to setup several Root level Controller services for developers that are typically DBCPConnectionPools and DistributedMapCacheClientService and maybe a few other. MOST importantly, these controller services cannot be down and cannot be disabled - must remain Enabled at all times.
We have now had a few outages where upon examination a Controller Service has encountered “something” that caused it to be Disabled or Down.
Is there a standard practice we can use to “Monitor” the controller services and ensure we get alerted if any one of them goes into a Disable state?
What do folks generally think is a good monitoring practice?
We are on
HDF Version 3.4.1.1.
Powered by Apache NiFi Version 1.9.0
1.9.0.3.4.1.1-4 built 05/01/2019 02:15:30 UTC
Tagged nifi-1.9.0-RC2
From 7410fa4 on branch UNKNOWN
Re: InvokeHTTP hangs after several successful calls
Posted by jeanne-herndon <je...@legalshieldcorp.com>.
Upgrading Nifi from 1.11.4 to 1.13.2 resolved this problem for us.
--
Sent from: http://apache-nifi-users-list.2361937.n4.nabble.com/
Re: InvokeHTTP hangs after several successful calls
Posted by Vijay Chhipa <vc...@apple.com>.
Jeanne,
This is not a JDK issue. It is specific to HTTP.
You can try to apply the fix for https://issues.apache.org/jira/browse/NIFI-8181 <https://issues.apache.org/jira/browse/NIFI-8181> as a patch to your current version.
HTH,
Vijay
> On Apr 14, 2021, at 8:20 AM, jeanne-herndon <je...@legalshieldcorp.com> wrote:
>
> I am having this same issue and do not have access to log files. Same
> pattern as the others. I need to terminate the invokeHttp processor to
> release threads. Is this a JDK issue or an HTTP2 issue? I am not able to
> find any solutions that work.
>
>
>
>
> --
> Sent from: http://apache-nifi-users-list.2361937.n4.nabble.com/
Re: InvokeHTTP hangs after several successful calls
Posted by jeanne-herndon <je...@legalshieldcorp.com>.
I am having this same issue and do not have access to log files. Same
pattern as the others. I need to terminate the invokeHttp processor to
release threads. Is this a JDK issue or an HTTP2 issue? I am not able to
find any solutions that work.
--
Sent from: http://apache-nifi-users-list.2361937.n4.nabble.com/
Re: InvokeHTTP hangs after several successful calls
Posted by "Jens M. Kofoed" <jm...@gmail.com>.
Hi
I have a similar issue. Sorry but at the moment I'm not able to copy
logfiles from the system. But I have a screendump from the dump file. The
only place in the dump file where there is a InvokeHTTP. See jpg.
We are using NIFI version 1.12.1 and Java-8-openjdk-amd64
[image: InvokeHTTP_Hung.JPG]
I've looked in the nifi-app.log and I can see the normal process is that
the InvokeHTTP is requesting the server, and the server response. But the
last entry in the log is a request. It seems the server does not reply.
We have set the "read timeout" to 1 min. and the idle timeout 5 min
(default). But the process never times out.
Kind regards
Jens M. Kofoed
Den man. 1. feb. 2021 kl. 05.16 skrev Vijay Chhipa <vc...@apple.com>:
> Mark,
>
> I changed the code to use HTTP 1.1 in InvokeHTTP, but it did not help.
>
> I looked into "Shutdown" part of this link
> https://square.github.io/okhttp/4.x/okhttp/okhttp3/-ok-http-client/#okhttpclient.
> After implementing this, the processor didn't hang.
>
> This defeats the purpose of the thread pool and connection reuse, but I
> was able to get past the issue.
>
> It is still a good idea to get
> https://issues.apache.org/jira/browse/NIFI-8181 implemented.
>
> Regards,
> Vijay
>
>
>
> On Jan 29, 2021, at 1:34 PM, Vijay Chhipa <vc...@apple.com> wrote:
>
> Joe,
>
> behavior is the same with the 1.12.1 version.
>
> Mark,
> I will make the change to the OkHttp client and try. I do know that this
> flow was working for a long time and there was possibly an upgrade done on
> the source endpoint. Thank you for opening the Jira.
>
> Regards,
> Vijay
>
> On Jan 28, 2021, at 4:37 PM, Mark Payne <ma...@hotmail.com> wrote:
>
> Hey Vijay,
>
> I’ve seen a few people lately running into issues with InvokeHTTP. The
> common thread for all of them is that they are hitting servers that are
> using HTTP 2. Reading threads from OkHttp (the underlying HTTP library that
> we use), I see that a lot of people are running into issues with it. In at
> least several of the cases, it ends up being the HTTP Server or a
> proxy/load balancer in the middle that is not properly supporting the HTTP
> 2 protocol. Unclear if there are HTTP 2.0 related bugs in OkHttp itself or
> not. In any case, users often comment that changing the protocol to HTTP
> 1.1 resolved the issue. I filed a Jira [1] to allow that to be exposed in
> the InvokeHTTP processor. Hopefully that will help.
>
> Thanks
> -Mark
>
> [1] https://issues.apache.org/jira/browse/NIFI-8181
>
> On Jan 28, 2021, at 4:46 PM, Joe Witt <jo...@gmail.com> wrote:
>
> The likely relevant thread is here
>
> "Timer-Driven Process Thread-9" #70 prio=5 os_prio=31 cpu=12025.30ms
> elapsed=4403.88s tid=0x00007fe44f16b000 nid=0xe103 in Object.wait()
> [0x000070000fed4000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(java.base@11.0.5/Native Method)
> - waiting on <no object reference available>
> at java.lang.Object.wait(java.base@11.0.5/Object.java:328)
> at okhttp3.internal.http2.Http2Stream.waitForIo(Http2Stream.java:577)
> at
> okhttp3.internal.http2.Http2Stream.takeResponseHeaders(Http2Stream.java:143)
> - waiting to re-lock in wait() <0x00000007e007d818> (a
> okhttp3.internal.http2.Http2Stream)
> at
> okhttp3.internal.http2.Http2Codec.readResponseHeaders(Http2Codec.java:120)
> at
> okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.java:75)
> at
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
> at
> okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:45)
> at
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
> at
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
> at
> okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93)
> at
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
> at
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
> at
> okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
> at
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
> at
> okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:120)
> at
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
> at
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
> at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:185)
> at okhttp3.RealCall.execute(RealCall.java:69)
> at
> org.apache.nifi.processors.standard.InvokeHTTP.onTrigger(InvokeHTTP.java:793)
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1176)
> at
> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:213)
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
> at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
> at java.util.concurrent.Executors$RunnableAdapter.call(java.base@11.0.5
> /Executors.java:515)
> at java.util.concurrent.FutureTask.runAndReset(java.base@11.0.5
> /FutureTask.java:305)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(java.base@11.0.5
> /ScheduledThreadPoolExecutor.java:305)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.5
> /ThreadPoolExecutor.java:1128)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.5
> /ThreadPoolExecutor.java:628)
> at java.lang.Thread.run(java.base@11.0.5/Thread.java:834)
>
> This implies that it is either genuinely waiting for IO that the remote
> server is failing to send or has gotten itself into a state it could never
> break out of. Can you please use this exact same flow on the latest NiFi
> release 1.12.1 to see if the issue remains?
>
> Thanks
>
> On Thu, Jan 28, 2021 at 2:43 PM Vijay Chhipa <vc...@apple.com> wrote:
>
>> Hi Joe,
>>
>> Thanks for looking into this issue.
>> We are on NiFi 1.10.0
>>
>> Please see the attached
>> 1. thread dump file
>> 2. nifi-app.log
>> 3. nifi-bootstrap.log
>> 4. nifi.properties
>>
>> The InvokeHTTP configuration is as shown below.
>>
>>
>>
>>
>>
>> On Jan 25, 2021, at 7:24 AM, Joe Witt <jo...@gmail.com> wrote:
>>
>> Hello
>>
>> If you suspect an actual stuck/hung thread then the best course of action
>> is to generate a thread dump. This can be done in a couple ways but one of
>> the easiest is to run 'bin/nifi.sh dump'. We would need to see the
>> nifi-app.log(s) and nifi-bootstrap.log(s). In addition the report should
>> include the specific nifi version being used specifics of the processor
>> configuration.
>>
>> Thanks
>>
>> On Mon, Jan 25, 2021 at 6:12 AM Jairo Henao <ja...@gmail.com>
>> wrote:
>>
>>> Hello,
>>>
>>> In my case something similar happens. I invoke a REST-JSON service that
>>> sometimes takes more than 10 minutes to respond (I have set the timeout to
>>> 15 minutes) and after the second or third call the service responds (I
>>> could check it on the server side) but the processor remains waiting for
>>> the response. I try to stop it and after "terminate" it but the thread
>>> seems to stay active, so I delete it, (disconnect it and remove it from the
>>> flow) and add a new InvokeHttp and it works again.
>>>
>>> Given the difficulty of finding steps to reproduce it, I have not
>>> reported it. I hope someone can help us.
>>>
>>> On Mon, Jan 25, 2021 at 12:00 AM Vijay Chhipa <vc...@apple.com> wrote:
>>>
>>>> Hi All,
>>>>
>>>> I am seeing a case where InvokeHTTP processor hangs after several
>>>> successful calls to an endpoint. (Processor is paginating over millions of
>>>> rows and pulling a few thousand rows at a time)
>>>>
>>>>
>>>> <Screen Shot 2021-01-24 at 10.37.14 PM.png>
>>>>
>>>> If I have more than one thread scheduled, all threads hang.
>>>> <Screen Shot 2021-01-24 at 10.37.55 PM.png>
>>>>
>>>>
>>>> When this happens InvokeHTTP does not release the current flowfile at
>>>> all. My only recourse is to try to stop the processor.
>>>> <Screen Shot 2021-01-17 at 9.36.08 PM.png>
>>>>
>>>> This "stop" does not complete, at which point I have to terminate the
>>>> processor.
>>>> <Screen Shot 2021-01-17 at 9.36.58 PM.png>
>>>>
>>>> After termination of this processor, starting of the processor does not
>>>> process any files. So I have to remove the processor and all its
>>>> connections, delete the processor and replace it with another InvokeHTTP
>>>> with same properties and connections.
>>>>
>>>> I have tried to set the "Connection: close" header but it didn't help.
>>>>
>>>> Has anyone seen this behaviors, how did you resolve it?
>>>>
>>>> Thanks
>>>>
>>>>
>>>
>>> --
>>> Jairo Henao
>>> @jairohenaorojas
>>>
>>>
>>
>
>
>
Re: InvokeHTTP hangs after several successful calls
Posted by Vijay Chhipa <vc...@apple.com>.
Mark,
I changed the code to use HTTP 1.1 in InvokeHTTP, but it did not help.
I looked into "Shutdown" part of this link https://square.github.io/okhttp/4.x/okhttp/okhttp3/-ok-http-client/#okhttpclient <https://square.github.io/okhttp/4.x/okhttp/okhttp3/-ok-http-client/#okhttpclient>. After implementing this, the processor didn't hang.
This defeats the purpose of the thread pool and connection reuse, but I was able to get past the issue.
It is still a good idea to get https://issues.apache.org/jira/browse/NIFI-8181 <https://issues.apache.org/jira/browse/NIFI-8181> implemented.
Regards,
Vijay
> On Jan 29, 2021, at 1:34 PM, Vijay Chhipa <vc...@apple.com> wrote:
>
> Joe,
>
> behavior is the same with the 1.12.1 version.
>
> Mark,
> I will make the change to the OkHttp client and try. I do know that this flow was working for a long time and there was possibly an upgrade done on the source endpoint. Thank you for opening the Jira.
>
> Regards,
> Vijay
>
>> On Jan 28, 2021, at 4:37 PM, Mark Payne <markap14@hotmail.com <ma...@hotmail.com>> wrote:
>>
>> Hey Vijay,
>>
>> I’ve seen a few people lately running into issues with InvokeHTTP. The common thread for all of them is that they are hitting servers that are using HTTP 2. Reading threads from OkHttp (the underlying HTTP library that we use), I see that a lot of people are running into issues with it. In at least several of the cases, it ends up being the HTTP Server or a proxy/load balancer in the middle that is not properly supporting the HTTP 2 protocol. Unclear if there are HTTP 2.0 related bugs in OkHttp itself or not. In any case, users often comment that changing the protocol to HTTP 1.1 resolved the issue. I filed a Jira [1] to allow that to be exposed in the InvokeHTTP processor. Hopefully that will help.
>>
>> Thanks
>> -Mark
>>
>> [1] https://issues.apache.org/jira/browse/NIFI-8181 <https://issues.apache.org/jira/browse/NIFI-8181>
>>
>>> On Jan 28, 2021, at 4:46 PM, Joe Witt <joe.witt@gmail.com <ma...@gmail.com>> wrote:
>>>
>>> The likely relevant thread is here
>>>
>>> "Timer-Driven Process Thread-9" #70 prio=5 os_prio=31 cpu=12025.30ms elapsed=4403.88s tid=0x00007fe44f16b000 nid=0xe103 in Object.wait() [0x000070000fed4000]
>>> java.lang.Thread.State: WAITING (on object monitor)
>>> at java.lang.Object.wait(java.base@11.0.5/Native Method)
>>> - waiting on <no object reference available>
>>> at java.lang.Object.wait(java.base@11.0.5/Object.java:328)
>>> at okhttp3.internal.http2.Http2Stream.waitForIo(Http2Stream.java:577)
>>> at okhttp3.internal.http2.Http2Stream.takeResponseHeaders(Http2Stream.java:143)
>>> - waiting to re-lock in wait() <0x00000007e007d818> (a okhttp3.internal.http2.Http2Stream)
>>> at okhttp3.internal.http2.Http2Codec.readResponseHeaders(Http2Codec.java:120)
>>> at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.java:75)
>>> at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
>>> at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:45)
>>> at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
>>> at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
>>> at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93)
>>> at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
>>> at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
>>> at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
>>> at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
>>> at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:120)
>>> at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
>>> at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
>>> at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:185)
>>> at okhttp3.RealCall.execute(RealCall.java:69)
>>> at org.apache.nifi.processors.standard.InvokeHTTP.onTrigger(InvokeHTTP.java:793)
>>> at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
>>> at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1176)
>>> at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:213)
>>> at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
>>> at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
>>> at java.util.concurrent.Executors$RunnableAdapter.call(java.base@11.0.5/Executors.java:515)
>>> at java.util.concurrent.FutureTask.runAndReset(java.base@11.0.5/FutureTask.java:305)
>>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(java.base@11.0.5/ScheduledThreadPoolExecutor.java:305)
>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.5/ThreadPoolExecutor.java:1128)
>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.5/ThreadPoolExecutor.java:628)
>>> at java.lang.Thread.run(java.base@11.0.5/Thread.java:834)
>>>
>>> This implies that it is either genuinely waiting for IO that the remote server is failing to send or has gotten itself into a state it could never break out of. Can you please use this exact same flow on the latest NiFi release 1.12.1 to see if the issue remains?
>>>
>>> Thanks
>>>
>>> On Thu, Jan 28, 2021 at 2:43 PM Vijay Chhipa <vchhipa@apple.com <ma...@apple.com>> wrote:
>>> Hi Joe,
>>>
>>> Thanks for looking into this issue.
>>> We are on NiFi 1.10.0
>>>
>>> Please see the attached
>>> 1. thread dump file
>>> 2. nifi-app.log
>>> 3. nifi-bootstrap.log
>>> 4. nifi.properties
>>>
>>> The InvokeHTTP configuration is as shown below.
>>>
>>>
>>>
>>>
>>>
>>>> On Jan 25, 2021, at 7:24 AM, Joe Witt <joe.witt@gmail.com <ma...@gmail.com>> wrote:
>>>>
>>>> Hello
>>>>
>>>> If you suspect an actual stuck/hung thread then the best course of action is to generate a thread dump. This can be done in a couple ways but one of the easiest is to run 'bin/nifi.sh dump'. We would need to see the nifi-app.log(s) and nifi-bootstrap.log(s). In addition the report should include the specific nifi version being used specifics of the processor configuration.
>>>>
>>>> Thanks
>>>>
>>>> On Mon, Jan 25, 2021 at 6:12 AM Jairo Henao <jairohenaorojas@gmail.com <ma...@gmail.com>> wrote:
>>>> Hello,
>>>>
>>>> In my case something similar happens. I invoke a REST-JSON service that sometimes takes more than 10 minutes to respond (I have set the timeout to 15 minutes) and after the second or third call the service responds (I could check it on the server side) but the processor remains waiting for the response. I try to stop it and after "terminate" it but the thread seems to stay active, so I delete it, (disconnect it and remove it from the flow) and add a new InvokeHttp and it works again.
>>>>
>>>> Given the difficulty of finding steps to reproduce it, I have not reported it. I hope someone can help us.
>>>>
>>>> On Mon, Jan 25, 2021 at 12:00 AM Vijay Chhipa <vchhipa@apple.com <ma...@apple.com>> wrote:
>>>> Hi All,
>>>>
>>>> I am seeing a case where InvokeHTTP processor hangs after several successful calls to an endpoint. (Processor is paginating over millions of rows and pulling a few thousand rows at a time)
>>>>
>>>>
>>>> <Screen Shot 2021-01-24 at 10.37.14 PM.png>
>>>>
>>>> If I have more than one thread scheduled, all threads hang.
>>>> <Screen Shot 2021-01-24 at 10.37.55 PM.png>
>>>>
>>>>
>>>> When this happens InvokeHTTP does not release the current flowfile at all. My only recourse is to try to stop the processor.
>>>> <Screen Shot 2021-01-17 at 9.36.08 PM.png>
>>>>
>>>> This "stop" does not complete, at which point I have to terminate the processor.
>>>> <Screen Shot 2021-01-17 at 9.36.58 PM.png>
>>>>
>>>> After termination of this processor, starting of the processor does not process any files. So I have to remove the processor and all its connections, delete the processor and replace it with another InvokeHTTP with same properties and connections.
>>>>
>>>> I have tried to set the "Connection: close" header but it didn't help.
>>>>
>>>> Has anyone seen this behaviors, how did you resolve it?
>>>>
>>>> Thanks
>>>>
>>>>
>>>>
>>>> --
>>>> Jairo Henao
>>>> @jairohenaorojas
>>>>
>>>
>>
>
Re: InvokeHTTP hangs after several successful calls
Posted by Vijay Chhipa <vc...@apple.com>.
Joe,
behavior is the same with the 1.12.1 version.
Mark,
I will make the change to the OkHttp client and try. I do know that this flow was working for a long time and there was possibly an upgrade done on the source endpoint. Thank you for opening the Jira.
Regards,
Vijay
> On Jan 28, 2021, at 4:37 PM, Mark Payne <ma...@hotmail.com> wrote:
>
> Hey Vijay,
>
> I’ve seen a few people lately running into issues with InvokeHTTP. The common thread for all of them is that they are hitting servers that are using HTTP 2. Reading threads from OkHttp (the underlying HTTP library that we use), I see that a lot of people are running into issues with it. In at least several of the cases, it ends up being the HTTP Server or a proxy/load balancer in the middle that is not properly supporting the HTTP 2 protocol. Unclear if there are HTTP 2.0 related bugs in OkHttp itself or not. In any case, users often comment that changing the protocol to HTTP 1.1 resolved the issue. I filed a Jira [1] to allow that to be exposed in the InvokeHTTP processor. Hopefully that will help.
>
> Thanks
> -Mark
>
> [1] https://issues.apache.org/jira/browse/NIFI-8181 <https://issues.apache.org/jira/browse/NIFI-8181>
>
>> On Jan 28, 2021, at 4:46 PM, Joe Witt <joe.witt@gmail.com <ma...@gmail.com>> wrote:
>>
>> The likely relevant thread is here
>>
>> "Timer-Driven Process Thread-9" #70 prio=5 os_prio=31 cpu=12025.30ms elapsed=4403.88s tid=0x00007fe44f16b000 nid=0xe103 in Object.wait() [0x000070000fed4000]
>> java.lang.Thread.State: WAITING (on object monitor)
>> at java.lang.Object.wait(java.base@11.0.5/Native Method)
>> - waiting on <no object reference available>
>> at java.lang.Object.wait(java.base@11.0.5/Object.java:328)
>> at okhttp3.internal.http2.Http2Stream.waitForIo(Http2Stream.java:577)
>> at okhttp3.internal.http2.Http2Stream.takeResponseHeaders(Http2Stream.java:143)
>> - waiting to re-lock in wait() <0x00000007e007d818> (a okhttp3.internal.http2.Http2Stream)
>> at okhttp3.internal.http2.Http2Codec.readResponseHeaders(Http2Codec.java:120)
>> at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.java:75)
>> at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
>> at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:45)
>> at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
>> at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
>> at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93)
>> at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
>> at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
>> at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
>> at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
>> at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:120)
>> at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
>> at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
>> at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:185)
>> at okhttp3.RealCall.execute(RealCall.java:69)
>> at org.apache.nifi.processors.standard.InvokeHTTP.onTrigger(InvokeHTTP.java:793)
>> at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
>> at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1176)
>> at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:213)
>> at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
>> at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
>> at java.util.concurrent.Executors$RunnableAdapter.call(java.base@11.0.5/Executors.java:515)
>> at java.util.concurrent.FutureTask.runAndReset(java.base@11.0.5/FutureTask.java:305)
>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(java.base@11.0.5/ScheduledThreadPoolExecutor.java:305)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.5/ThreadPoolExecutor.java:1128)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.5/ThreadPoolExecutor.java:628)
>> at java.lang.Thread.run(java.base@11.0.5/Thread.java:834)
>>
>> This implies that it is either genuinely waiting for IO that the remote server is failing to send or has gotten itself into a state it could never break out of. Can you please use this exact same flow on the latest NiFi release 1.12.1 to see if the issue remains?
>>
>> Thanks
>>
>> On Thu, Jan 28, 2021 at 2:43 PM Vijay Chhipa <vchhipa@apple.com <ma...@apple.com>> wrote:
>> Hi Joe,
>>
>> Thanks for looking into this issue.
>> We are on NiFi 1.10.0
>>
>> Please see the attached
>> 1. thread dump file
>> 2. nifi-app.log
>> 3. nifi-bootstrap.log
>> 4. nifi.properties
>>
>> The InvokeHTTP configuration is as shown below.
>>
>>
>>
>>
>>
>>> On Jan 25, 2021, at 7:24 AM, Joe Witt <joe.witt@gmail.com <ma...@gmail.com>> wrote:
>>>
>>> Hello
>>>
>>> If you suspect an actual stuck/hung thread then the best course of action is to generate a thread dump. This can be done in a couple ways but one of the easiest is to run 'bin/nifi.sh dump'. We would need to see the nifi-app.log(s) and nifi-bootstrap.log(s). In addition the report should include the specific nifi version being used specifics of the processor configuration.
>>>
>>> Thanks
>>>
>>> On Mon, Jan 25, 2021 at 6:12 AM Jairo Henao <jairohenaorojas@gmail.com <ma...@gmail.com>> wrote:
>>> Hello,
>>>
>>> In my case something similar happens. I invoke a REST-JSON service that sometimes takes more than 10 minutes to respond (I have set the timeout to 15 minutes) and after the second or third call the service responds (I could check it on the server side) but the processor remains waiting for the response. I try to stop it and after "terminate" it but the thread seems to stay active, so I delete it, (disconnect it and remove it from the flow) and add a new InvokeHttp and it works again.
>>>
>>> Given the difficulty of finding steps to reproduce it, I have not reported it. I hope someone can help us.
>>>
>>> On Mon, Jan 25, 2021 at 12:00 AM Vijay Chhipa <vchhipa@apple.com <ma...@apple.com>> wrote:
>>> Hi All,
>>>
>>> I am seeing a case where InvokeHTTP processor hangs after several successful calls to an endpoint. (Processor is paginating over millions of rows and pulling a few thousand rows at a time)
>>>
>>>
>>> <Screen Shot 2021-01-24 at 10.37.14 PM.png>
>>>
>>> If I have more than one thread scheduled, all threads hang.
>>> <Screen Shot 2021-01-24 at 10.37.55 PM.png>
>>>
>>>
>>> When this happens InvokeHTTP does not release the current flowfile at all. My only recourse is to try to stop the processor.
>>> <Screen Shot 2021-01-17 at 9.36.08 PM.png>
>>>
>>> This "stop" does not complete, at which point I have to terminate the processor.
>>> <Screen Shot 2021-01-17 at 9.36.58 PM.png>
>>>
>>> After termination of this processor, starting of the processor does not process any files. So I have to remove the processor and all its connections, delete the processor and replace it with another InvokeHTTP with same properties and connections.
>>>
>>> I have tried to set the "Connection: close" header but it didn't help.
>>>
>>> Has anyone seen this behaviors, how did you resolve it?
>>>
>>> Thanks
>>>
>>>
>>>
>>> --
>>> Jairo Henao
>>> @jairohenaorojas
>>>
>>
>
Re: InvokeHTTP hangs after several successful calls
Posted by Mark Payne <ma...@hotmail.com>.
Hey Vijay,
I’ve seen a few people lately running into issues with InvokeHTTP. The common thread for all of them is that they are hitting servers that are using HTTP 2. Reading threads from OkHttp (the underlying HTTP library that we use), I see that a lot of people are running into issues with it. In at least several of the cases, it ends up being the HTTP Server or a proxy/load balancer in the middle that is not properly supporting the HTTP 2 protocol. Unclear if there are HTTP 2.0 related bugs in OkHttp itself or not. In any case, users often comment that changing the protocol to HTTP 1.1 resolved the issue. I filed a Jira [1] to allow that to be exposed in the InvokeHTTP processor. Hopefully that will help.
Thanks
-Mark
[1] https://issues.apache.org/jira/browse/NIFI-8181
On Jan 28, 2021, at 4:46 PM, Joe Witt <jo...@gmail.com>> wrote:
The likely relevant thread is here
"Timer-Driven Process Thread-9" #70 prio=5 os_prio=31 cpu=12025.30ms elapsed=4403.88s tid=0x00007fe44f16b000 nid=0xe103 in Object.wait() [0x000070000fed4000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(java.base@11.0.5/Native Method)
- waiting on <no object reference available>
at java.lang.Object.wait(java.base@11.0.5/Object.java:328)
at okhttp3.internal.http2.Http2Stream.waitForIo(Http2Stream.java:577)
at okhttp3.internal.http2.Http2Stream.takeResponseHeaders(Http2Stream.java:143)
- waiting to re-lock in wait() <0x00000007e007d818> (a okhttp3.internal.http2.Http2Stream)
at okhttp3.internal.http2.Http2Codec.readResponseHeaders(Http2Codec.java:120)
at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.java:75)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:45)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:120)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:185)
at okhttp3.RealCall.execute(RealCall.java:69)
at org.apache.nifi.processors.standard.InvokeHTTP.onTrigger(InvokeHTTP.java:793)
at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1176)
at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:213)
at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
at java.util.concurrent.Executors$RunnableAdapter.call(java.base@11.0.5/Executors.java:515)
at java.util.concurrent.FutureTask.runAndReset(java.base@11.0.5/FutureTask.java:305)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(java.base@11.0.5/ScheduledThreadPoolExecutor.java:305)
at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.5/ThreadPoolExecutor.java:1128)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.5/ThreadPoolExecutor.java:628)
at java.lang.Thread.run(java.base@11.0.5/Thread.java:834)
This implies that it is either genuinely waiting for IO that the remote server is failing to send or has gotten itself into a state it could never break out of. Can you please use this exact same flow on the latest NiFi release 1.12.1 to see if the issue remains?
Thanks
On Thu, Jan 28, 2021 at 2:43 PM Vijay Chhipa <vc...@apple.com>> wrote:
Hi Joe,
Thanks for looking into this issue.
We are on NiFi 1.10.0
Please see the attached
1. thread dump file
2. nifi-app.log
3. nifi-bootstrap.log
4. nifi.properties
The InvokeHTTP configuration is as shown below.
On Jan 25, 2021, at 7:24 AM, Joe Witt <jo...@gmail.com>> wrote:
Hello
If you suspect an actual stuck/hung thread then the best course of action is to generate a thread dump. This can be done in a couple ways but one of the easiest is to run 'bin/nifi.sh dump'. We would need to see the nifi-app.log(s) and nifi-bootstrap.log(s). In addition the report should include the specific nifi version being used specifics of the processor configuration.
Thanks
On Mon, Jan 25, 2021 at 6:12 AM Jairo Henao <ja...@gmail.com>> wrote:
Hello,
In my case something similar happens. I invoke a REST-JSON service that sometimes takes more than 10 minutes to respond (I have set the timeout to 15 minutes) and after the second or third call the service responds (I could check it on the server side) but the processor remains waiting for the response. I try to stop it and after "terminate" it but the thread seems to stay active, so I delete it, (disconnect it and remove it from the flow) and add a new InvokeHttp and it works again.
Given the difficulty of finding steps to reproduce it, I have not reported it. I hope someone can help us.
On Mon, Jan 25, 2021 at 12:00 AM Vijay Chhipa <vc...@apple.com>> wrote:
Hi All,
I am seeing a case where InvokeHTTP processor hangs after several successful calls to an endpoint. (Processor is paginating over millions of rows and pulling a few thousand rows at a time)
<Screen Shot 2021-01-24 at 10.37.14 PM.png>
If I have more than one thread scheduled, all threads hang.
<Screen Shot 2021-01-24 at 10.37.55 PM.png>
When this happens InvokeHTTP does not release the current flowfile at all. My only recourse is to try to stop the processor.
<Screen Shot 2021-01-17 at 9.36.08 PM.png>
This "stop" does not complete, at which point I have to terminate the processor.
<Screen Shot 2021-01-17 at 9.36.58 PM.png>
After termination of this processor, starting of the processor does not process any files. So I have to remove the processor and all its connections, delete the processor and replace it with another InvokeHTTP with same properties and connections.
I have tried to set the "Connection: close" header but it didn't help.
Has anyone seen this behaviors, how did you resolve it?
Thanks
--
Jairo Henao
@jairohenaorojas
Re: InvokeHTTP hangs after several successful calls
Posted by Joe Witt <jo...@gmail.com>.
The likely relevant thread is here
"Timer-Driven Process Thread-9" #70 prio=5 os_prio=31 cpu=12025.30ms
elapsed=4403.88s tid=0x00007fe44f16b000 nid=0xe103 in Object.wait()
[0x000070000fed4000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(java.base@11.0.5/Native Method)
- waiting on <no object reference available>
at java.lang.Object.wait(java.base@11.0.5/Object.java:328)
at okhttp3.internal.http2.Http2Stream.waitForIo(Http2Stream.java:577)
at
okhttp3.internal.http2.Http2Stream.takeResponseHeaders(Http2Stream.java:143)
- waiting to re-lock in wait() <0x00000007e007d818> (a
okhttp3.internal.http2.Http2Stream)
at
okhttp3.internal.http2.Http2Codec.readResponseHeaders(Http2Codec.java:120)
at
okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.java:75)
at
okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
at
okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:45)
at
okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
at
okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
at
okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93)
at
okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
at
okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
at
okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
at
okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
at
okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:120)
at
okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
at
okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:185)
at okhttp3.RealCall.execute(RealCall.java:69)
at
org.apache.nifi.processors.standard.InvokeHTTP.onTrigger(InvokeHTTP.java:793)
at
org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
at
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1176)
at
org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:213)
at
org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
at java.util.concurrent.Executors$RunnableAdapter.call(java.base@11.0.5
/Executors.java:515)
at java.util.concurrent.FutureTask.runAndReset(java.base@11.0.5
/FutureTask.java:305)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(java.base@11.0.5
/ScheduledThreadPoolExecutor.java:305)
at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.5
/ThreadPoolExecutor.java:1128)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.5
/ThreadPoolExecutor.java:628)
at java.lang.Thread.run(java.base@11.0.5/Thread.java:834)
This implies that it is either genuinely waiting for IO that the remote
server is failing to send or has gotten itself into a state it could never
break out of. Can you please use this exact same flow on the latest NiFi
release 1.12.1 to see if the issue remains?
Thanks
On Thu, Jan 28, 2021 at 2:43 PM Vijay Chhipa <vc...@apple.com> wrote:
> Hi Joe,
>
> Thanks for looking into this issue.
> We are on NiFi 1.10.0
>
> Please see the attached
> 1. thread dump file
> 2. nifi-app.log
> 3. nifi-bootstrap.log
> 4. nifi.properties
>
> The InvokeHTTP configuration is as shown below.
>
>
>
>
>
> On Jan 25, 2021, at 7:24 AM, Joe Witt <jo...@gmail.com> wrote:
>
> Hello
>
> If you suspect an actual stuck/hung thread then the best course of action
> is to generate a thread dump. This can be done in a couple ways but one of
> the easiest is to run 'bin/nifi.sh dump'. We would need to see the
> nifi-app.log(s) and nifi-bootstrap.log(s). In addition the report should
> include the specific nifi version being used specifics of the processor
> configuration.
>
> Thanks
>
> On Mon, Jan 25, 2021 at 6:12 AM Jairo Henao <ja...@gmail.com>
> wrote:
>
>> Hello,
>>
>> In my case something similar happens. I invoke a REST-JSON service that
>> sometimes takes more than 10 minutes to respond (I have set the timeout to
>> 15 minutes) and after the second or third call the service responds (I
>> could check it on the server side) but the processor remains waiting for
>> the response. I try to stop it and after "terminate" it but the thread
>> seems to stay active, so I delete it, (disconnect it and remove it from the
>> flow) and add a new InvokeHttp and it works again.
>>
>> Given the difficulty of finding steps to reproduce it, I have not
>> reported it. I hope someone can help us.
>>
>> On Mon, Jan 25, 2021 at 12:00 AM Vijay Chhipa <vc...@apple.com> wrote:
>>
>>> Hi All,
>>>
>>> I am seeing a case where InvokeHTTP processor hangs after several
>>> successful calls to an endpoint. (Processor is paginating over millions of
>>> rows and pulling a few thousand rows at a time)
>>>
>>>
>>> <Screen Shot 2021-01-24 at 10.37.14 PM.png>
>>>
>>> If I have more than one thread scheduled, all threads hang.
>>> <Screen Shot 2021-01-24 at 10.37.55 PM.png>
>>>
>>>
>>> When this happens InvokeHTTP does not release the current flowfile at
>>> all. My only recourse is to try to stop the processor.
>>> <Screen Shot 2021-01-17 at 9.36.08 PM.png>
>>>
>>> This "stop" does not complete, at which point I have to terminate the
>>> processor.
>>> <Screen Shot 2021-01-17 at 9.36.58 PM.png>
>>>
>>> After termination of this processor, starting of the processor does not
>>> process any files. So I have to remove the processor and all its
>>> connections, delete the processor and replace it with another InvokeHTTP
>>> with same properties and connections.
>>>
>>> I have tried to set the "Connection: close" header but it didn't help.
>>>
>>> Has anyone seen this behaviors, how did you resolve it?
>>>
>>> Thanks
>>>
>>>
>>
>> --
>> Jairo Henao
>> @jairohenaorojas
>>
>>
>
Re: InvokeHTTP hangs after several successful calls
Posted by Vijay Chhipa <vc...@apple.com>.
Hi Joe,
Thanks for looking into this issue.
We are on NiFi 1.10.0
Please see the attached
1. thread dump file
2. nifi-app.log
3. nifi-bootstrap.log
4. nifi.properties
The InvokeHTTP configuration is as shown below.
> On Jan 25, 2021, at 7:24 AM, Joe Witt <jo...@gmail.com> wrote:
>
> Hello
>
> If you suspect an actual stuck/hung thread then the best course of action is to generate a thread dump. This can be done in a couple ways but one of the easiest is to run 'bin/nifi.sh dump'. We would need to see the nifi-app.log(s) and nifi-bootstrap.log(s). In addition the report should include the specific nifi version being used specifics of the processor configuration.
>
> Thanks
>
> On Mon, Jan 25, 2021 at 6:12 AM Jairo Henao <jairohenaorojas@gmail.com <ma...@gmail.com>> wrote:
> Hello,
>
> In my case something similar happens. I invoke a REST-JSON service that sometimes takes more than 10 minutes to respond (I have set the timeout to 15 minutes) and after the second or third call the service responds (I could check it on the server side) but the processor remains waiting for the response. I try to stop it and after "terminate" it but the thread seems to stay active, so I delete it, (disconnect it and remove it from the flow) and add a new InvokeHttp and it works again.
>
> Given the difficulty of finding steps to reproduce it, I have not reported it. I hope someone can help us.
>
> On Mon, Jan 25, 2021 at 12:00 AM Vijay Chhipa <vchhipa@apple.com <ma...@apple.com>> wrote:
> Hi All,
>
> I am seeing a case where InvokeHTTP processor hangs after several successful calls to an endpoint. (Processor is paginating over millions of rows and pulling a few thousand rows at a time)
>
>
> <Screen Shot 2021-01-24 at 10.37.14 PM.png>
>
> If I have more than one thread scheduled, all threads hang.
> <Screen Shot 2021-01-24 at 10.37.55 PM.png>
>
>
> When this happens InvokeHTTP does not release the current flowfile at all. My only recourse is to try to stop the processor.
> <Screen Shot 2021-01-17 at 9.36.08 PM.png>
>
> This "stop" does not complete, at which point I have to terminate the processor.
> <Screen Shot 2021-01-17 at 9.36.58 PM.png>
>
> After termination of this processor, starting of the processor does not process any files. So I have to remove the processor and all its connections, delete the processor and replace it with another InvokeHTTP with same properties and connections.
>
> I have tried to set the "Connection: close" header but it didn't help.
>
> Has anyone seen this behaviors, how did you resolve it?
>
> Thanks
>
>
>
> --
> Jairo Henao
> @jairohenaorojas
>
Re: InvokeHTTP hangs after several successful calls
Posted by Martin Ebert <ma...@gmx.de>.
Hi community,
one more thing I have to mention. If you deactivate the hanging invokehttp
processor and start it again, then all FlowFiles run into an error.
{code":401,"status":"Unauthorized","message":"Unauthorized"}
Only copying the processor and deleting the hanging processor helps.
However, this is not a practicable solution.
Thanks
Am Mi., 27. Jan. 2021 um 09:24 Uhr schrieb Martin Ebert <
martin.irgang@gmx.de>:
> Hello community,
>
> my standard InvokeHTTP processor (1.12.1) hangs after a while. I tried it
> with different configurations (concurrent tasks 1-20, run duration
> 0ms-2sec). So far I have not been able to find a workaround or identify and
> solve the problem.
>
> NiFi Settings: Maximum Timer Driven Thread Count 32, Maximum Event Driven
> Thread Count 1
> NiFi Version: 1.12.109/23/2020 14:27:38 CEST
> Tagged nifi-1.12.1-RC2
>
> I followed the instructions: nifi.sh dump.
>
> Thanks
>
> Am Mo., 25. Jan. 2021 um 14:24 Uhr schrieb Joe Witt <jo...@gmail.com>:
>
>> Hello
>>
>> If you suspect an actual stuck/hung thread then the best course of action
>> is to generate a thread dump. This can be done in a couple ways but one of
>> the easiest is to run 'bin/nifi.sh dump'. We would need to see the
>> nifi-app.log(s) and nifi-bootstrap.log(s). In addition the report should
>> include the specific nifi version being used specifics of the processor
>> configuration.
>>
>> Thanks
>>
>> On Mon, Jan 25, 2021 at 6:12 AM Jairo Henao <ja...@gmail.com>
>> wrote:
>>
>>> Hello,
>>>
>>> In my case something similar happens. I invoke a REST-JSON service that
>>> sometimes takes more than 10 minutes to respond (I have set the timeout to
>>> 15 minutes) and after the second or third call the service responds (I
>>> could check it on the server side) but the processor remains waiting for
>>> the response. I try to stop it and after "terminate" it but the thread
>>> seems to stay active, so I delete it, (disconnect it and remove it from the
>>> flow) and add a new InvokeHttp and it works again.
>>>
>>> Given the difficulty of finding steps to reproduce it, I have not
>>> reported it. I hope someone can help us.
>>>
>>> On Mon, Jan 25, 2021 at 12:00 AM Vijay Chhipa <vc...@apple.com> wrote:
>>>
>>>> Hi All,
>>>>
>>>> I am seeing a case where InvokeHTTP processor hangs after several
>>>> successful calls to an endpoint. (Processor is paginating over millions of
>>>> rows and pulling a few thousand rows at a time)
>>>>
>>>>
>>>>
>>>> If I have more than one thread scheduled, all threads hang.
>>>>
>>>>
>>>> When this happens InvokeHTTP does not release the current flowfile at
>>>> all. My only recourse is to try to stop the processor.
>>>>
>>>> This "stop" does not complete, at which point I have to terminate the
>>>> processor.
>>>>
>>>> After termination of this processor, starting of the processor does not
>>>> process any files. So I have to remove the processor and all its
>>>> connections, delete the processor and replace it with another InvokeHTTP
>>>> with same properties and connections.
>>>>
>>>> I have tried to set the "Connection: close" header but it didn't help.
>>>>
>>>> Has anyone seen this behaviors, how did you resolve it?
>>>>
>>>> Thanks
>>>>
>>>>
>>>
>>> --
>>> Jairo Henao
>>> @jairohenaorojas
>>>
>>>
Re: InvokeHTTP hangs after several successful calls
Posted by Martin Ebert <ma...@gmx.de>.
Hello community,
my standard InvokeHTTP processor (1.12.1) hangs after a while. I tried it
with different configurations (concurrent tasks 1-20, run duration
0ms-2sec). So far I have not been able to find a workaround or identify and
solve the problem.
NiFi Settings: Maximum Timer Driven Thread Count 32, Maximum Event Driven
Thread Count 1
NiFi Version: 1.12.109/23/2020 14:27:38 CEST
Tagged nifi-1.12.1-RC2
I followed the instructions: nifi.sh dump.
Thanks
Am Mo., 25. Jan. 2021 um 14:24 Uhr schrieb Joe Witt <jo...@gmail.com>:
> Hello
>
> If you suspect an actual stuck/hung thread then the best course of action
> is to generate a thread dump. This can be done in a couple ways but one of
> the easiest is to run 'bin/nifi.sh dump'. We would need to see the
> nifi-app.log(s) and nifi-bootstrap.log(s). In addition the report should
> include the specific nifi version being used specifics of the processor
> configuration.
>
> Thanks
>
> On Mon, Jan 25, 2021 at 6:12 AM Jairo Henao <ja...@gmail.com>
> wrote:
>
>> Hello,
>>
>> In my case something similar happens. I invoke a REST-JSON service that
>> sometimes takes more than 10 minutes to respond (I have set the timeout to
>> 15 minutes) and after the second or third call the service responds (I
>> could check it on the server side) but the processor remains waiting for
>> the response. I try to stop it and after "terminate" it but the thread
>> seems to stay active, so I delete it, (disconnect it and remove it from the
>> flow) and add a new InvokeHttp and it works again.
>>
>> Given the difficulty of finding steps to reproduce it, I have not
>> reported it. I hope someone can help us.
>>
>> On Mon, Jan 25, 2021 at 12:00 AM Vijay Chhipa <vc...@apple.com> wrote:
>>
>>> Hi All,
>>>
>>> I am seeing a case where InvokeHTTP processor hangs after several
>>> successful calls to an endpoint. (Processor is paginating over millions of
>>> rows and pulling a few thousand rows at a time)
>>>
>>>
>>>
>>> If I have more than one thread scheduled, all threads hang.
>>>
>>>
>>> When this happens InvokeHTTP does not release the current flowfile at
>>> all. My only recourse is to try to stop the processor.
>>>
>>> This "stop" does not complete, at which point I have to terminate the
>>> processor.
>>>
>>> After termination of this processor, starting of the processor does not
>>> process any files. So I have to remove the processor and all its
>>> connections, delete the processor and replace it with another InvokeHTTP
>>> with same properties and connections.
>>>
>>> I have tried to set the "Connection: close" header but it didn't help.
>>>
>>> Has anyone seen this behaviors, how did you resolve it?
>>>
>>> Thanks
>>>
>>>
>>
>> --
>> Jairo Henao
>> @jairohenaorojas
>>
>>
Re: InvokeHTTP hangs after several successful calls
Posted by Joe Witt <jo...@gmail.com>.
Hello
If you suspect an actual stuck/hung thread then the best course of action
is to generate a thread dump. This can be done in a couple ways but one of
the easiest is to run 'bin/nifi.sh dump'. We would need to see the
nifi-app.log(s) and nifi-bootstrap.log(s). In addition the report should
include the specific nifi version being used specifics of the processor
configuration.
Thanks
On Mon, Jan 25, 2021 at 6:12 AM Jairo Henao <ja...@gmail.com>
wrote:
> Hello,
>
> In my case something similar happens. I invoke a REST-JSON service that
> sometimes takes more than 10 minutes to respond (I have set the timeout to
> 15 minutes) and after the second or third call the service responds (I
> could check it on the server side) but the processor remains waiting for
> the response. I try to stop it and after "terminate" it but the thread
> seems to stay active, so I delete it, (disconnect it and remove it from the
> flow) and add a new InvokeHttp and it works again.
>
> Given the difficulty of finding steps to reproduce it, I have not reported
> it. I hope someone can help us.
>
> On Mon, Jan 25, 2021 at 12:00 AM Vijay Chhipa <vc...@apple.com> wrote:
>
>> Hi All,
>>
>> I am seeing a case where InvokeHTTP processor hangs after several
>> successful calls to an endpoint. (Processor is paginating over millions of
>> rows and pulling a few thousand rows at a time)
>>
>>
>>
>> If I have more than one thread scheduled, all threads hang.
>>
>>
>> When this happens InvokeHTTP does not release the current flowfile at
>> all. My only recourse is to try to stop the processor.
>>
>> This "stop" does not complete, at which point I have to terminate the
>> processor.
>>
>> After termination of this processor, starting of the processor does not
>> process any files. So I have to remove the processor and all its
>> connections, delete the processor and replace it with another InvokeHTTP
>> with same properties and connections.
>>
>> I have tried to set the "Connection: close" header but it didn't help.
>>
>> Has anyone seen this behaviors, how did you resolve it?
>>
>> Thanks
>>
>>
>
> --
> Jairo Henao
> @jairohenaorojas
>
>
Re: InvokeHTTP hangs after several successful calls
Posted by Jairo Henao <ja...@gmail.com>.
Hello,
In my case something similar happens. I invoke a REST-JSON service that
sometimes takes more than 10 minutes to respond (I have set the timeout to
15 minutes) and after the second or third call the service responds (I
could check it on the server side) but the processor remains waiting for
the response. I try to stop it and after "terminate" it but the thread
seems to stay active, so I delete it, (disconnect it and remove it from the
flow) and add a new InvokeHttp and it works again.
Given the difficulty of finding steps to reproduce it, I have not reported
it. I hope someone can help us.
On Mon, Jan 25, 2021 at 12:00 AM Vijay Chhipa <vc...@apple.com> wrote:
> Hi All,
>
> I am seeing a case where InvokeHTTP processor hangs after several
> successful calls to an endpoint. (Processor is paginating over millions of
> rows and pulling a few thousand rows at a time)
>
>
>
> If I have more than one thread scheduled, all threads hang.
>
>
> When this happens InvokeHTTP does not release the current flowfile at all.
> My only recourse is to try to stop the processor.
>
> This "stop" does not complete, at which point I have to terminate the
> processor.
>
> After termination of this processor, starting of the processor does not
> process any files. So I have to remove the processor and all its
> connections, delete the processor and replace it with another InvokeHTTP
> with same properties and connections.
>
> I have tried to set the "Connection: close" header but it didn't help.
>
> Has anyone seen this behaviors, how did you resolve it?
>
> Thanks
>
>
--
Jairo Henao
@jairohenaorojas