Long Polling Client

[Clay Teahouse <cl...@gmail.com>]

Does NiFi have a processor that can act as a client for a long polling
server, for example an SSE server?
More specifically, I want a client that can issue a HTTP GET request to a
long polling server and accept stream of messages from the server (on the
same connection).
If there isn't one, which processor is best to extend to achieve this goal?

thanks.

Re: Long Polling Client

[Clay Teahouse <cl...@gmail.com>]

Thanks Joe for the pointers.

On Wed, Oct 14, 2020 at 3:31 PM Joe Witt <jo...@gmail.com> wrote:

> Right makes sense.  They're waiting for the completion of the response as
> the payload to pass on.  If we need incremental handling of the response
> body then we need to factor that into a given processor.  InvokeHttp or
> something like it is a good candidate.  Almost a sort of 'InvokeHttpRecord'
> kind of processor which uses the format/schema awareness of records to
> frame the response objects and then based on some configurable value passes
> those records on as a flow file.
>
> Thanks
>
> On Wed, Oct 14, 2020 at 1:25 PM Clay Teahouse <cl...@gmail.com>
> wrote:
>
>> Hello,
>> I tried both getHTTP and invokeHTTP (but didn't try all options). What I
>> need is to deal with the cases, such as SSE (Server Sent Events) which
>> works with long polling. Meaning the client initiates a connection (via a
>> HTTP request) to the SSE server, the server keeps the connection open (for
>> as long as possible) and streams data back to the client on the same
>> connection. I am able to have the data streamed to the client if I set up a
>> stand alone HTTP client and issue HTTP requests but I am not able to get
>> getHTTP or invokeHTTP work (I don't get any messages back).
>>
>> thanks.
>>
>> On Wed, Oct 14, 2020 at 2:50 PM Joe Witt <jo...@gmail.com> wrote:
>>
>>> Clay
>>>
>>> Have you evaluated whether InvokeHTTP will give you the desired behavior
>>> for your case - in particular with a long timeout perhaps?  If you have and
>>> it doesn't do the job do you mean something which initiates a request to an
>>> HTTP server then assumes the response will remain open and it should take
>>> portions of the response and treat each as its own flowfile/object to pass
>>> along?
>>>
>>> Thanks
>>>
>>> On Tue, Oct 13, 2020 at 6:31 PM Clay Teahouse <cl...@gmail.com>
>>> wrote:
>>>
>>>> Does NiFi have a processor that can act as a client for a long polling
>>>> server, for example an SSE server?
>>>> More specifically, I want a client that can issue a HTTP GET request to
>>>> a long polling server and accept stream of messages from the server (on the
>>>> same connection).
>>>> If there isn't one, which processor is best to extend to achieve this
>>>> goal?
>>>>
>>>> thanks.
>>>>
>>>

Re: Long Polling Client

[Joe Witt <jo...@gmail.com>]

Right makes sense.  They're waiting for the completion of the response as
the payload to pass on.  If we need incremental handling of the response
body then we need to factor that into a given processor.  InvokeHttp or
something like it is a good candidate.  Almost a sort of 'InvokeHttpRecord'
kind of processor which uses the format/schema awareness of records to
frame the response objects and then based on some configurable value passes
those records on as a flow file.

Thanks

On Wed, Oct 14, 2020 at 1:25 PM Clay Teahouse <cl...@gmail.com>
wrote:

> Hello,
> I tried both getHTTP and invokeHTTP (but didn't try all options). What I
> need is to deal with the cases, such as SSE (Server Sent Events) which
> works with long polling. Meaning the client initiates a connection (via a
> HTTP request) to the SSE server, the server keeps the connection open (for
> as long as possible) and streams data back to the client on the same
> connection. I am able to have the data streamed to the client if I set up a
> stand alone HTTP client and issue HTTP requests but I am not able to get
> getHTTP or invokeHTTP work (I don't get any messages back).
>
> thanks.
>
> On Wed, Oct 14, 2020 at 2:50 PM Joe Witt <jo...@gmail.com> wrote:
>
>> Clay
>>
>> Have you evaluated whether InvokeHTTP will give you the desired behavior
>> for your case - in particular with a long timeout perhaps?  If you have and
>> it doesn't do the job do you mean something which initiates a request to an
>> HTTP server then assumes the response will remain open and it should take
>> portions of the response and treat each as its own flowfile/object to pass
>> along?
>>
>> Thanks
>>
>> On Tue, Oct 13, 2020 at 6:31 PM Clay Teahouse <cl...@gmail.com>
>> wrote:
>>
>>> Does NiFi have a processor that can act as a client for a long polling
>>> server, for example an SSE server?
>>> More specifically, I want a client that can issue a HTTP GET request to
>>> a long polling server and accept stream of messages from the server (on the
>>> same connection).
>>> If there isn't one, which processor is best to extend to achieve this
>>> goal?
>>>
>>> thanks.
>>>
>>

Re: Long Polling Client

[Clay Teahouse <cl...@gmail.com>]

Hello,
I tried both getHTTP and invokeHTTP (but didn't try all options). What I
need is to deal with the cases, such as SSE (Server Sent Events) which
works with long polling. Meaning the client initiates a connection (via a
HTTP request) to the SSE server, the server keeps the connection open (for
as long as possible) and streams data back to the client on the same
connection. I am able to have the data streamed to the client if I set up a
stand alone HTTP client and issue HTTP requests but I am not able to get
getHTTP or invokeHTTP work (I don't get any messages back).

thanks.

On Wed, Oct 14, 2020 at 2:50 PM Joe Witt <jo...@gmail.com> wrote:

> Clay
>
> Have you evaluated whether InvokeHTTP will give you the desired behavior
> for your case - in particular with a long timeout perhaps?  If you have and
> it doesn't do the job do you mean something which initiates a request to an
> HTTP server then assumes the response will remain open and it should take
> portions of the response and treat each as its own flowfile/object to pass
> along?
>
> Thanks
>
> On Tue, Oct 13, 2020 at 6:31 PM Clay Teahouse <cl...@gmail.com>
> wrote:
>
>> Does NiFi have a processor that can act as a client for a long polling
>> server, for example an SSE server?
>> More specifically, I want a client that can issue a HTTP GET request to a
>> long polling server and accept stream of messages from the server (on the
>> same connection).
>> If there isn't one, which processor is best to extend to achieve this
>> goal?
>>
>> thanks.
>>
>

Re: Long Polling Client

[Joe Witt <jo...@gmail.com>]

Clay

Have you evaluated whether InvokeHTTP will give you the desired behavior
for your case - in particular with a long timeout perhaps?  If you have and
it doesn't do the job do you mean something which initiates a request to an
HTTP server then assumes the response will remain open and it should take
portions of the response and treat each as its own flowfile/object to pass
along?

Thanks

On Tue, Oct 13, 2020 at 6:31 PM Clay Teahouse <cl...@gmail.com>
wrote:

> Does NiFi have a processor that can act as a client for a long polling
> server, for example an SSE server?
> More specifically, I want a client that can issue a HTTP GET request to a
> long polling server and accept stream of messages from the server (on the
> same connection).
> If there isn't one, which processor is best to extend to achieve this goal?
>
> thanks.
>