You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by James McMahon <js...@gmail.com> on 2016/09/04 11:40:08 UTC
Posting input files to NiFi using REST
I have a series of applications, each of which I need to redesign so that
they post raw data to NiFi. Is there an example showing how to to construct
and execute RESTful calls that push data to NiFi? My assumption is that I
would establish a ListenHTTP processor in my workflow to receive the
incoming content. Is that correct, or should this be handled differently?
My apps are in java and are simple, so I can readily port them to nodeJS,
Python, Groovy, etc. Thanks in advance for your help.
Re: Posting input files to NiFi using REST
Posted by Matt Burgess <ma...@apache.org>.
I have some undisciplined samples [1] in Groovy with HttpBuilder, also
you could use httplib in Python [2] (some 2.4-based examples at [3]).
For Java you can use OkHttp [4] as some of the NiFi processors do, or
Apache HttpComponents (example [5]), etc.
Regards,
Matt
[1] https://github.com/mattyb149/nifi-client/tree/master/src/main/groovy/nifi/client
[2] https://docs.python.org/2/library/httplib.html
[3] https://docs.python.org/2.4/lib/httplib-examples.html
[4] http://square.github.io/okhttp/
[5] http://stackoverflow.com/questions/3324717/sending-http-post-request-in-java
On Mon, Sep 5, 2016 at 7:29 PM, James McMahon <js...@gmail.com> wrote:
> Thank you Matt. This helps me configure NiFi to field the request. Is there
> an example you can point me towards that shows how to build, issue, and send
> such a request - in Python or java, for example?
>
> On Sun, Sep 4, 2016 at 8:22 AM, Matt Burgess <ma...@apache.org> wrote:
>>
>> James,
>>
>> For simple calls that return immediately, ListenHttp probably works
>> fine. For more flexible and powerful processing of HTTP requests (and
>> responses), you might be better off with HandleHttpRequest and
>> HandleHttpResponse. There is an example of this under
>> Hello_NiFi_Web_Service [1].
>>
>> Regards,
>> Matt
>>
>> [1]
>> https://cwiki.apache.org/confluence/display/NIFI/Example+Dataflow+Templates
>>
>> On Sun, Sep 4, 2016 at 7:40 AM, James McMahon <js...@gmail.com>
>> wrote:
>> > I have a series of applications, each of which I need to redesign so
>> > that
>> > they post raw data to NiFi. Is there an example showing how to to
>> > construct
>> > and execute RESTful calls that push data to NiFi? My assumption is that
>> > I
>> > would establish a ListenHTTP processor in my workflow to receive the
>> > incoming content. Is that correct, or should this be handled
>> > differently? My
>> > apps are in java and are simple, so I can readily port them to nodeJS,
>> > Python, Groovy, etc. Thanks in advance for your help.
>
>
Re: Posting input files to NiFi using REST
Posted by James McMahon <js...@gmail.com>.
Thank you Matt. This helps me configure NiFi to field the request. Is there
an example you can point me towards that shows how to build, issue, and
send such a request - in Python or java, for example?
On Sun, Sep 4, 2016 at 8:22 AM, Matt Burgess <ma...@apache.org> wrote:
> James,
>
> For simple calls that return immediately, ListenHttp probably works
> fine. For more flexible and powerful processing of HTTP requests (and
> responses), you might be better off with HandleHttpRequest and
> HandleHttpResponse. There is an example of this under
> Hello_NiFi_Web_Service [1].
>
> Regards,
> Matt
>
> [1] https://cwiki.apache.org/confluence/display/NIFI/
> Example+Dataflow+Templates
>
> On Sun, Sep 4, 2016 at 7:40 AM, James McMahon <js...@gmail.com>
> wrote:
> > I have a series of applications, each of which I need to redesign so that
> > they post raw data to NiFi. Is there an example showing how to to
> construct
> > and execute RESTful calls that push data to NiFi? My assumption is that I
> > would establish a ListenHTTP processor in my workflow to receive the
> > incoming content. Is that correct, or should this be handled
> differently? My
> > apps are in java and are simple, so I can readily port them to nodeJS,
> > Python, Groovy, etc. Thanks in advance for your help.
>
Re: Posting input files to NiFi using REST
Posted by Joe Skora <js...@gmail.com>.
That's great news!
I hope the rest goes smoothly, but if not let us know!
On Thu, Sep 8, 2016 at 8:30 AM, James McMahon <js...@gmail.com> wrote:
> Worked like a charm once I added the SSLContextService. With the basics in
> place for the HTTP exchange now I can press on and add my NiFi flow logic.
> Thanks again.
>
> On Wed, Sep 7, 2016 at 5:26 PM, Joe Skora <js...@gmail.com> wrote:
>
>> James,
>>
>> I believe you need both. StandardHttpContextMap caches request
>> information between a pair of HandleHttpRequest and HandleHttpResponse
>> processors and the StandardSSLContextService provides server side
>> encryption of the HTTPS connection.
>>
>> I don't think you mentioned a HandleHttpResponse processor, do you have
>> one to provide the web service response? If not, look at
>> "Hello_NiFi_Web_Service" template on the Example Dataflow Templates
>> <https://cwiki.apache.org/confluence/display/NIFI/Example+Dataflow+Templates>
>> page.
>>
>> Regards,
>> Joe S
>>
>> On Wed, Sep 7, 2016 at 3:07 PM, James McMahon <js...@gmail.com>
>> wrote:
>>
>>> I tried to POST to HandleHTTPRequest from Python, but it is not working.
>>> I noticed that I've got StandardHttpContextMap configured in this
>>> processor. I am using a secure https NiFi instance; instead of
>>> StandardHttpContextMap, should I be configuring this processor with
>>> StandardSSLContextService? Thank you.
>>>
>>> On Sun, Sep 4, 2016 at 8:22 AM, Matt Burgess <ma...@apache.org>
>>> wrote:
>>>
>>>> James,
>>>>
>>>> For simple calls that return immediately, ListenHttp probably works
>>>> fine. For more flexible and powerful processing of HTTP requests (and
>>>> responses), you might be better off with HandleHttpRequest and
>>>> HandleHttpResponse. There is an example of this under
>>>> Hello_NiFi_Web_Service [1].
>>>>
>>>> Regards,
>>>> Matt
>>>>
>>>> [1] https://cwiki.apache.org/confluence/display/NIFI/Example+Dat
>>>> aflow+Templates
>>>>
>>>> On Sun, Sep 4, 2016 at 7:40 AM, James McMahon <js...@gmail.com>
>>>> wrote:
>>>> > I have a series of applications, each of which I need to redesign so
>>>> that
>>>> > they post raw data to NiFi. Is there an example showing how to to
>>>> construct
>>>> > and execute RESTful calls that push data to NiFi? My assumption is
>>>> that I
>>>> > would establish a ListenHTTP processor in my workflow to receive the
>>>> > incoming content. Is that correct, or should this be handled
>>>> differently? My
>>>> > apps are in java and are simple, so I can readily port them to nodeJS,
>>>> > Python, Groovy, etc. Thanks in advance for your help.
>>>>
>>>
>>>
>>
>
Re: Posting input files to NiFi using REST
Posted by James McMahon <js...@gmail.com>.
Worked like a charm once I added the SSLContextService. With the basics in
place for the HTTP exchange now I can press on and add my NiFi flow logic.
Thanks again.
On Wed, Sep 7, 2016 at 5:26 PM, Joe Skora <js...@gmail.com> wrote:
> James,
>
> I believe you need both. StandardHttpContextMap caches request
> information between a pair of HandleHttpRequest and HandleHttpResponse
> processors and the StandardSSLContextService provides server side
> encryption of the HTTPS connection.
>
> I don't think you mentioned a HandleHttpResponse processor, do you have
> one to provide the web service response? If not, look at
> "Hello_NiFi_Web_Service" template on the Example Dataflow Templates
> <https://cwiki.apache.org/confluence/display/NIFI/Example+Dataflow+Templates>
> page.
>
> Regards,
> Joe S
>
> On Wed, Sep 7, 2016 at 3:07 PM, James McMahon <js...@gmail.com>
> wrote:
>
>> I tried to POST to HandleHTTPRequest from Python, but it is not working.
>> I noticed that I've got StandardHttpContextMap configured in this
>> processor. I am using a secure https NiFi instance; instead of
>> StandardHttpContextMap, should I be configuring this processor with
>> StandardSSLContextService? Thank you.
>>
>> On Sun, Sep 4, 2016 at 8:22 AM, Matt Burgess <ma...@apache.org>
>> wrote:
>>
>>> James,
>>>
>>> For simple calls that return immediately, ListenHttp probably works
>>> fine. For more flexible and powerful processing of HTTP requests (and
>>> responses), you might be better off with HandleHttpRequest and
>>> HandleHttpResponse. There is an example of this under
>>> Hello_NiFi_Web_Service [1].
>>>
>>> Regards,
>>> Matt
>>>
>>> [1] https://cwiki.apache.org/confluence/display/NIFI/Example+Dat
>>> aflow+Templates
>>>
>>> On Sun, Sep 4, 2016 at 7:40 AM, James McMahon <js...@gmail.com>
>>> wrote:
>>> > I have a series of applications, each of which I need to redesign so
>>> that
>>> > they post raw data to NiFi. Is there an example showing how to to
>>> construct
>>> > and execute RESTful calls that push data to NiFi? My assumption is
>>> that I
>>> > would establish a ListenHTTP processor in my workflow to receive the
>>> > incoming content. Is that correct, or should this be handled
>>> differently? My
>>> > apps are in java and are simple, so I can readily port them to nodeJS,
>>> > Python, Groovy, etc. Thanks in advance for your help.
>>>
>>
>>
>
Re: Posting input files to NiFi using REST
Posted by James McMahon <js...@gmail.com>.
Thank you very much Joe. I'll work on configuring my SSLContextService. I
am indeed using Hello_NiFi_Web_Service as my starting point. I do have that
HandleHttpResponse processor in there but haven't yet configured and
enabled that. I will do that too. My initial thought was this: get
something to POST from Python over to the processor in NiFi first, worry
about the response next. Perhaps both are required for it to work; I'm not
sure, being that I'm just now trying this for the first time. I'll give it
another try. Thanks again, Matt and Joe. Cheers, Jim
On Wed, Sep 7, 2016 at 5:26 PM, Joe Skora <js...@gmail.com> wrote:
> James,
>
> I believe you need both. StandardHttpContextMap caches request
> information between a pair of HandleHttpRequest and HandleHttpResponse
> processors and the StandardSSLContextService provides server side
> encryption of the HTTPS connection.
>
> I don't think you mentioned a HandleHttpResponse processor, do you have
> one to provide the web service response? If not, look at
> "Hello_NiFi_Web_Service" template on the Example Dataflow Templates
> <https://cwiki.apache.org/confluence/display/NIFI/Example+Dataflow+Templates>
> page.
>
> Regards,
> Joe S
>
> On Wed, Sep 7, 2016 at 3:07 PM, James McMahon <js...@gmail.com>
> wrote:
>
>> I tried to POST to HandleHTTPRequest from Python, but it is not working.
>> I noticed that I've got StandardHttpContextMap configured in this
>> processor. I am using a secure https NiFi instance; instead of
>> StandardHttpContextMap, should I be configuring this processor with
>> StandardSSLContextService? Thank you.
>>
>> On Sun, Sep 4, 2016 at 8:22 AM, Matt Burgess <ma...@apache.org>
>> wrote:
>>
>>> James,
>>>
>>> For simple calls that return immediately, ListenHttp probably works
>>> fine. For more flexible and powerful processing of HTTP requests (and
>>> responses), you might be better off with HandleHttpRequest and
>>> HandleHttpResponse. There is an example of this under
>>> Hello_NiFi_Web_Service [1].
>>>
>>> Regards,
>>> Matt
>>>
>>> [1] https://cwiki.apache.org/confluence/display/NIFI/Example+Dat
>>> aflow+Templates
>>>
>>> On Sun, Sep 4, 2016 at 7:40 AM, James McMahon <js...@gmail.com>
>>> wrote:
>>> > I have a series of applications, each of which I need to redesign so
>>> that
>>> > they post raw data to NiFi. Is there an example showing how to to
>>> construct
>>> > and execute RESTful calls that push data to NiFi? My assumption is
>>> that I
>>> > would establish a ListenHTTP processor in my workflow to receive the
>>> > incoming content. Is that correct, or should this be handled
>>> differently? My
>>> > apps are in java and are simple, so I can readily port them to nodeJS,
>>> > Python, Groovy, etc. Thanks in advance for your help.
>>>
>>
>>
>
Re: Posting input files to NiFi using REST
Posted by Joe Skora <js...@gmail.com>.
James,
I believe you need both. StandardHttpContextMap caches request information
between a pair of HandleHttpRequest and HandleHttpResponse processors and
the StandardSSLContextService provides server side encryption of the HTTPS
connection.
I don't think you mentioned a HandleHttpResponse processor, do you have one
to provide the web service response? If not, look at
"Hello_NiFi_Web_Service" template on the Example Dataflow Templates
<https://cwiki.apache.org/confluence/display/NIFI/Example+Dataflow+Templates>
page.
Regards,
Joe S
On Wed, Sep 7, 2016 at 3:07 PM, James McMahon <js...@gmail.com> wrote:
> I tried to POST to HandleHTTPRequest from Python, but it is not working. I
> noticed that I've got StandardHttpContextMap configured in this processor.
> I am using a secure https NiFi instance; instead of StandardHttpContextMap,
> should I be configuring this processor with StandardSSLContextService?
> Thank you.
>
> On Sun, Sep 4, 2016 at 8:22 AM, Matt Burgess <ma...@apache.org> wrote:
>
>> James,
>>
>> For simple calls that return immediately, ListenHttp probably works
>> fine. For more flexible and powerful processing of HTTP requests (and
>> responses), you might be better off with HandleHttpRequest and
>> HandleHttpResponse. There is an example of this under
>> Hello_NiFi_Web_Service [1].
>>
>> Regards,
>> Matt
>>
>> [1] https://cwiki.apache.org/confluence/display/NIFI/Example+
>> Dataflow+Templates
>>
>> On Sun, Sep 4, 2016 at 7:40 AM, James McMahon <js...@gmail.com>
>> wrote:
>> > I have a series of applications, each of which I need to redesign so
>> that
>> > they post raw data to NiFi. Is there an example showing how to to
>> construct
>> > and execute RESTful calls that push data to NiFi? My assumption is that
>> I
>> > would establish a ListenHTTP processor in my workflow to receive the
>> > incoming content. Is that correct, or should this be handled
>> differently? My
>> > apps are in java and are simple, so I can readily port them to nodeJS,
>> > Python, Groovy, etc. Thanks in advance for your help.
>>
>
>
Re: Posting input files to NiFi using REST
Posted by James McMahon <js...@gmail.com>.
I tried to POST to HandleHTTPRequest from Python, but it is not working. I
noticed that I've got StandardHttpContextMap configured in this processor.
I am using a secure https NiFi instance; instead of StandardHttpContextMap,
should I be configuring this processor with StandardSSLContextService?
Thank you.
On Sun, Sep 4, 2016 at 8:22 AM, Matt Burgess <ma...@apache.org> wrote:
> James,
>
> For simple calls that return immediately, ListenHttp probably works
> fine. For more flexible and powerful processing of HTTP requests (and
> responses), you might be better off with HandleHttpRequest and
> HandleHttpResponse. There is an example of this under
> Hello_NiFi_Web_Service [1].
>
> Regards,
> Matt
>
> [1] https://cwiki.apache.org/confluence/display/NIFI/
> Example+Dataflow+Templates
>
> On Sun, Sep 4, 2016 at 7:40 AM, James McMahon <js...@gmail.com>
> wrote:
> > I have a series of applications, each of which I need to redesign so that
> > they post raw data to NiFi. Is there an example showing how to to
> construct
> > and execute RESTful calls that push data to NiFi? My assumption is that I
> > would establish a ListenHTTP processor in my workflow to receive the
> > incoming content. Is that correct, or should this be handled
> differently? My
> > apps are in java and are simple, so I can readily port them to nodeJS,
> > Python, Groovy, etc. Thanks in advance for your help.
>
Re: Posting input files to NiFi using REST
Posted by Matt Burgess <ma...@apache.org>.
James,
For simple calls that return immediately, ListenHttp probably works
fine. For more flexible and powerful processing of HTTP requests (and
responses), you might be better off with HandleHttpRequest and
HandleHttpResponse. There is an example of this under
Hello_NiFi_Web_Service [1].
Regards,
Matt
[1] https://cwiki.apache.org/confluence/display/NIFI/Example+Dataflow+Templates
On Sun, Sep 4, 2016 at 7:40 AM, James McMahon <js...@gmail.com> wrote:
> I have a series of applications, each of which I need to redesign so that
> they post raw data to NiFi. Is there an example showing how to to construct
> and execute RESTful calls that push data to NiFi? My assumption is that I
> would establish a ListenHTTP processor in my workflow to receive the
> incoming content. Is that correct, or should this be handled differently? My
> apps are in java and are simple, so I can readily port them to nodeJS,
> Python, Groovy, etc. Thanks in advance for your help.