You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by steveM <st...@lmco.com> on 2015/08/03 19:46:30 UTC
Route Original Flow File Base on InvokeHTTP Response
I have a use case where I need to call a web service based on a flow file
attribute then route the original flow file based on the response from the
web service call. Can this be done using the standard processors and if so,
how?
--
View this message in context: http://apache-nifi-incubating-developer-list.39713.n7.nabble.com/Route-Original-Flow-File-Base-on-InvokeHTTP-Response-tp2317.html
Sent from the Apache NiFi (incubating) Developer List mailing list archive at Nabble.com.
Re: Route Original Flow File Base on InvokeHTTP Response
Posted by Joe Witt <jo...@gmail.com>.
https://issues.apache.org/jira/browse/NIFI-812
On Tue, Aug 4, 2015 at 12:52 PM, Joe Witt <jo...@gmail.com> wrote:
> Again with adam. Will make a jira and knock that out. Will also pursue
> what bryan mentioned though that will take longer
>
> On Aug 4, 2015 12:44 PM, "Adam Taft" <ad...@adamtaft.com> wrote:
>>
>> One option I think we kicked around at some point was to capture the
>> response body as a flowfile attribute in the original flowfile. For
>> reasonably sized response bodies, this would work OK. It would be a nice
>> way to handle your situation, because then the response becomes an
>> attribute of the request.
>>
>> This would obviously take a code change, but adding a property to the
>> effect of "Capture response body as flowfile attribute" might be a nice
>> feature.
>>
>>
>> On Tue, Aug 4, 2015 at 11:57 AM, steveM <st...@lmco.com>
>> wrote:
>>
>> > My use case is I pull the doc id from the file, call a web service with
>> > that
>> > id. The service responds with json that I would then parse to determine
>> > where to route the document next. Sometimes the document might be new,
>> > sometimes an update is allowed, sometimes duplicates need to be put
>> > somewhere else.
>> > I was hoping I was missing something that allowed you to handle a
>> > response
>> > and just add an attribute to the original file (or something similar to
>> > handle this case).
>> >
>> >
>> >
>> > --
>> > View this message in context:
>> >
>> > http://apache-nifi-incubating-developer-list.39713.n7.nabble.com/Route-Original-Flow-File-Base-on-InvokeHTTP-Response-tp2317p2343.html
>> > Sent from the Apache NiFi (incubating) Developer List mailing list
>> > archive
>> > at Nabble.com.
>> >
Re: Route Original Flow File Base on InvokeHTTP Response
Posted by Joe Witt <jo...@gmail.com>.
Again with adam. Will make a jira and knock that out. Will also pursue
what bryan mentioned though that will take longer
On Aug 4, 2015 12:44 PM, "Adam Taft" <ad...@adamtaft.com> wrote:
> One option I think we kicked around at some point was to capture the
> response body as a flowfile attribute in the original flowfile. For
> reasonably sized response bodies, this would work OK. It would be a nice
> way to handle your situation, because then the response becomes an
> attribute of the request.
>
> This would obviously take a code change, but adding a property to the
> effect of "Capture response body as flowfile attribute" might be a nice
> feature.
>
>
> On Tue, Aug 4, 2015 at 11:57 AM, steveM <st...@lmco.com>
> wrote:
>
> > My use case is I pull the doc id from the file, call a web service with
> > that
> > id. The service responds with json that I would then parse to determine
> > where to route the document next. Sometimes the document might be new,
> > sometimes an update is allowed, sometimes duplicates need to be put
> > somewhere else.
> > I was hoping I was missing something that allowed you to handle a
> response
> > and just add an attribute to the original file (or something similar to
> > handle this case).
> >
> >
> >
> > --
> > View this message in context:
> >
> http://apache-nifi-incubating-developer-list.39713.n7.nabble.com/Route-Original-Flow-File-Base-on-InvokeHTTP-Response-tp2317p2343.html
> > Sent from the Apache NiFi (incubating) Developer List mailing list
> archive
> > at Nabble.com.
> >
>
Re: Route Original Flow File Base on InvokeHTTP Response
Posted by Adam Taft <ad...@adamtaft.com>.
One option I think we kicked around at some point was to capture the
response body as a flowfile attribute in the original flowfile. For
reasonably sized response bodies, this would work OK. It would be a nice
way to handle your situation, because then the response becomes an
attribute of the request.
This would obviously take a code change, but adding a property to the
effect of "Capture response body as flowfile attribute" might be a nice
feature.
On Tue, Aug 4, 2015 at 11:57 AM, steveM <st...@lmco.com> wrote:
> My use case is I pull the doc id from the file, call a web service with
> that
> id. The service responds with json that I would then parse to determine
> where to route the document next. Sometimes the document might be new,
> sometimes an update is allowed, sometimes duplicates need to be put
> somewhere else.
> I was hoping I was missing something that allowed you to handle a response
> and just add an attribute to the original file (or something similar to
> handle this case).
>
>
>
> --
> View this message in context:
> http://apache-nifi-incubating-developer-list.39713.n7.nabble.com/Route-Original-Flow-File-Base-on-InvokeHTTP-Response-tp2317p2343.html
> Sent from the Apache NiFi (incubating) Developer List mailing list archive
> at Nabble.com.
>
Re: Route Original Flow File Base on InvokeHTTP Response
Posted by steveM <st...@lmco.com>.
My use case is I pull the doc id from the file, call a web service with that
id. The service responds with json that I would then parse to determine
where to route the document next. Sometimes the document might be new,
sometimes an update is allowed, sometimes duplicates need to be put
somewhere else.
I was hoping I was missing something that allowed you to handle a response
and just add an attribute to the original file (or something similar to
handle this case).
--
View this message in context: http://apache-nifi-incubating-developer-list.39713.n7.nabble.com/Route-Original-Flow-File-Base-on-InvokeHTTP-Response-tp2317p2343.html
Sent from the Apache NiFi (incubating) Developer List mailing list archive at Nabble.com.
RE: Route Original Flow File Base on InvokeHTTP Response
Posted by Mark Payne <ma...@hotmail.com>.
Steve,
I certainly agree with both Joe & Adam here. I do think that this could in fact be done by using
the MergeContent processor. However, it would get pretty confusing and would be inefficient
and a bit 'unclean', as you would be really be using some processors to achieve a goal very different
than what they were designed for.
That being said, a member of the community, Joe Gresock, submitted a ticket a while back [1] that
I think would be a great solution for this use case.
Unfortunately, it's not a processor that is in the baseline right now, but it's being worked on. We should
be able to get it into the build pretty soon.
I hope this helps!
-Mark
[1] HoldFile Processor: https://issues.apache.org/jira/browse/NIFI-190
----------------------------------------
> Date: Tue, 4 Aug 2015 10:45:44 -0400
> Subject: Re: Route Original Flow File Base on InvokeHTTP Response
> From: joe.witt@gmail.com
> To: dev@nifi.apache.org
> CC: dev@nifi.incubator.apache.org
>
> Yep - i'm with Adam's interpretation here.
>
> Steve: For your case can you elaborate on what you'd want to do with
> the content of the web response in deciding how to handle the orig? I
> do think a custom processor would be necessary but if perhaps we can
> add something simple/consistent with the purpose of InvokeHTTP it can
> be avoided. Just need to understand the use case a bit better
>
> Thanks
> Joe
>
> On Tue, Aug 4, 2015 at 10:36 AM, Adam Taft <ad...@adamtaft.com> wrote:
>> Right. Fundamentally, with InvokeHTTP you have two flowfiles involved.
>> The original flowfile which represents the HTTP request, and a newly
>> created "response" flowfile which captures the message body from the
>> response.
>>
>> After invoke HTTP does its thing, you are effectively left with two
>> flowfiles. They have a common "transaction id" associated to both
>> flowfiles, so it might be possible to use this with, for example,
>> MergeContent and associate the two files back together.
>>
>> If MergeContent can't be made to work, it might require a new custom
>> processor to take the two flowfiles coming out from InvokeHTTP and evaluate
>> them as a single unit.
>>
>> Adam
>>
>>
>> On Tue, Aug 4, 2015 at 10:27 AM, Joe Witt <jo...@gmail.com> wrote:
>>
>>> Aldrin, Bryan
>>>
>>> I think the point is to route the orig flowfile based on analyzing the
>>> content of the web response. The setup would be more involved
>>> On Aug 4, 2015 8:52 AM, "Aldrin Piri" <al...@gmail.com> wrote:
>>>
>>>> Steve,
>>>>
>>>> Building on the template Bryan provided you can route a successful
>>> response
>>>> to perform further inspection. Depending on content any of RouteOnContent
>>>> or EvaluateJsonPath/EvaluateXPath in conjunction with RouteOnAtrribute
>>>> could be viable options.
>>>>
>>>> Let us know if this helps with the use case you are tackling or if there
>>>> are any other questions.
>>>>
>>>> --
>>>> Sent from my mobile device.
>>>> On Tue, Aug 4, 2015 at 07:32 steveM <st...@lmco.com> wrote:
>>>>
>>>>> Thanks for the quick response. However, my use case requires that I
>>> parse
>>>>> the
>>>>> actual response (not just make sure the response code is 200) to make a
>>>>> decision on routing.
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> View this message in context:
>>>>>
>>>>
>>> http://apache-nifi-incubating-developer-list.39713.n7.nabble.com/Route-Original-Flow-File-Base-on-InvokeHTTP-Response-tp2317p2324.html
>>>>> Sent from the Apache NiFi (incubating) Developer List mailing list
>>>> archive
>>>>> at Nabble.com.
>>>>>
>>>>
>>>
Re: Route Original Flow File Base on InvokeHTTP Response
Posted by Joe Witt <jo...@gmail.com>.
Yep - i'm with Adam's interpretation here.
Steve: For your case can you elaborate on what you'd want to do with
the content of the web response in deciding how to handle the orig? I
do think a custom processor would be necessary but if perhaps we can
add something simple/consistent with the purpose of InvokeHTTP it can
be avoided. Just need to understand the use case a bit better
Thanks
Joe
On Tue, Aug 4, 2015 at 10:36 AM, Adam Taft <ad...@adamtaft.com> wrote:
> Right. Fundamentally, with InvokeHTTP you have two flowfiles involved.
> The original flowfile which represents the HTTP request, and a newly
> created "response" flowfile which captures the message body from the
> response.
>
> After invoke HTTP does its thing, you are effectively left with two
> flowfiles. They have a common "transaction id" associated to both
> flowfiles, so it might be possible to use this with, for example,
> MergeContent and associate the two files back together.
>
> If MergeContent can't be made to work, it might require a new custom
> processor to take the two flowfiles coming out from InvokeHTTP and evaluate
> them as a single unit.
>
> Adam
>
>
> On Tue, Aug 4, 2015 at 10:27 AM, Joe Witt <jo...@gmail.com> wrote:
>
>> Aldrin, Bryan
>>
>> I think the point is to route the orig flowfile based on analyzing the
>> content of the web response. The setup would be more involved
>> On Aug 4, 2015 8:52 AM, "Aldrin Piri" <al...@gmail.com> wrote:
>>
>> > Steve,
>> >
>> > Building on the template Bryan provided you can route a successful
>> response
>> > to perform further inspection. Depending on content any of RouteOnContent
>> > or EvaluateJsonPath/EvaluateXPath in conjunction with RouteOnAtrribute
>> > could be viable options.
>> >
>> > Let us know if this helps with the use case you are tackling or if there
>> > are any other questions.
>> >
>> > --
>> > Sent from my mobile device.
>> > On Tue, Aug 4, 2015 at 07:32 steveM <st...@lmco.com> wrote:
>> >
>> > > Thanks for the quick response. However, my use case requires that I
>> parse
>> > > the
>> > > actual response (not just make sure the response code is 200) to make a
>> > > decision on routing.
>> > >
>> > >
>> > >
>> > > --
>> > > View this message in context:
>> > >
>> >
>> http://apache-nifi-incubating-developer-list.39713.n7.nabble.com/Route-Original-Flow-File-Base-on-InvokeHTTP-Response-tp2317p2324.html
>> > > Sent from the Apache NiFi (incubating) Developer List mailing list
>> > archive
>> > > at Nabble.com.
>> > >
>> >
>>
Re: Route Original Flow File Base on InvokeHTTP Response
Posted by Adam Taft <ad...@adamtaft.com>.
Right. Fundamentally, with InvokeHTTP you have two flowfiles involved.
The original flowfile which represents the HTTP request, and a newly
created "response" flowfile which captures the message body from the
response.
After invoke HTTP does its thing, you are effectively left with two
flowfiles. They have a common "transaction id" associated to both
flowfiles, so it might be possible to use this with, for example,
MergeContent and associate the two files back together.
If MergeContent can't be made to work, it might require a new custom
processor to take the two flowfiles coming out from InvokeHTTP and evaluate
them as a single unit.
Adam
On Tue, Aug 4, 2015 at 10:27 AM, Joe Witt <jo...@gmail.com> wrote:
> Aldrin, Bryan
>
> I think the point is to route the orig flowfile based on analyzing the
> content of the web response. The setup would be more involved
> On Aug 4, 2015 8:52 AM, "Aldrin Piri" <al...@gmail.com> wrote:
>
> > Steve,
> >
> > Building on the template Bryan provided you can route a successful
> response
> > to perform further inspection. Depending on content any of RouteOnContent
> > or EvaluateJsonPath/EvaluateXPath in conjunction with RouteOnAtrribute
> > could be viable options.
> >
> > Let us know if this helps with the use case you are tackling or if there
> > are any other questions.
> >
> > --
> > Sent from my mobile device.
> > On Tue, Aug 4, 2015 at 07:32 steveM <st...@lmco.com> wrote:
> >
> > > Thanks for the quick response. However, my use case requires that I
> parse
> > > the
> > > actual response (not just make sure the response code is 200) to make a
> > > decision on routing.
> > >
> > >
> > >
> > > --
> > > View this message in context:
> > >
> >
> http://apache-nifi-incubating-developer-list.39713.n7.nabble.com/Route-Original-Flow-File-Base-on-InvokeHTTP-Response-tp2317p2324.html
> > > Sent from the Apache NiFi (incubating) Developer List mailing list
> > archive
> > > at Nabble.com.
> > >
> >
>
Re: Route Original Flow File Base on InvokeHTTP Response
Posted by Joe Witt <jo...@gmail.com>.
Aldrin, Bryan
I think the point is to route the orig flowfile based on analyzing the
content of the web response. The setup would be more involved
On Aug 4, 2015 8:52 AM, "Aldrin Piri" <al...@gmail.com> wrote:
> Steve,
>
> Building on the template Bryan provided you can route a successful response
> to perform further inspection. Depending on content any of RouteOnContent
> or EvaluateJsonPath/EvaluateXPath in conjunction with RouteOnAtrribute
> could be viable options.
>
> Let us know if this helps with the use case you are tackling or if there
> are any other questions.
>
> --
> Sent from my mobile device.
> On Tue, Aug 4, 2015 at 07:32 steveM <st...@lmco.com> wrote:
>
> > Thanks for the quick response. However, my use case requires that I parse
> > the
> > actual response (not just make sure the response code is 200) to make a
> > decision on routing.
> >
> >
> >
> > --
> > View this message in context:
> >
> http://apache-nifi-incubating-developer-list.39713.n7.nabble.com/Route-Original-Flow-File-Base-on-InvokeHTTP-Response-tp2317p2324.html
> > Sent from the Apache NiFi (incubating) Developer List mailing list
> archive
> > at Nabble.com.
> >
>
Re: Route Original Flow File Base on InvokeHTTP Response
Posted by Aldrin Piri <al...@gmail.com>.
Steve,
Building on the template Bryan provided you can route a successful response
to perform further inspection. Depending on content any of RouteOnContent
or EvaluateJsonPath/EvaluateXPath in conjunction with RouteOnAtrribute
could be viable options.
Let us know if this helps with the use case you are tackling or if there
are any other questions.
--
Sent from my mobile device.
On Tue, Aug 4, 2015 at 07:32 steveM <st...@lmco.com> wrote:
> Thanks for the quick response. However, my use case requires that I parse
> the
> actual response (not just make sure the response code is 200) to make a
> decision on routing.
>
>
>
> --
> View this message in context:
> http://apache-nifi-incubating-developer-list.39713.n7.nabble.com/Route-Original-Flow-File-Base-on-InvokeHTTP-Response-tp2317p2324.html
> Sent from the Apache NiFi (incubating) Developer List mailing list archive
> at Nabble.com.
>
Re: Route Original Flow File Base on InvokeHTTP Response
Posted by steveM <st...@lmco.com>.
Thanks for the quick response. However, my use case requires that I parse the
actual response (not just make sure the response code is 200) to make a
decision on routing.
--
View this message in context: http://apache-nifi-incubating-developer-list.39713.n7.nabble.com/Route-Original-Flow-File-Base-on-InvokeHTTP-Response-tp2317p2324.html
Sent from the Apache NiFi (incubating) Developer List mailing list archive at Nabble.com.
Re: Route Original Flow File Base on InvokeHTTP Response
Posted by Bryan Bende <bb...@gmail.com>.
Steve,
I agree with Adam's idea of using InvokeHTTP.
I created an example flow that I believe demonstrates what you are trying
to do. You can download the template here
(InvokeHttp_And_Route_Original_On_Status.xml):
https://cwiki.apache.org/confluence/display/NIFI/Example+Dataflow+Templates
The example searches Google every 30 seconds for a query that was specified
in a FlowFile attribute (in this case q=nifi), and then routes the original
FlowFile to a HashContent processor whenever a 200 was received from
Google. The HashContent would be replaced with whatever you wanted to do.
Let us know if you have any questions.
-Bryan
On Mon, Aug 3, 2015 at 2:52 PM, Adam Taft <ad...@adamtaft.com> wrote:
> If you're needing to read the HTTP response code, InvokeHTTP should do this
> for you. It basically stores the HTTP response code into the original
> request flowfile. With a 2xx response code, it will route the request
> flowfile (with the response code attributes) to the "Original"
> relationship. From there, the flowfile should have a
> "invokehttp.status.code" attribute which you can act on.
>
>
>
>
> On Mon, Aug 3, 2015 at 1:46 PM, steveM <st...@lmco.com> wrote:
>
> > I have a use case where I need to call a web service based on a flow file
> > attribute then route the original flow file based on the response from
> the
> > web service call. Can this be done using the standard processors and if
> so,
> > how?
> >
> >
> >
> > --
> > View this message in context:
> >
> http://apache-nifi-incubating-developer-list.39713.n7.nabble.com/Route-Original-Flow-File-Base-on-InvokeHTTP-Response-tp2317.html
> > Sent from the Apache NiFi (incubating) Developer List mailing list
> archive
> > at Nabble.com.
> >
>
Re: Route Original Flow File Base on InvokeHTTP Response
Posted by Bryan Bende <bb...@gmail.com>.
Steve,
I agree with Adam's idea of using InvokeHTTP.
I created an example flow that I believe demonstrates what you are trying
to do. You can download the template here
(InvokeHttp_And_Route_Original_On_Status.xml):
https://cwiki.apache.org/confluence/display/NIFI/Example+Dataflow+Templates
The example searches Google every 30 seconds for a query that was specified
in a FlowFile attribute (in this case q=nifi), and then routes the original
FlowFile to a HashContent processor whenever a 200 was received from
Google. The HashContent would be replaced with whatever you wanted to do.
Let us know if you have any questions.
-Bryan
On Mon, Aug 3, 2015 at 2:52 PM, Adam Taft <ad...@adamtaft.com> wrote:
> If you're needing to read the HTTP response code, InvokeHTTP should do this
> for you. It basically stores the HTTP response code into the original
> request flowfile. With a 2xx response code, it will route the request
> flowfile (with the response code attributes) to the "Original"
> relationship. From there, the flowfile should have a
> "invokehttp.status.code" attribute which you can act on.
>
>
>
>
> On Mon, Aug 3, 2015 at 1:46 PM, steveM <st...@lmco.com> wrote:
>
> > I have a use case where I need to call a web service based on a flow file
> > attribute then route the original flow file based on the response from
> the
> > web service call. Can this be done using the standard processors and if
> so,
> > how?
> >
> >
> >
> > --
> > View this message in context:
> >
> http://apache-nifi-incubating-developer-list.39713.n7.nabble.com/Route-Original-Flow-File-Base-on-InvokeHTTP-Response-tp2317.html
> > Sent from the Apache NiFi (incubating) Developer List mailing list
> archive
> > at Nabble.com.
> >
>
Re: Route Original Flow File Base on InvokeHTTP Response
Posted by Adam Taft <ad...@adamtaft.com>.
If you're needing to read the HTTP response code, InvokeHTTP should do this
for you. It basically stores the HTTP response code into the original
request flowfile. With a 2xx response code, it will route the request
flowfile (with the response code attributes) to the "Original"
relationship. From there, the flowfile should have a
"invokehttp.status.code" attribute which you can act on.
On Mon, Aug 3, 2015 at 1:46 PM, steveM <st...@lmco.com> wrote:
> I have a use case where I need to call a web service based on a flow file
> attribute then route the original flow file based on the response from the
> web service call. Can this be done using the standard processors and if so,
> how?
>
>
>
> --
> View this message in context:
> http://apache-nifi-incubating-developer-list.39713.n7.nabble.com/Route-Original-Flow-File-Base-on-InvokeHTTP-Response-tp2317.html
> Sent from the Apache NiFi (incubating) Developer List mailing list archive
> at Nabble.com.
>