You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Michael Di Domenico <md...@gmail.com> on 2019/05/16 14:17:43 UTC
flowfile through invokehttp put
i'm pretty new to nifi, so i'm not sure this worded correctly or
possible. i have a flow which ends with an invokehttp process that
does a PUT to web server.
my naive understanding of this process is that when the flowfile hits
invokehttp, the flow-attributes can be sent as key:value pairs via the
http headers and the file-content gets PUT to the remote webserver via
the http content-body
is there a way to send the flow-attr and the flow-content (ie the
entire flowfile) via the PUT?
i have nifi on two ends of a connection, but in order to send the data
from one to another i have to send it through a proxy like device.
the near end of the device receives data from the PUT and far end does
a POST to a remote nifi.
this process works fine, but on the far end i only receive the file
and none of the attributes (as expected). i cannot configure the
proxy device to pass key:value pairs in the http headers. i'm
presuming all the flow information would have to be sent via the PUT
and then decoded by the nifi server on the far end. i'm hoping
there's a simple connector to do this rather then having me munge all
the data together into something like an xml or json file and send
that.
Re: flowfile through invokehttp put
Posted by Michael Di Domenico <md...@gmail.com>.
On Thu, May 16, 2019 at 10:48 AM Bryan Bende <bb...@gmail.com> wrote:
>
> Normally this would all be handled for you with site-to-site, but not
> sure that can be made to work with your custom proxy. Before
> site-to-site there was PostHttp and ListenHttp, and PostHttp has an
> options for "Send As Flow File". PostHTTP is marked as deprecated
> because generally you should use site-to-site or InvokeHttp, but
> InvokeHttp doesn't have the send as flow file concept. So you could
> try using PostHttp as long as you realize its deprecated, and maybe
> someone can add the send as flow file concept to InvokeHttp, or maybe
> a PackageContent processor.
thanks. i may try the posthttp just to see if it works, but this is a
new process (likely longlived) and i don't want to use a deprecated
module.
> The only other option I can think of is to use MergeContent with the
> Merge Format set to "FlowFile Stream v3" and configure the max and min
> entries to 1 so it doesn't really actually merge anything, but just
> writes out the new flow content. On the receiving side use
> UnpackContent also configured with format as "FlowFile Stream v3".
this seems like it'll merge the flow-contents and flow-attr from
multiple flows rather then merge the flow-attr and flow-content
together. which isn't really what i want, or am i reading the doc
page wrong
Re: flowfile through invokehttp put
Posted by Michael Di Domenico <md...@gmail.com>.
On Thu, May 16, 2019 at 10:48 AM Bryan Bende <bb...@gmail.com> wrote:
> The only other option I can think of is to use MergeContent with the
> Merge Format set to "FlowFile Stream v3" and configure the max and min
> entries to 1 so it doesn't really actually merge anything, but just
> writes out the new flow content. On the receiving side use
> UnpackContent also configured with format as "FlowFile Stream v3".
just to follow up, i was able to get this to work. the files are
merge with the attributes and i'm able to pull them back apart on the
other end. thanks for the help
Re: flowfile through invokehttp put
Posted by Bryan Bende <bb...@gmail.com>.
Normally this would all be handled for you with site-to-site, but not
sure that can be made to work with your custom proxy. Before
site-to-site there was PostHttp and ListenHttp, and PostHttp has an
options for "Send As Flow File". PostHTTP is marked as deprecated
because generally you should use site-to-site or InvokeHttp, but
InvokeHttp doesn't have the send as flow file concept. So you could
try using PostHttp as long as you realize its deprecated, and maybe
someone can add the send as flow file concept to InvokeHttp, or maybe
a PackageContent processor.
The only other option I can think of is to use MergeContent with the
Merge Format set to "FlowFile Stream v3" and configure the max and min
entries to 1 so it doesn't really actually merge anything, but just
writes out the new flow content. On the receiving side use
UnpackContent also configured with format as "FlowFile Stream v3".
On Thu, May 16, 2019 at 10:18 AM Michael Di Domenico
<md...@gmail.com> wrote:
>
> i'm pretty new to nifi, so i'm not sure this worded correctly or
> possible. i have a flow which ends with an invokehttp process that
> does a PUT to web server.
>
> my naive understanding of this process is that when the flowfile hits
> invokehttp, the flow-attributes can be sent as key:value pairs via the
> http headers and the file-content gets PUT to the remote webserver via
> the http content-body
>
> is there a way to send the flow-attr and the flow-content (ie the
> entire flowfile) via the PUT?
>
> i have nifi on two ends of a connection, but in order to send the data
> from one to another i have to send it through a proxy like device.
> the near end of the device receives data from the PUT and far end does
> a POST to a remote nifi.
>
> this process works fine, but on the far end i only receive the file
> and none of the attributes (as expected). i cannot configure the
> proxy device to pass key:value pairs in the http headers. i'm
> presuming all the flow information would have to be sent via the PUT
> and then decoded by the nifi server on the far end. i'm hoping
> there's a simple connector to do this rather then having me munge all
> the data together into something like an xml or json file and send
> that.
Re: flowfile through invokehttp put
Posted by Michael Di Domenico <md...@gmail.com>.
On Thu, May 16, 2019 at 3:38 PM Andy LoPresto <al...@apache.org> wrote:
>
> Bryan’s comments are definitely helpful around the additional work separation, but revisiting your initial question, did you try using Site To Site through the proxy? Koji Kawamura has done a lot of work on this [1][2] and I believe it’s pretty successful. There are example configurations and diagrams in the Admin Guide [3][4][5].
interesting, but i'm not sure it's applicable in my case. my proxy is
not a traditional web proxy, where a client can query the server and
get a response. my proxy is a byte pass through, but only in one
direction. i can send it some bytes and those bytes get magically
transported from one network to another (ie air-gapped)
Re: flowfile through invokehttp put
Posted by Andy LoPresto <al...@apache.org>.
Bryan’s comments are definitely helpful around the additional work separation, but revisiting your initial question, did you try using Site To Site through the proxy? Koji Kawamura has done a lot of work on this [1][2] and I believe it’s pretty successful. There are example configurations and diagrams in the Admin Guide [3][4][5].
[1] https://cwiki.apache.org/confluence/display/NIFI/Support+HTTP%28S%29+as+a+transport+mechanism+for+Site-to-Site#SupportHTTP(S)asatransportmechanismforSite-to-Site-ToStandaloneNiFi:HTTPusingProxy <https://cwiki.apache.org/confluence/display/NIFI/Support+HTTP(S)+as+a+transport+mechanism+for+Site-to-Site#SupportHTTP(S)asatransportmechanismforSite-to-Site-ToStandaloneNiFi:HTTPusingProxy>
[2] https://issues.apache.org/jira/browse/NIFI-1857 <https://issues.apache.org/jira/browse/NIFI-1857>
[3] https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#proxy_configuration <https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#proxy_configuration>
[4] https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#site_to_site_reverse_proxy_properties <https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#site_to_site_reverse_proxy_properties>
[5] https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#site-to-site-and-reverse-proxy-examples <https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#site-to-site-and-reverse-proxy-examples>
Andy LoPresto
alopresto@apache.org
alopresto.apache@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> On May 16, 2019, at 8:31 AM, Bryan Bende <bb...@gmail.com> wrote:
>
> I think it might be simpler to implement it as a separate
> PackageContent processor that you would use right before InvokeHttp.
> Only because InvokeHttp is already quite complex, and really the
> packaging is a separate function from the http interaction.
>
> PackageContent would be the reverse of UnpackContent.
>
> On Thu, May 16, 2019 at 11:26 AM Michael Di Domenico
> <md...@gmail.com> wrote:
>>
>> On Thu, May 16, 2019 at 11:20 AM Bryan Bende <bb...@gmail.com> wrote:
>>>
>>> Well MergeContent in general is meant to take many flow files and
>>> merge them together, so typically if you were using the flow file
>>> format, the idea would be to create a single flow file where the
>>> content contained (flow1 attrs, content)(flow 2 attrs, content) etc,
>>> but what I was suggesting was to try to configure the processor in a
>>> way that it never actually merges multiple flow files and just acts on
>>> 1 flow file. Essentially trying to use the packaging functionality of
>>> MergeContent without the merging, since there is no corresponding
>>> PackageContent processor to go with the UnpackContent processor.
>>
>> Oh, i see. I'll give it a try and see.
>>
>> Do you know how hard it would be to get the "Send flow" added into invokehttp?
Re: flowfile through invokehttp put
Posted by Bryan Bende <bb...@gmail.com>.
I think it might be simpler to implement it as a separate
PackageContent processor that you would use right before InvokeHttp.
Only because InvokeHttp is already quite complex, and really the
packaging is a separate function from the http interaction.
PackageContent would be the reverse of UnpackContent.
On Thu, May 16, 2019 at 11:26 AM Michael Di Domenico
<md...@gmail.com> wrote:
>
> On Thu, May 16, 2019 at 11:20 AM Bryan Bende <bb...@gmail.com> wrote:
> >
> > Well MergeContent in general is meant to take many flow files and
> > merge them together, so typically if you were using the flow file
> > format, the idea would be to create a single flow file where the
> > content contained (flow1 attrs, content)(flow 2 attrs, content) etc,
> > but what I was suggesting was to try to configure the processor in a
> > way that it never actually merges multiple flow files and just acts on
> > 1 flow file. Essentially trying to use the packaging functionality of
> > MergeContent without the merging, since there is no corresponding
> > PackageContent processor to go with the UnpackContent processor.
>
> Oh, i see. I'll give it a try and see.
>
> Do you know how hard it would be to get the "Send flow" added into invokehttp?
Re: flowfile through invokehttp put
Posted by Michael Di Domenico <md...@gmail.com>.
On Thu, May 16, 2019 at 11:20 AM Bryan Bende <bb...@gmail.com> wrote:
>
> Well MergeContent in general is meant to take many flow files and
> merge them together, so typically if you were using the flow file
> format, the idea would be to create a single flow file where the
> content contained (flow1 attrs, content)(flow 2 attrs, content) etc,
> but what I was suggesting was to try to configure the processor in a
> way that it never actually merges multiple flow files and just acts on
> 1 flow file. Essentially trying to use the packaging functionality of
> MergeContent without the merging, since there is no corresponding
> PackageContent processor to go with the UnpackContent processor.
Oh, i see. I'll give it a try and see.
Do you know how hard it would be to get the "Send flow" added into invokehttp?
Re: flowfile through invokehttp put
Posted by Bryan Bende <bb...@gmail.com>.
Well MergeContent in general is meant to take many flow files and
merge them together, so typically if you were using the flow file
format, the idea would be to create a single flow file where the
content contained (flow1 attrs, content)(flow 2 attrs, content) etc,
but what I was suggesting was to try to configure the processor in a
way that it never actually merges multiple flow files and just acts on
1 flow file. Essentially trying to use the packaging functionality of
MergeContent without the merging, since there is no corresponding
PackageContent processor to go with the UnpackContent processor.
On Thu, May 16, 2019 at 11:16 AM lcarmona@openpartner.cl
<lc...@openpartner.cl> wrote:
>
>
>
> Enviado desde mi HUAWEI P20 Pro
> Hi Michael,
>
> You might put all the relevant attributes down mixed with the content in a special tag (Jolt)
> It is a kind of tricky because if there is Json structured data you will have to flatten it first and then remove the special characters through ReplaceText, but you get the content with the desired attributes included.
>
> Hope this tip helps.
>
> Regards,
>
> LC
>
> -------- Mensaje original --------
> Asunto: flowfile through invokehttp put
> De: Michael Di Domenico
> Para: users@nifi.apache.org
> CC:
>
> i'm pretty new to nifi, so i'm not sure this worded correctly or
> possible. i have a flow which ends with an invokehttp process that
> does a PUT to web server.
>
> my naive understanding of this process is that when the flowfile hits
> invokehttp, the flow-attributes can be sent as key:value pairs via the
> http headers and the file-content gets PUT to the remote webserver via
> the http content-body
>
> is there a way to send the flow-attr and the flow-content (ie the
> entire flowfile) via the PUT?
>
> i have nifi on two ends of a connection, but in order to send the data
> from one to another i have to send it through a proxy like device.
> the near end of the device receives data from the PUT and far end does
> a POST to a remote nifi.
>
> this process works fine, but on the far end i only receive the file
> and none of the attributes (as expected). i cannot configure the
> proxy device to pass key:value pairs in the http headers. i'm
> presuming all the flow information would have to be sent via the PUT
> and then decoded by the nifi server on the far end. i'm hoping
> there's a simple connector to do this rather then having me munge all
> the data together into something like an xml or json file and send
> that.
Re: flowfile through invokehttp put
Posted by "lcarmona@openpartner.cl" <lc...@openpartner.cl>.
Enviado desde mi HUAWEI P20 Pro
Hi Michael,
You might put all the relevant attributes down mixed with the content in a
special tag (Jolt)
It is a kind of tricky because if there is Json structured data you will have
to flatten it first and then remove the special characters through
ReplaceText, but you get the content with the desired attributes included.
Hope this tip helps.
Regards,
LC
\-------- Mensaje original --------
Asunto: flowfile through invokehttp put
De: Michael Di Domenico
Para: users@nifi.apache.org
CC:
> > i'm pretty new to nifi, so i'm not sure this worded correctly or
> possible. i have a flow which ends with an invokehttp process that
> does a PUT to web server.
>
> my naive understanding of this process is that when the flowfile hits
> invokehttp, the flow-attributes can be sent as key:value pairs via the
> http headers and the file-content gets PUT to the remote webserver via
> the http content-body
>
> is there a way to send the flow-attr and the flow-content (ie the
> entire flowfile) via the PUT?
>
> i have nifi on two ends of a connection, but in order to send the data
> from one to another i have to send it through a proxy like device.
> the near end of the device receives data from the PUT and far end does
> a POST to a remote nifi.
>
> this process works fine, but on the far end i only receive the file
> and none of the attributes (as expected). i cannot configure the
> proxy device to pass key:value pairs in the http headers. i'm
> presuming all the flow information would have to be sent via the PUT
> and then decoded by the nifi server on the far end. i'm hoping
> there's a simple connector to do this rather then having me munge all
> the data together into something like an xml or json file and send
> that.
>