You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by mohammed shambakey <sh...@gmail.com> on 2017/02/27 14:15:41 UTC
send http request to nifi input port
Hi
Is it possible to send http request (including form data and uploaded
files) to NIFI input port instead of "HandleHTTPRequest" processor?
Regards
--
Mohammed
Re: send http request to nifi input port
Posted by mohammed shambakey <sh...@gmail.com>.
Thanks Andy.
I think I should go for Jetty servlet as suggested in Koji's JIRA request "
https://issues.apache.org/jira/browse/NIFI-3469"
Regards
On Tue, Feb 28, 2017 at 12:09 PM, Andy LoPresto <al...@apache.org>
wrote:
> Mohammed,
>
> NiFi Input Ports are definitely not the answer you are looking for here.
> As stated, they are specifically for NiFi to NiFi communication using Site
> to Site protocol, either over raw sockets or HTTPS. They do not accept
> arbitrary HTTP requests from other entities.
>
> Andy LoPresto
> alopresto@apache.org
> *alopresto.apache@gmail.com <al...@gmail.com>*
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>
> On Feb 28, 2017, at 12:04 PM, mohammed shambakey <sh...@gmail.com>
> wrote:
>
> Thanks Koji
>
> On Tue, Feb 28, 2017 at 8:44 AM, Koji Kawamura <ij...@gmail.com>
> wrote:
>
>> Mohammed,
>>
>> NiFi REST API is for managing NiFi itself and data flow running on it,
>> as Andy described earlier, it's not for data transfer. It does have
>> endpoints to receive POST requests but those are for adding Processors
>> or Templates ... etc. Please refer NiFi REST API docs for details.
>> https://nifi.apache.org/docs/nifi-docs/rest-api/index.html
>>
>> Thanks,
>> Koji
>>
>> On Tue, Feb 28, 2017 at 9:25 PM, mohammed shambakey
>> <sh...@gmail.com> wrote:
>> > Thank you all for your help
>> >
>> > I think Koji's thread at JIRA will be the best solution when it is
>> > implemented
>> >
>> > Just a quick question, Andy mentioned REST APIs for NIFI. I didn't find
>> much
>> > examples about using NIFI REST APIs. Is it possible to POST the WEB UI
>> > (including all inputs from multipart form and uploaded files in a POST
>> > request to a NIFI input port of a remote NIFI instance? because, I
>> think,
>> > this will avoid using "handlehttprequest" processor and allow users to
>> > communicate with the remote NIFI instance both from WEB UI and/or NIFI
>> > instance (e.g., site-to-site).
>> >
>> > Regards
>> >
>> > On Mon, Feb 27, 2017 at 6:54 PM, Koji Kawamura <ij...@gmail.com>
>> > wrote:
>> >>
>> >> If the multipart contains only text data, it might be possible to use
>> >> those processors Andy suggested. However, if we support multipart
>> >> natively, it can be done much easier and safer. I created a JIRA for
>> >> this improvement before, but haven't been able to find a time to
>> >> implement it myself.
>> >> If anyone interested in tackling this enhancement, it'd be greatly
>> >> appreciated.
>> >> https://issues.apache.org/jira/browse/NIFI-3469
>> >>
>> >> Thanks,
>> >> Koji
>> >>
>> >> On Tue, Feb 28, 2017 at 4:59 AM, Andy LoPresto <al...@apache.org>
>> >> wrote:
>> >> > Absolutely. Use some combination of SplitText, SplitContent,
>> SplitJSON,
>> >> > and
>> >> > ExtractText to split the content into multiple flowfiles on regular
>> >> > expressions, new lines, and/or byte sequences and then
>> RouteOnContent to
>> >> > send the flowfiles which reference a file to a processor to
>> >> > retrieve/parse
>> >> > the file contents.
>> >> >
>> >> > Andy LoPresto
>> >> > alopresto@apache.org
>> >> > alopresto.apache@gmail.com
>> >> > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>> >> >
>> >> > On Feb 27, 2017, at 2:08 PM, mohammed shambakey <
>> shambakey1@gmail.com>
>> >> > wrote:
>> >> >
>> >> > This is an example of the output file from "handlehttprequest":
>> >> >
>> >> > "
>> >> > -----------------------------831805444208556434014280960^M
>> >> > Content-Disposition: form-data; name="userid"^M
>> >> > ^M
>> >> > jpl_user^M
>> >> > -----------------------------831805444208556434014280960^M
>> >> > Content-Disposition: form-data; name="scripts"; filename="a.py"^M
>> >> > Content-Type: text/x-python^M
>> >> > ^M
>> >> > ^M
>> >> > -----------------------------831805444208556434014280960^M
>> >> > Content-Disposition: form-data; name="scripts"; filename="b.py"^M
>> >> > Content-Type: text/x-python^M
>> >> > ^M
>> >> > ^M
>> >> > -----------------------------831805444208556434014280960^M
>> >> > Content-Disposition: form-data; name="scripts";
>> filename="conf.json"^M
>> >> > Content-Type: application/json^M
>> >> > ^M
>> >> > {
>> >> > "userid":"jpl_user_1",
>> >> > "scripts":["a.py","b.py"]
>> >> > }
>> >> > ^M
>> >> > -----------------------------831805444208556434014280960--^M
>> >> > "
>> >> >
>> >> > Can I extract different components of this file without script at
>> server
>> >> > side?
>> >> >
>> >> > Regards
>> >> >
>> >> > On Mon, Feb 27, 2017 at 10:52 AM, Andy LoPresto <
>> alopresto@apache.org>
>> >> > wrote:
>> >> >>
>> >> >> Mohammed,
>> >> >>
>> >> >> So from what I understand you have a separate web application which
>> >> >> accepts a variety of data (a combination of form fields and file
>> >> >> uploads)
>> >> >> and submits that (I’m assuming via POST) to NiFi. I’m not sure what
>> >> >> problem
>> >> >> you had extracting these “components” from the output flowfile from
>> the
>> >> >> HandleHTTPRequest processor. Can you change your web interface to
>> send
>> >> >> separate POST requests to NiFi and logically separate these
>> components
>> >> >> from
>> >> >> the beginning? Can you modify the format of the request (i.e. make
>> it
>> >> >> JSON,
>> >> >> etc.) so it is properly structured for NiFi to parse and operate on
>> >> >> this
>> >> >> data?
>> >> >>
>> >> >> NiFi treats the content of the flowfile the same once it is in the
>> >> >> system
>> >> >> regardless of the source processor, so you should be able to split,
>> >> >> extract,
>> >> >> or manipulate the contents of a single flowfile containing multiple
>> >> >> form
>> >> >> elements and file bodies.
>> >> >>
>> >> >> If you are referencing NiFi’s Input Port components as a destination
>> >> >> for
>> >> >> an arbitrary HTTP request, this will not work. The input ports are
>> >> >> designed
>> >> >> to work with NiFi’s Site to Site protocol (either over raw sockets
>> or
>> >> >> HTTP(S)).
>> >> >>
>> >> >>
>> >> >> Andy LoPresto
>> >> >> alopresto@apache.org
>> >> >> alopresto.apache@gmail.com
>> >> >> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>> >> >>
>> >> >> On Feb 27, 2017, at 11:09 AM, mohammed shambakey <
>> shambakey1@gmail.com>
>> >> >> wrote:
>> >> >>
>> >> >> Thanks Andy
>> >> >>
>> >> >> I have a user web interface that send some form information (e.g.,
>> user
>> >> >> id), as well as upload multiple files, to a remote NIFI instance.
>> When
>> >> >> using
>> >> >> "handlhttprequest" processor, it results in a single file that
>> contains
>> >> >> "form inputs" and bodies of uploaded files.
>> >> >>
>> >> >> I want to extract individual components of this file (i.e., download
>> >> >> all
>> >> >> uploaded files and extract other form information (e.g., userid) in
>> a
>> >> >> JSON
>> >> >> or other text file. I had a similar problem before and I learnt that
>> >> >> "httphandlerequest" has a problem to extract individual compoents
>> from,
>> >> >> so I
>> >> >> have to write my own scripts to extract these components (form
>> inputs
>> >> >> and
>> >> >> uploaded files).
>> >> >>
>> >> >> As NIFI input ports does not face this problem of
>> "handlehttprequest",
>> >> >> I
>> >> >> though if I can send form inputs and upload files directly to "NIFI
>> >> >> input
>> >> >> port" at remote site, then I don't have to write my own scripts to
>> >> >> extract
>> >> >> individual components.
>> >> >>
>> >> >> Regards
>> >> >>
>> >> >> On Mon, Feb 27, 2017 at 10:02 AM, Andy LoPresto <
>> alopresto@apache.org>
>> >> >> wrote:
>> >> >>>
>> >> >>> Mohammed,
>> >> >>>
>> >> >>> I’m not sure I understand your question. NiFi has a REST API which
>> >> >>> supports the UI and allows scripts/other applications to make
>> requests
>> >> >>> of
>> >> >>> the platform. This runs on the input port specified in
>> >> >>> nifi.properties.
>> >> >>>
>> >> >>> If you want to submit form data and uploaded files as data to be
>> >> >>> ingested
>> >> >>> by NiFi, HandleHTTPRequest or (less likely) ListenHTTP are the
>> >> >>> processors
>> >> >>> you would want to use.
>> >> >>>
>> >> >>> If you can clarify your use case, we should be able to offer more
>> >> >>> assistance.
>> >> >>>
>> >> >>> Andy LoPresto
>> >> >>> alopresto@apache.org
>> >> >>> alopresto.apache@gmail.com
>> >> >>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>> >> >>>
>> >> >>> On Feb 27, 2017, at 9:15 AM, mohammed shambakey <
>> shambakey1@gmail.com>
>> >> >>> wrote:
>> >> >>>
>> >> >>> Hi
>> >> >>>
>> >> >>> Is it possible to send http request (including form data and
>> uploaded
>> >> >>> files) to NIFI input port instead of "HandleHTTPRequest" processor?
>> >> >>>
>> >> >>> Regards
>> >> >>>
>> >> >>> --
>> >> >>> Mohammed
>> >> >>>
>> >> >>>
>> >> >>
>> >> >>
>> >> >>
>> >> >> --
>> >> >> Mohammed
>> >> >>
>> >> >>
>> >> >
>> >> >
>> >> >
>> >> > --
>> >> > Mohammed
>> >> >
>> >> >
>> >
>> >
>> >
>> >
>> > --
>> > Mohammed
>>
>
>
>
> --
> Mohammed
>
>
>
--
Mohammed
Re: send http request to nifi input port
Posted by Andy LoPresto <al...@apache.org>.
Mohammed,
NiFi Input Ports are definitely not the answer you are looking for here. As stated, they are specifically for NiFi to NiFi communication using Site to Site protocol, either over raw sockets or HTTPS. They do not accept arbitrary HTTP requests from other entities.
Andy LoPresto
alopresto@apache.org
alopresto.apache@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> On Feb 28, 2017, at 12:04 PM, mohammed shambakey <sh...@gmail.com> wrote:
>
> Thanks Koji
>
> On Tue, Feb 28, 2017 at 8:44 AM, Koji Kawamura <ijokarumawak@gmail.com <ma...@gmail.com>> wrote:
> Mohammed,
>
> NiFi REST API is for managing NiFi itself and data flow running on it,
> as Andy described earlier, it's not for data transfer. It does have
> endpoints to receive POST requests but those are for adding Processors
> or Templates ... etc. Please refer NiFi REST API docs for details.
> https://nifi.apache.org/docs/nifi-docs/rest-api/index.html <https://nifi.apache.org/docs/nifi-docs/rest-api/index.html>
>
> Thanks,
> Koji
>
> On Tue, Feb 28, 2017 at 9:25 PM, mohammed shambakey
> <shambakey1@gmail.com <ma...@gmail.com>> wrote:
> > Thank you all for your help
> >
> > I think Koji's thread at JIRA will be the best solution when it is
> > implemented
> >
> > Just a quick question, Andy mentioned REST APIs for NIFI. I didn't find much
> > examples about using NIFI REST APIs. Is it possible to POST the WEB UI
> > (including all inputs from multipart form and uploaded files in a POST
> > request to a NIFI input port of a remote NIFI instance? because, I think,
> > this will avoid using "handlehttprequest" processor and allow users to
> > communicate with the remote NIFI instance both from WEB UI and/or NIFI
> > instance (e.g., site-to-site).
> >
> > Regards
> >
> > On Mon, Feb 27, 2017 at 6:54 PM, Koji Kawamura <ijokarumawak@gmail.com <ma...@gmail.com>>
> > wrote:
> >>
> >> If the multipart contains only text data, it might be possible to use
> >> those processors Andy suggested. However, if we support multipart
> >> natively, it can be done much easier and safer. I created a JIRA for
> >> this improvement before, but haven't been able to find a time to
> >> implement it myself.
> >> If anyone interested in tackling this enhancement, it'd be greatly
> >> appreciated.
> >> https://issues.apache.org/jira/browse/NIFI-3469 <https://issues.apache.org/jira/browse/NIFI-3469>
> >>
> >> Thanks,
> >> Koji
> >>
> >> On Tue, Feb 28, 2017 at 4:59 AM, Andy LoPresto <alopresto@apache.org <ma...@apache.org>>
> >> wrote:
> >> > Absolutely. Use some combination of SplitText, SplitContent, SplitJSON,
> >> > and
> >> > ExtractText to split the content into multiple flowfiles on regular
> >> > expressions, new lines, and/or byte sequences and then RouteOnContent to
> >> > send the flowfiles which reference a file to a processor to
> >> > retrieve/parse
> >> > the file contents.
> >> >
> >> > Andy LoPresto
> >> > alopresto@apache.org <ma...@apache.org>
> >> > alopresto.apache@gmail.com <ma...@gmail.com>
> >> > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> >> >
> >> > On Feb 27, 2017, at 2:08 PM, mohammed shambakey <shambakey1@gmail.com <ma...@gmail.com>>
> >> > wrote:
> >> >
> >> > This is an example of the output file from "handlehttprequest":
> >> >
> >> > "
> >> > -----------------------------831805444208556434014280960^M
> >> > Content-Disposition: form-data; name="userid"^M
> >> > ^M
> >> > jpl_user^M
> >> > -----------------------------831805444208556434014280960^M
> >> > Content-Disposition: form-data; name="scripts"; filename="a.py"^M
> >> > Content-Type: text/x-python^M
> >> > ^M
> >> > ^M
> >> > -----------------------------831805444208556434014280960^M
> >> > Content-Disposition: form-data; name="scripts"; filename="b.py"^M
> >> > Content-Type: text/x-python^M
> >> > ^M
> >> > ^M
> >> > -----------------------------831805444208556434014280960^M
> >> > Content-Disposition: form-data; name="scripts"; filename="conf.json"^M
> >> > Content-Type: application/json^M
> >> > ^M
> >> > {
> >> > "userid":"jpl_user_1",
> >> > "scripts":["a.py","b.py"]
> >> > }
> >> > ^M
> >> > -----------------------------831805444208556434014280960--^M
> >> > "
> >> >
> >> > Can I extract different components of this file without script at server
> >> > side?
> >> >
> >> > Regards
> >> >
> >> > On Mon, Feb 27, 2017 at 10:52 AM, Andy LoPresto <alopresto@apache.org <ma...@apache.org>>
> >> > wrote:
> >> >>
> >> >> Mohammed,
> >> >>
> >> >> So from what I understand you have a separate web application which
> >> >> accepts a variety of data (a combination of form fields and file
> >> >> uploads)
> >> >> and submits that (I’m assuming via POST) to NiFi. I’m not sure what
> >> >> problem
> >> >> you had extracting these “components” from the output flowfile from the
> >> >> HandleHTTPRequest processor. Can you change your web interface to send
> >> >> separate POST requests to NiFi and logically separate these components
> >> >> from
> >> >> the beginning? Can you modify the format of the request (i.e. make it
> >> >> JSON,
> >> >> etc.) so it is properly structured for NiFi to parse and operate on
> >> >> this
> >> >> data?
> >> >>
> >> >> NiFi treats the content of the flowfile the same once it is in the
> >> >> system
> >> >> regardless of the source processor, so you should be able to split,
> >> >> extract,
> >> >> or manipulate the contents of a single flowfile containing multiple
> >> >> form
> >> >> elements and file bodies.
> >> >>
> >> >> If you are referencing NiFi’s Input Port components as a destination
> >> >> for
> >> >> an arbitrary HTTP request, this will not work. The input ports are
> >> >> designed
> >> >> to work with NiFi’s Site to Site protocol (either over raw sockets or
> >> >> HTTP(S)).
> >> >>
> >> >>
> >> >> Andy LoPresto
> >> >> alopresto@apache.org <ma...@apache.org>
> >> >> alopresto.apache@gmail.com <ma...@gmail.com>
> >> >> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> >> >>
> >> >> On Feb 27, 2017, at 11:09 AM, mohammed shambakey <shambakey1@gmail.com <ma...@gmail.com>>
> >> >> wrote:
> >> >>
> >> >> Thanks Andy
> >> >>
> >> >> I have a user web interface that send some form information (e.g., user
> >> >> id), as well as upload multiple files, to a remote NIFI instance. When
> >> >> using
> >> >> "handlhttprequest" processor, it results in a single file that contains
> >> >> "form inputs" and bodies of uploaded files.
> >> >>
> >> >> I want to extract individual components of this file (i.e., download
> >> >> all
> >> >> uploaded files and extract other form information (e.g., userid) in a
> >> >> JSON
> >> >> or other text file. I had a similar problem before and I learnt that
> >> >> "httphandlerequest" has a problem to extract individual compoents from,
> >> >> so I
> >> >> have to write my own scripts to extract these components (form inputs
> >> >> and
> >> >> uploaded files).
> >> >>
> >> >> As NIFI input ports does not face this problem of "handlehttprequest",
> >> >> I
> >> >> though if I can send form inputs and upload files directly to "NIFI
> >> >> input
> >> >> port" at remote site, then I don't have to write my own scripts to
> >> >> extract
> >> >> individual components.
> >> >>
> >> >> Regards
> >> >>
> >> >> On Mon, Feb 27, 2017 at 10:02 AM, Andy LoPresto <alopresto@apache.org <ma...@apache.org>>
> >> >> wrote:
> >> >>>
> >> >>> Mohammed,
> >> >>>
> >> >>> I’m not sure I understand your question. NiFi has a REST API which
> >> >>> supports the UI and allows scripts/other applications to make requests
> >> >>> of
> >> >>> the platform. This runs on the input port specified in
> >> >>> nifi.properties.
> >> >>>
> >> >>> If you want to submit form data and uploaded files as data to be
> >> >>> ingested
> >> >>> by NiFi, HandleHTTPRequest or (less likely) ListenHTTP are the
> >> >>> processors
> >> >>> you would want to use.
> >> >>>
> >> >>> If you can clarify your use case, we should be able to offer more
> >> >>> assistance.
> >> >>>
> >> >>> Andy LoPresto
> >> >>> alopresto@apache.org <ma...@apache.org>
> >> >>> alopresto.apache@gmail.com <ma...@gmail.com>
> >> >>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> >> >>>
> >> >>> On Feb 27, 2017, at 9:15 AM, mohammed shambakey <shambakey1@gmail.com <ma...@gmail.com>>
> >> >>> wrote:
> >> >>>
> >> >>> Hi
> >> >>>
> >> >>> Is it possible to send http request (including form data and uploaded
> >> >>> files) to NIFI input port instead of "HandleHTTPRequest" processor?
> >> >>>
> >> >>> Regards
> >> >>>
> >> >>> --
> >> >>> Mohammed
> >> >>>
> >> >>>
> >> >>
> >> >>
> >> >>
> >> >> --
> >> >> Mohammed
> >> >>
> >> >>
> >> >
> >> >
> >> >
> >> > --
> >> > Mohammed
> >> >
> >> >
> >
> >
> >
> >
> > --
> > Mohammed
>
>
>
> --
> Mohammed
Re: send http request to nifi input port
Posted by mohammed shambakey <sh...@gmail.com>.
Thanks Koji
On Tue, Feb 28, 2017 at 8:44 AM, Koji Kawamura <ij...@gmail.com>
wrote:
> Mohammed,
>
> NiFi REST API is for managing NiFi itself and data flow running on it,
> as Andy described earlier, it's not for data transfer. It does have
> endpoints to receive POST requests but those are for adding Processors
> or Templates ... etc. Please refer NiFi REST API docs for details.
> https://nifi.apache.org/docs/nifi-docs/rest-api/index.html
>
> Thanks,
> Koji
>
> On Tue, Feb 28, 2017 at 9:25 PM, mohammed shambakey
> <sh...@gmail.com> wrote:
> > Thank you all for your help
> >
> > I think Koji's thread at JIRA will be the best solution when it is
> > implemented
> >
> > Just a quick question, Andy mentioned REST APIs for NIFI. I didn't find
> much
> > examples about using NIFI REST APIs. Is it possible to POST the WEB UI
> > (including all inputs from multipart form and uploaded files in a POST
> > request to a NIFI input port of a remote NIFI instance? because, I think,
> > this will avoid using "handlehttprequest" processor and allow users to
> > communicate with the remote NIFI instance both from WEB UI and/or NIFI
> > instance (e.g., site-to-site).
> >
> > Regards
> >
> > On Mon, Feb 27, 2017 at 6:54 PM, Koji Kawamura <ij...@gmail.com>
> > wrote:
> >>
> >> If the multipart contains only text data, it might be possible to use
> >> those processors Andy suggested. However, if we support multipart
> >> natively, it can be done much easier and safer. I created a JIRA for
> >> this improvement before, but haven't been able to find a time to
> >> implement it myself.
> >> If anyone interested in tackling this enhancement, it'd be greatly
> >> appreciated.
> >> https://issues.apache.org/jira/browse/NIFI-3469
> >>
> >> Thanks,
> >> Koji
> >>
> >> On Tue, Feb 28, 2017 at 4:59 AM, Andy LoPresto <al...@apache.org>
> >> wrote:
> >> > Absolutely. Use some combination of SplitText, SplitContent,
> SplitJSON,
> >> > and
> >> > ExtractText to split the content into multiple flowfiles on regular
> >> > expressions, new lines, and/or byte sequences and then RouteOnContent
> to
> >> > send the flowfiles which reference a file to a processor to
> >> > retrieve/parse
> >> > the file contents.
> >> >
> >> > Andy LoPresto
> >> > alopresto@apache.org
> >> > alopresto.apache@gmail.com
> >> > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> >> >
> >> > On Feb 27, 2017, at 2:08 PM, mohammed shambakey <shambakey1@gmail.com
> >
> >> > wrote:
> >> >
> >> > This is an example of the output file from "handlehttprequest":
> >> >
> >> > "
> >> > -----------------------------831805444208556434014280960^M
> >> > Content-Disposition: form-data; name="userid"^M
> >> > ^M
> >> > jpl_user^M
> >> > -----------------------------831805444208556434014280960^M
> >> > Content-Disposition: form-data; name="scripts"; filename="a.py"^M
> >> > Content-Type: text/x-python^M
> >> > ^M
> >> > ^M
> >> > -----------------------------831805444208556434014280960^M
> >> > Content-Disposition: form-data; name="scripts"; filename="b.py"^M
> >> > Content-Type: text/x-python^M
> >> > ^M
> >> > ^M
> >> > -----------------------------831805444208556434014280960^M
> >> > Content-Disposition: form-data; name="scripts"; filename="conf.json"^M
> >> > Content-Type: application/json^M
> >> > ^M
> >> > {
> >> > "userid":"jpl_user_1",
> >> > "scripts":["a.py","b.py"]
> >> > }
> >> > ^M
> >> > -----------------------------831805444208556434014280960--^M
> >> > "
> >> >
> >> > Can I extract different components of this file without script at
> server
> >> > side?
> >> >
> >> > Regards
> >> >
> >> > On Mon, Feb 27, 2017 at 10:52 AM, Andy LoPresto <alopresto@apache.org
> >
> >> > wrote:
> >> >>
> >> >> Mohammed,
> >> >>
> >> >> So from what I understand you have a separate web application which
> >> >> accepts a variety of data (a combination of form fields and file
> >> >> uploads)
> >> >> and submits that (I’m assuming via POST) to NiFi. I’m not sure what
> >> >> problem
> >> >> you had extracting these “components” from the output flowfile from
> the
> >> >> HandleHTTPRequest processor. Can you change your web interface to
> send
> >> >> separate POST requests to NiFi and logically separate these
> components
> >> >> from
> >> >> the beginning? Can you modify the format of the request (i.e. make it
> >> >> JSON,
> >> >> etc.) so it is properly structured for NiFi to parse and operate on
> >> >> this
> >> >> data?
> >> >>
> >> >> NiFi treats the content of the flowfile the same once it is in the
> >> >> system
> >> >> regardless of the source processor, so you should be able to split,
> >> >> extract,
> >> >> or manipulate the contents of a single flowfile containing multiple
> >> >> form
> >> >> elements and file bodies.
> >> >>
> >> >> If you are referencing NiFi’s Input Port components as a destination
> >> >> for
> >> >> an arbitrary HTTP request, this will not work. The input ports are
> >> >> designed
> >> >> to work with NiFi’s Site to Site protocol (either over raw sockets or
> >> >> HTTP(S)).
> >> >>
> >> >>
> >> >> Andy LoPresto
> >> >> alopresto@apache.org
> >> >> alopresto.apache@gmail.com
> >> >> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> >> >>
> >> >> On Feb 27, 2017, at 11:09 AM, mohammed shambakey <
> shambakey1@gmail.com>
> >> >> wrote:
> >> >>
> >> >> Thanks Andy
> >> >>
> >> >> I have a user web interface that send some form information (e.g.,
> user
> >> >> id), as well as upload multiple files, to a remote NIFI instance.
> When
> >> >> using
> >> >> "handlhttprequest" processor, it results in a single file that
> contains
> >> >> "form inputs" and bodies of uploaded files.
> >> >>
> >> >> I want to extract individual components of this file (i.e., download
> >> >> all
> >> >> uploaded files and extract other form information (e.g., userid) in a
> >> >> JSON
> >> >> or other text file. I had a similar problem before and I learnt that
> >> >> "httphandlerequest" has a problem to extract individual compoents
> from,
> >> >> so I
> >> >> have to write my own scripts to extract these components (form inputs
> >> >> and
> >> >> uploaded files).
> >> >>
> >> >> As NIFI input ports does not face this problem of
> "handlehttprequest",
> >> >> I
> >> >> though if I can send form inputs and upload files directly to "NIFI
> >> >> input
> >> >> port" at remote site, then I don't have to write my own scripts to
> >> >> extract
> >> >> individual components.
> >> >>
> >> >> Regards
> >> >>
> >> >> On Mon, Feb 27, 2017 at 10:02 AM, Andy LoPresto <
> alopresto@apache.org>
> >> >> wrote:
> >> >>>
> >> >>> Mohammed,
> >> >>>
> >> >>> I’m not sure I understand your question. NiFi has a REST API which
> >> >>> supports the UI and allows scripts/other applications to make
> requests
> >> >>> of
> >> >>> the platform. This runs on the input port specified in
> >> >>> nifi.properties.
> >> >>>
> >> >>> If you want to submit form data and uploaded files as data to be
> >> >>> ingested
> >> >>> by NiFi, HandleHTTPRequest or (less likely) ListenHTTP are the
> >> >>> processors
> >> >>> you would want to use.
> >> >>>
> >> >>> If you can clarify your use case, we should be able to offer more
> >> >>> assistance.
> >> >>>
> >> >>> Andy LoPresto
> >> >>> alopresto@apache.org
> >> >>> alopresto.apache@gmail.com
> >> >>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> >> >>>
> >> >>> On Feb 27, 2017, at 9:15 AM, mohammed shambakey <
> shambakey1@gmail.com>
> >> >>> wrote:
> >> >>>
> >> >>> Hi
> >> >>>
> >> >>> Is it possible to send http request (including form data and
> uploaded
> >> >>> files) to NIFI input port instead of "HandleHTTPRequest" processor?
> >> >>>
> >> >>> Regards
> >> >>>
> >> >>> --
> >> >>> Mohammed
> >> >>>
> >> >>>
> >> >>
> >> >>
> >> >>
> >> >> --
> >> >> Mohammed
> >> >>
> >> >>
> >> >
> >> >
> >> >
> >> > --
> >> > Mohammed
> >> >
> >> >
> >
> >
> >
> >
> > --
> > Mohammed
>
--
Mohammed
Re: send http request to nifi input port
Posted by Koji Kawamura <ij...@gmail.com>.
Mohammed,
NiFi REST API is for managing NiFi itself and data flow running on it,
as Andy described earlier, it's not for data transfer. It does have
endpoints to receive POST requests but those are for adding Processors
or Templates ... etc. Please refer NiFi REST API docs for details.
https://nifi.apache.org/docs/nifi-docs/rest-api/index.html
Thanks,
Koji
On Tue, Feb 28, 2017 at 9:25 PM, mohammed shambakey
<sh...@gmail.com> wrote:
> Thank you all for your help
>
> I think Koji's thread at JIRA will be the best solution when it is
> implemented
>
> Just a quick question, Andy mentioned REST APIs for NIFI. I didn't find much
> examples about using NIFI REST APIs. Is it possible to POST the WEB UI
> (including all inputs from multipart form and uploaded files in a POST
> request to a NIFI input port of a remote NIFI instance? because, I think,
> this will avoid using "handlehttprequest" processor and allow users to
> communicate with the remote NIFI instance both from WEB UI and/or NIFI
> instance (e.g., site-to-site).
>
> Regards
>
> On Mon, Feb 27, 2017 at 6:54 PM, Koji Kawamura <ij...@gmail.com>
> wrote:
>>
>> If the multipart contains only text data, it might be possible to use
>> those processors Andy suggested. However, if we support multipart
>> natively, it can be done much easier and safer. I created a JIRA for
>> this improvement before, but haven't been able to find a time to
>> implement it myself.
>> If anyone interested in tackling this enhancement, it'd be greatly
>> appreciated.
>> https://issues.apache.org/jira/browse/NIFI-3469
>>
>> Thanks,
>> Koji
>>
>> On Tue, Feb 28, 2017 at 4:59 AM, Andy LoPresto <al...@apache.org>
>> wrote:
>> > Absolutely. Use some combination of SplitText, SplitContent, SplitJSON,
>> > and
>> > ExtractText to split the content into multiple flowfiles on regular
>> > expressions, new lines, and/or byte sequences and then RouteOnContent to
>> > send the flowfiles which reference a file to a processor to
>> > retrieve/parse
>> > the file contents.
>> >
>> > Andy LoPresto
>> > alopresto@apache.org
>> > alopresto.apache@gmail.com
>> > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>> >
>> > On Feb 27, 2017, at 2:08 PM, mohammed shambakey <sh...@gmail.com>
>> > wrote:
>> >
>> > This is an example of the output file from "handlehttprequest":
>> >
>> > "
>> > -----------------------------831805444208556434014280960^M
>> > Content-Disposition: form-data; name="userid"^M
>> > ^M
>> > jpl_user^M
>> > -----------------------------831805444208556434014280960^M
>> > Content-Disposition: form-data; name="scripts"; filename="a.py"^M
>> > Content-Type: text/x-python^M
>> > ^M
>> > ^M
>> > -----------------------------831805444208556434014280960^M
>> > Content-Disposition: form-data; name="scripts"; filename="b.py"^M
>> > Content-Type: text/x-python^M
>> > ^M
>> > ^M
>> > -----------------------------831805444208556434014280960^M
>> > Content-Disposition: form-data; name="scripts"; filename="conf.json"^M
>> > Content-Type: application/json^M
>> > ^M
>> > {
>> > "userid":"jpl_user_1",
>> > "scripts":["a.py","b.py"]
>> > }
>> > ^M
>> > -----------------------------831805444208556434014280960--^M
>> > "
>> >
>> > Can I extract different components of this file without script at server
>> > side?
>> >
>> > Regards
>> >
>> > On Mon, Feb 27, 2017 at 10:52 AM, Andy LoPresto <al...@apache.org>
>> > wrote:
>> >>
>> >> Mohammed,
>> >>
>> >> So from what I understand you have a separate web application which
>> >> accepts a variety of data (a combination of form fields and file
>> >> uploads)
>> >> and submits that (I’m assuming via POST) to NiFi. I’m not sure what
>> >> problem
>> >> you had extracting these “components” from the output flowfile from the
>> >> HandleHTTPRequest processor. Can you change your web interface to send
>> >> separate POST requests to NiFi and logically separate these components
>> >> from
>> >> the beginning? Can you modify the format of the request (i.e. make it
>> >> JSON,
>> >> etc.) so it is properly structured for NiFi to parse and operate on
>> >> this
>> >> data?
>> >>
>> >> NiFi treats the content of the flowfile the same once it is in the
>> >> system
>> >> regardless of the source processor, so you should be able to split,
>> >> extract,
>> >> or manipulate the contents of a single flowfile containing multiple
>> >> form
>> >> elements and file bodies.
>> >>
>> >> If you are referencing NiFi’s Input Port components as a destination
>> >> for
>> >> an arbitrary HTTP request, this will not work. The input ports are
>> >> designed
>> >> to work with NiFi’s Site to Site protocol (either over raw sockets or
>> >> HTTP(S)).
>> >>
>> >>
>> >> Andy LoPresto
>> >> alopresto@apache.org
>> >> alopresto.apache@gmail.com
>> >> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>> >>
>> >> On Feb 27, 2017, at 11:09 AM, mohammed shambakey <sh...@gmail.com>
>> >> wrote:
>> >>
>> >> Thanks Andy
>> >>
>> >> I have a user web interface that send some form information (e.g., user
>> >> id), as well as upload multiple files, to a remote NIFI instance. When
>> >> using
>> >> "handlhttprequest" processor, it results in a single file that contains
>> >> "form inputs" and bodies of uploaded files.
>> >>
>> >> I want to extract individual components of this file (i.e., download
>> >> all
>> >> uploaded files and extract other form information (e.g., userid) in a
>> >> JSON
>> >> or other text file. I had a similar problem before and I learnt that
>> >> "httphandlerequest" has a problem to extract individual compoents from,
>> >> so I
>> >> have to write my own scripts to extract these components (form inputs
>> >> and
>> >> uploaded files).
>> >>
>> >> As NIFI input ports does not face this problem of "handlehttprequest",
>> >> I
>> >> though if I can send form inputs and upload files directly to "NIFI
>> >> input
>> >> port" at remote site, then I don't have to write my own scripts to
>> >> extract
>> >> individual components.
>> >>
>> >> Regards
>> >>
>> >> On Mon, Feb 27, 2017 at 10:02 AM, Andy LoPresto <al...@apache.org>
>> >> wrote:
>> >>>
>> >>> Mohammed,
>> >>>
>> >>> I’m not sure I understand your question. NiFi has a REST API which
>> >>> supports the UI and allows scripts/other applications to make requests
>> >>> of
>> >>> the platform. This runs on the input port specified in
>> >>> nifi.properties.
>> >>>
>> >>> If you want to submit form data and uploaded files as data to be
>> >>> ingested
>> >>> by NiFi, HandleHTTPRequest or (less likely) ListenHTTP are the
>> >>> processors
>> >>> you would want to use.
>> >>>
>> >>> If you can clarify your use case, we should be able to offer more
>> >>> assistance.
>> >>>
>> >>> Andy LoPresto
>> >>> alopresto@apache.org
>> >>> alopresto.apache@gmail.com
>> >>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>> >>>
>> >>> On Feb 27, 2017, at 9:15 AM, mohammed shambakey <sh...@gmail.com>
>> >>> wrote:
>> >>>
>> >>> Hi
>> >>>
>> >>> Is it possible to send http request (including form data and uploaded
>> >>> files) to NIFI input port instead of "HandleHTTPRequest" processor?
>> >>>
>> >>> Regards
>> >>>
>> >>> --
>> >>> Mohammed
>> >>>
>> >>>
>> >>
>> >>
>> >>
>> >> --
>> >> Mohammed
>> >>
>> >>
>> >
>> >
>> >
>> > --
>> > Mohammed
>> >
>> >
>
>
>
>
> --
> Mohammed
Re: send http request to nifi input port
Posted by mohammed shambakey <sh...@gmail.com>.
Thank you all for your help
I think Koji's thread at JIRA will be the best solution when it is
implemented
Just a quick question, Andy mentioned REST APIs for NIFI. I didn't find
much examples about using NIFI REST APIs. Is it possible to POST the WEB UI
(including all inputs from multipart form and uploaded files in a POST
request to a NIFI input port of a remote NIFI instance? because, I think,
this will avoid using "handlehttprequest" processor and allow users to
communicate with the remote NIFI instance both from WEB UI and/or NIFI
instance (e.g., site-to-site).
Regards
On Mon, Feb 27, 2017 at 6:54 PM, Koji Kawamura <ij...@gmail.com>
wrote:
> If the multipart contains only text data, it might be possible to use
> those processors Andy suggested. However, if we support multipart
> natively, it can be done much easier and safer. I created a JIRA for
> this improvement before, but haven't been able to find a time to
> implement it myself.
> If anyone interested in tackling this enhancement, it'd be greatly
> appreciated.
> https://issues.apache.org/jira/browse/NIFI-3469
>
> Thanks,
> Koji
>
> On Tue, Feb 28, 2017 at 4:59 AM, Andy LoPresto <al...@apache.org>
> wrote:
> > Absolutely. Use some combination of SplitText, SplitContent, SplitJSON,
> and
> > ExtractText to split the content into multiple flowfiles on regular
> > expressions, new lines, and/or byte sequences and then RouteOnContent to
> > send the flowfiles which reference a file to a processor to
> retrieve/parse
> > the file contents.
> >
> > Andy LoPresto
> > alopresto@apache.org
> > alopresto.apache@gmail.com
> > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> >
> > On Feb 27, 2017, at 2:08 PM, mohammed shambakey <sh...@gmail.com>
> > wrote:
> >
> > This is an example of the output file from "handlehttprequest":
> >
> > "
> > -----------------------------831805444208556434014280960^M
> > Content-Disposition: form-data; name="userid"^M
> > ^M
> > jpl_user^M
> > -----------------------------831805444208556434014280960^M
> > Content-Disposition: form-data; name="scripts"; filename="a.py"^M
> > Content-Type: text/x-python^M
> > ^M
> > ^M
> > -----------------------------831805444208556434014280960^M
> > Content-Disposition: form-data; name="scripts"; filename="b.py"^M
> > Content-Type: text/x-python^M
> > ^M
> > ^M
> > -----------------------------831805444208556434014280960^M
> > Content-Disposition: form-data; name="scripts"; filename="conf.json"^M
> > Content-Type: application/json^M
> > ^M
> > {
> > "userid":"jpl_user_1",
> > "scripts":["a.py","b.py"]
> > }
> > ^M
> > -----------------------------831805444208556434014280960--^M
> > "
> >
> > Can I extract different components of this file without script at server
> > side?
> >
> > Regards
> >
> > On Mon, Feb 27, 2017 at 10:52 AM, Andy LoPresto <al...@apache.org>
> > wrote:
> >>
> >> Mohammed,
> >>
> >> So from what I understand you have a separate web application which
> >> accepts a variety of data (a combination of form fields and file
> uploads)
> >> and submits that (I’m assuming via POST) to NiFi. I’m not sure what
> problem
> >> you had extracting these “components” from the output flowfile from the
> >> HandleHTTPRequest processor. Can you change your web interface to send
> >> separate POST requests to NiFi and logically separate these components
> from
> >> the beginning? Can you modify the format of the request (i.e. make it
> JSON,
> >> etc.) so it is properly structured for NiFi to parse and operate on this
> >> data?
> >>
> >> NiFi treats the content of the flowfile the same once it is in the
> system
> >> regardless of the source processor, so you should be able to split,
> extract,
> >> or manipulate the contents of a single flowfile containing multiple form
> >> elements and file bodies.
> >>
> >> If you are referencing NiFi’s Input Port components as a destination for
> >> an arbitrary HTTP request, this will not work. The input ports are
> designed
> >> to work with NiFi’s Site to Site protocol (either over raw sockets or
> >> HTTP(S)).
> >>
> >>
> >> Andy LoPresto
> >> alopresto@apache.org
> >> alopresto.apache@gmail.com
> >> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> >>
> >> On Feb 27, 2017, at 11:09 AM, mohammed shambakey <sh...@gmail.com>
> >> wrote:
> >>
> >> Thanks Andy
> >>
> >> I have a user web interface that send some form information (e.g., user
> >> id), as well as upload multiple files, to a remote NIFI instance. When
> using
> >> "handlhttprequest" processor, it results in a single file that contains
> >> "form inputs" and bodies of uploaded files.
> >>
> >> I want to extract individual components of this file (i.e., download all
> >> uploaded files and extract other form information (e.g., userid) in a
> JSON
> >> or other text file. I had a similar problem before and I learnt that
> >> "httphandlerequest" has a problem to extract individual compoents from,
> so I
> >> have to write my own scripts to extract these components (form inputs
> and
> >> uploaded files).
> >>
> >> As NIFI input ports does not face this problem of "handlehttprequest", I
> >> though if I can send form inputs and upload files directly to "NIFI
> input
> >> port" at remote site, then I don't have to write my own scripts to
> extract
> >> individual components.
> >>
> >> Regards
> >>
> >> On Mon, Feb 27, 2017 at 10:02 AM, Andy LoPresto <al...@apache.org>
> >> wrote:
> >>>
> >>> Mohammed,
> >>>
> >>> I’m not sure I understand your question. NiFi has a REST API which
> >>> supports the UI and allows scripts/other applications to make requests
> of
> >>> the platform. This runs on the input port specified in nifi.properties.
> >>>
> >>> If you want to submit form data and uploaded files as data to be
> ingested
> >>> by NiFi, HandleHTTPRequest or (less likely) ListenHTTP are the
> processors
> >>> you would want to use.
> >>>
> >>> If you can clarify your use case, we should be able to offer more
> >>> assistance.
> >>>
> >>> Andy LoPresto
> >>> alopresto@apache.org
> >>> alopresto.apache@gmail.com
> >>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> >>>
> >>> On Feb 27, 2017, at 9:15 AM, mohammed shambakey <sh...@gmail.com>
> >>> wrote:
> >>>
> >>> Hi
> >>>
> >>> Is it possible to send http request (including form data and uploaded
> >>> files) to NIFI input port instead of "HandleHTTPRequest" processor?
> >>>
> >>> Regards
> >>>
> >>> --
> >>> Mohammed
> >>>
> >>>
> >>
> >>
> >>
> >> --
> >> Mohammed
> >>
> >>
> >
> >
> >
> > --
> > Mohammed
> >
> >
>
--
Mohammed
Re: send http request to nifi input port
Posted by Koji Kawamura <ij...@gmail.com>.
If the multipart contains only text data, it might be possible to use
those processors Andy suggested. However, if we support multipart
natively, it can be done much easier and safer. I created a JIRA for
this improvement before, but haven't been able to find a time to
implement it myself.
If anyone interested in tackling this enhancement, it'd be greatly appreciated.
https://issues.apache.org/jira/browse/NIFI-3469
Thanks,
Koji
On Tue, Feb 28, 2017 at 4:59 AM, Andy LoPresto <al...@apache.org> wrote:
> Absolutely. Use some combination of SplitText, SplitContent, SplitJSON, and
> ExtractText to split the content into multiple flowfiles on regular
> expressions, new lines, and/or byte sequences and then RouteOnContent to
> send the flowfiles which reference a file to a processor to retrieve/parse
> the file contents.
>
> Andy LoPresto
> alopresto@apache.org
> alopresto.apache@gmail.com
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>
> On Feb 27, 2017, at 2:08 PM, mohammed shambakey <sh...@gmail.com>
> wrote:
>
> This is an example of the output file from "handlehttprequest":
>
> "
> -----------------------------831805444208556434014280960^M
> Content-Disposition: form-data; name="userid"^M
> ^M
> jpl_user^M
> -----------------------------831805444208556434014280960^M
> Content-Disposition: form-data; name="scripts"; filename="a.py"^M
> Content-Type: text/x-python^M
> ^M
> ^M
> -----------------------------831805444208556434014280960^M
> Content-Disposition: form-data; name="scripts"; filename="b.py"^M
> Content-Type: text/x-python^M
> ^M
> ^M
> -----------------------------831805444208556434014280960^M
> Content-Disposition: form-data; name="scripts"; filename="conf.json"^M
> Content-Type: application/json^M
> ^M
> {
> "userid":"jpl_user_1",
> "scripts":["a.py","b.py"]
> }
> ^M
> -----------------------------831805444208556434014280960--^M
> "
>
> Can I extract different components of this file without script at server
> side?
>
> Regards
>
> On Mon, Feb 27, 2017 at 10:52 AM, Andy LoPresto <al...@apache.org>
> wrote:
>>
>> Mohammed,
>>
>> So from what I understand you have a separate web application which
>> accepts a variety of data (a combination of form fields and file uploads)
>> and submits that (I’m assuming via POST) to NiFi. I’m not sure what problem
>> you had extracting these “components” from the output flowfile from the
>> HandleHTTPRequest processor. Can you change your web interface to send
>> separate POST requests to NiFi and logically separate these components from
>> the beginning? Can you modify the format of the request (i.e. make it JSON,
>> etc.) so it is properly structured for NiFi to parse and operate on this
>> data?
>>
>> NiFi treats the content of the flowfile the same once it is in the system
>> regardless of the source processor, so you should be able to split, extract,
>> or manipulate the contents of a single flowfile containing multiple form
>> elements and file bodies.
>>
>> If you are referencing NiFi’s Input Port components as a destination for
>> an arbitrary HTTP request, this will not work. The input ports are designed
>> to work with NiFi’s Site to Site protocol (either over raw sockets or
>> HTTP(S)).
>>
>>
>> Andy LoPresto
>> alopresto@apache.org
>> alopresto.apache@gmail.com
>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>>
>> On Feb 27, 2017, at 11:09 AM, mohammed shambakey <sh...@gmail.com>
>> wrote:
>>
>> Thanks Andy
>>
>> I have a user web interface that send some form information (e.g., user
>> id), as well as upload multiple files, to a remote NIFI instance. When using
>> "handlhttprequest" processor, it results in a single file that contains
>> "form inputs" and bodies of uploaded files.
>>
>> I want to extract individual components of this file (i.e., download all
>> uploaded files and extract other form information (e.g., userid) in a JSON
>> or other text file. I had a similar problem before and I learnt that
>> "httphandlerequest" has a problem to extract individual compoents from, so I
>> have to write my own scripts to extract these components (form inputs and
>> uploaded files).
>>
>> As NIFI input ports does not face this problem of "handlehttprequest", I
>> though if I can send form inputs and upload files directly to "NIFI input
>> port" at remote site, then I don't have to write my own scripts to extract
>> individual components.
>>
>> Regards
>>
>> On Mon, Feb 27, 2017 at 10:02 AM, Andy LoPresto <al...@apache.org>
>> wrote:
>>>
>>> Mohammed,
>>>
>>> I’m not sure I understand your question. NiFi has a REST API which
>>> supports the UI and allows scripts/other applications to make requests of
>>> the platform. This runs on the input port specified in nifi.properties.
>>>
>>> If you want to submit form data and uploaded files as data to be ingested
>>> by NiFi, HandleHTTPRequest or (less likely) ListenHTTP are the processors
>>> you would want to use.
>>>
>>> If you can clarify your use case, we should be able to offer more
>>> assistance.
>>>
>>> Andy LoPresto
>>> alopresto@apache.org
>>> alopresto.apache@gmail.com
>>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>>>
>>> On Feb 27, 2017, at 9:15 AM, mohammed shambakey <sh...@gmail.com>
>>> wrote:
>>>
>>> Hi
>>>
>>> Is it possible to send http request (including form data and uploaded
>>> files) to NIFI input port instead of "HandleHTTPRequest" processor?
>>>
>>> Regards
>>>
>>> --
>>> Mohammed
>>>
>>>
>>
>>
>>
>> --
>> Mohammed
>>
>>
>
>
>
> --
> Mohammed
>
>
Re: send http request to nifi input port
Posted by Andy LoPresto <al...@apache.org>.
Absolutely. Use some combination of SplitText, SplitContent, SplitJSON, and ExtractText to split the content into multiple flowfiles on regular expressions, new lines, and/or byte sequences and then RouteOnContent to send the flowfiles which reference a file to a processor to retrieve/parse the file contents.
Andy LoPresto
alopresto@apache.org
alopresto.apache@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> On Feb 27, 2017, at 2:08 PM, mohammed shambakey <sh...@gmail.com> wrote:
>
> This is an example of the output file from "handlehttprequest":
>
> "
> -----------------------------831805444208556434014280960^M
> Content-Disposition: form-data; name="userid"^M
> ^M
> jpl_user^M
> -----------------------------831805444208556434014280960^M
> Content-Disposition: form-data; name="scripts"; filename="a.py"^M
> Content-Type: text/x-python^M
> ^M
> ^M
> -----------------------------831805444208556434014280960^M
> Content-Disposition: form-data; name="scripts"; filename="b.py"^M
> Content-Type: text/x-python^M
> ^M
> ^M
> -----------------------------831805444208556434014280960^M
> Content-Disposition: form-data; name="scripts"; filename="conf.json"^M
> Content-Type: application/json^M
> ^M
> {
> "userid":"jpl_user_1",
> "scripts":["a.py","b.py"]
> }
> ^M
> -----------------------------831805444208556434014280960--^M
> "
>
> Can I extract different components of this file without script at server side?
>
> Regards
>
> On Mon, Feb 27, 2017 at 10:52 AM, Andy LoPresto <alopresto@apache.org <ma...@apache.org>> wrote:
> Mohammed,
>
> So from what I understand you have a separate web application which accepts a variety of data (a combination of form fields and file uploads) and submits that (I’m assuming via POST) to NiFi. I’m not sure what problem you had extracting these “components” from the output flowfile from the HandleHTTPRequest processor. Can you change your web interface to send separate POST requests to NiFi and logically separate these components from the beginning? Can you modify the format of the request (i.e. make it JSON, etc.) so it is properly structured for NiFi to parse and operate on this data?
>
> NiFi treats the content of the flowfile the same once it is in the system regardless of the source processor, so you should be able to split, extract, or manipulate the contents of a single flowfile containing multiple form elements and file bodies.
>
> If you are referencing NiFi’s Input Port components as a destination for an arbitrary HTTP request, this will not work. The input ports are designed to work with NiFi’s Site to Site protocol (either over raw sockets or HTTP(S)).
>
>
> Andy LoPresto
> alopresto@apache.org <ma...@apache.org>
> alopresto.apache@gmail.com <ma...@gmail.com>
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>
>> On Feb 27, 2017, at 11:09 AM, mohammed shambakey <shambakey1@gmail.com <ma...@gmail.com>> wrote:
>>
>> Thanks Andy
>>
>> I have a user web interface that send some form information (e.g., user id), as well as upload multiple files, to a remote NIFI instance. When using "handlhttprequest" processor, it results in a single file that contains "form inputs" and bodies of uploaded files.
>>
>> I want to extract individual components of this file (i.e., download all uploaded files and extract other form information (e.g., userid) in a JSON or other text file. I had a similar problem before and I learnt that "httphandlerequest" has a problem to extract individual compoents from, so I have to write my own scripts to extract these components (form inputs and uploaded files).
>>
>> As NIFI input ports does not face this problem of "handlehttprequest", I though if I can send form inputs and upload files directly to "NIFI input port" at remote site, then I don't have to write my own scripts to extract individual components.
>>
>> Regards
>>
>> On Mon, Feb 27, 2017 at 10:02 AM, Andy LoPresto <alopresto@apache.org <ma...@apache.org>> wrote:
>> Mohammed,
>>
>> I’m not sure I understand your question. NiFi has a REST API which supports the UI and allows scripts/other applications to make requests of the platform. This runs on the input port specified in nifi.properties.
>>
>> If you want to submit form data and uploaded files as data to be ingested by NiFi, HandleHTTPRequest or (less likely) ListenHTTP are the processors you would want to use.
>>
>> If you can clarify your use case, we should be able to offer more assistance.
>>
>> Andy LoPresto
>> alopresto@apache.org <ma...@apache.org>
>> alopresto.apache@gmail.com <ma...@gmail.com>
>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>>
>>> On Feb 27, 2017, at 9:15 AM, mohammed shambakey <shambakey1@gmail.com <ma...@gmail.com>> wrote:
>>>
>>> Hi
>>>
>>> Is it possible to send http request (including form data and uploaded files) to NIFI input port instead of "HandleHTTPRequest" processor?
>>>
>>> Regards
>>>
>>> --
>>> Mohammed
>>
>>
>>
>>
>> --
>> Mohammed
>
>
>
>
> --
> Mohammed
Re: send http request to nifi input port
Posted by mohammed shambakey <sh...@gmail.com>.
This is an example of the output file from "handlehttprequest":
"
-----------------------------831805444208556434014280960^M
Content-Disposition: form-data; name="userid"^M
^M
jpl_user^M
-----------------------------831805444208556434014280960^M
Content-Disposition: form-data; name="scripts"; filename="a.py"^M
Content-Type: text/x-python^M
^M
^M
-----------------------------831805444208556434014280960^M
Content-Disposition: form-data; name="scripts"; filename="b.py"^M
Content-Type: text/x-python^M
^M
^M
-----------------------------831805444208556434014280960^M
Content-Disposition: form-data; name="scripts"; filename="conf.json"^M
Content-Type: application/json^M
^M
{
"userid":"jpl_user_1",
"scripts":["a.py","b.py"]
}
^M
-----------------------------831805444208556434014280960--^M
"
Can I extract different components of this file without script at server
side?
Regards
On Mon, Feb 27, 2017 at 10:52 AM, Andy LoPresto <al...@apache.org>
wrote:
> Mohammed,
>
> So from what I understand you have a separate web application which
> accepts a variety of data (a combination of form fields and file uploads)
> and submits that (I’m assuming via POST) to NiFi. I’m not sure what problem
> you had extracting these “components” from the output flowfile from the
> HandleHTTPRequest processor. Can you change your web interface to send
> separate POST requests to NiFi and logically separate these components from
> the beginning? Can you modify the format of the request (i.e. make it JSON,
> etc.) so it is properly structured for NiFi to parse and operate on this
> data?
>
> NiFi treats the content of the flowfile the same once it is in the system
> regardless of the source processor, so you should be able to split,
> extract, or manipulate the contents of a single flowfile containing
> multiple form elements and file bodies.
>
> If you are referencing NiFi’s Input Port components as a destination for
> an arbitrary HTTP request, this will not work. The input ports are designed
> to work with NiFi’s Site to Site protocol (either over raw sockets or
> HTTP(S)).
>
>
> Andy LoPresto
> alopresto@apache.org
> *alopresto.apache@gmail.com <al...@gmail.com>*
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>
> On Feb 27, 2017, at 11:09 AM, mohammed shambakey <sh...@gmail.com>
> wrote:
>
> Thanks Andy
>
> I have a user web interface that send some form information (e.g., user
> id), as well as upload multiple files, to a remote NIFI instance. When
> using "handlhttprequest" processor, it results in a single file that
> contains "form inputs" and bodies of uploaded files.
>
> I want to extract individual components of this file (i.e., download all
> uploaded files and extract other form information (e.g., userid) in a JSON
> or other text file. I had a similar problem before and I learnt that
> "httphandlerequest" has a problem to extract individual compoents from, so
> I have to write my own scripts to extract these components (form inputs and
> uploaded files).
>
> As NIFI input ports does not face this problem of "handlehttprequest", I
> though if I can send form inputs and upload files directly to "NIFI input
> port" at remote site, then I don't have to write my own scripts to extract
> individual components.
>
> Regards
>
> On Mon, Feb 27, 2017 at 10:02 AM, Andy LoPresto <al...@apache.org>
> wrote:
>
>> Mohammed,
>>
>> I’m not sure I understand your question. NiFi has a REST API which
>> supports the UI and allows scripts/other applications to make requests of
>> the platform. This runs on the input port specified in nifi.properties.
>>
>> If you want to submit form data and uploaded files as data to be ingested
>> by NiFi, HandleHTTPRequest or (less likely) ListenHTTP are the processors
>> you would want to use.
>>
>> If you can clarify your use case, we should be able to offer more
>> assistance.
>>
>> Andy LoPresto
>> alopresto@apache.org
>> *alopresto.apache@gmail.com <al...@gmail.com>*
>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>>
>> On Feb 27, 2017, at 9:15 AM, mohammed shambakey <sh...@gmail.com>
>> wrote:
>>
>> Hi
>>
>> Is it possible to send http request (including form data and uploaded
>> files) to NIFI input port instead of "HandleHTTPRequest" processor?
>>
>> Regards
>>
>> --
>> Mohammed
>>
>>
>>
>
>
> --
> Mohammed
>
>
>
--
Mohammed
Re: send http request to nifi input port
Posted by Andy LoPresto <al...@apache.org>.
Mohammed,
So from what I understand you have a separate web application which accepts a variety of data (a combination of form fields and file uploads) and submits that (I’m assuming via POST) to NiFi. I’m not sure what problem you had extracting these “components” from the output flowfile from the HandleHTTPRequest processor. Can you change your web interface to send separate POST requests to NiFi and logically separate these components from the beginning? Can you modify the format of the request (i.e. make it JSON, etc.) so it is properly structured for NiFi to parse and operate on this data?
NiFi treats the content of the flowfile the same once it is in the system regardless of the source processor, so you should be able to split, extract, or manipulate the contents of a single flowfile containing multiple form elements and file bodies.
If you are referencing NiFi’s Input Port components as a destination for an arbitrary HTTP request, this will not work. The input ports are designed to work with NiFi’s Site to Site protocol (either over raw sockets or HTTP(S)).
Andy LoPresto
alopresto@apache.org
alopresto.apache@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> On Feb 27, 2017, at 11:09 AM, mohammed shambakey <sh...@gmail.com> wrote:
>
> Thanks Andy
>
> I have a user web interface that send some form information (e.g., user id), as well as upload multiple files, to a remote NIFI instance. When using "handlhttprequest" processor, it results in a single file that contains "form inputs" and bodies of uploaded files.
>
> I want to extract individual components of this file (i.e., download all uploaded files and extract other form information (e.g., userid) in a JSON or other text file. I had a similar problem before and I learnt that "httphandlerequest" has a problem to extract individual compoents from, so I have to write my own scripts to extract these components (form inputs and uploaded files).
>
> As NIFI input ports does not face this problem of "handlehttprequest", I though if I can send form inputs and upload files directly to "NIFI input port" at remote site, then I don't have to write my own scripts to extract individual components.
>
> Regards
>
> On Mon, Feb 27, 2017 at 10:02 AM, Andy LoPresto <alopresto@apache.org <ma...@apache.org>> wrote:
> Mohammed,
>
> I’m not sure I understand your question. NiFi has a REST API which supports the UI and allows scripts/other applications to make requests of the platform. This runs on the input port specified in nifi.properties.
>
> If you want to submit form data and uploaded files as data to be ingested by NiFi, HandleHTTPRequest or (less likely) ListenHTTP are the processors you would want to use.
>
> If you can clarify your use case, we should be able to offer more assistance.
>
> Andy LoPresto
> alopresto@apache.org <ma...@apache.org>
> alopresto.apache@gmail.com <ma...@gmail.com>
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>
>> On Feb 27, 2017, at 9:15 AM, mohammed shambakey <shambakey1@gmail.com <ma...@gmail.com>> wrote:
>>
>> Hi
>>
>> Is it possible to send http request (including form data and uploaded files) to NIFI input port instead of "HandleHTTPRequest" processor?
>>
>> Regards
>>
>> --
>> Mohammed
>
>
>
>
> --
> Mohammed
Re: send http request to nifi input port
Posted by mohammed shambakey <sh...@gmail.com>.
Thanks Andy
I have a user web interface that send some form information (e.g., user
id), as well as upload multiple files, to a remote NIFI instance. When
using "handlhttprequest" processor, it results in a single file that
contains "form inputs" and bodies of uploaded files.
I want to extract individual components of this file (i.e., download all
uploaded files and extract other form information (e.g., userid) in a JSON
or other text file. I had a similar problem before and I learnt that
"httphandlerequest" has a problem to extract individual compoents from, so
I have to write my own scripts to extract these components (form inputs and
uploaded files).
As NIFI input ports does not face this problem of "handlehttprequest", I
though if I can send form inputs and upload files directly to "NIFI input
port" at remote site, then I don't have to write my own scripts to extract
individual components.
Regards
On Mon, Feb 27, 2017 at 10:02 AM, Andy LoPresto <al...@apache.org>
wrote:
> Mohammed,
>
> I’m not sure I understand your question. NiFi has a REST API which
> supports the UI and allows scripts/other applications to make requests of
> the platform. This runs on the input port specified in nifi.properties.
>
> If you want to submit form data and uploaded files as data to be ingested
> by NiFi, HandleHTTPRequest or (less likely) ListenHTTP are the processors
> you would want to use.
>
> If you can clarify your use case, we should be able to offer more
> assistance.
>
> Andy LoPresto
> alopresto@apache.org
> *alopresto.apache@gmail.com <al...@gmail.com>*
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>
> On Feb 27, 2017, at 9:15 AM, mohammed shambakey <sh...@gmail.com>
> wrote:
>
> Hi
>
> Is it possible to send http request (including form data and uploaded
> files) to NIFI input port instead of "HandleHTTPRequest" processor?
>
> Regards
>
> --
> Mohammed
>
>
>
--
Mohammed
Re: send http request to nifi input port
Posted by Andy LoPresto <al...@apache.org>.
Mohammed,
I’m not sure I understand your question. NiFi has a REST API which supports the UI and allows scripts/other applications to make requests of the platform. This runs on the input port specified in nifi.properties.
If you want to submit form data and uploaded files as data to be ingested by NiFi, HandleHTTPRequest or (less likely) ListenHTTP are the processors you would want to use.
If you can clarify your use case, we should be able to offer more assistance.
Andy LoPresto
alopresto@apache.org
alopresto.apache@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> On Feb 27, 2017, at 9:15 AM, mohammed shambakey <sh...@gmail.com> wrote:
>
> Hi
>
> Is it possible to send http request (including form data and uploaded files) to NIFI input port instead of "HandleHTTPRequest" processor?
>
> Regards
>
> --
> Mohammed