You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Martijn Dekkers <ma...@dekkers.org.uk> on 2019/04/26 11:27:38 UTC
InvokeHTTP POST
Hi,
I am working to implement a flow where we need to POST some data to another system using multipart/form-data. This system requires that one of the form fields acts as an attached file. Using curl, this works fine:
curl -F <field1>=<data1> -F <field2>=<data2> -F <field3>=@[filename] <URL>
I created a flowfile that mimics what curl sends, including the boundaries etc. Using InvokeHTTP POST I receive a web-server error from the receiving end that mentions a missing parameter.
Running with curl --trace-ascii or POSTing to a running netcat with both curl and nifi shows exactly the same results, with the exception of the actual boundary marker (which is always different) and some minor header stuff (UA, etc.) The payload is simple text.
I understand that there is some difference between sending the data and uploading a file, but am unable to figure out what this is. for example, sending the same data with curl using -F<field3>=<[filename] (instead of @[filename]) also results in an error.
I have read a few different messages about POSTing multipart-form data where files need to be uploaded, and most appear to end up with writing a scripted processor or invoking shell. This is particularly undesirable for us given the volume of files we must process, and the speed at which we should be uploading those files.
Is there any way in which to mimic the uploading/attaching of a file, instead of sending the data as text?
Thanks
Martijn
Re: InvokeHTTP POST
Posted by Vijay Chhipa <vc...@apple.com>.
Martjin,
Can you show what the HTTP error code and the message is?
Also, please list all the properties you have set on the InvokeHTTP processor.
A screenshot works too,
Vijay
> On Apr 26, 2019, at 6:27 AM, Martijn Dekkers <ma...@dekkers.org.uk> wrote:
>
> Hi,
>
> I am working to implement a flow where we need to POST some data to another system using multipart/form-data. This system requires that one of the form fields acts as an attached file. Using curl, this works fine:
>
> curl -F <field1>=<data1> -F <field2>=<data2> -F <field3>=@[filename] <URL>
>
> I created a flowfile that mimics what curl sends, including the boundaries etc. Using InvokeHTTP POST I receive a web-server error from the receiving end that mentions a missing parameter.
>
> Running with curl --trace-ascii or POSTing to a running netcat with both curl and nifi shows exactly the same results, with the exception of the actual boundary marker (which is always different) and some minor header stuff (UA, etc.) The payload is simple text.
>
> I understand that there is some difference between sending the data and uploading a file, but am unable to figure out what this is. for example, sending the same data with curl using -F<field3>=<[filename] (instead of @[filename]) also results in an error.
>
> I have read a few different messages about POSTing multipart-form data where files need to be uploaded, and most appear to end up with writing a scripted processor or invoking shell. This is particularly undesirable for us given the volume of files we must process, and the speed at which we should be uploading those files.
>
> Is there any way in which to mimic the uploading/attaching of a file, instead of sending the data as text?
>
> Thanks
>
> Martijn
Re: InvokeHTTP POST
Posted by Otto Fowler <ot...@gmail.com>.
Not all replies will be to you and the list. Did you see this?
On April 28, 2019 at 07:41:44, Vijay Chhipa (vchhipa@apple.com) wrote:
Martjin,
Can you show what the HTTP error code and the message is?
Also, please list all the properties you have set on the InvokeHTTP
processor.
A screenshot works too,
Vijay
On May 3, 2019 at 02:52:48, Martijn Dekkers (martijn@dekkers.org.uk) wrote:
Hi all,
This is a last-ditch attempt to resolve the issue of POSTing
multipart/form-data using InvokeHTTP. Our alternatives would be to either
not using NiFi for this use-case - which would be a pity - or writing our
own processor that can properly submit forms.
Our flow, NiFi 1.9.2:
Issues found so far:
[HandleHTTPRequest] - [ExtractText] - [ReplaceText] - [InvokeHTTP] -
[HandleHTTPResponse]
ExtractText pulls the flowfile data into an attribute;
ReplaceText builds a message as per RFC2388 (tried several variations of
this, including with/without newlines in the expression editor):
--------------------------555791708927339309556823${literal(‘\r\n’)}
Content-Disposition: form-data; name=“data_src”${literal(‘\r\n’)}
${literal(‘\r\n’)}
TST${literal(‘\r\n’)}
--------------------------555791708927339309556823${literal(‘\r\n’)}
Content-Disposition: form-data; name=“cmd”${literal(‘\r\n’)}
${literal(‘\r\n’)}
sendResultDetails${literal(‘\r\n’)}
--------------------------555791708927339309556823${literal(‘\r\n’)}
Content-Disposition: form-data; name=“cmd_data”;
filename=“${filename}“${literal(‘\r\n’)}
Content-Type: text/plain${literal(‘\r\n’)}${literal(‘\r\n’)}
${cmd_data}
${literal(‘\r\n’)}
--------------------------555791708927339309556823--${literal(‘\r\n’)}
InvokeHTTP POSTs message to httpbin.org for testing:
Set URL, etc.
Set Content-Type to: multipart/form-data;
boundary=--------------------------555791708927339309556823
(We also tried setting Content-Type to text/plain and
application/octet-stream, and adding Content-Type: multipart/form-data;
boundary=--------------------------555791708927339309556823 to the top of
the body, none of these options worked)
Add "Accept: */*" Header
Where we use the Content-Type: multipart/form-data configuration,
httpbin.org doesn't see any data at all. Where we use a different
Content-Type, httpdin.org just parses the complete message body as a
standard message body.
If we build the exact same message with, for example, Postman, we receive
valid results. Sending both NiFi and Postman POSTs yields identical output
when captured on a listening netcat port.
The only answer I have managed to find on the question of "How to POST
form-data with InvokeHTTP is "build your own message body". Most of these
questions are met with silence. It would be extremely helpful if anyone
could comment on this actually being possible with InvokeHTTP.
Any feedback or assistance would be appreciated.
On Fri, 26 Apr 2019, at 13:27, Martijn Dekkers wrote:
Hi,
I am working to implement a flow where we need to POST some data to another
system using multipart/form-data. This system requires that one of the form
fields acts as an attached file. Using curl, this works fine:
curl -F <field1>=<data1> -F <field2>=<data2> -F <field3>=@[filename] <URL>
I created a flowfile that mimics what curl sends, including the boundaries
etc. Using InvokeHTTP POST I receive a web-server error from the receiving
end that mentions a missing parameter.
Running with curl --trace-ascii or POSTing to a running netcat with both
curl and nifi shows exactly the same results, with the exception of the
actual boundary marker (which is always different) and some minor header
stuff (UA, etc.) The payload is simple text.
I understand that there is some difference between sending the data and
uploading a file, but am unable to figure out what this is. for example,
sending the same data with curl using -F<field3>=<[filename] (instead of
@[filename]) also results in an error.
I have read a few different messages about POSTing multipart-form data
where files need to be uploaded, and most appear to end up with writing a
scripted processor or invoking shell. This is particularly undesirable for
us given the volume of files we must process, and the speed at which we
should be uploading those files.
Is there any way in which to mimic the uploading/attaching of a file,
instead of sending the data as text?
Thanks
Martijn
Re: InvokeHTTP POST
Posted by Martijn Dekkers <ma...@dekkers.org.uk>.
Hi all,
This is a last-ditch attempt to resolve the issue of POSTing multipart/form-data using InvokeHTTP. Our alternatives would be to either not using NiFi for this use-case - which would be a pity - or writing our own processor that can properly submit forms.
Our flow, NiFi 1.9.2:
Issues found so far:
[HandleHTTPRequest] - [ExtractText] - [ReplaceText] - [InvokeHTTP] - [HandleHTTPResponse]
ExtractText pulls the flowfile data into an attribute;
ReplaceText builds a message as per RFC2388 (tried several variations of this, including with/without newlines in the expression editor):
--------------------------555791708927339309556823${literal(‘\r\n’)}
Content-Disposition: form-data; name=“data_src”${literal(‘\r\n’)}
${literal(‘\r\n’)}
TST${literal(‘\r\n’)}
--------------------------555791708927339309556823${literal(‘\r\n’)}
Content-Disposition: form-data; name=“cmd”${literal(‘\r\n’)}
${literal(‘\r\n’)}
sendResultDetails${literal(‘\r\n’)}
--------------------------555791708927339309556823${literal(‘\r\n’)}
Content-Disposition: form-data; name=“cmd_data”; filename=“${filename}“${literal(‘\r\n’)}
Content-Type: text/plain${literal(‘\r\n’)}${literal(‘\r\n’)}
${cmd_data}
${literal(‘\r\n’)}
--------------------------555791708927339309556823--${literal(‘\r\n’)}
InvokeHTTP POSTs message to httpbin.org for testing:
Set URL, etc.
Set Content-Type to: multipart/form-data; boundary=--------------------------555791708927339309556823
(We also tried setting Content-Type to text/plain and application/octet-stream, and adding Content-Type: multipart/form-data; boundary=--------------------------555791708927339309556823 to the top of the body, none of these options worked)
Add "Accept: */*" Header
Where we use the Content-Type: multipart/form-data configuration, httpbin.org doesn't see any data at all. Where we use a different Content-Type, httpdin.org just parses the complete message body as a standard message body.
If we build the exact same message with, for example, Postman, we receive valid results. Sending both NiFi and Postman POSTs yields identical output when captured on a listening netcat port.
The only answer I have managed to find on the question of "How to POST form-data with InvokeHTTP is "build your own message body". Most of these questions are met with silence. It would be extremely helpful if anyone could comment on this actually being possible with InvokeHTTP.
Any feedback or assistance would be appreciated.
On Fri, 26 Apr 2019, at 13:27, Martijn Dekkers wrote:
> Hi,
>
> I am working to implement a flow where we need to POST some data to another system using multipart/form-data. This system requires that one of the form fields acts as an attached file. Using curl, this works fine:
>
> curl -F <field1>=<data1> -F <field2>=<data2> -F <field3>=@[filename] <URL>
>
> I created a flowfile that mimics what curl sends, including the boundaries etc. Using InvokeHTTP POST I receive a web-server error from the receiving end that mentions a missing parameter.
>
> Running with curl --trace-ascii or POSTing to a running netcat with both curl and nifi shows exactly the same results, with the exception of the actual boundary marker (which is always different) and some minor header stuff (UA, etc.) The payload is simple text.
>
> I understand that there is some difference between sending the data and uploading a file, but am unable to figure out what this is. for example, sending the same data with curl using -F<field3>=<[filename] (instead of @[filename]) also results in an error.
>
> I have read a few different messages about POSTing multipart-form data where files need to be uploaded, and most appear to end up with writing a scripted processor or invoking shell. This is particularly undesirable for us given the volume of files we must process, and the speed at which we should be uploading those files.
>
> Is there any way in which to mimic the uploading/attaching of a file, instead of sending the data as text?
>
> Thanks
>
> Martijn