You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@jmeter.apache.org by Eric Siegerman <er...@telepres.com> on 2002/08/13 06:07:11 UTC
HTTP Proxy URL-encoding woes
Ok, so this business of the HTTP proxy URL-encoding stuff is
biting me now.
Why does this happen? It seems to me that the proxy should pass
along exactly what it got, unless there's a good reason to do
otherwise. That invites the question: is there a good reason
for URL-encoding the data? What purpose is served by modifying
it, in however supposedly-benign a way?
On Fri, Aug 09, 2002 at 10:25:30AM -0400, Mike Stover wrote (over on jmeter-dev):
> It seems that what is needed is a new type of proxy server that generates SOAP
> Samplers, instead of normal HTTP Samplers. Not that your communications are
> using SOAP necessarily, but the form is the same (XML through HTTP).
I wasn't following the quoted thread closely, but this suggestion
seems to be a response to someone else's URL-encoding problems.
If so, it wouldn't help me. My problem is not with XML-formatted
data, but with bog-standard HTML forms. My browser encodes a
particular form as "multipart/form-data", but JMeter URL-encodes
the whole request body (all of its MIME parts crunched into one
huge long line) and zaps the encoding to
"application/x-www-form-urlencoded". Not surprisingly, the app
server hasn't a clue what to do with this.
Of course, a special case could be added for
"multipart/form-data" forms too (perhaps not as visibly as a
separate subclass). But better than multiplying kludges, it
seems to me, would be to just make the existing proxy
transparent.
Here are the ways in which the proxy incorrectly modifies the
data (perhaps only a partial list; this is based only on the form
submission that's causing me grief):
- form body munging as described above
- the response headers are changed to be terminated by LF;
they're supposed to be CRLF's, and are before the proxy gets
hold of them [1]
There are other modifications, but they're all correct -- either
necessary, or else unnecessary but innocuous (e.g. reordering
headers). If a list of these would be useful, I can post one.
I'm willing to take a quick stab at this (but haven't time for
more, so I can't promise anything). But before I do, I'd like to
know that I'm not going off on the wrong track. So, is there a
good reason for the current behaviour, or is it a bug /
implementation artifact?
Thanks much.
[1] "[...] a bare CR or LF MUST NOT be substituted for CRLF
within any of the HTTP control structures (such as header
fields and multipart boundaries)."
- RFC 2068, p. 25, emphasis in original
P.S. Sorry if this sounds a bit snarky. Normally I'd hold onto
it overnight and revise it in the morning. But deadline pressure
kind of prevents that :-(
--
| | /\
|-_|/ > Eric Siegerman, Toronto, Ont. erics@telepres.com
| | /
Anyone who swims with the current will reach the big music steamship;
whoever swims against the current will perhaps reach the source.
- Paul Schneider-Esleben
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: Multipart problems (was Re: HTTP Proxy URL-encoding woes)
Posted by Mike Stover <ms...@apache.org>.
I had a thought on this - instead of making a subclass of hashmap, I could instead
use the Header object that JMeter has as the Key in the Map, and change the
Header objects "equals()" and "hashCode()" methods. I think that would work well.
-Mike
On 14 Aug 2002 at 13:26, Eric Siegerman wrote:
> On Tue, Aug 13, 2002 at 07:46:31PM -0400, Mike Stover wrote:
> > On 13 Aug 2002 at 18:56, Eric Siegerman wrote:
> > I could change all headers to lower case without a subclass of hashmap - but that would be
> > altering the data as it goes through the proxy, wouldn't it - something you were complaining
> > about earlier.
>
> Doh! I guess I don't know the code well enough to be discussing
> it in such detail.
>
> --
>
> | | /\
> |-_|/ > Eric Siegerman, Toronto, Ont. erics@telepres.com
> | | /
> Anyone who swims with the current will reach the big music steamship;
> whoever swims against the current will perhaps reach the source.
> - Paul Schneider-Esleben
>
> --
> To unsubscribe, e-mail: <ma...@jakarta.apache.org>
> For additional commands, e-mail: <ma...@jakarta.apache.org>
>
--
Michael Stover
mstover1@apache.org
Yahoo IM: mstover_ya
ICQ: 152975688
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: Multipart problems (was Re: HTTP Proxy URL-encoding woes)
Posted by Eric Siegerman <er...@telepres.com>.
On Tue, Aug 13, 2002 at 07:46:31PM -0400, Mike Stover wrote:
> On 13 Aug 2002 at 18:56, Eric Siegerman wrote:
> I could change all headers to lower case without a subclass of hashmap - but that would be
> altering the data as it goes through the proxy, wouldn't it - something you were complaining
> about earlier.
Doh! I guess I don't know the code well enough to be discussing
it in such detail.
--
| | /\
|-_|/ > Eric Siegerman, Toronto, Ont. erics@telepres.com
| | /
Anyone who swims with the current will reach the big music steamship;
whoever swims against the current will perhaps reach the source.
- Paul Schneider-Esleben
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: Multipart problems (was Re: HTTP Proxy URL-encoding woes)
Posted by Mike Stover <ms...@apache.org>.
On 13 Aug 2002 at 18:56, Eric Siegerman wrote:
>[snip]
> Of course, this is only one of a large number of cases. E.g. The
> above was generated by ver. 6.2.3. I just tried it with Netscape
> 4.8; it spells "Content-type" as above, and also does
> "Content-length" the same way. Rather than patching case by
> case, it'd be safer to write a case-insensitive HashMap subclass
> (of course, keys would have to be String's, not Object's; each
> method just needs to zap any keys to lower case before delegating
> to HashMap).
I could change all headers to lower case without a subclass of hashmap - but that would be
altering the data as it goes through the proxy, wouldn't it - something you were complaining
about earlier. What we need is a hashmap subclass that compares values in a case-
insensitive way, but stores them faithfully. This is what I understood Jason Height's
suggestion to be. Anyone who wants to take that on, feel free. I'll use it when you send it in!
--
Michael Stover
mstover1@apache.org
Yahoo IM: mstover_ya
ICQ: 152975688
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: Multipart problems (was Re: HTTP Proxy URL-encoding woes)
Posted by Eric Siegerman <er...@telepres.com>.
On Tue, Aug 13, 2002 at 06:19:35PM -0400, Mike Stover wrote:
> I notice something strange from the log you sent. Look at the following headers:
>
> Keep-Alive: 300
> Proxy-Connection: keep-alive
> Referer:
> http://lawrencium.telepres.com:23000/easypub/access?objName=ep15.C.PGMGR.com_att_A
> nnouncement_PGMGR_PageManager&action=edit&editMessage=createMessage&%24%24S
> ESSION%24%24=%7BauthID%7D%3B%7BuserID%7D%3B%7Brole%7D%3B%7BprintNam
> e%7D&role=member&printName=Eric&userID=eks2&authID=2269eb1c%3Aef8c4b857a%3A-
> 7fee2f80e0
> Content-type: multipart/form-data; boundary=---------------------------
> 135049002710252023621189641421
> Content-Length: 960
>
> Notice that "Content-type" is the only one that doesn't capitalize the word after the "-".
That's Linux Netscape doing that. Slightly odd, perhaps, but
it's within spec: "Field names are case-insensitive." [RFC 2068,
first paragraph of Sec. 4.2]
> JMeter
> looks for the value of "Content-Type", and I wonder if it's missing it in your case. I can update
> JMeter to be more inclusive in looking for it and see if that solves your problem
Yup; it works. Thanks much!
Of course, this is only one of a large number of cases. E.g. The
above was generated by ver. 6.2.3. I just tried it with Netscape
4.8; it spells "Content-type" as above, and also does
"Content-length" the same way. Rather than patching case by
case, it'd be safer to write a case-insensitive HashMap subclass
(of course, keys would have to be String's, not Object's; each
method just needs to zap any keys to lower case before delegating
to HashMap).
--
| | /\
|-_|/ > Eric Siegerman, Toronto, Ont. erics@telepres.com
| | /
Anyone who swims with the current will reach the big music steamship;
whoever swims against the current will perhaps reach the source.
- Paul Schneider-Esleben
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: Multipart problems (was Re: HTTP Proxy URL-encoding woes)
Posted by Mike Stover <ms...@apache.org>.
I notice something strange from the log you sent. Look at the following headers:
Keep-Alive: 300
Proxy-Connection: keep-alive
Referer:
http://lawrencium.telepres.com:23000/easypub/access?objName=ep15.C.PGMGR.com_att_A
nnouncement_PGMGR_PageManager&action=edit&editMessage=createMessage&%24%24S
ESSION%24%24=%7BauthID%7D%3B%7BuserID%7D%3B%7Brole%7D%3B%7BprintNam
e%7D&role=member&printName=Eric&userID=eks2&authID=2269eb1c%3Aef8c4b857a%3A-
7fee2f80e0
Content-type: multipart/form-data; boundary=---------------------------
135049002710252023621189641421
Content-Length: 960
Notice that "Content-type" is the only one that doesn't capitalize the word after the "-". JMeter
looks for the value of "Content-Type", and I wonder if it's missing it in your case. I can update
JMeter to be more inclusive in looking for it and see if that solves your problem
-Mike
On 13 Aug 2002 at 17:36, Eric Siegerman wrote:
> On Tue, Aug 13, 2002 at 03:53:58PM -0400, Mike Stover wrote:
> > If JMeter isn't doing multiparts right, that's news. Have you tried again since I committed
> > today's fix?
>
> I have. It's still URL-encoding them :-( I'll enter it into bugzilla.
>
> Here's a further clue. When I look at the resulting HTTP Request element
> in JMeter, it shows a single parameter, whose name is:
> -----------------------------135049002710252023621189641421
> Content-Disposition: form-data; name
>
> and whose value is:
> "headline"
>
> yada yada
> -----------------------------135049002710252023621189641421
> Content-Disposition: form-data; name="dirText"
>
> blah blah blah
>
> -----------------------------135049002710252023621189641421
> [many more parts deleted]
>
> Looks as though it missed the Content-Type: header, and parsed
> the body as though it were application/x-www-form-urlencoded.
> The whole thing was treated as a single name/value pair, split on
> the first "=" -- the one in:
> Content-Disposition: form-data; name="headline"
>
> I guess the body happened happened not to contain any "&"s. Sure
> enough, if I resubmit the form with an "&" typed into one of the
> text fields, JMeter shows me two name/value pairs with all the
> boundaries at the expected (though of course still bogus) places.
>
> --
>
> | | /\
> |-_|/ > Eric Siegerman, Toronto, Ont. erics@telepres.com
> | | /
> Anyone who swims with the current will reach the big music steamship;
> whoever swims against the current will perhaps reach the source.
> - Paul Schneider-Esleben
>
> --
> To unsubscribe, e-mail: <ma...@jakarta.apache.org>
> For additional commands, e-mail: <ma...@jakarta.apache.org>
>
--
Michael Stover
mstover1@apache.org
Yahoo IM: mstover_ya
ICQ: 152975688
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Multipart problems (was Re: HTTP Proxy URL-encoding woes)
Posted by Eric Siegerman <er...@telepres.com>.
On Tue, Aug 13, 2002 at 03:53:58PM -0400, Mike Stover wrote:
> If JMeter isn't doing multiparts right, that's news. Have you tried again since I committed
> today's fix?
I have. It's still URL-encoding them :-( I'll enter it into bugzilla.
Here's a further clue. When I look at the resulting HTTP Request element
in JMeter, it shows a single parameter, whose name is:
-----------------------------135049002710252023621189641421
Content-Disposition: form-data; name
and whose value is:
"headline"
yada yada
-----------------------------135049002710252023621189641421
Content-Disposition: form-data; name="dirText"
blah blah blah
-----------------------------135049002710252023621189641421
[many more parts deleted]
Looks as though it missed the Content-Type: header, and parsed
the body as though it were application/x-www-form-urlencoded.
The whole thing was treated as a single name/value pair, split on
the first "=" -- the one in:
Content-Disposition: form-data; name="headline"
I guess the body happened happened not to contain any "&"s. Sure
enough, if I resubmit the form with an "&" typed into one of the
text fields, JMeter shows me two name/value pairs with all the
boundaries at the expected (though of course still bogus) places.
--
| | /\
|-_|/ > Eric Siegerman, Toronto, Ont. erics@telepres.com
| | /
Anyone who swims with the current will reach the big music steamship;
whoever swims against the current will perhaps reach the source.
- Paul Schneider-Esleben
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: HTTP Proxy URL-encoding woes
Posted by Eric Siegerman <er...@telepres.com>.
Apologies for following up my own post; I forgot the attachments.
The dummy reply body as shown has LF line terminators, not CRLFs.
The real body had mixed LFs and CRLFs. I think that's what the
server is generating. If so, that's a problem; but I don't think
it's JMeter's fault.
The reply *headers* on the other hand...
--
| | /\
|-_|/ > Eric Siegerman, Toronto, Ont. erics@telepres.com
| | /
Anyone who swims with the current will reach the big music steamship;
whoever swims against the current will perhaps reach the source.
- Paul Schneider-Esleben
Re: HTTP Proxy URL-encoding woes
Posted by Eric Siegerman <er...@telepres.com>.
Apologies for following up my own post; I forgot the attachments.
The dummy reply body as shown has LF line terminators, not CRLFs.
The real body had mixed LFs and CRLFs. I think that's what the
server is generating. If so, that's a problem; but I don't think
it's JMeter's fault.
The reply *headers* on the other hand...
--
| | /\
|-_|/ > Eric Siegerman, Toronto, Ont. erics@telepres.com
| | /
Anyone who swims with the current will reach the big music steamship;
whoever swims against the current will perhaps reach the source.
- Paul Schneider-Esleben
Re: HTTP Proxy URL-encoding woes
Posted by Mike Stover <ms...@apache.org>.
Since you're on the dev list, let's keep it there. feedback below.
On 13 Aug 2002 at 15:26, Eric Siegerman wrote:
> [Crossposting to jmeter-user and jmeter-dev, following Mike S's
> lead. Mike, was it your intention to move this to only
> jmeter-dev?]
>
> On Tue, Aug 13, 2002 at 09:13:28AM -0400, Mike Stover wrote:
> > I'm a little concerned about the multipart form issue you raise - JMeter has always in the past
> > correctly parsed multipart forms as multipart forms - so I'm wondering why your request
> > doesn't work. Are you doing something weird? Normally, mulitpart forms are created when
> > browsers upload files, with HTML like so: <form action="upload.jsp" method="POST"
> > enctype="multipart/form-data"><input type="File" ...></form>.
>
> It is indeed a file upload. I don't know whether the application
> does anything weird. I didn't write it; I was just told,
> "load-test it". I just dove into the RFCs last night while
> tracking this down.
>
> Maybe what's weird is that I checked a radio button that doesn't
> require a file to actually be uploaded ("use previous image", as
> opposed to "use the image I'm about to upload")...
>
> I've attached a log of the transaction (hooray for Ethereal!),
> and also a hex dump for anyone who wants to read it directly in
> their mail client. Byte-for-byte, except that I've removed the
> reply body (dunno if there's anything proprietary in there; don't
> want to take the time to ask; and it's not important anyway, just
> a Java stack trace plus elaboration from the application itself).
If JMeter isn't doing multiparts right, that's news. Have you tried again since I committed
today's fix?
>
>
> On Tue, Aug 13, 2002 at 10:31:16AM -0400, Mike Stover wrote:
> > I've made a change that prevents JMeter from encoding the values sent by the browser.
>
> Cool! Thanks!
>
> > It's
> > not a perfect solution yet - if you change things in the GUI (after recording is done), it can
> > mess things up.
>
> Which GUI; JMeter's or the target app's? If the latter, I've
> been assuming that; isn't it *necessarily* the case? In either
> case, how could merely URL-encoding prevent it? Could you give
> an example?
JMeter's GUI. If you change an argument in the GUI after you've recorded it, it will re-encode
the value, which may or may not be what you want.
>
> > The problem is, some values
> > need to be encoded, and some should not be, but detecting the difference will require some
> > work.
>
> Why? Surely the browser has already encoded the values that need
> it -- and only those. Why does the proxy have to even think
> about it?!? Why is it not sufficient -- and indeed, the only
> theoretically correct approach -- for the proxy to pass along
> precisely the bytes it received?
The proxy server as I've changed it should be passing values on as it received them with no
encoding. The problem is what to do with them in the GUI? If I just leave them as encoded
values, it looks ugly, and how many people speak URL-encoded? People want to see the
unencoded values. But sometimes there is no difference between the normal value and the
encoded value, such as in sending XML data. But, if you slip a '&' in your xml, it will need to
be encoded if the server is reading it as a normal POST request. Thus, it gets complicated,
and the best solution will probably end up being to display both normal and encoded value for
the user to examine.
>
> --
>
> | | /\
> |-_|/ > Eric Siegerman, Toronto, Ont. erics@telepres.com
> | | /
> Anyone who swims with the current will reach the big music steamship;
> whoever swims against the current will perhaps reach the source.
> - Paul Schneider-Esleben
>
> --
> To unsubscribe, e-mail: <ma...@jakarta.apache.org>
> For additional commands, e-mail: <ma...@jakarta.apache.org>
>
--
Michael Stover
mstover1@apache.org
Yahoo IM: mstover_ya
ICQ: 152975688
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: HTTP Proxy URL-encoding woes
Posted by Eric Siegerman <er...@telepres.com>.
[Crossposting to jmeter-user and jmeter-dev, following Mike S's
lead. Mike, was it your intention to move this to only
jmeter-dev?]
On Tue, Aug 13, 2002 at 09:13:28AM -0400, Mike Stover wrote:
> I'm a little concerned about the multipart form issue you raise - JMeter has always in the past
> correctly parsed multipart forms as multipart forms - so I'm wondering why your request
> doesn't work. Are you doing something weird? Normally, mulitpart forms are created when
> browsers upload files, with HTML like so: <form action="upload.jsp" method="POST"
> enctype="multipart/form-data"><input type="File" ...></form>.
It is indeed a file upload. I don't know whether the application
does anything weird. I didn't write it; I was just told,
"load-test it". I just dove into the RFCs last night while
tracking this down.
Maybe what's weird is that I checked a radio button that doesn't
require a file to actually be uploaded ("use previous image", as
opposed to "use the image I'm about to upload")...
I've attached a log of the transaction (hooray for Ethereal!),
and also a hex dump for anyone who wants to read it directly in
their mail client. Byte-for-byte, except that I've removed the
reply body (dunno if there's anything proprietary in there; don't
want to take the time to ask; and it's not important anyway, just
a Java stack trace plus elaboration from the application itself).
On Tue, Aug 13, 2002 at 10:31:16AM -0400, Mike Stover wrote:
> I've made a change that prevents JMeter from encoding the values sent by the browser.
Cool! Thanks!
> It's
> not a perfect solution yet - if you change things in the GUI (after recording is done), it can
> mess things up.
Which GUI; JMeter's or the target app's? If the latter, I've
been assuming that; isn't it *necessarily* the case? In either
case, how could merely URL-encoding prevent it? Could you give
an example?
> The problem is, some values
> need to be encoded, and some should not be, but detecting the difference will require some
> work.
Why? Surely the browser has already encoded the values that need
it -- and only those. Why does the proxy have to even think
about it?!? Why is it not sufficient -- and indeed, the only
theoretically correct approach -- for the proxy to pass along
precisely the bytes it received?
--
| | /\
|-_|/ > Eric Siegerman, Toronto, Ont. erics@telepres.com
| | /
Anyone who swims with the current will reach the big music steamship;
whoever swims against the current will perhaps reach the source.
- Paul Schneider-Esleben
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: HTTP Proxy URL-encoding woes
Posted by Eric Siegerman <er...@telepres.com>.
[Crossposting to jmeter-user and jmeter-dev, following Mike S's
lead. Mike, was it your intention to move this to only
jmeter-dev?]
On Tue, Aug 13, 2002 at 09:13:28AM -0400, Mike Stover wrote:
> I'm a little concerned about the multipart form issue you raise - JMeter has always in the past
> correctly parsed multipart forms as multipart forms - so I'm wondering why your request
> doesn't work. Are you doing something weird? Normally, mulitpart forms are created when
> browsers upload files, with HTML like so: <form action="upload.jsp" method="POST"
> enctype="multipart/form-data"><input type="File" ...></form>.
It is indeed a file upload. I don't know whether the application
does anything weird. I didn't write it; I was just told,
"load-test it". I just dove into the RFCs last night while
tracking this down.
Maybe what's weird is that I checked a radio button that doesn't
require a file to actually be uploaded ("use previous image", as
opposed to "use the image I'm about to upload")...
I've attached a log of the transaction (hooray for Ethereal!),
and also a hex dump for anyone who wants to read it directly in
their mail client. Byte-for-byte, except that I've removed the
reply body (dunno if there's anything proprietary in there; don't
want to take the time to ask; and it's not important anyway, just
a Java stack trace plus elaboration from the application itself).
On Tue, Aug 13, 2002 at 10:31:16AM -0400, Mike Stover wrote:
> I've made a change that prevents JMeter from encoding the values sent by the browser.
Cool! Thanks!
> It's
> not a perfect solution yet - if you change things in the GUI (after recording is done), it can
> mess things up.
Which GUI; JMeter's or the target app's? If the latter, I've
been assuming that; isn't it *necessarily* the case? In either
case, how could merely URL-encoding prevent it? Could you give
an example?
> The problem is, some values
> need to be encoded, and some should not be, but detecting the difference will require some
> work.
Why? Surely the browser has already encoded the values that need
it -- and only those. Why does the proxy have to even think
about it?!? Why is it not sufficient -- and indeed, the only
theoretically correct approach -- for the proxy to pass along
precisely the bytes it received?
--
| | /\
|-_|/ > Eric Siegerman, Toronto, Ont. erics@telepres.com
| | /
Anyone who swims with the current will reach the big music steamship;
whoever swims against the current will perhaps reach the source.
- Paul Schneider-Esleben
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: HTTP Proxy URL-encoding woes
Posted by Mike Stover <ms...@apache.org>.
Some good ideas - I respond below.
On 14 Aug 2002 at 0:57, Jordi Salvat i Alabart wrote:
> Hi Mike.
>
> While editing a script which used a form with possibly 100 parameters
> (!) I thought it would be kool to have the HTTP parameters be "elements"
> hanging from the HTTP Sampler in the test plan tree.
>
> In this way we would have much more screen real-state available to edit
> the individual parameters, which would help a lot with long
> parameter-names and/or values -- such as those which appear when using
> the regexp function.
Making alternate GUI's to create test elements is something I've been wanting to get around to
ever since I re-architected JMeter from 1.7 to 1.7.1. You could easily make a GUI class that
creates an Arguments object, and it could be attached to HTTP Requests in the test tree, as
you describe. JMeter already works this way, and the only thing anyone out there needs to
do is create a GUI class, and implement JMeterGUIComponent.
I would like to create a very simple GUI alternative to the HTTP Request specifically for basic
GET requests. It would be a single text field that held the URL. The current GUI is kind of
overkill for such simple requests.
But, bugs and trying to get 1.7.3 have all my attention right now :-(
>
> The thing is that you usually don't need to edit lots of parameters:
> just one or two. The kind of interface I'm describing would make this
> easier than it is now.
>
> I also gave some thought to the URL-encoding issue. I observed JMeter
> was too greedy when URL-encoding stuff -- even though this didn't cause
> me too much damage, I thought it could eventually become an issue.
>
> It may actually be better not to URL-encode at all. After all, you can
> always URL-encode "by hand" when entering your values, or use an
> embedded function.
Some of us can do it by hand. Many users would want JMeter to do it for them. I could,
however, pass on encoding it unless a value held an ampersand or space. What I'm persuing
now is an alteration in the GUI to give a choice for each argument on whether JMeter should
encode it automatically, or leave it to the user to do. You should be able to see it tomorrow,
and give me feedback on it :-)
-Mike
>
> Another, intermediate, approach would be to do as little encoding as
> possible: just encode spaces, ampersands, equal signs, and non-ASCII
> characters. (The URLEncoder class encodes all non-alphanumeric characters)
>
> Salut,
>
> Jordi.
>
> Mike Stover wrote:
> > I've made a change that prevents JMeter from encoding the values sent by the browser. It's
> > not a perfect solution yet - if you change things in the GUI (after recording is done), it can
> > mess things up. Thus, my next step will be to add checkboxes to the HTTP Request gui to
> > indicate whether each argument should be encoded or not. The problem is, some values
> > need to be encoded, and some should not be, but detecting the difference will require some
> > work.
> >
> > To the parameter panel, I am thinking of adding a third and fourth column. The third column
> > will hold the value to actually be sent to the server (ie the encoded value). The fourth column
> > will hold a checkbox indicating whether URL encoding should be performed. Thus, you type
> > in the second column, and the encoded (or not encoded, depending) value appears in the
> > third column as you type. You'll also be able to edit the third column directly, to maximize
> > your opportunities to screw things up.
> >
> > Um, I might just make a whole different GUI for all this - call it the "Advanced" HTTP Request,
> > or something... Any suggestions would be greatly appreciated :-)
> >
> > -Mike
> >
> >
> > On 13 Aug 2002 at 0:07, Eric Siegerman wrote:
> >
> >
> >>Ok, so this business of the HTTP proxy URL-encoding stuff is
> >>biting me now.
> >>
> >>Why does this happen? It seems to me that the proxy should pass
> >>along exactly what it got, unless there's a good reason to do
> >>otherwise. That invites the question: is there a good reason
> >>for URL-encoding the data? What purpose is served by modifying
> >>it, in however supposedly-benign a way?
> >>
> >>On Fri, Aug 09, 2002 at 10:25:30AM -0400, Mike Stover wrote (over on jmeter-dev):
> >>
> >>>It seems that what is needed is a new type of proxy server that generates SOAP
> >>>Samplers, instead of normal HTTP Samplers. Not that your communications are
> >>>using SOAP necessarily, but the form is the same (XML through HTTP).
> >>
> >>I wasn't following the quoted thread closely, but this suggestion
> >>seems to be a response to someone else's URL-encoding problems.
> >>If so, it wouldn't help me. My problem is not with XML-formatted
> >>data, but with bog-standard HTML forms. My browser encodes a
> >>particular form as "multipart/form-data", but JMeter URL-encodes
> >>the whole request body (all of its MIME parts crunched into one
> >>huge long line) and zaps the encoding to
> >>"application/x-www-form-urlencoded". Not surprisingly, the app
> >>server hasn't a clue what to do with this.
> >>
> >>Of course, a special case could be added for
> >>"multipart/form-data" forms too (perhaps not as visibly as a
> >>separate subclass). But better than multiplying kludges, it
> >>seems to me, would be to just make the existing proxy
> >>transparent.
> >>
> >>Here are the ways in which the proxy incorrectly modifies the
> >>data (perhaps only a partial list; this is based only on the form
> >>submission that's causing me grief):
> >> - form body munging as described above
> >> - the response headers are changed to be terminated by LF;
> >> they're supposed to be CRLF's, and are before the proxy gets
> >> hold of them [1]
> >>
> >>There are other modifications, but they're all correct -- either
> >>necessary, or else unnecessary but innocuous (e.g. reordering
> >>headers). If a list of these would be useful, I can post one.
> >>
> >>I'm willing to take a quick stab at this (but haven't time for
> >>more, so I can't promise anything). But before I do, I'd like to
> >>know that I'm not going off on the wrong track. So, is there a
> >>good reason for the current behaviour, or is it a bug /
> >>implementation artifact?
> >>
> >>Thanks much.
> >>
> >>
> >>[1] "[...] a bare CR or LF MUST NOT be substituted for CRLF
> >> within any of the HTTP control structures (such as header
> >> fields and multipart boundaries)."
> >> - RFC 2068, p. 25, emphasis in original
> >>
> >>
> >>P.S. Sorry if this sounds a bit snarky. Normally I'd hold onto
> >>it overnight and revise it in the morning. But deadline pressure
> >>kind of prevents that :-(
> >>
> >>--
> >>
> >>| | /\
> >>|-_|/ > Eric Siegerman, Toronto, Ont. erics@telepres.com
> >>| | /
> >>Anyone who swims with the current will reach the big music steamship;
> >>whoever swims against the current will perhaps reach the source.
> >> - Paul Schneider-Esleben
> >>
> >>--
> >>To unsubscribe, e-mail: <ma...@jakarta.apache.org>
> >>For additional commands, e-mail: <ma...@jakarta.apache.org>
> >>
> >
> >
> >
> > --
> > Michael Stover
> > mstover1@apache.org
> > Yahoo IM: mstover_ya
> > ICQ: 152975688
> >
> > --
> > To unsubscribe, e-mail: <ma...@jakarta.apache.org>
> > For additional commands, e-mail: <ma...@jakarta.apache.org>
> >
> >
>
>
>
>
> --
> To unsubscribe, e-mail: <ma...@jakarta.apache.org>
> For additional commands, e-mail: <ma...@jakarta.apache.org>
>
--
Michael Stover
mstover1@apache.org
Yahoo IM: mstover_ya
ICQ: 152975688
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: HTTP Proxy URL-encoding woes
Posted by Jordi Salvat i Alabart <js...@atg.com>.
Hi Mike.
While editing a script which used a form with possibly 100 parameters
(!) I thought it would be kool to have the HTTP parameters be "elements"
hanging from the HTTP Sampler in the test plan tree.
In this way we would have much more screen real-state available to edit
the individual parameters, which would help a lot with long
parameter-names and/or values -- such as those which appear when using
the regexp function.
The thing is that you usually don't need to edit lots of parameters:
just one or two. The kind of interface I'm describing would make this
easier than it is now.
I also gave some thought to the URL-encoding issue. I observed JMeter
was too greedy when URL-encoding stuff -- even though this didn't cause
me too much damage, I thought it could eventually become an issue.
It may actually be better not to URL-encode at all. After all, you can
always URL-encode "by hand" when entering your values, or use an
embedded function.
Another, intermediate, approach would be to do as little encoding as
possible: just encode spaces, ampersands, equal signs, and non-ASCII
characters. (The URLEncoder class encodes all non-alphanumeric characters)
Salut,
Jordi.
Mike Stover wrote:
> I've made a change that prevents JMeter from encoding the values sent by the browser. It's
> not a perfect solution yet - if you change things in the GUI (after recording is done), it can
> mess things up. Thus, my next step will be to add checkboxes to the HTTP Request gui to
> indicate whether each argument should be encoded or not. The problem is, some values
> need to be encoded, and some should not be, but detecting the difference will require some
> work.
>
> To the parameter panel, I am thinking of adding a third and fourth column. The third column
> will hold the value to actually be sent to the server (ie the encoded value). The fourth column
> will hold a checkbox indicating whether URL encoding should be performed. Thus, you type
> in the second column, and the encoded (or not encoded, depending) value appears in the
> third column as you type. You'll also be able to edit the third column directly, to maximize
> your opportunities to screw things up.
>
> Um, I might just make a whole different GUI for all this - call it the "Advanced" HTTP Request,
> or something... Any suggestions would be greatly appreciated :-)
>
> -Mike
>
>
> On 13 Aug 2002 at 0:07, Eric Siegerman wrote:
>
>
>>Ok, so this business of the HTTP proxy URL-encoding stuff is
>>biting me now.
>>
>>Why does this happen? It seems to me that the proxy should pass
>>along exactly what it got, unless there's a good reason to do
>>otherwise. That invites the question: is there a good reason
>>for URL-encoding the data? What purpose is served by modifying
>>it, in however supposedly-benign a way?
>>
>>On Fri, Aug 09, 2002 at 10:25:30AM -0400, Mike Stover wrote (over on jmeter-dev):
>>
>>>It seems that what is needed is a new type of proxy server that generates SOAP
>>>Samplers, instead of normal HTTP Samplers. Not that your communications are
>>>using SOAP necessarily, but the form is the same (XML through HTTP).
>>
>>I wasn't following the quoted thread closely, but this suggestion
>>seems to be a response to someone else's URL-encoding problems.
>>If so, it wouldn't help me. My problem is not with XML-formatted
>>data, but with bog-standard HTML forms. My browser encodes a
>>particular form as "multipart/form-data", but JMeter URL-encodes
>>the whole request body (all of its MIME parts crunched into one
>>huge long line) and zaps the encoding to
>>"application/x-www-form-urlencoded". Not surprisingly, the app
>>server hasn't a clue what to do with this.
>>
>>Of course, a special case could be added for
>>"multipart/form-data" forms too (perhaps not as visibly as a
>>separate subclass). But better than multiplying kludges, it
>>seems to me, would be to just make the existing proxy
>>transparent.
>>
>>Here are the ways in which the proxy incorrectly modifies the
>>data (perhaps only a partial list; this is based only on the form
>>submission that's causing me grief):
>> - form body munging as described above
>> - the response headers are changed to be terminated by LF;
>> they're supposed to be CRLF's, and are before the proxy gets
>> hold of them [1]
>>
>>There are other modifications, but they're all correct -- either
>>necessary, or else unnecessary but innocuous (e.g. reordering
>>headers). If a list of these would be useful, I can post one.
>>
>>I'm willing to take a quick stab at this (but haven't time for
>>more, so I can't promise anything). But before I do, I'd like to
>>know that I'm not going off on the wrong track. So, is there a
>>good reason for the current behaviour, or is it a bug /
>>implementation artifact?
>>
>>Thanks much.
>>
>>
>>[1] "[...] a bare CR or LF MUST NOT be substituted for CRLF
>> within any of the HTTP control structures (such as header
>> fields and multipart boundaries)."
>> - RFC 2068, p. 25, emphasis in original
>>
>>
>>P.S. Sorry if this sounds a bit snarky. Normally I'd hold onto
>>it overnight and revise it in the morning. But deadline pressure
>>kind of prevents that :-(
>>
>>--
>>
>>| | /\
>>|-_|/ > Eric Siegerman, Toronto, Ont. erics@telepres.com
>>| | /
>>Anyone who swims with the current will reach the big music steamship;
>>whoever swims against the current will perhaps reach the source.
>> - Paul Schneider-Esleben
>>
>>--
>>To unsubscribe, e-mail: <ma...@jakarta.apache.org>
>>For additional commands, e-mail: <ma...@jakarta.apache.org>
>>
>
>
>
> --
> Michael Stover
> mstover1@apache.org
> Yahoo IM: mstover_ya
> ICQ: 152975688
>
> --
> To unsubscribe, e-mail: <ma...@jakarta.apache.org>
> For additional commands, e-mail: <ma...@jakarta.apache.org>
>
>
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: HTTP Proxy URL-encoding woes
Posted by Mike Stover <ms...@apache.org>.
I've made a change that prevents JMeter from encoding the values sent by the browser. It's
not a perfect solution yet - if you change things in the GUI (after recording is done), it can
mess things up. Thus, my next step will be to add checkboxes to the HTTP Request gui to
indicate whether each argument should be encoded or not. The problem is, some values
need to be encoded, and some should not be, but detecting the difference will require some
work.
To the parameter panel, I am thinking of adding a third and fourth column. The third column
will hold the value to actually be sent to the server (ie the encoded value). The fourth column
will hold a checkbox indicating whether URL encoding should be performed. Thus, you type
in the second column, and the encoded (or not encoded, depending) value appears in the
third column as you type. You'll also be able to edit the third column directly, to maximize
your opportunities to screw things up.
Um, I might just make a whole different GUI for all this - call it the "Advanced" HTTP Request,
or something... Any suggestions would be greatly appreciated :-)
-Mike
On 13 Aug 2002 at 0:07, Eric Siegerman wrote:
> Ok, so this business of the HTTP proxy URL-encoding stuff is
> biting me now.
>
> Why does this happen? It seems to me that the proxy should pass
> along exactly what it got, unless there's a good reason to do
> otherwise. That invites the question: is there a good reason
> for URL-encoding the data? What purpose is served by modifying
> it, in however supposedly-benign a way?
>
> On Fri, Aug 09, 2002 at 10:25:30AM -0400, Mike Stover wrote (over on jmeter-dev):
> > It seems that what is needed is a new type of proxy server that generates SOAP
> > Samplers, instead of normal HTTP Samplers. Not that your communications are
> > using SOAP necessarily, but the form is the same (XML through HTTP).
>
> I wasn't following the quoted thread closely, but this suggestion
> seems to be a response to someone else's URL-encoding problems.
> If so, it wouldn't help me. My problem is not with XML-formatted
> data, but with bog-standard HTML forms. My browser encodes a
> particular form as "multipart/form-data", but JMeter URL-encodes
> the whole request body (all of its MIME parts crunched into one
> huge long line) and zaps the encoding to
> "application/x-www-form-urlencoded". Not surprisingly, the app
> server hasn't a clue what to do with this.
>
> Of course, a special case could be added for
> "multipart/form-data" forms too (perhaps not as visibly as a
> separate subclass). But better than multiplying kludges, it
> seems to me, would be to just make the existing proxy
> transparent.
>
> Here are the ways in which the proxy incorrectly modifies the
> data (perhaps only a partial list; this is based only on the form
> submission that's causing me grief):
> - form body munging as described above
> - the response headers are changed to be terminated by LF;
> they're supposed to be CRLF's, and are before the proxy gets
> hold of them [1]
>
> There are other modifications, but they're all correct -- either
> necessary, or else unnecessary but innocuous (e.g. reordering
> headers). If a list of these would be useful, I can post one.
>
> I'm willing to take a quick stab at this (but haven't time for
> more, so I can't promise anything). But before I do, I'd like to
> know that I'm not going off on the wrong track. So, is there a
> good reason for the current behaviour, or is it a bug /
> implementation artifact?
>
> Thanks much.
>
>
> [1] "[...] a bare CR or LF MUST NOT be substituted for CRLF
> within any of the HTTP control structures (such as header
> fields and multipart boundaries)."
> - RFC 2068, p. 25, emphasis in original
>
>
> P.S. Sorry if this sounds a bit snarky. Normally I'd hold onto
> it overnight and revise it in the morning. But deadline pressure
> kind of prevents that :-(
>
> --
>
> | | /\
> |-_|/ > Eric Siegerman, Toronto, Ont. erics@telepres.com
> | | /
> Anyone who swims with the current will reach the big music steamship;
> whoever swims against the current will perhaps reach the source.
> - Paul Schneider-Esleben
>
> --
> To unsubscribe, e-mail: <ma...@jakarta.apache.org>
> For additional commands, e-mail: <ma...@jakarta.apache.org>
>
--
Michael Stover
mstover1@apache.org
Yahoo IM: mstover_ya
ICQ: 152975688
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: HTTP Proxy URL-encoding woes
Posted by Mike Stover <ms...@apache.org>.
Ok, I got an idea while reading this on how it (the url-encoding issue) might be fixed. I'll let
you know when I'm done.
I'm a little concerned about the multipart form issue you raise - JMeter has always in the past
correctly parsed multipart forms as multipart forms - so I'm wondering why your request
doesn't work. Are you doing something weird? Normally, mulitpart forms are created when
browsers upload files, with HTML like so: <form action="upload.jsp" method="POST"
enctype="multipart/form-data"><input type="File" ...></form>. Is this what you are doing, or
is it different somehow?
-Mike
On 13 Aug 2002 at 0:07, Eric Siegerman wrote:
> Ok, so this business of the HTTP proxy URL-encoding stuff is
> biting me now.
>
> Why does this happen? It seems to me that the proxy should pass
> along exactly what it got, unless there's a good reason to do
> otherwise. That invites the question: is there a good reason
> for URL-encoding the data? What purpose is served by modifying
> it, in however supposedly-benign a way?
>
> On Fri, Aug 09, 2002 at 10:25:30AM -0400, Mike Stover wrote (over on jmeter-dev):
> > It seems that what is needed is a new type of proxy server that generates SOAP
> > Samplers, instead of normal HTTP Samplers. Not that your communications are
> > using SOAP necessarily, but the form is the same (XML through HTTP).
>
> I wasn't following the quoted thread closely, but this suggestion
> seems to be a response to someone else's URL-encoding problems.
> If so, it wouldn't help me. My problem is not with XML-formatted
> data, but with bog-standard HTML forms. My browser encodes a
> particular form as "multipart/form-data", but JMeter URL-encodes
> the whole request body (all of its MIME parts crunched into one
> huge long line) and zaps the encoding to
> "application/x-www-form-urlencoded". Not surprisingly, the app
> server hasn't a clue what to do with this.
>
> Of course, a special case could be added for
> "multipart/form-data" forms too (perhaps not as visibly as a
> separate subclass). But better than multiplying kludges, it
> seems to me, would be to just make the existing proxy
> transparent.
>
> Here are the ways in which the proxy incorrectly modifies the
> data (perhaps only a partial list; this is based only on the form
> submission that's causing me grief):
> - form body munging as described above
> - the response headers are changed to be terminated by LF;
> they're supposed to be CRLF's, and are before the proxy gets
> hold of them [1]
>
> There are other modifications, but they're all correct -- either
> necessary, or else unnecessary but innocuous (e.g. reordering
> headers). If a list of these would be useful, I can post one.
>
> I'm willing to take a quick stab at this (but haven't time for
> more, so I can't promise anything). But before I do, I'd like to
> know that I'm not going off on the wrong track. So, is there a
> good reason for the current behaviour, or is it a bug /
> implementation artifact?
>
> Thanks much.
>
>
> [1] "[...] a bare CR or LF MUST NOT be substituted for CRLF
> within any of the HTTP control structures (such as header
> fields and multipart boundaries)."
> - RFC 2068, p. 25, emphasis in original
>
>
> P.S. Sorry if this sounds a bit snarky. Normally I'd hold onto
> it overnight and revise it in the morning. But deadline pressure
> kind of prevents that :-(
>
> --
>
> | | /\
> |-_|/ > Eric Siegerman, Toronto, Ont. erics@telepres.com
> | | /
> Anyone who swims with the current will reach the big music steamship;
> whoever swims against the current will perhaps reach the source.
> - Paul Schneider-Esleben
>
> --
> To unsubscribe, e-mail: <ma...@jakarta.apache.org>
> For additional commands, e-mail: <ma...@jakarta.apache.org>
>
--
Michael Stover
mstover1@apache.org
Yahoo IM: mstover_ya
ICQ: 152975688
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: HTTP Proxy URL-encoding woes
Posted by Mike Stover <ms...@apache.org>.
I've made a change that prevents JMeter from encoding the values sent by the browser. It's
not a perfect solution yet - if you change things in the GUI (after recording is done), it can
mess things up. Thus, my next step will be to add checkboxes to the HTTP Request gui to
indicate whether each argument should be encoded or not. The problem is, some values
need to be encoded, and some should not be, but detecting the difference will require some
work.
To the parameter panel, I am thinking of adding a third and fourth column. The third column
will hold the value to actually be sent to the server (ie the encoded value). The fourth column
will hold a checkbox indicating whether URL encoding should be performed. Thus, you type
in the second column, and the encoded (or not encoded, depending) value appears in the
third column as you type. You'll also be able to edit the third column directly, to maximize
your opportunities to screw things up.
Um, I might just make a whole different GUI for all this - call it the "Advanced" HTTP Request,
or something... Any suggestions would be greatly appreciated :-)
-Mike
On 13 Aug 2002 at 0:07, Eric Siegerman wrote:
> Ok, so this business of the HTTP proxy URL-encoding stuff is
> biting me now.
>
> Why does this happen? It seems to me that the proxy should pass
> along exactly what it got, unless there's a good reason to do
> otherwise. That invites the question: is there a good reason
> for URL-encoding the data? What purpose is served by modifying
> it, in however supposedly-benign a way?
>
> On Fri, Aug 09, 2002 at 10:25:30AM -0400, Mike Stover wrote (over on jmeter-dev):
> > It seems that what is needed is a new type of proxy server that generates SOAP
> > Samplers, instead of normal HTTP Samplers. Not that your communications are
> > using SOAP necessarily, but the form is the same (XML through HTTP).
>
> I wasn't following the quoted thread closely, but this suggestion
> seems to be a response to someone else's URL-encoding problems.
> If so, it wouldn't help me. My problem is not with XML-formatted
> data, but with bog-standard HTML forms. My browser encodes a
> particular form as "multipart/form-data", but JMeter URL-encodes
> the whole request body (all of its MIME parts crunched into one
> huge long line) and zaps the encoding to
> "application/x-www-form-urlencoded". Not surprisingly, the app
> server hasn't a clue what to do with this.
>
> Of course, a special case could be added for
> "multipart/form-data" forms too (perhaps not as visibly as a
> separate subclass). But better than multiplying kludges, it
> seems to me, would be to just make the existing proxy
> transparent.
>
> Here are the ways in which the proxy incorrectly modifies the
> data (perhaps only a partial list; this is based only on the form
> submission that's causing me grief):
> - form body munging as described above
> - the response headers are changed to be terminated by LF;
> they're supposed to be CRLF's, and are before the proxy gets
> hold of them [1]
>
> There are other modifications, but they're all correct -- either
> necessary, or else unnecessary but innocuous (e.g. reordering
> headers). If a list of these would be useful, I can post one.
>
> I'm willing to take a quick stab at this (but haven't time for
> more, so I can't promise anything). But before I do, I'd like to
> know that I'm not going off on the wrong track. So, is there a
> good reason for the current behaviour, or is it a bug /
> implementation artifact?
>
> Thanks much.
>
>
> [1] "[...] a bare CR or LF MUST NOT be substituted for CRLF
> within any of the HTTP control structures (such as header
> fields and multipart boundaries)."
> - RFC 2068, p. 25, emphasis in original
>
>
> P.S. Sorry if this sounds a bit snarky. Normally I'd hold onto
> it overnight and revise it in the morning. But deadline pressure
> kind of prevents that :-(
>
> --
>
> | | /\
> |-_|/ > Eric Siegerman, Toronto, Ont. erics@telepres.com
> | | /
> Anyone who swims with the current will reach the big music steamship;
> whoever swims against the current will perhaps reach the source.
> - Paul Schneider-Esleben
>
> --
> To unsubscribe, e-mail: <ma...@jakarta.apache.org>
> For additional commands, e-mail: <ma...@jakarta.apache.org>
>
--
Michael Stover
mstover1@apache.org
Yahoo IM: mstover_ya
ICQ: 152975688
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>