You are viewing a plain text version of this content. The canonical link for it is here.
Posted to httpclient-users@hc.apache.org by Steve Labarbera <st...@whitehatsec.com> on 2011/04/02 23:15:58 UTC
Trying to send raw requests without URL encodings?
I'm trying to send requests out without any URL encoding. This is necessary because we need to simulate sending requests bypassing the browser encoding the parameters. I have done a bunch of reading and it appears that one user mentioned implementing my own RequestEntity class. Is this the best option?
- Ideally, I need to be able to send anything in the url or post parameters without any URL encoding being done on them before being sent to the server.
- Would HttpClient be best approach or will I need to use a lower level library like HttpCore.
Any input that can help me move forward would be very helpful.
Thanks,
Steve
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: Trying to send raw requests without URL encodings?
Posted by Oleg Kalnichevski <ol...@apache.org>.
On Sun, 2011-04-03 at 08:22 -0700, Ken Krugler wrote:
> On Apr 3, 2011, at 8:14am, Sam Crawford wrote:
>
> > Steve,
> >
> > I've had a dig around the code this afternoon and understand the
> > issues you've run into. The java.net.URI class is fairly well embedded
> > throughout, and overriding HttpRequestBase.getRequestLine() alone will
> > not be enough.
> >
> > One thought: Have you considered copying and modifying the source for
> > java.net.URI to not throw the URISyntaxException? (Sadly you can't
> > just extend it, as it's declared final). Patching HttpClient /
> > HttpCore to use your new class would be very straightforward.
> >
> > You can take a look at some source for java.net.URI at
> > http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/6-b14/java/net/URI.java/?v=source
>
> Oddly enough, I was just looking at the same thing.
>
> I'd run into a similar issue, though with hostnames that are allowed by DNS servers but rejected by URI (e.g. "-sub.domain.com")
>
> One issue I notice is that you might need to handle is having these query parameters re-encoded when say URI.toURL() is called. E.g.
>
> java.net.URI olduri = new java.net.URI("http", "example.com", null, "var=<xss>", null);
> System.out.println(olduri.toURL());
>
> will print out "http://example.com?var=%3Cxss%3E"
Steve, Sam and Ken
There are several ways to approach the problem:
(1) HttpCore uses plain String to represent request-URIs. If core
functionality is enough, one can use just about anything for a
request-URI
(2) If giving up HttpClient if not an option, as long as a resource can
be represented by an java.net.URI instance, one can still convert it to
a different representation / format when writing the request out. One
would need a custom connection manager with a custom MessageWriter,
though.
Hope this helps
Oleg
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: Trying to send raw requests without URL encodings?
Posted by Ken Krugler <kk...@transpac.com>.
On Apr 3, 2011, at 8:14am, Sam Crawford wrote:
> Steve,
>
> I've had a dig around the code this afternoon and understand the
> issues you've run into. The java.net.URI class is fairly well embedded
> throughout, and overriding HttpRequestBase.getRequestLine() alone will
> not be enough.
>
> One thought: Have you considered copying and modifying the source for
> java.net.URI to not throw the URISyntaxException? (Sadly you can't
> just extend it, as it's declared final). Patching HttpClient /
> HttpCore to use your new class would be very straightforward.
>
> You can take a look at some source for java.net.URI at
> http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/6-b14/java/net/URI.java/?v=source
Oddly enough, I was just looking at the same thing.
I'd run into a similar issue, though with hostnames that are allowed by DNS servers but rejected by URI (e.g. "-sub.domain.com")
One issue I notice is that you might need to handle is having these query parameters re-encoded when say URI.toURL() is called. E.g.
java.net.URI olduri = new java.net.URI("http", "example.com", null, "var=<xss>", null);
System.out.println(olduri.toURL());
will print out "http://example.com?var=%3Cxss%3E"
-- Ken
> On 3 April 2011 06:00, Steve Labarbera <st...@whitehatsec.com> wrote:
>> Thanks for the reply but actually that was me asking that question a few months back. I got so frustrated I stepped away from the problem. Now I'm back determined to find a solution. As what was mentioned in that response the HttpCore is built on URI and I have not found a way to bypass the java.net.URISyntaxException: Illegal character in query at index exception that is thrown when illegal characters are added to the query.
>>
>> So, as an example, I need to send a request like http://example.com?var=<xss> <====== not url encoded
>>
>> Problem is no matter what I try the URI class throws an exception. I'm trying to avoid having to have to re-invent the wheel and build my own HTTPClient class.You think I have another option?
>>
>> Thanks in advance for the help,
>>
>> Steve
>>
>>
>>
>>
>>
>>
>>
>>
>> _____
>> From: Sam Crawford [samcrawford@gmail.com]
>> Sent: Saturday, April 02, 2011 3:00 PM
>> To: HttpClient User Discussion
>> Cc: Steve Labarbera
>> Subject: Re: Trying to send raw requests without URL encodings?
>>
>> Hi Steve,
>>
>> A very similar question was asked here:
>> http://old.nabble.com/Apache's-HttpClient-don't-want-URL-Encoding-any-ideas-how-I-can-bypass--td25762297.html
>>
>> This only deals with the request URL, but it would mean you don't need
>> to move to HttpCore.
>>
>> Implementing your own RequestEntity makes the most sense to me for
>> handling the POST scenario.
>>
>> Hope this helps,
>>
>> Sam
>>
>>
>>
>> On 2 April 2011 22:15, Steve Labarbera <st...@whitehatsec.com> wrote:
>>> I'm trying to send requests out without any URL encoding. This is necessary because we need to simulate sending requests bypassing the browser encoding the parameters. I have done a bunch of reading and it appears that one user mentioned implementing my own RequestEntity class. Is this the best option?
>>>
>>> - Ideally, I need to be able to send anything in the url or post parameters without any URL encoding being done on them before being sent to the server.
>>>
>>> - Would HttpClient be best approach or will I need to use a lower level library like HttpCore.
>>>
>>> Any input that can help me move forward would be very helpful.
>>>
>>> Thanks,
>>>
>>> Steve
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
>>> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>>>
>>>
>>
>> ________________________________________
>> From: Sam Crawford [samcrawford@gmail.com]
>> Sent: Saturday, April 02, 2011 3:00 PM
>> To: HttpClient User Discussion
>> Cc: Steve Labarbera
>> Subject: Re: Trying to send raw requests without URL encodings?
>>
>> Hi Steve,
>>
>> A very similar question was asked here:
>> http://old.nabble.com/Apache's-HttpClient-don't-want-URL-Encoding-any-ideas-how-I-can-bypass--td25762297.html
>>
>> This only deals with the request URL, but it would mean you don't need
>> to move to HttpCore.
>>
>> Implementing your own RequestEntity makes the most sense to me for
>> handling the POST scenario.
>>
>> Hope this helps,
>>
>> Sam
>>
>>
>>
>> On 2 April 2011 22:15, Steve Labarbera <st...@whitehatsec.com> wrote:
>>> I'm trying to send requests out without any URL encoding. This is necessary because we need to simulate sending requests bypassing the browser encoding the parameters. I have done a bunch of reading and it appears that one user mentioned implementing my own RequestEntity class. Is this the best option?
>>>
>>> - Ideally, I need to be able to send anything in the url or post parameters without any URL encoding being done on them before being sent to the server.
>>>
>>> - Would HttpClient be best approach or will I need to use a lower level library like HttpCore.
>>>
>>> Any input that can help me move forward would be very helpful.
>>>
>>> Thanks,
>>>
>>> Steve
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
>>> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>>>
>>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
--------------------------
Ken Krugler
+1 530-210-6378
http://bixolabs.com
e l a s t i c w e b m i n i n g
Re: Trying to send raw requests without URL encodings?
Posted by Sam Crawford <sa...@gmail.com>.
Steve,
I've had a dig around the code this afternoon and understand the
issues you've run into. The java.net.URI class is fairly well embedded
throughout, and overriding HttpRequestBase.getRequestLine() alone will
not be enough.
One thought: Have you considered copying and modifying the source for
java.net.URI to not throw the URISyntaxException? (Sadly you can't
just extend it, as it's declared final). Patching HttpClient /
HttpCore to use your new class would be very straightforward.
You can take a look at some source for java.net.URI at
http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/6-b14/java/net/URI.java/?v=source
Thanks,
Sam
On 3 April 2011 06:00, Steve Labarbera <st...@whitehatsec.com> wrote:
> Thanks for the reply but actually that was me asking that question a few months back. I got so frustrated I stepped away from the problem. Now I'm back determined to find a solution. As what was mentioned in that response the HttpCore is built on URI and I have not found a way to bypass the java.net.URISyntaxException: Illegal character in query at index exception that is thrown when illegal characters are added to the query.
>
> So, as an example, I need to send a request like http://example.com?var=<xss> <====== not url encoded
>
> Problem is no matter what I try the URI class throws an exception. I'm trying to avoid having to have to re-invent the wheel and build my own HTTPClient class.You think I have another option?
>
> Thanks in advance for the help,
>
> Steve
>
>
>
>
>
>
>
>
> _____
> From: Sam Crawford [samcrawford@gmail.com]
> Sent: Saturday, April 02, 2011 3:00 PM
> To: HttpClient User Discussion
> Cc: Steve Labarbera
> Subject: Re: Trying to send raw requests without URL encodings?
>
> Hi Steve,
>
> A very similar question was asked here:
> http://old.nabble.com/Apache's-HttpClient-don't-want-URL-Encoding-any-ideas-how-I-can-bypass--td25762297.html
>
> This only deals with the request URL, but it would mean you don't need
> to move to HttpCore.
>
> Implementing your own RequestEntity makes the most sense to me for
> handling the POST scenario.
>
> Hope this helps,
>
> Sam
>
>
>
> On 2 April 2011 22:15, Steve Labarbera <st...@whitehatsec.com> wrote:
>> I'm trying to send requests out without any URL encoding. This is necessary because we need to simulate sending requests bypassing the browser encoding the parameters. I have done a bunch of reading and it appears that one user mentioned implementing my own RequestEntity class. Is this the best option?
>>
>> - Ideally, I need to be able to send anything in the url or post parameters without any URL encoding being done on them before being sent to the server.
>>
>> - Would HttpClient be best approach or will I need to use a lower level library like HttpCore.
>>
>> Any input that can help me move forward would be very helpful.
>>
>> Thanks,
>>
>> Steve
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
>> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>>
>>
>
> ________________________________________
> From: Sam Crawford [samcrawford@gmail.com]
> Sent: Saturday, April 02, 2011 3:00 PM
> To: HttpClient User Discussion
> Cc: Steve Labarbera
> Subject: Re: Trying to send raw requests without URL encodings?
>
> Hi Steve,
>
> A very similar question was asked here:
> http://old.nabble.com/Apache's-HttpClient-don't-want-URL-Encoding-any-ideas-how-I-can-bypass--td25762297.html
>
> This only deals with the request URL, but it would mean you don't need
> to move to HttpCore.
>
> Implementing your own RequestEntity makes the most sense to me for
> handling the POST scenario.
>
> Hope this helps,
>
> Sam
>
>
>
> On 2 April 2011 22:15, Steve Labarbera <st...@whitehatsec.com> wrote:
>> I'm trying to send requests out without any URL encoding. This is necessary because we need to simulate sending requests bypassing the browser encoding the parameters. I have done a bunch of reading and it appears that one user mentioned implementing my own RequestEntity class. Is this the best option?
>>
>> - Ideally, I need to be able to send anything in the url or post parameters without any URL encoding being done on them before being sent to the server.
>>
>> - Would HttpClient be best approach or will I need to use a lower level library like HttpCore.
>>
>> Any input that can help me move forward would be very helpful.
>>
>> Thanks,
>>
>> Steve
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
>> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>>
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
RE: Trying to send raw requests without URL encodings?
Posted by Steve Labarbera <st...@whitehatsec.com>.
Thanks for the reply but actually that was me asking that question a few months back. I got so frustrated I stepped away from the problem. Now I'm back determined to find a solution. As what was mentioned in that response the HttpCore is built on URI and I have not found a way to bypass the java.net.URISyntaxException: Illegal character in query at index exception that is thrown when illegal characters are added to the query.
So, as an example, I need to send a request like http://example.com?var=<xss> <====== not url encoded
Problem is no matter what I try the URI class throws an exception. I'm trying to avoid having to have to re-invent the wheel and build my own HTTPClient class.You think I have another option?
Thanks in advance for the help,
Steve
_____
From: Sam Crawford [samcrawford@gmail.com]
Sent: Saturday, April 02, 2011 3:00 PM
To: HttpClient User Discussion
Cc: Steve Labarbera
Subject: Re: Trying to send raw requests without URL encodings?
Hi Steve,
A very similar question was asked here:
http://old.nabble.com/Apache's-HttpClient-don't-want-URL-Encoding-any-ideas-how-I-can-bypass--td25762297.html
This only deals with the request URL, but it would mean you don't need
to move to HttpCore.
Implementing your own RequestEntity makes the most sense to me for
handling the POST scenario.
Hope this helps,
Sam
On 2 April 2011 22:15, Steve Labarbera <st...@whitehatsec.com> wrote:
> I'm trying to send requests out without any URL encoding. This is necessary because we need to simulate sending requests bypassing the browser encoding the parameters. I have done a bunch of reading and it appears that one user mentioned implementing my own RequestEntity class. Is this the best option?
>
> - Ideally, I need to be able to send anything in the url or post parameters without any URL encoding being done on them before being sent to the server.
>
> - Would HttpClient be best approach or will I need to use a lower level library like HttpCore.
>
> Any input that can help me move forward would be very helpful.
>
> Thanks,
>
> Steve
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
________________________________________
From: Sam Crawford [samcrawford@gmail.com]
Sent: Saturday, April 02, 2011 3:00 PM
To: HttpClient User Discussion
Cc: Steve Labarbera
Subject: Re: Trying to send raw requests without URL encodings?
Hi Steve,
A very similar question was asked here:
http://old.nabble.com/Apache's-HttpClient-don't-want-URL-Encoding-any-ideas-how-I-can-bypass--td25762297.html
This only deals with the request URL, but it would mean you don't need
to move to HttpCore.
Implementing your own RequestEntity makes the most sense to me for
handling the POST scenario.
Hope this helps,
Sam
On 2 April 2011 22:15, Steve Labarbera <st...@whitehatsec.com> wrote:
> I'm trying to send requests out without any URL encoding. This is necessary because we need to simulate sending requests bypassing the browser encoding the parameters. I have done a bunch of reading and it appears that one user mentioned implementing my own RequestEntity class. Is this the best option?
>
> - Ideally, I need to be able to send anything in the url or post parameters without any URL encoding being done on them before being sent to the server.
>
> - Would HttpClient be best approach or will I need to use a lower level library like HttpCore.
>
> Any input that can help me move forward would be very helpful.
>
> Thanks,
>
> Steve
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: Trying to send raw requests without URL encodings?
Posted by Sam Crawford <sa...@gmail.com>.
Hi Steve,
A very similar question was asked here:
http://old.nabble.com/Apache's-HttpClient-don't-want-URL-Encoding-any-ideas-how-I-can-bypass--td25762297.html
This only deals with the request URL, but it would mean you don't need
to move to HttpCore.
Implementing your own RequestEntity makes the most sense to me for
handling the POST scenario.
Hope this helps,
Sam
On 2 April 2011 22:15, Steve Labarbera <st...@whitehatsec.com> wrote:
> I'm trying to send requests out without any URL encoding. This is necessary because we need to simulate sending requests bypassing the browser encoding the parameters. I have done a bunch of reading and it appears that one user mentioned implementing my own RequestEntity class. Is this the best option?
>
> - Ideally, I need to be able to send anything in the url or post parameters without any URL encoding being done on them before being sent to the server.
>
> - Would HttpClient be best approach or will I need to use a lower level library like HttpCore.
>
> Any input that can help me move forward would be very helpful.
>
> Thanks,
>
> Steve
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org