You are viewing a plain text version of this content. The canonical link for it is here.
Posted to httpclient-users@hc.apache.org by R Smith <ry...@gmail.com> on 2013/05/01 15:58:29 UTC
Proxy chaining
Hello,
I have a question on proxy chaining.
I know that httpclient supports a single hop proxy, (Source -> P1 ->
Target) out of the box by using TUNNEL_TARGET. However, when I provide an
extra proxy in the chain using a custom HttpRoutePlanner#determineRoute(),
(Source -> P0 -> P1 -> Target) I use TUNNEL_PROXY and need to implement a
custom DefaultRequestDirector#createTunnelToProxy(). Im just not sure what
all should be done in this method. My route object looks correct, I can
see both proxies and target but Im just not sure if I need to make a
connection to the 1st proxy using a custom HttpConnect method object. I
noticed there is no Http "CONNECT" method, there is HttpGet, HttpPost and
HttpDelete. Should I create my own HttpConnect object for proxy chaining
or can I accomplish the "Connect" command on the 1st proxy in the chain
some other way?
Any example code or advice would be appreicated, thank you.
-Ryan
Re: Proxy chaining
Posted by arcivanov <ar...@ivanov.biz>.
Oleg Kalnichevski wrote
> On Tue, 2014-06-17 at 20:44 -0700, arcivanov wrote:
>> Oleg Kalnichevski wrote
>> > Unfortunately at this point there is no way around forking HttpClient
>> if
>> > you want to implement proxy chaining. We can however consider making it
>> > possible to replace MainClientExec with a custom implementation.
>> >
>> > Oleg
>>
>> Thanks Oleg, fair enough. Would you consider committing the below patch
>> to
>> improve extensibility? Criticism is, obviously, welcome. :)
>>
>> httpclient.patch
>> <http://httpcomponents.10934.n7.nabble.com/file/n23646/httpclient.patch>
>>
>> - Arcadiy
>>
>
> Arcadiy
>
> Could you please raise a change request in JIRA and attach the patch to
> it?
>
> As far as the patch goes I would rather avoid creating additional
> interfaces that are unlikely to be used often. A non-final protected
> factory method in HttpClientBuilder should be sufficient in my opinion.
>
> Oleg
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:
> httpclient-users-unsubscribe@.apache
> For additional commands, e-mail:
> httpclient-users-help@.apache
Good point. Jira issue HTTPCLIENT-1521 created
<https://issues.apache.org/jira/browse/HTTPCLIENT-1521> with attached
patched modified according to your recommendations.
Thanks a lot for your help!
--
View this message in context: http://httpcomponents.10934.n7.nabble.com/Proxy-chaining-tp19965p23658.html
Sent from the HttpClient-User mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: Proxy chaining
Posted by Oleg Kalnichevski <ol...@apache.org>.
On Tue, 2014-06-17 at 20:44 -0700, arcivanov wrote:
> Oleg Kalnichevski wrote
> > Unfortunately at this point there is no way around forking HttpClient if
> > you want to implement proxy chaining. We can however consider making it
> > possible to replace MainClientExec with a custom implementation.
> >
> > Oleg
>
> Thanks Oleg, fair enough. Would you consider committing the below patch to
> improve extensibility? Criticism is, obviously, welcome. :)
>
> httpclient.patch
> <http://httpcomponents.10934.n7.nabble.com/file/n23646/httpclient.patch>
>
> - Arcadiy
>
Arcadiy
Could you please raise a change request in JIRA and attach the patch to
it?
As far as the patch goes I would rather avoid creating additional
interfaces that are unlikely to be used often. A non-final protected
factory method in HttpClientBuilder should be sufficient in my opinion.
Oleg
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: Proxy chaining
Posted by arcivanov <ar...@ivanov.biz>.
Oleg Kalnichevski wrote
> Unfortunately at this point there is no way around forking HttpClient if
> you want to implement proxy chaining. We can however consider making it
> possible to replace MainClientExec with a custom implementation.
>
> Oleg
Thanks Oleg, fair enough. Would you consider committing the below patch to
improve extensibility? Criticism is, obviously, welcome. :)
httpclient.patch
<http://httpcomponents.10934.n7.nabble.com/file/n23646/httpclient.patch>
- Arcadiy
--
View this message in context: http://httpcomponents.10934.n7.nabble.com/Proxy-chaining-tp19965p23646.html
Sent from the HttpClient-User mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: Proxy chaining
Posted by Oleg Kalnichevski <ol...@apache.org>.
On Mon, 2014-06-16 at 11:32 -0700, arcivanov wrote:
> Oleg Kalnichevski wrote
> > On Sun, 2014-06-15 at 18:24 -0700, arcivanov wrote:
> >> Hi Oleg,
> >>
> >> RequestDirector and respectively DefaultRequestDirector have been
> >> deprecated
> >> as of 4.3 with no replacement suggested/provided.
> >
> > That bit of logic got moved to MainClientExec
> >
> > http://hc.apache.org/httpcomponents-client-4.3.x/httpclient/xref/org/apache/http/impl/execchain/MainClientExec.html#346
>
> Thanks a lot, that's what I thought, however I'm somewhat lost looking at
> how to extend this class, without actually trying to branch the
> HttpComponents, but rather introducing a new proxy provider behavior. My
> problem is as follows:
>
> * MainClientExec.establishRoute is package-private, so I would have to place
> my overriding components in impl.execchain and I'd have to create bundle
> patches given that I'm using OSGI.
> * MainClientExec.createTunnelToProxy is private and cannot be overwritten at
> all
> * MainClientExec is hard-instantiated in
> http://hc.apache.org/httpcomponents-client-4.3.x/httpclient/xref/org/apache/http/impl/client/HttpClientBuilder.html#787
> - it's not a pluggable service. I could override
> HttpClientBuilder.decorateMainExec and created some decorator ClientExec but
> for that I would need to copy all the logic from the MainClientExec related
> to route handling and delegate the rest - seems quite cumbersome and would
> require quite a lot of validation testing.
>
> Am I missing something major?
>
> Thanks for your help!
>
> - Arcadiy
>
Hi Arcadiy
Unfortunately at this point there is no way around forking HttpClient if
you want to implement proxy chaining. We can however consider making it
possible to replace MainClientExec with a custom implementation.
Oleg
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: Proxy chaining
Posted by arcivanov <ar...@ivanov.biz>.
Oleg Kalnichevski wrote
> On Sun, 2014-06-15 at 18:24 -0700, arcivanov wrote:
>> Hi Oleg,
>>
>> RequestDirector and respectively DefaultRequestDirector have been
>> deprecated
>> as of 4.3 with no replacement suggested/provided.
>
> That bit of logic got moved to MainClientExec
>
> http://hc.apache.org/httpcomponents-client-4.3.x/httpclient/xref/org/apache/http/impl/execchain/MainClientExec.html#346
Thanks a lot, that's what I thought, however I'm somewhat lost looking at
how to extend this class, without actually trying to branch the
HttpComponents, but rather introducing a new proxy provider behavior. My
problem is as follows:
* MainClientExec.establishRoute is package-private, so I would have to place
my overriding components in impl.execchain and I'd have to create bundle
patches given that I'm using OSGI.
* MainClientExec.createTunnelToProxy is private and cannot be overwritten at
all
* MainClientExec is hard-instantiated in
http://hc.apache.org/httpcomponents-client-4.3.x/httpclient/xref/org/apache/http/impl/client/HttpClientBuilder.html#787
- it's not a pluggable service. I could override
HttpClientBuilder.decorateMainExec and created some decorator ClientExec but
for that I would need to copy all the logic from the MainClientExec related
to route handling and delegate the rest - seems quite cumbersome and would
require quite a lot of validation testing.
Am I missing something major?
Thanks for your help!
- Arcadiy
--
View this message in context: http://httpcomponents.10934.n7.nabble.com/Proxy-chaining-tp19965p23630.html
Sent from the HttpClient-User mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: Proxy chaining
Posted by Oleg Kalnichevski <ol...@apache.org>.
On Sun, 2014-06-15 at 18:24 -0700, arcivanov wrote:
> Hi Oleg,
>
> RequestDirector and respectively DefaultRequestDirector have been deprecated
> as of 4.3 with no replacement suggested/provided.
>
> We need to implement chained proxies with SOCKS5 support and it would be
> great if you could briefly outline what path should be taken under your
> architectural vision in order to accomplish that.
>
> Thank you!
>
That bit of logic got moved to MainClientExec
http://hc.apache.org/httpcomponents-client-4.3.x/httpclient/xref/org/apache/http/impl/execchain/MainClientExec.html#346
Hope this helps
Oleg
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: Proxy chaining
Posted by arcivanov <ar...@ivanov.biz>.
Hi Oleg,
RequestDirector and respectively DefaultRequestDirector have been deprecated
as of 4.3 with no replacement suggested/provided.
We need to implement chained proxies with SOCKS5 support and it would be
great if you could briefly outline what path should be taken under your
architectural vision in order to accomplish that.
Thank you!
--
View this message in context: http://httpcomponents.10934.n7.nabble.com/Proxy-chaining-tp19965p23618.html
Sent from the HttpClient-User mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: Proxy chaining
Posted by galengt <ga...@talentbin.com>.
Hi Ryan,
That is the strange thing, I can connect with telnet to both. Also, I have
been testing with a unix utility called proxychains and I am able to chain
them together with that. With HttpClient, I can use either proxy as a one
hop, but it doesn't work as a two hop regardless of the order of the
proxies. The proxies I am testing with are just http proxies from one of
the various sites online that list them.
On Fri, Jun 6, 2014 at 4:40 AM, R Smith [via HttpComponents] <
ml-node+s10934n23558h38@n7.nabble.com> wrote:
> It looks like your tunnel proxy might not allow "CONNECT" statements. You
> should test first manually, using a telnet client. If it works in telnet,
> it should work in httpclient. hth.
>
>
> On Thu, Jun 5, 2014 at 9:01 PM, galengt <[hidden email]
> <http://user/SendEmail.jtp?type=node&node=23558&i=0>> wrote:
>
> > Hi Ryan and Oleg,
> >
> > Thanks for this thread, has been very helpful. Like Ryan I have 4.2
> locally
> > and have overwritten createTunnelToProxy, but I think I am missing
> > something.
> >
> > Execution looks like this:
> > DefaultHttpClient client = new DefaultHttpClient();
> > HttpRoutePlanner routePlanner = new TwoHopRoutePlanner(host, port,
> host2,
> > port2);
> > client.setRoutePlanner(routePlanner);
> > client.execute(get)
> >
> > TwoHopRoutePlanner is super simple, implements HttpRoutePlanner and has
> the
> > override below:
> > @Override public HttpRoute determineRoute(HttpHost target,
> > HttpRequest request, HttpContext context) throws HttpException {
> > HttpHost proxy1 = new HttpHost(this.host1, this.port1);
> > HttpHost proxy2 = new HttpHost(this.host2, this.port2);
> > HttpHost[] proxies = new HttpHost[] {proxy1, proxy2};
> > return new HttpRoute(target, null, proxies, false,
> > TunnelType.TUNNELLED, LayerType.PLAIN);
> > }
> > }
> >
> >
> > Here is my attempt at createTunnelToProxy:
> >
> > protected boolean createTunnelToProxy(HttpRoute route, int hop,
> > HttpContext context) throws HttpException, IOException {
> > //final HttpHost proxy = route.getProxyHost();
> > final HttpHost target = route.getTargetHost();
> > HttpResponse response = null;
> > for (int i = 0; i < route.getHopCount(); i++) {
> > final HttpHost proxy = route.getHopTarget(i);
> > if (!this.managedConn.isOpen()) {
> > this.managedConn.open(route, context, this.params);
> > }
> >
> > final HttpRequest connect = createConnectRequest(route,
> > context);
> > connect.setParams(this.params);
> >
> > // Populate the execution context
> > context.setAttribute(ExecutionContext.HTTP_TARGET_HOST,
> > target);
> > context.setAttribute(ExecutionContext.HTTP_PROXY_HOST,
> proxy);
> > context.setAttribute(ExecutionContext.HTTP_CONNECTION,
> > managedConn);
> > context.setAttribute(ExecutionContext.HTTP_REQUEST,
> connect);
> >
> > this.requestExec.preProcess(connect, this.httpProcessor,
> > context);
> >
> > System.out.println("CONNECT Issued: " + "host:" +
> > proxy.getHostName() + " port: " + proxy.getPort() + " scheme: " +
> > proxy.getSchemeName());
> > response = this.requestExec.execute(connect,
> this.managedConn,
> > context);
> >
> > response.setParams(this.params);
> > this.requestExec.postProcess(response, this.httpProcessor,
> > context);
> >
> > final int status = response.getStatusLine().getStatusCode();
> > System.out.println("CONNECT status code: " + status);
> > if (status < 200) {
> > throw new HttpException("Unexpected response to CONNECT
> > request: " + response.getStatusLine());
> > }
> > }
> > final int status = response.getStatusLine().getStatusCode();
> > if (status > 299) {
> >
> > // Buffer response content
> > final HttpEntity entity = response.getEntity();
> > if (entity != null) {
> > response.setEntity(new BufferedHttpEntity(entity));
> > }
> >
> > this.managedConn.close();
> > throw new TunnelRefusedException("CONNECT refused by proxy:
> " +
> > response.getStatusLine(), response);
> > }
> >
> > this.managedConn.markReusable();
> > return false;
> > }
> >
> >
> > I added a couple printlns in there, here is an example output:
> > CONNECT Issued: host:119.2.41.98 port: 8080 scheme: http
> > CONNECT status code: 200
> > CONNECT Issued: host:119.2.41.98 port: 8080 scheme: http
> > CONNECT status code: 405
> > CONNECT Issued: host:www.xhaus.com port: -1 scheme: http
> > CONNECT status code: 405
> >
> > The body comes back with an error that includes: "The requested method
> > CONNECT is not allowed for the URL /index.html". I am guessing I missed
> > something in createTunnelToProxy, any help would be appreciated!
> >
> >
> >
> > --
> > View this message in context:
> >
> http://httpcomponents.10934.n7.nabble.com/Proxy-chaining-tp19965p23546.html
> > Sent from the HttpClient-User mailing list archive at Nabble.com.
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: [hidden email]
> <http://user/SendEmail.jtp?type=node&node=23558&i=1>
> > For additional commands, e-mail: [hidden email]
> <http://user/SendEmail.jtp?type=node&node=23558&i=2>
> >
> >
>
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
> http://httpcomponents.10934.n7.nabble.com/Proxy-chaining-tp19965p23558.html
> To unsubscribe from Proxy chaining, click here
> <http://httpcomponents.10934.n7.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=19965&code=Z2FsZW5AdGFsZW50YmluLmNvbXwxOTk2NXwxNTA1MDEwNDM=>
> .
> NAML
> <http://httpcomponents.10934.n7.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
--
View this message in context: http://httpcomponents.10934.n7.nabble.com/Proxy-chaining-tp19965p23563.html
Sent from the HttpClient-User mailing list archive at Nabble.com.
Re: Proxy chaining
Posted by R Smith <ry...@gmail.com>.
It looks like your tunnel proxy might not allow "CONNECT" statements. You
should test first manually, using a telnet client. If it works in telnet,
it should work in httpclient. hth.
On Thu, Jun 5, 2014 at 9:01 PM, galengt <ga...@talentbin.com> wrote:
> Hi Ryan and Oleg,
>
> Thanks for this thread, has been very helpful. Like Ryan I have 4.2 locally
> and have overwritten createTunnelToProxy, but I think I am missing
> something.
>
> Execution looks like this:
> DefaultHttpClient client = new DefaultHttpClient();
> HttpRoutePlanner routePlanner = new TwoHopRoutePlanner(host, port, host2,
> port2);
> client.setRoutePlanner(routePlanner);
> client.execute(get)
>
> TwoHopRoutePlanner is super simple, implements HttpRoutePlanner and has the
> override below:
> @Override public HttpRoute determineRoute(HttpHost target,
> HttpRequest request, HttpContext context) throws HttpException {
> HttpHost proxy1 = new HttpHost(this.host1, this.port1);
> HttpHost proxy2 = new HttpHost(this.host2, this.port2);
> HttpHost[] proxies = new HttpHost[] {proxy1, proxy2};
> return new HttpRoute(target, null, proxies, false,
> TunnelType.TUNNELLED, LayerType.PLAIN);
> }
> }
>
>
> Here is my attempt at createTunnelToProxy:
>
> protected boolean createTunnelToProxy(HttpRoute route, int hop,
> HttpContext context) throws HttpException, IOException {
> //final HttpHost proxy = route.getProxyHost();
> final HttpHost target = route.getTargetHost();
> HttpResponse response = null;
> for (int i = 0; i < route.getHopCount(); i++) {
> final HttpHost proxy = route.getHopTarget(i);
> if (!this.managedConn.isOpen()) {
> this.managedConn.open(route, context, this.params);
> }
>
> final HttpRequest connect = createConnectRequest(route,
> context);
> connect.setParams(this.params);
>
> // Populate the execution context
> context.setAttribute(ExecutionContext.HTTP_TARGET_HOST,
> target);
> context.setAttribute(ExecutionContext.HTTP_PROXY_HOST, proxy);
> context.setAttribute(ExecutionContext.HTTP_CONNECTION,
> managedConn);
> context.setAttribute(ExecutionContext.HTTP_REQUEST, connect);
>
> this.requestExec.preProcess(connect, this.httpProcessor,
> context);
>
> System.out.println("CONNECT Issued: " + "host:" +
> proxy.getHostName() + " port: " + proxy.getPort() + " scheme: " +
> proxy.getSchemeName());
> response = this.requestExec.execute(connect, this.managedConn,
> context);
>
> response.setParams(this.params);
> this.requestExec.postProcess(response, this.httpProcessor,
> context);
>
> final int status = response.getStatusLine().getStatusCode();
> System.out.println("CONNECT status code: " + status);
> if (status < 200) {
> throw new HttpException("Unexpected response to CONNECT
> request: " + response.getStatusLine());
> }
> }
> final int status = response.getStatusLine().getStatusCode();
> if (status > 299) {
>
> // Buffer response content
> final HttpEntity entity = response.getEntity();
> if (entity != null) {
> response.setEntity(new BufferedHttpEntity(entity));
> }
>
> this.managedConn.close();
> throw new TunnelRefusedException("CONNECT refused by proxy: " +
> response.getStatusLine(), response);
> }
>
> this.managedConn.markReusable();
> return false;
> }
>
>
> I added a couple printlns in there, here is an example output:
> CONNECT Issued: host:119.2.41.98 port: 8080 scheme: http
> CONNECT status code: 200
> CONNECT Issued: host:119.2.41.98 port: 8080 scheme: http
> CONNECT status code: 405
> CONNECT Issued: host:www.xhaus.com port: -1 scheme: http
> CONNECT status code: 405
>
> The body comes back with an error that includes: "The requested method
> CONNECT is not allowed for the URL /index.html". I am guessing I missed
> something in createTunnelToProxy, any help would be appreciated!
>
>
>
> --
> View this message in context:
> http://httpcomponents.10934.n7.nabble.com/Proxy-chaining-tp19965p23546.html
> Sent from the HttpClient-User mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
Re: Proxy chaining
Posted by galengt <ga...@talentbin.com>.
Hi Ryan and Oleg,
Thanks for this thread, has been very helpful. Like Ryan I have 4.2 locally
and have overwritten createTunnelToProxy, but I think I am missing
something.
Execution looks like this:
DefaultHttpClient client = new DefaultHttpClient();
HttpRoutePlanner routePlanner = new TwoHopRoutePlanner(host, port, host2,
port2);
client.setRoutePlanner(routePlanner);
client.execute(get)
TwoHopRoutePlanner is super simple, implements HttpRoutePlanner and has the
override below:
@Override public HttpRoute determineRoute(HttpHost target,
HttpRequest request, HttpContext context) throws HttpException {
HttpHost proxy1 = new HttpHost(this.host1, this.port1);
HttpHost proxy2 = new HttpHost(this.host2, this.port2);
HttpHost[] proxies = new HttpHost[] {proxy1, proxy2};
return new HttpRoute(target, null, proxies, false,
TunnelType.TUNNELLED, LayerType.PLAIN);
}
}
Here is my attempt at createTunnelToProxy:
protected boolean createTunnelToProxy(HttpRoute route, int hop,
HttpContext context) throws HttpException, IOException {
//final HttpHost proxy = route.getProxyHost();
final HttpHost target = route.getTargetHost();
HttpResponse response = null;
for (int i = 0; i < route.getHopCount(); i++) {
final HttpHost proxy = route.getHopTarget(i);
if (!this.managedConn.isOpen()) {
this.managedConn.open(route, context, this.params);
}
final HttpRequest connect = createConnectRequest(route,
context);
connect.setParams(this.params);
// Populate the execution context
context.setAttribute(ExecutionContext.HTTP_TARGET_HOST, target);
context.setAttribute(ExecutionContext.HTTP_PROXY_HOST, proxy);
context.setAttribute(ExecutionContext.HTTP_CONNECTION,
managedConn);
context.setAttribute(ExecutionContext.HTTP_REQUEST, connect);
this.requestExec.preProcess(connect, this.httpProcessor,
context);
System.out.println("CONNECT Issued: " + "host:" +
proxy.getHostName() + " port: " + proxy.getPort() + " scheme: " +
proxy.getSchemeName());
response = this.requestExec.execute(connect, this.managedConn,
context);
response.setParams(this.params);
this.requestExec.postProcess(response, this.httpProcessor,
context);
final int status = response.getStatusLine().getStatusCode();
System.out.println("CONNECT status code: " + status);
if (status < 200) {
throw new HttpException("Unexpected response to CONNECT
request: " + response.getStatusLine());
}
}
final int status = response.getStatusLine().getStatusCode();
if (status > 299) {
// Buffer response content
final HttpEntity entity = response.getEntity();
if (entity != null) {
response.setEntity(new BufferedHttpEntity(entity));
}
this.managedConn.close();
throw new TunnelRefusedException("CONNECT refused by proxy: " +
response.getStatusLine(), response);
}
this.managedConn.markReusable();
return false;
}
I added a couple printlns in there, here is an example output:
CONNECT Issued: host:119.2.41.98 port: 8080 scheme: http
CONNECT status code: 200
CONNECT Issued: host:119.2.41.98 port: 8080 scheme: http
CONNECT status code: 405
CONNECT Issued: host:www.xhaus.com port: -1 scheme: http
CONNECT status code: 405
The body comes back with an error that includes: "The requested method
CONNECT is not allowed for the URL /index.html". I am guessing I missed
something in createTunnelToProxy, any help would be appreciated!
--
View this message in context: http://httpcomponents.10934.n7.nabble.com/Proxy-chaining-tp19965p23546.html
Sent from the HttpClient-User mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: Proxy chaining
Posted by R Smith <ry...@gmail.com>.
Thanks as always Oleg!
On Fri, May 3, 2013 at 1:05 PM, Oleg Kalnichevski <ol...@apache.org> wrote:
> On Fri, 2013-05-03 at 12:45 -0400, R Smith wrote:
> > Thanks Oleg. Great help. Ive been using the logic from
> > createTunnelToTarget() initially but wasnt sure if I needed a "CONNECT"
> > method. Now I see thats what the tunnelToTarget is doing, making a
> > connect. Thanks for pointing me back.
> >
> > A question about auth support for proxies, would it make more sense to
> fork
> > 4.3.x instead of 4.2.x if I wanted to attempt auth support for proxy
> chains?
> >
>
> Yes, absolutely. The most fundamental change that would have to make to
> achieve that is to support multiple proxy AuthStates, one per proxy hop.
>
> Oleg
>
>
> > Thanks again for all the help.
> > -Ryan
> >
> > On Fri, May 3, 2013 at 12:16 PM, Oleg Kalnichevski <ol...@apache.org>
> wrote:
> >
> > >
> > > On Thu, 2013-05-02 at 15:58 -0400, R Smith wrote:
> > > > Thanks Oleg. Thanks for the pointer on CONNECT. Makes sense. Ive
> forked
> > > > the httpclient 4.2.x branch locally so I have my own copy to work
> with. I
> > > > dont mind submitting patches if I have something usable and generic
> > > enough
> > > > when Im done.
> > > >
> > > > So am I correct in thinking I should still use a "CONNECT" method for
> > > > chaining but I should make HttpConnect() a private inner class of my
> > > > custom RequestDirector class so end users wont call it? Does
> HttpClient
> > > > use the "CONNECT" method anywhere internally that I am missing so I
> can
> > > see
> > > > it being used as an example?
> > > >
> > > > Just trying to figure out if Im barking up the right tree with the
> > > CONNECT
> > > > method for proxy chaining. I can proxy chain using command line
> telnet,
> > > so
> > > > I am just trying to recreate the same calls in httpclient at this
> point.
> > > >
> > > > Thanks again for all the help,
> > > > -Ryan
> > > >
> > > >
> > >
> > >
> > > Ryan
> > >
> > > I think you are not the right track. Have a look at the
> > > #createTunnelToTarget. It basically contains all the necessary
> machinery
> > > to establish a simple (one proxy hop) route. You will need to do
> > > something very similar once per proxy hop. Things will get _very_
> > > difficult though if you decide to support different authentication
> > > contexts per individual proxy hops. This will likely to require a major
> > > refactoring of authentication state management in HttpClient.
> > >
> > > Oleg
> > >
> > >
> > > >
> > > > On Thu, May 2, 2013 at 3:06 PM, Oleg Kalnichevski <ol...@apache.org>
> > > wrote:
> > > >
> > > > > On Wed, 2013-05-01 at 09:58 -0400, R Smith wrote:
> > > > > > Hello,
> > > > > >
> > > > > > I have a question on proxy chaining.
> > > > > >
> > > > > > I know that httpclient supports a single hop proxy, (Source ->
> P1 ->
> > > > > > Target) out of the box by using TUNNEL_TARGET. However, when I
> > > provide
> > > > > an
> > > > > > extra proxy in the chain using a custom
> > > > > HttpRoutePlanner#determineRoute(),
> > > > > > (Source -> P0 -> P1 -> Target) I use TUNNEL_PROXY and need to
> > > implement
> > > > > a
> > > > > > custom DefaultRequestDirector#createTunnelToProxy(). Im just not
> > > sure
> > > > > what
> > > > > > all should be done in this method. My route object looks
> correct, I
> > > can
> > > > > > see both proxies and target but Im just not sure if I need to
> make a
> > > > > > connection to the 1st proxy using a custom HttpConnect method
> > > object. I
> > > > > > noticed there is no Http "CONNECT" method, there is HttpGet,
> > > HttpPost and
> > > > > > HttpDelete. Should I create my own HttpConnect object for proxy
> > > chaining
> > > > > > or can I accomplish the "Connect" command on the 1st proxy in the
> > > chain
> > > > > > some other way?
> > > > > >
> > > > > > Any example code or advice would be appreicated, thank you.
> > > > > > -Ryan
> > > > >
> > > > > Ryan,
> > > > >
> > > > > HttpClient routing API is fully proxy chaining aware. However,
> request
> > > > > execution code presently only supports connection tunneling
> through a
> > > > > single proxy only. Protocol logic for proxy chaining still needs
> to be
> > > > > implemented.
> > > > >
> > > > > There is no class representing CONNECT method in HttpClient
> because it
> > > > > is a special purpose method and as such it should never be
> executed by
> > > > > the end users of HttpClient.
> > > > >
> > > > > Hope this makes some sense.
> > > > >
> > > > > Oleg
> > > > >
> > > > >
> > > > >
> ---------------------------------------------------------------------
> > > > > To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > > > > For additional commands, e-mail:
> httpclient-users-help@hc.apache.org
> > > > >
> > > > >
> > >
> > >
> > >
> > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> > >
> > >
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
Re: Proxy chaining
Posted by Oleg Kalnichevski <ol...@apache.org>.
On Fri, 2013-05-03 at 12:45 -0400, R Smith wrote:
> Thanks Oleg. Great help. Ive been using the logic from
> createTunnelToTarget() initially but wasnt sure if I needed a "CONNECT"
> method. Now I see thats what the tunnelToTarget is doing, making a
> connect. Thanks for pointing me back.
>
> A question about auth support for proxies, would it make more sense to fork
> 4.3.x instead of 4.2.x if I wanted to attempt auth support for proxy chains?
>
Yes, absolutely. The most fundamental change that would have to make to
achieve that is to support multiple proxy AuthStates, one per proxy hop.
Oleg
> Thanks again for all the help.
> -Ryan
>
> On Fri, May 3, 2013 at 12:16 PM, Oleg Kalnichevski <ol...@apache.org> wrote:
>
> >
> > On Thu, 2013-05-02 at 15:58 -0400, R Smith wrote:
> > > Thanks Oleg. Thanks for the pointer on CONNECT. Makes sense. Ive forked
> > > the httpclient 4.2.x branch locally so I have my own copy to work with. I
> > > dont mind submitting patches if I have something usable and generic
> > enough
> > > when Im done.
> > >
> > > So am I correct in thinking I should still use a "CONNECT" method for
> > > chaining but I should make HttpConnect() a private inner class of my
> > > custom RequestDirector class so end users wont call it? Does HttpClient
> > > use the "CONNECT" method anywhere internally that I am missing so I can
> > see
> > > it being used as an example?
> > >
> > > Just trying to figure out if Im barking up the right tree with the
> > CONNECT
> > > method for proxy chaining. I can proxy chain using command line telnet,
> > so
> > > I am just trying to recreate the same calls in httpclient at this point.
> > >
> > > Thanks again for all the help,
> > > -Ryan
> > >
> > >
> >
> >
> > Ryan
> >
> > I think you are not the right track. Have a look at the
> > #createTunnelToTarget. It basically contains all the necessary machinery
> > to establish a simple (one proxy hop) route. You will need to do
> > something very similar once per proxy hop. Things will get _very_
> > difficult though if you decide to support different authentication
> > contexts per individual proxy hops. This will likely to require a major
> > refactoring of authentication state management in HttpClient.
> >
> > Oleg
> >
> >
> > >
> > > On Thu, May 2, 2013 at 3:06 PM, Oleg Kalnichevski <ol...@apache.org>
> > wrote:
> > >
> > > > On Wed, 2013-05-01 at 09:58 -0400, R Smith wrote:
> > > > > Hello,
> > > > >
> > > > > I have a question on proxy chaining.
> > > > >
> > > > > I know that httpclient supports a single hop proxy, (Source -> P1 ->
> > > > > Target) out of the box by using TUNNEL_TARGET. However, when I
> > provide
> > > > an
> > > > > extra proxy in the chain using a custom
> > > > HttpRoutePlanner#determineRoute(),
> > > > > (Source -> P0 -> P1 -> Target) I use TUNNEL_PROXY and need to
> > implement
> > > > a
> > > > > custom DefaultRequestDirector#createTunnelToProxy(). Im just not
> > sure
> > > > what
> > > > > all should be done in this method. My route object looks correct, I
> > can
> > > > > see both proxies and target but Im just not sure if I need to make a
> > > > > connection to the 1st proxy using a custom HttpConnect method
> > object. I
> > > > > noticed there is no Http "CONNECT" method, there is HttpGet,
> > HttpPost and
> > > > > HttpDelete. Should I create my own HttpConnect object for proxy
> > chaining
> > > > > or can I accomplish the "Connect" command on the 1st proxy in the
> > chain
> > > > > some other way?
> > > > >
> > > > > Any example code or advice would be appreicated, thank you.
> > > > > -Ryan
> > > >
> > > > Ryan,
> > > >
> > > > HttpClient routing API is fully proxy chaining aware. However, request
> > > > execution code presently only supports connection tunneling through a
> > > > single proxy only. Protocol logic for proxy chaining still needs to be
> > > > implemented.
> > > >
> > > > There is no class representing CONNECT method in HttpClient because it
> > > > is a special purpose method and as such it should never be executed by
> > > > the end users of HttpClient.
> > > >
> > > > Hope this makes some sense.
> > > >
> > > > Oleg
> > > >
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > > > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> > > >
> > > >
> >
> >
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> >
> >
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: Proxy chaining
Posted by R Smith <ry...@gmail.com>.
Thanks Oleg. Great help. Ive been using the logic from
createTunnelToTarget() initially but wasnt sure if I needed a "CONNECT"
method. Now I see thats what the tunnelToTarget is doing, making a
connect. Thanks for pointing me back.
A question about auth support for proxies, would it make more sense to fork
4.3.x instead of 4.2.x if I wanted to attempt auth support for proxy chains?
Thanks again for all the help.
-Ryan
On Fri, May 3, 2013 at 12:16 PM, Oleg Kalnichevski <ol...@apache.org> wrote:
>
> On Thu, 2013-05-02 at 15:58 -0400, R Smith wrote:
> > Thanks Oleg. Thanks for the pointer on CONNECT. Makes sense. Ive forked
> > the httpclient 4.2.x branch locally so I have my own copy to work with. I
> > dont mind submitting patches if I have something usable and generic
> enough
> > when Im done.
> >
> > So am I correct in thinking I should still use a "CONNECT" method for
> > chaining but I should make HttpConnect() a private inner class of my
> > custom RequestDirector class so end users wont call it? Does HttpClient
> > use the "CONNECT" method anywhere internally that I am missing so I can
> see
> > it being used as an example?
> >
> > Just trying to figure out if Im barking up the right tree with the
> CONNECT
> > method for proxy chaining. I can proxy chain using command line telnet,
> so
> > I am just trying to recreate the same calls in httpclient at this point.
> >
> > Thanks again for all the help,
> > -Ryan
> >
> >
>
>
> Ryan
>
> I think you are not the right track. Have a look at the
> #createTunnelToTarget. It basically contains all the necessary machinery
> to establish a simple (one proxy hop) route. You will need to do
> something very similar once per proxy hop. Things will get _very_
> difficult though if you decide to support different authentication
> contexts per individual proxy hops. This will likely to require a major
> refactoring of authentication state management in HttpClient.
>
> Oleg
>
>
> >
> > On Thu, May 2, 2013 at 3:06 PM, Oleg Kalnichevski <ol...@apache.org>
> wrote:
> >
> > > On Wed, 2013-05-01 at 09:58 -0400, R Smith wrote:
> > > > Hello,
> > > >
> > > > I have a question on proxy chaining.
> > > >
> > > > I know that httpclient supports a single hop proxy, (Source -> P1 ->
> > > > Target) out of the box by using TUNNEL_TARGET. However, when I
> provide
> > > an
> > > > extra proxy in the chain using a custom
> > > HttpRoutePlanner#determineRoute(),
> > > > (Source -> P0 -> P1 -> Target) I use TUNNEL_PROXY and need to
> implement
> > > a
> > > > custom DefaultRequestDirector#createTunnelToProxy(). Im just not
> sure
> > > what
> > > > all should be done in this method. My route object looks correct, I
> can
> > > > see both proxies and target but Im just not sure if I need to make a
> > > > connection to the 1st proxy using a custom HttpConnect method
> object. I
> > > > noticed there is no Http "CONNECT" method, there is HttpGet,
> HttpPost and
> > > > HttpDelete. Should I create my own HttpConnect object for proxy
> chaining
> > > > or can I accomplish the "Connect" command on the 1st proxy in the
> chain
> > > > some other way?
> > > >
> > > > Any example code or advice would be appreicated, thank you.
> > > > -Ryan
> > >
> > > Ryan,
> > >
> > > HttpClient routing API is fully proxy chaining aware. However, request
> > > execution code presently only supports connection tunneling through a
> > > single proxy only. Protocol logic for proxy chaining still needs to be
> > > implemented.
> > >
> > > There is no class representing CONNECT method in HttpClient because it
> > > is a special purpose method and as such it should never be executed by
> > > the end users of HttpClient.
> > >
> > > Hope this makes some sense.
> > >
> > > Oleg
> > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> > >
> > >
>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
Re: Proxy chaining
Posted by Oleg Kalnichevski <ol...@apache.org>.
On Thu, 2013-05-02 at 15:58 -0400, R Smith wrote:
> Thanks Oleg. Thanks for the pointer on CONNECT. Makes sense. Ive forked
> the httpclient 4.2.x branch locally so I have my own copy to work with. I
> dont mind submitting patches if I have something usable and generic enough
> when Im done.
>
> So am I correct in thinking I should still use a "CONNECT" method for
> chaining but I should make HttpConnect() a private inner class of my
> custom RequestDirector class so end users wont call it? Does HttpClient
> use the "CONNECT" method anywhere internally that I am missing so I can see
> it being used as an example?
>
> Just trying to figure out if Im barking up the right tree with the CONNECT
> method for proxy chaining. I can proxy chain using command line telnet, so
> I am just trying to recreate the same calls in httpclient at this point.
>
> Thanks again for all the help,
> -Ryan
>
>
Ryan
I think you are not the right track. Have a look at the
#createTunnelToTarget. It basically contains all the necessary machinery
to establish a simple (one proxy hop) route. You will need to do
something very similar once per proxy hop. Things will get _very_
difficult though if you decide to support different authentication
contexts per individual proxy hops. This will likely to require a major
refactoring of authentication state management in HttpClient.
Oleg
>
> On Thu, May 2, 2013 at 3:06 PM, Oleg Kalnichevski <ol...@apache.org> wrote:
>
> > On Wed, 2013-05-01 at 09:58 -0400, R Smith wrote:
> > > Hello,
> > >
> > > I have a question on proxy chaining.
> > >
> > > I know that httpclient supports a single hop proxy, (Source -> P1 ->
> > > Target) out of the box by using TUNNEL_TARGET. However, when I provide
> > an
> > > extra proxy in the chain using a custom
> > HttpRoutePlanner#determineRoute(),
> > > (Source -> P0 -> P1 -> Target) I use TUNNEL_PROXY and need to implement
> > a
> > > custom DefaultRequestDirector#createTunnelToProxy(). Im just not sure
> > what
> > > all should be done in this method. My route object looks correct, I can
> > > see both proxies and target but Im just not sure if I need to make a
> > > connection to the 1st proxy using a custom HttpConnect method object. I
> > > noticed there is no Http "CONNECT" method, there is HttpGet, HttpPost and
> > > HttpDelete. Should I create my own HttpConnect object for proxy chaining
> > > or can I accomplish the "Connect" command on the 1st proxy in the chain
> > > some other way?
> > >
> > > Any example code or advice would be appreicated, thank you.
> > > -Ryan
> >
> > Ryan,
> >
> > HttpClient routing API is fully proxy chaining aware. However, request
> > execution code presently only supports connection tunneling through a
> > single proxy only. Protocol logic for proxy chaining still needs to be
> > implemented.
> >
> > There is no class representing CONNECT method in HttpClient because it
> > is a special purpose method and as such it should never be executed by
> > the end users of HttpClient.
> >
> > Hope this makes some sense.
> >
> > Oleg
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> >
> >
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: Proxy chaining
Posted by R Smith <ry...@gmail.com>.
Thanks Oleg. Thanks for the pointer on CONNECT. Makes sense. Ive forked
the httpclient 4.2.x branch locally so I have my own copy to work with. I
dont mind submitting patches if I have something usable and generic enough
when Im done.
So am I correct in thinking I should still use a "CONNECT" method for
chaining but I should make HttpConnect() a private inner class of my
custom RequestDirector class so end users wont call it? Does HttpClient
use the "CONNECT" method anywhere internally that I am missing so I can see
it being used as an example?
Just trying to figure out if Im barking up the right tree with the CONNECT
method for proxy chaining. I can proxy chain using command line telnet, so
I am just trying to recreate the same calls in httpclient at this point.
Thanks again for all the help,
-Ryan
On Thu, May 2, 2013 at 3:06 PM, Oleg Kalnichevski <ol...@apache.org> wrote:
> On Wed, 2013-05-01 at 09:58 -0400, R Smith wrote:
> > Hello,
> >
> > I have a question on proxy chaining.
> >
> > I know that httpclient supports a single hop proxy, (Source -> P1 ->
> > Target) out of the box by using TUNNEL_TARGET. However, when I provide
> an
> > extra proxy in the chain using a custom
> HttpRoutePlanner#determineRoute(),
> > (Source -> P0 -> P1 -> Target) I use TUNNEL_PROXY and need to implement
> a
> > custom DefaultRequestDirector#createTunnelToProxy(). Im just not sure
> what
> > all should be done in this method. My route object looks correct, I can
> > see both proxies and target but Im just not sure if I need to make a
> > connection to the 1st proxy using a custom HttpConnect method object. I
> > noticed there is no Http "CONNECT" method, there is HttpGet, HttpPost and
> > HttpDelete. Should I create my own HttpConnect object for proxy chaining
> > or can I accomplish the "Connect" command on the 1st proxy in the chain
> > some other way?
> >
> > Any example code or advice would be appreicated, thank you.
> > -Ryan
>
> Ryan,
>
> HttpClient routing API is fully proxy chaining aware. However, request
> execution code presently only supports connection tunneling through a
> single proxy only. Protocol logic for proxy chaining still needs to be
> implemented.
>
> There is no class representing CONNECT method in HttpClient because it
> is a special purpose method and as such it should never be executed by
> the end users of HttpClient.
>
> Hope this makes some sense.
>
> Oleg
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
Re: Proxy chaining
Posted by Oleg Kalnichevski <ol...@apache.org>.
On Wed, 2013-05-01 at 09:58 -0400, R Smith wrote:
> Hello,
>
> I have a question on proxy chaining.
>
> I know that httpclient supports a single hop proxy, (Source -> P1 ->
> Target) out of the box by using TUNNEL_TARGET. However, when I provide an
> extra proxy in the chain using a custom HttpRoutePlanner#determineRoute(),
> (Source -> P0 -> P1 -> Target) I use TUNNEL_PROXY and need to implement a
> custom DefaultRequestDirector#createTunnelToProxy(). Im just not sure what
> all should be done in this method. My route object looks correct, I can
> see both proxies and target but Im just not sure if I need to make a
> connection to the 1st proxy using a custom HttpConnect method object. I
> noticed there is no Http "CONNECT" method, there is HttpGet, HttpPost and
> HttpDelete. Should I create my own HttpConnect object for proxy chaining
> or can I accomplish the "Connect" command on the 1st proxy in the chain
> some other way?
>
> Any example code or advice would be appreicated, thank you.
> -Ryan
Ryan,
HttpClient routing API is fully proxy chaining aware. However, request
execution code presently only supports connection tunneling through a
single proxy only. Protocol logic for proxy chaining still needs to be
implemented.
There is no class representing CONNECT method in HttpClient because it
is a special purpose method and as such it should never be executed by
the end users of HttpClient.
Hope this makes some sense.
Oleg
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org