Tomcat as https proxy

[Alessandro Novarini <an...@gmail.com>]

Hello all,

My current client has web application deployed on tomcat that acts as an
http proxy recording all the responses from remote web services in order to
work "off-line" when needed.

The problem they asked me to solve is with a service that uses the https
protocol.
When I configure the application to talk via https to the "custom" proxy,
it gets in return a return code of 400 (Bad Request).

Enabling the access log on tomcat, done setting the valve for the host, I
can see that the verb used during the first call is CONNECT.
This is fine, as the https proxy should first ask the proxy for a port and
then tunnel the communication encrypted.

I tried to debug the HttpConnector and neighbourhood, and as far as I can
understand, the class org.apache.catalina.connector.CoyoteAdapter, gets an
url like www.gmail.com:443, that it can't normalize.

In other parts of the code (org.apache.coyote.http11.Http11Processor maybe?
I don't have the code with me right now) I see checks on the http verb used
in the communication, but it only checks whether this is GET or POST.

Is it correct to think that this kind of usage of tomcat is improper? To me
it would be better to add the custom behaviour to a stand-alone proxy
instead of trying to work (or even worse patching the code) with tomcat.

What do you think? Does anybody have some experience to share about it?

Thanks in advance
Alessandro Novarini

Re: Tomcat as https proxy

[Alessandro Novarini <an...@gmail.com>]

On 15/12/2011 16:46, Christopher Schultz wrote:
> Okay... just not when the client is making an HTTPS request because of 
> the non-HTTPS CONNECT request? 
Exactly
>> What we have done at the moment is to switch the https connection to 
>> the http one, and that works fine 
> But it doesn't produce the desired result :)
No, it doesn't, but for the client was ok as a temporary patch; this 
proxy is meant to work for development environments, as you can detach 
the application from all the remote services and work "offline".
> Honestly, I would look at this as my first option: if all goes well,
> you replace one component (your broken proxy) with another (a
> commercial-grade proxy) and you no longer have to maintain that proxy
> code yourself. You just need to learn how to configure and protect
> that new component.
Ok, it'll be difficult to push for that but it that makes sense.

Thanks a lot!
Alessandro

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org

Re: Tomcat as https proxy

[Christopher Schultz <ch...@christopherschultz.net>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Alessandro,

On 12/15/11 11:37 AM, Alessandro Novarini wrote:
> On Thu, Dec 15, 2011 at 4:15 PM, Christopher Schultz 
> <ch...@christopherschultz.net> wrote:
> 
>> No, when I said "server" I meant "server". If you are writing a
>> proxy, you really ought to understand these terms:
>> 
>> 1. User Agent - the ultimate client -- the one making the HTTP
>> request
>> 
>> 2. Proxy Server - the server you are babysitting
>> 
>> 3. Origin Server - the server that actually provides the true
>> response
> 
> Yes, probably, but in my case I have three servers, and with Origin
> I meant the one that originates the request. Sorry, lesson learnt

No, the origin server is the server that produces the response. It's
definitely a server, not a client. The process that originates the
request is called the client :)

>> Oh... you are using a SOCKS proxy? That's a different thing.
> 
> Well, yes, but I guess that when you set http.proxyHost and 
> https.proxyHost for the jvm, the same happens, or it doesn't?

I think that's what you are talking about.

> Not at all, the communication from the proxy to the https server
> works with no problem

Okay... just not when the client is making an HTTPS request because of
the non-HTTPS CONNECT request?

>> Is this something that you have written yourself, or are you
>> expecting Tomcat to handle CONNECT requests and then respond with
>> this secure port, etc. Tomcat generally doesn't bind to
>> additional ports on-the-fly.
> 
> Here you can find the specification, if you have time to give it a
> quick look 
> http://curl.haxx.se/rfc/draft-luotonen-web-proxy-tunneling-01.txt

Yeah, Tomcat isn't a good solution for this kind of thing. HTTP
proxying is not the same as TCP/IP tunneling, which is what the above
RFC is about. I think you're barking up the wrong tree.

>> Does this ever work? Or, are you saying that HTTPS does not work
>> but HTTP does?
> 
> What we have done at the moment is to switch the https connection
> to the http one, and that works fine

But it doesn't produce the desired result :)

>> Why? If the application speaks HTTP and can be configured with
>> an HTTP/S proxy, why not simply replace your Tomcat solution with
>> a "real" HTTP proxy that will do all this out of the box? Then
>> you won't have to babysit all that code anymore. :)
> 
> Yes, it's not the client application that will need to be
> partially rewritten, I was referring to the proxy, they need to
> change it moving from a web application to a bunch of
> listeners/plugins/whatever that can be connected to the proxy
> implementation.
> 
>> Hmm. I wonder if other commercial-grade proxies (seriously, look
>> into Squid, it might do what you want) can provide the type of
>> logging that you require.
> 
> Ok, I'll give it a try, but just only as a last resort.

Honestly, I would look at this as my first option: if all goes well,
you replace one component (your broken proxy) with another (a
commercial-grade proxy) and you no longer have to maintain that proxy
code yourself. You just need to learn how to configure and protect
that new component.

- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk7qJFQACgkQ9CaO5/Lv0PBcjQCghvFIX0WSW5HQIh6dQApNEWds
QsYAoIBTebEYOqb9zoRfZhUFmCJBLGs1
=taQj
-----END PGP SIGNATURE-----

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org

Re: Tomcat as https proxy

[Alessandro Novarini <an...@gmail.com>]

Chris,

On Thu, Dec 15, 2011 at 4:15 PM, Christopher Schultz
<ch...@christopherschultz.net> wrote:

> No, when I said "server" I meant "server". If you are writing a proxy,
> you really ought to understand these terms:
>
> 1. User Agent - the ultimate client -- the one making the HTTP request
>
> 2. Proxy Server - the server you are babysitting
>
> 3. Origin Server - the server that actually provides the true response

Yes, probably, but in my case I have three servers, and with Origin I
meant the one that originates the request. Sorry, lesson learnt

> Oh... you are using a SOCKS proxy? That's a different thing.

Well, yes, but I guess that when you set http.proxyHost and
https.proxyHost for the jvm, the same happens, or it doesn't?

>> They're using apache common http client.
>
> Okay. But that's not the problem, is it: the proxy and the origin
> server are communicating properly: it's the client that's getting a
> 400 Bad Request, right?

Not at all, the communication from the proxy to the https server works
with no problem

> Is this something that you have written yourself, or are you expecting
> Tomcat to handle CONNECT requests and then respond with this secure
> port, etc. Tomcat generally doesn't bind to additional ports on-the-fly.

Here you can find the specification, if you have time to give it a quick look
http://curl.haxx.se/rfc/draft-luotonen-web-proxy-tunneling-01.txt

> Does this ever work? Or, are you saying that HTTPS does not work but
> HTTP does?

What we have done at the moment is to switch the https connection to
the http one, and that works fine

> If I correctly understand what you are trying to do, my guess is that
> Tomcat itself would require significant changes to support these types
> of requests. Tomcat itself would have to act as a proxy server rather
> than delegating the request to a servlet that would do such things.
> The reason is that the request itself can't be properly mapped to a
> host/context/servlet unless Tomcat could be instructed to just ignore
> everything and pass it through to a single, deputized servlet that
> would essentially handle everything. At that point, Tomcat would be
> acting as a TCP/IP server and nothing more... you'd be wasting all the
> infrastructure that Tomcat provides.

Ok, this note has been saved for the record, thanks

> Yup, and probably none of them are running within a servlet container.

Indeed, none of the proxy servers I've seen so far work within a
servlet container.

> Why? If the application speaks HTTP and can be configured with an
> HTTP/S proxy, why not simply replace your Tomcat solution with a
> "real" HTTP proxy that will do all this out of the box? Then you won't
> have to babysit all that code anymore. :)

Yes, it's not the client application that will need to be partially
rewritten, I was referring to the proxy, they need to change it moving
from a web application to a bunch of listeners/plugins/whatever that
can be connected to the proxy implementation.

> Hmm. I wonder if other commercial-grade proxies (seriously, look into
> Squid, it might do what you want) can provide the type of logging that
> you require.

Ok, I'll give it a try, but just only as a last resort.

Thanks again
Alessandro

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org

Re: Tomcat as https proxy

[Christopher Schultz <ch...@christopherschultz.net>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Alessandro,

On 12/15/11 10:56 AM, Alessandro Novarini wrote:
> On Thu, Dec 15, 2011 at 3:07 PM, Christopher Schultz 
> <ch...@christopherschultz.net> wrote:
>> Is the Origin server also running Tomcat, or is this exclusively
>> a problem with the proxy?
> 
> If with Origin server you mean the "client", then yes

No, when I said "server" I meant "server". If you are writing a proxy,
you really ought to understand these terms:

1. User Agent - the ultimate client -- the one making the HTTP request

2. Proxy Server - the server you are babysitting

3. Origin Server - the server that actually provides the true response

> [The client is] another tomcat on another machine, but at the
> moment I'm using a browser configured to use as https proxy the
> tomcat with the java proxy installed.

Oh... you are using a SOCKS proxy? That's a different thing.

> If with Origin server you mean the "destination", then we don't
> really know what kind of servers are, they just exposes web
> services

Okay. I just wasn't sure if you meant that the request the proxy sends
to the origin was getting a 400 Bad Request or if it was the client's
request to the proxy that was getting a 400 Bad Request.

If you are expecting Tomcat to act as a SOCKS proxy, then I'm not sure
if that's possible or a good idea. I don't understand SOCKS well
enough to give you any advise.

>> How are you connecting from your proxy webapp to the origin
>> server?
> 
> They're using apache common http client.

Okay. But that's not the problem, is it: the proxy and the origin
server are communicating properly: it's the client that's getting a
400 Bad Request, right?

>> Does the request come to your proxy over HTTPS? Do you call-out
>> over HTTPS or is the request to the origin server over HTTP?
> 
> Well, it seems more complicated than that, as far as I know.
> Reading how the https proxy protocol works, I get that the first
> connection between source and proxy is in clear http, sending the
> CONNECT verb; then, the proxy replies with a secure port that the
> client uses to send the https requests.

Is this something that you have written yourself, or are you expecting
Tomcat to handle CONNECT requests and then respond with this secure
port, etc. Tomcat generally doesn't bind to additional ports on-the-fly.

> This is also the reason why you usually set the http and https
> proxy with same host and port, because the "real" secure
> communication is done over a port decided at communication time.
> 
>> The client gets 400 Bad Request? Can you post the complete HTTP 
>> conversation for this client->proxy request/response?
> 
> The client gets a 400 Bad Request, yes. This is what happens:
> 
> 
> CLIENT -> PROXY -------------------------------------- CONNECT
> home.netscape.com:443 HTTP/1.0 User-agent: Mozilla/4.0 <<< empty
> line >>>

I don't think that's a valid HTTP 1.0 request... the request headers
will be ignored IIRC.

> PROXY -> CLIENT -------------------------------------- HTTP/1.0 400
> Bad Request
> 
>> 
>> That URL is trivially normalizable (and not really complete, as
>> it has neither a protocol nor a path) -- Tomcat should have no
>> problem with that.
> 
> It should, I agree with you. I don't know why during the normalize
> it seems the code doesn't expect something with colons and port to
> me

Yeah, Tomcat has no idea what to do with that request. It's trying to
map home.netscape.com:443 to a host/context/servlet and the URL
doesn't even start with a '/'. I'm not surprised you are getting a 400
response.

Does this ever work? Or, are you saying that HTTPS does not work but
HTTP does?

If HTTP does work but HTTPS does not, I'm sure it's because Tomcat
does not understand the semantics of accepting proxy-style-HTTPS
requests in this way.

>> So, are you asking how to make it work inside of Tomcat or how to
>> make it work outside of Tomcat?
> 
> I'm asking whether there's any chance to make it work in tomcat, 
> before claiming that we need to switch to another kind of 
> architecture.

If I correctly understand what you are trying to do, my guess is that
Tomcat itself would require significant changes to support these types
of requests. Tomcat itself would have to act as a proxy server rather
than delegating the request to a servlet that would do such things.
The reason is that the request itself can't be properly mapped to a
host/context/servlet unless Tomcat could be instructed to just ignore
everything and pass it through to a single, deputized servlet that
would essentially handle everything. At that point, Tomcat would be
acting as a TCP/IP server and nothing more... you'd be wasting all the
infrastructure that Tomcat provides.

> Outside tomcat it seems so straightforward, there are lot of java 
> proxies that work with https flawlessly.

Yup, and probably none of them are running within a servlet container.

> The problem is that for telling the client this I'd like to be
> 100% sure, as this means rewrite basically at least on layer of
> the application.

Why? If the application speaks HTTP and can be configured with an
HTTP/S proxy, why not simply replace your Tomcat solution with a
"real" HTTP proxy that will do all this out of the box? Then you won't
have to babysit all that code anymore. :)

>> If you don't need Java for anything else (do you?), then why are
>> you using Tomcat at all? There are plenty of high-performance
>> proxy servers available like squid that will probably do this job
>> better.
> 
> No, we need Java because this proxy does also as a recorder and
> some other stuff, and here there are only java developers.

Hmm. I wonder if other commercial-grade proxies (seriously, look into
Squid, it might do what you want) can provide the type of logging that
you require.

- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk7qHTwACgkQ9CaO5/Lv0PBzAACfczABe1XXlDM0+nAs8FAtsqOe
Z58An3bRbJtJkmNAVmzqaHOkcWCIJNtG
=18EX
-----END PGP SIGNATURE-----

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org

Re: Tomcat as https proxy

[Alessandro Novarini <an...@gmail.com>]

Hi Chris,

On Thu, Dec 15, 2011 at 3:07 PM, Christopher Schultz
<ch...@christopherschultz.net> wrote:
>
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Alessandro,
>
> On 12/14/11 6:02 PM, Alessandro Novarini wrote:
> > My current client has web application deployed on tomcat that acts
> > as an http proxy recording all the responses from remote web
> > services in order to work "off-line" when needed.
>
> Is the Origin server also running Tomcat, or is this exclusively a
> problem with the proxy?


If with Origin server you mean the "client", then yes, it's another
tomcat on another machine, but at the moment I'm using a browser
configured to use as https proxy the tomcat with the java proxy
installed.
If with Origin server you mean the "destination", then we don't really
know what kind of servers are, they just exposes web services

>
> How are you connecting from your proxy webapp to the origin server?

They're using apache common http client.

>
> Does the request come to your proxy over HTTPS? Do you call-out over
> HTTPS or is the request to the origin server over HTTP?

Well, it seems more complicated than that, as far as I know. Reading
how the https proxy protocol works, I get that the first connection
between source and proxy is in clear http, sending the CONNECT verb;
then, the proxy replies with a secure port that the client uses to
send the https requests.
This is also the reason why you usually set the http and https proxy
with same host and port, because the "real" secure communication is
done over a port decided at communication time.

> The client gets 400 Bad Request? Can you post the complete HTTP
> conversation for this client->proxy request/response?

The client gets a 400 Bad Request, yes.
This is what happens:


CLIENT -> PROXY
--------------------------------------
CONNECT home.netscape.com:443 HTTP/1.0
User-agent: Mozilla/4.0
<<< empty line >>>

PROXY -> CLIENT
--------------------------------------
HTTP/1.0 400 Bad Request

>
> That URL is trivially normalizable (and not really complete, as it has
> neither a protocol nor a path) -- Tomcat should have no problem with that.

It should, I agree with you. I don't know why during the normalize it
seems the code doesn't expect something with colons and port to me

>
> > In other parts of the code
> > (org.apache.coyote.http11.Http11Processor maybe? I don't have the
> > code with me right now) I see checks on the http verb used in the
> > communication, but it only checks whether this is GET or POST.
>
> That will be only for certain operations where GET and POST are
> relevant. If you can't remember what you saw, please re-check it and
> post with more details.

In the Http11Processor I see this for example

MessageBytes methodMB = request.method();
if (methodMB.equals(Constants.GET)) {
    methodMB.setString(Constants.GET);
} else if (methodMB.equals(Constants.POST)) {
    methodMB.setString(Constants.POST);
}

and I guess some of the job is done by the method parseHost in the
same class, but I can't debug at the moment

> Making an HTTPS request to Tomcat is not unusual.

Yes, but I'm not making a direct https connection to tomcat, that's
the point. I already tried to enable https connection with
certificates, but it doesn't just work in that way.

> So, are you asking how to make it work inside of Tomcat or how to make
> it work outside of Tomcat?

I'm asking whether there's any chance to make it work in tomcat,
before claiming that we need to switch to another kind of
architecture.
Outside tomcat it seems so straightforward, there are lot of java
proxies that work with https flawlessly.
The problem is that for telling the client this I'd like to be 100%
sure, as this means rewrite basically at least on layer of the
application.

> If you don't need Java for anything else (do you?), then why are you
> using Tomcat at all? There are plenty of high-performance proxy
> servers available like squid that will probably do this job better.

No, we need Java because this proxy does also as a recorder and some
other stuff, and here there are only java developers.

> Nobody that I know of has bothered to write an HTTP proxy module for
> Tomcat probably because there are perfectly good solutions out there
> already that don't require Tomcat at all.

Unfortunately - for the client - I agree

Thanks for your time
Alessandro

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org

Re: Tomcat as https proxy

[Christopher Schultz <ch...@christopherschultz.net>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Alessandro,

On 12/14/11 6:02 PM, Alessandro Novarini wrote:
> My current client has web application deployed on tomcat that acts
> as an http proxy recording all the responses from remote web
> services in order to work "off-line" when needed.

Is the Origin server also running Tomcat, or is this exclusively a
problem with the proxy?

How are you connecting from your proxy webapp to the origin server?
Tomcat does not include any call-out-HTTP connectors so you must be
using either Java's URLConnection class or some kind of wrapper for it
(or, I suppose, something that is entirely separate, thought I'm not
sure why anyone would bother with that).

> The problem they asked me to solve is with a service that uses the
> https protocol.

Does the request come to your proxy over HTTPS? Do you call-out over
HTTPS or is the request to the origin server over HTTP?

> When I configure the application to talk via https to the "custom"
> proxy, it gets in return a return code of 400 (Bad Request).

The client gets 400 Bad Request? Can you post the complete HTTP
conversation for this client->proxy request/response?

> Enabling the access log on tomcat, done setting the valve for the
> host, I can see that the verb used during the first call is
> CONNECT. This is fine, as the https proxy should first ask the
> proxy for a port and then tunnel the communication encrypted.

Okay.

> I tried to debug the HttpConnector and neighbourhood, and as far as
> I can understand, the class
> org.apache.catalina.connector.CoyoteAdapter, gets an url like
> www.gmail.com:443, that it can't normalize.

That URL is trivially normalizable (and not really complete, as it has
neither a protocol nor a path) -- Tomcat should have no problem with that.

> In other parts of the code
> (org.apache.coyote.http11.Http11Processor maybe? I don't have the
> code with me right now) I see checks on the http verb used in the
> communication, but it only checks whether this is GET or POST.

That will be only for certain operations where GET and POST are
relevant. If you can't remember what you saw, please re-check it and
post with more details.

> Is it correct to think that this kind of usage of tomcat is
> improper?

Making an HTTPS request to Tomcat is not unusual.

> To me it would be better to add the custom behaviour to a
> stand-alone proxy instead of trying to work (or even worse patching
> the code) with tomcat.

So, are you asking how to make it work inside of Tomcat or how to make
it work outside of Tomcat?

If you don't need Java for anything else (do you?), then why are you
using Tomcat at all? There are plenty of high-performance proxy
servers available like squid that will probably do this job better.

> What do you think? Does anybody have some experience to share about
> it?

Nobody that I know of has bothered to write an HTTP proxy module for
Tomcat probably because there are perfectly good solutions out there
already that don't require Tomcat at all.

- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk7qDRUACgkQ9CaO5/Lv0PDVQgCeJt/A7VUaRn7k/8XGbGgABdu+
TrIAn3uWP+iugaQHtYK6UtnGUEebevwg
=zZ/M
-----END PGP SIGNATURE-----

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org