You are viewing a plain text version of this content. The canonical link for it is here.
Posted to httpclient-users@hc.apache.org by Doug Lochart <dl...@gmail.com> on 2006/08/30 20:07:03 UTC
Persistent connections ?
I have a few more simple questions:
1) Is this list still the appropriate place to ask HttpComponents questions
(vs HttpClient) and will there be a new List created for the HttpComponents
project?
I just finished reading most of the HTTP 1.1 RFC and I tried to pay
attention to the Persistent Connections as I was curious as to their impact
on my application.
So I have a simple proof-of-concept proxy server (based on the example
ElementalHttpServer) and it is receiving connections. I understand that
persistent connections are the default for all HTTP 1.1 communication. I
see the HttpCore code refer to keeping the connections alive (even debug
messages). So here are my questions ...
2) When I use Firefox to go grab a webpage that contains URI's to several
other resources (images, css files etc) Firefox seems to make a new
connection to port 8080 everytime and does not reuse the connection that it
originally established. To my knowlege I am not explicitly closing the
connection as I am expecting more connections to come. If you look at the
ElementalHttpServer code you will see pretty much exactly what I am doing
when dealing with the client. Eventually the ConnectionProcessorThread will
try to read from the socket again as the connections was kept open and I
will get a Socket Read error.
So is there something I might be doing wrong that makes Firefox create a new
connection?
Is there something I might be missing or misunderstanding?
3) Is it because its a proxy server and the browsers make new connections
to proxies vs reuse existing connections to web servers?
4) When it comes to connection persistence and I am the client talking to a
server will the underlying framework handle the "close" connection header
for me and close the connection or is that something I will have to look for
? Also if I explicitly close a connection to a server on a persistent
connection does that action send a connection "close" header?
thanks for all your help
I hope my endaevors will provide some good stress testing of this
framework. I really like what I see so far
Doug
--
What profits a man if he gains the whole world yet loses his soul?
Re: Persistent connections ?
Posted by Oleg Kalnichevski <ol...@apache.org>.
On Wed, 2006-08-30 at 20:45 +0000, Doug Lochart wrote:
> On 8/30/06, Oleg Kalnichevski <ol...@apache.org> wrote:
> >
> >
> > Actually as per RFC2616 Firefox always opens two connections with the
> > target host if there's more than one entity to be retrieved.
>
>
> So if there are say 7 entities then Firefox would use both connections to
> get all 7? Or should I expect to see more connections?
>
>
No, you should not. RFC2616 mandates that at a time there may be at most
two simultaneous connection to the same target host, and as far as I
know Firefox is spec compliant in this regard.
> > To my knowlege I am not explicitly closing the
> > > connection as I am expecting more connections to come.
> >
> > You should be using DefaultConnectionReuseStrategy to determine if a
> > connection should be kept alive or not. There are cases when HTTP agents
> > may explicitly request the connection to be closed.
>
>
> I am using DefaultConnectionReuseStrategy in both places. (proxy ->
> webserver) When I receive the response from the web server and then again
> (proxy -> client) when the response is serialized. The HttpService I
> extended does it for me for the (proxy -> client) clase.
>
>
Then it is all right.
>
> > > If you look at the
> > > ElementalHttpServer code you will see pretty much exactly what I am
> > doing
> > > when dealing with the client. Eventually the ConnectionProcessorThread
> > will
> > > try to read from the socket again as the connections was kept open and I
> > > will get a Socket Read error.
> > >
> >
> > You are talking about a socket connected to the client host (incoming
> > connection) or to the target host (outgoing connection)?
>
>
> Here is the code snippet (its connection between the client and the proxy)
> I realize this is just example code and that may be my current problem but
> my question is still valid as In this case I expect the connection to be
> reused and I do not believe that handleRequest() will do a blocking read
> (thus my timeout error) any suggestions?
Actually #handleRequest() will be doing blocking I/O.
>
> static class ConnectionProcessorThread extends Thread {
>
> private final HttpService httpservice;
>
> public ConnectionProcessorThread(final HttpService httpservice) {
> super();
> this.httpservice = httpservice;
> }
>
> public void run() {
> _logger.debug("New connection thread");
> try {
> while (!Thread.interrupted() &&
> !this.httpservice.isDestroyed() && this.httpservice.isActive()) {
> _logger.debug("handleRequest() begin");
> this.httpservice.handleRequest();
> _logger.debug("handleRequest() end");
> }
> } finally {
> _logger.debug("calling httpservice destroy()");
> this.httpservice.destroy();
> }
> _logger.debug("connection thread exiting");
> }
> }
>
>
> In this case what should I do if my ResponseConnControl adds a connection
> header to keep this connection alive?
>
You have to decide whether you want to keep that connection alive or not
and set the Connection header accordingly. This decision I cannot make
for you.
>
>
>
> >
> > > 3) Is it because its a proxy server and the browsers make new
> > connections
> > > to proxies vs reuse existing connections to web servers?
> > >
> >
> > Are you implementing a transparent (reverse) proxy or a standard
> > (caching) proxy? Did you actually have to tell the browser to connect
> > via the proxy of yours or not?
>
>
> Actually it is a content stream filtering proxy that will probably sit in
> front of squid cache when I am done. I am doing content filtering, tag
> stripping etc as well as providing a flexible template design for specifying
> classes of urls and being able to apply different rules to those classes.
> We were using the Sun Proxy Server which claims to do all of this but after
> I discovered and reported 3 bugs (2 of which were show stoppers and proved
> the code is hardly tested) we decided to write our own and grow it over
> time.
>
Writing a decent and most importantly scalable proxy server is not
trivial. Do make sure your management does not have inflated
expectations. The fact that ultimately you want to have a chain of
proxies makes your life even more difficult.
>
> I do see a ResponseConnControl which answers my other question.
>
> Now is there a little write up or anything that discusses the various
> interceptors or at least describes what is required and what is not and if
> any ordering of them is required? Any design cheat sheets as it were?
>
>
There's no good documentation at this point as we are not ready to
freeze the API yet. The best thing to do for you would be to get very
familiar with RFC2616. And I'll be around to answer HttpComponents
specific questions.
Oleg
> thanks
>
> Doug
>
> ---------------------------------------------------------------------
> > To unsubscribe, e-mail: httpclient-user-unsubscribe@jakarta.apache.org
> > For additional commands, e-mail: httpclient-user-help@jakarta.apache.org
> >
> >
>
>
> --
> What profits a man if he gains the whole world yet loses his soul?
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: httpclient-user-help@jakarta.apache.org
Re: Persistent connections ?
Posted by Doug Lochart <dl...@gmail.com>.
On 8/30/06, Oleg Kalnichevski <ol...@apache.org> wrote:
>
>
> Actually as per RFC2616 Firefox always opens two connections with the
> target host if there's more than one entity to be retrieved.
So if there are say 7 entities then Firefox would use both connections to
get all 7? Or should I expect to see more connections?
> To my knowlege I am not explicitly closing the
> > connection as I am expecting more connections to come.
>
> You should be using DefaultConnectionReuseStrategy to determine if a
> connection should be kept alive or not. There are cases when HTTP agents
> may explicitly request the connection to be closed.
I am using DefaultConnectionReuseStrategy in both places. (proxy ->
webserver) When I receive the response from the web server and then again
(proxy -> client) when the response is serialized. The HttpService I
extended does it for me for the (proxy -> client) clase.
> > If you look at the
> > ElementalHttpServer code you will see pretty much exactly what I am
> doing
> > when dealing with the client. Eventually the ConnectionProcessorThread
> will
> > try to read from the socket again as the connections was kept open and I
> > will get a Socket Read error.
> >
>
> You are talking about a socket connected to the client host (incoming
> connection) or to the target host (outgoing connection)?
Here is the code snippet (its connection between the client and the proxy)
I realize this is just example code and that may be my current problem but
my question is still valid as In this case I expect the connection to be
reused and I do not believe that handleRequest() will do a blocking read
(thus my timeout error) any suggestions?
static class ConnectionProcessorThread extends Thread {
private final HttpService httpservice;
public ConnectionProcessorThread(final HttpService httpservice) {
super();
this.httpservice = httpservice;
}
public void run() {
_logger.debug("New connection thread");
try {
while (!Thread.interrupted() &&
!this.httpservice.isDestroyed() && this.httpservice.isActive()) {
_logger.debug("handleRequest() begin");
this.httpservice.handleRequest();
_logger.debug("handleRequest() end");
}
} finally {
_logger.debug("calling httpservice destroy()");
this.httpservice.destroy();
}
_logger.debug("connection thread exiting");
}
}
In this case what should I do if my ResponseConnControl adds a connection
header to keep this connection alive?
>
> > 3) Is it because its a proxy server and the browsers make new
> connections
> > to proxies vs reuse existing connections to web servers?
> >
>
> Are you implementing a transparent (reverse) proxy or a standard
> (caching) proxy? Did you actually have to tell the browser to connect
> via the proxy of yours or not?
Actually it is a content stream filtering proxy that will probably sit in
front of squid cache when I am done. I am doing content filtering, tag
stripping etc as well as providing a flexible template design for specifying
classes of urls and being able to apply different rules to those classes.
We were using the Sun Proxy Server which claims to do all of this but after
I discovered and reported 3 bugs (2 of which were show stoppers and proved
the code is hardly tested) we decided to write our own and grow it over
time.
I do see a ResponseConnControl which answers my other question.
Now is there a little write up or anything that discusses the various
interceptors or at least describes what is required and what is not and if
any ordering of them is required? Any design cheat sheets as it were?
thanks
Doug
---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: httpclient-user-help@jakarta.apache.org
>
>
--
What profits a man if he gains the whole world yet loses his soul?
Re: Persistent connections ?
Posted by Oleg Kalnichevski <ol...@apache.org>.
On Wed, 2006-08-30 at 18:07 +0000, Doug Lochart wrote:
> I have a few more simple questions:
>
> 1) Is this list still the appropriate place to ask HttpComponents questions
> (vs HttpClient) and will there be a new List created for the HttpComponents
> project?
>
Yes, it is. HttpComponents project will be maintaining Commons
HttpClient 3.x until Jakarta HttpClient 4.0 is stable enough to
supersede it. At some point we'll just ask the Apache infrastructure
folks to rename the lists. Probably it is about time we did it.
> I just finished reading most of the HTTP 1.1 RFC and I tried to pay
> attention to the Persistent Connections as I was curious as to their impact
> on my application.
> So I have a simple proof-of-concept proxy server (based on the example
> ElementalHttpServer) and it is receiving connections. I understand that
> persistent connections are the default for all HTTP 1.1 communication. I
> see the HttpCore code refer to keeping the connections alive (even debug
> messages). So here are my questions ...
>
> 2) When I use Firefox to go grab a webpage that contains URI's to several
> other resources (images, css files etc) Firefox seems to make a new
> connection to port 8080 everytime and does not reuse the connection that it
> originally established.
Actually as per RFC2616 Firefox always opens two connections with the
target host if there's more than one entity to be retrieved.
> To my knowlege I am not explicitly closing the
> connection as I am expecting more connections to come.
You should be using DefaultConnectionReuseStrategy to determine if a
connection should be kept alive or not. There are cases when HTTP agents
may explicitly request the connection to be closed.
> If you look at the
> ElementalHttpServer code you will see pretty much exactly what I am doing
> when dealing with the client. Eventually the ConnectionProcessorThread will
> try to read from the socket again as the connections was kept open and I
> will get a Socket Read error.
>
You are talking about a socket connected to the client host (incoming
connection) or to the target host (outgoing connection)?
> So is there something I might be doing wrong that makes Firefox create a new
> connection?
> Is there something I might be missing or misunderstanding?
>
Quite likely
> 3) Is it because its a proxy server and the browsers make new connections
> to proxies vs reuse existing connections to web servers?
>
Are you implementing a transparent (reverse) proxy or a standard
(caching) proxy? Did you actually have to tell the browser to connect
via the proxy of yours or not?
> 4) When it comes to connection persistence and I am the client talking to a
> server will the underlying framework handle the "close" connection header
> for me and close the connection or is that something I will have to look for
> ?
It will not. However, we provide the DefaultConnectionReuseStrategy
class to aid you in implementing connection persistence.
HttpClientConnection conn = new DefaultHttpClientConnection(host);
try {
for (int i = 0; i < targets.length; i++) {
HttpGet request = new HttpGet(targets[i]);
...
HttpResponse response = httpexecutor.execute(request, conn);
...
if (!connStrategy.keepAlive(conn, response)) {
conn.close();
}
}
} finally {
conn.close();
}
> Also if I explicitly close a connection to a server on a persistent
> connection does that action send a connection "close" header?
>
No. But if you use HttpRequestExecutor with the RequestConnControl
protocol interceptor it will add the appropriate 'Connection' header for
you.
> thanks for all your help
>
> I hope my endaevors will provide some good stress testing of this
> framework.
They do. I am glad you are helping us by reviewing the API
> I really like what I see so far
>
Oleg
>
> Doug
>
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: httpclient-user-help@jakarta.apache.org