Using MultiThreadedHttpConnectionManager in a Servlet

[Josh Fix <jo...@yahoo.com>]

I have a servlet that reads some parameters from a URL, uses an HttpURLConnection to query and retrieve a web page, parse the text, and finally construct some XML as output.  I know using an HTTP connection as the primary back-end data source is horribly bad, but we have to do it for now and the web pages it is scraping are all internal to our large company.

The remote server sometimes takes a long time to respond.  The HttpURLConnection/BufferedReader methods seem to act as if they are synchronized--only one connection/request can be made at a time.  This can cause significant congestion when multiple requests come in at once.

My goal was to use HttpClient with the MultiThreadedHttpConnectionManager to handle all of these back-end connections, but so far it seems like the behavior is the same.  Is it possible to have multiple threads from my servlet open simultaneous connections to an external web site with different requests and read the results?  I would really like to get around the blocking aspect of the read statements.  My servlet declares the MTHCM as a static object in the servlet class, so that once instance is shared amongst all incoming servlet requests, and a static{} block is used in the class to set timeouts/etc.  The HttpClient is newed in the doGet method.

If I build a test servlet that does nothing but return a string after about 20 seconds to simulate the remote server responding slowly and point my real servlet to the test servlet, no other requests/threads to the real servlet are allowed to make a connection until the test servlet returns it's result.  What am I doing wrong?  How can I allow other threads to continue to query the server, even when one request is very slow to respond?  Thank you in advance for you help!

Josh




       
____________________________________________________________________________________
Building a website is a piece of cake. Yahoo! Small Business gives you all the tools to get online.
http://smallbusiness.yahoo.com/webhosting 

Re: Using MultiThreadedHttpConnectionManager in a Servlet

[Roland Weber <os...@dubioso.net>]

Hello Josh,

Josh Fix wrote:
> Is it possible to have multiple threads from my servlet
> open simultaneous connections to an external web site with
> different requests and read the results?
Sure. That's what the MultiThreadedHttpConnectionManager
is there for.
> I would really like to get around the blocking aspect of
> the read statements.

What are your settings for the connections-per-host and
total-connections limits? Default for CPH is 2, as that
is mentioned in RFC 2616. Since you're operating some
kind of proxy agent, you are entitled to use 2 per client
for which your servlet is acting.
http://jakarta.apache.org/httpcomponents/httpclient-3.x/threading.html

> My servlet declares the MTHCM as a static object in the
> servlet class, so that once instance is shared amongst all
> incoming servlet requests, and a static{} block is used in
> the class to set timeouts/etc.

Bad style, but probably irrelevant for the problem at hand.
You should declare shared objects as non-static attributes of
the servlet and initialize them in the Servlet.init() method.

> The HttpClient is newed in the doGet method.

Unless you need different parameters for the instances,
that is not a good idea. Use one client object and set
specific parameters at the method you are executing.
Our preferences hierarchies do not respond well to using
multiple clients with the same MTHCM. There can be only
one parent to the connection manager params.

hope that helps,
  Roland


---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: httpclient-user-help@jakarta.apache.org