You are viewing a plain text version of this content. The canonical link for it is here.
Posted to httpclient-users@hc.apache.org by "AP Adam Patt (5094)" <ap...@ag.com> on 2008/10/09 21:05:06 UTC
HttpClient and Threads
I am trying to use HttpClient with threads and while it works, it is
only sending one request at a time such that having multiple threads to
make my requests does not get any advantage from having multiple
threads.
I am using the same HttpClient instance among all of the threads, am
using the MultiThreadedHttpConnectionManager when creating the instance
and have set setDefaultMaxConnectionsPerHost to 20 and
setMaxTotalConnections to 30 and I still cannot get more than one
connection at the same time.
I am sending files over http. I am also setting some cookies (I do the
cookie handling myself) when doing the post and I do a releaseConnection
after I am done with each request.
I have been looking for examples and incorperated things I have seen but
none of them have many any change in being able to actually get multiple
requests to be happening at the same time.
here are the relavant parts of code (err is a window for debug messages
that is passed in):
public class SessionHttpRequest {
private static HttpClient client;
private static HttpConnectionManagerParams params = new
HttpConnectionManagerParams();
private static HttpConnectionManager connectionManager = new
MultiThreadedHttpConnectionManager();
static {
// Configure params for the Connection Manager
params.setDefaultMaxConnectionsPerHost( 20 );
params.setMaxTotalConnections( 30 );
// This next line may not be necessary since we specified
default 2 lines ago, but here it is anyway
params.setMaxConnectionsPerHost(
HostConfiguration.ANY_HOST_CONFIGURATION, 20 );
// Set up the connection manager
connectionManager.setParams( params );
// Finally set up the static multithreaded HttpClient
client = new HttpClient( connectionManager );
}
public SessionHttpRequest(ErrorWindow err) {
this.err = err;
}
public SessionHttpRequest(ErrorWindow err, String host, int port,
String protocol)
{
this(err);
client.getHostConfiguration().setHost(host, port, protocol);
//new
HttpConnectionManagerParams().setMaxConnectionsPerHost(client.getHostCon
figuration(), 20);
}
// each thread calls this method to get a clone. client is static, so
it will come for "free"
// I just need the session information
public SessionHttpRequest clone()
{
SessionHttpRequest request = new SessionHttpRequest(err);
request.setSaveSession(saveSession);
request.setSession(session);
return request;
}
public void setSession(HashMap<String,String> session)
{
this.session = (HashMap<String,String>)session.clone();
}
public void setSaveSession(boolean saveSession) {
this.saveSession = saveSession;
}
// This is the method that the threads call to do the file upload
public String post(boolean safe, String url, boolean clearParameters,
String requestMethod, int timeout) throws IOException
{
if (timeout > 0)
client.getParams().setParameter(HttpMethodParams.SO_TIMEOUT, new
Integer(timeout));
HttpMethodBase method;
if (requestMethod.equals("POST"))
method = new PostMethod(url);
else
method = new GetMethod(url);
if (saveSession)
method.getParams().setCookiePolicy(CookiePolicy.IGNORE_COOKIES);
// set all of the saved cookies
if (saveSession)
{
String setCookie = new String();
String key;
String value;
for (Iterator it=session.entrySet().iterator(); it.hasNext(); )
{
Map.Entry entry = (Map.Entry)it.next();
key = (String)entry.getKey();
value = (String)entry.getValue();
if (!setCookie.equals(""))
setCookie += " ";
setCookie += key + "=" + value + ";";
}
method.setRequestHeader("Cookie", setCookie);
if (debug)
err.println("sending cookie: " + setCookie);
}
if (requestMethod.equals("POST"))
{
String key;
Object value;
ArrayList<Part> parts = new ArrayList<Part>();
for (Iterator it=parameters.entrySet().iterator(); it.hasNext();
)
{
Map.Entry entry = (Map.Entry)it.next();
key = (String)entry.getKey();
value = entry.getValue();
if (value instanceof File)
parts.add(new FilePart(key , (File)value));
else
parts.add(new StringPart(key, value.toString()));
}
Part[] dummy = {};
((PostMethod)method).setRequestEntity(new
MultipartRequestEntity(parts.toArray(dummy), method.getParams()) );
}
String result="";
// make the request
int responseCode;
try
{
responseCode = client.executeMethod(method);
// read all just takes the stream and turns it into a string.
// I was getting warnings when trying to use
getResponseBodyAsString
result =
StaticMethods.readAll(method.getResponseBodyAsStream());
String location = "";
//catch all of the cookies being set now
//TODO need to turn cookies into a map and pay attention to the
name if things are being overwritten
if (saveSession)
{
String header;
String cookie;
int cookieEnd;
int cookieMiddle;
String cookieName;
String cookieValue;
Header[] headers = method.getResponseHeaders("Set-Cookie");
for(int x=0; x < headers.length ; x++)
{
cookie = headers[x].getValue();
cookieEnd = cookie.indexOf(";");
cookie = cookie.substring(0,cookieEnd);
cookieMiddle = cookie.indexOf("=");
cookieName = cookie.substring(0,cookieMiddle);
cookieValue = cookie.substring(cookieMiddle+1);
if (debug)
err.println("saving Cookie: " + cookieName + " = " +
cookieValue);
session.put(cookieName, cookieValue);
}
}
if (clearParameters) {
parameters.clear();
}
if ((responseCode == 302 || responseCode == 301) && location !=
"") {
location = url.substring(0, url.lastIndexOf("/")+1) +
method.getResponseHeader("Location").getValue();
if (debug)
err.println("redirecting to " + location);
result = post(safe, location, clearParameters, "GET");
}
}
catch (java.io.IOException e)
{
StaticMethods.prettyException(e, err);
}
finally
{
method.releaseConnection();
}
return result;
}
What am I doing wrong? Thanks.
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
RE: HttpClient and Threads
Posted by Oleg Kalnichevski <ol...@apache.org>.
On Fri, 2008-10-10 at 14:22 -0400, AP Adam Patt (5094) wrote:
> > Essentially you _assume_ HttpClient is the culprit, rather
> > synchronization logic in your code, don't you?
> >
> > To see what exactly what is happening inside HttpClient
> > components, run
> > it with both wire and context logging on. You should also use
> > a logging
> > toolkit that can provide you with the thread context information (such
> > as Log4j), since this problem appears to be threading related. If you
> > need help interpreting the logs feel free to post them to this list.
>
> When I was using the built in HttpURLConnection I could clearly see that
> multiple threads were actually working at the same time. The only
> change I made was to use the HttpClient library in place of the
> HttpURLConnection library and I found this problem.
>
> Just to be sure that there wasn't something else causing things to act
> this way, I put a sleep that adjusted based on the size of the file in
> place of where the executeMethod and getResponseBodyAsStream calls go.
> When I use the sleep, I clearly see that 4 threads are working and
> completing at different times relative to the size of the file. When I
> add the code back to actually make the network call, it goes back to
> putting one request after another.
>
> I wish I could give you a working version of the program, but it depends
> on internal server side stuff to work correctly. I was hoping that I
> was making a call incorrectly, but it seems like something stranger is
> going on. I need to actually switch and work on something else right
> now but would like to create a simple program that can show this problem
> clearly without the other dependencies. Maybe I can do that next week
> and post it.
>
> When do you mean by context logging? Is there something in the
> HttpClient library that I should enable to see what is going on? And
> are you referring to a packet tracking tool like Wireshark for wire
> logging, or is that another HttpClient option that I should turn on?
>
Please refer to the instructions given in the HttpClient logging guide:
http://hc.apache.org/httpclient-3.x/logging.html
This should give you enough details about request execution and the
internal state of HttpClient.
Oleg
> I have a debug window that I write log messages to that basically acts
> as my log to see what is going on and I have message give me thread
> information in there when it's relevant.
>
> Thanks much for the responses. Even though I haven't gotten the problem
> solved, I feel I am making some progress. If you have any other
> suggestions I would love to hear them, otherwise I will try and work on
> a sample program that I can actually post to demonstrate my problem.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
RE: HttpClient and Threads
Posted by "AP Adam Patt (5094)" <ap...@ag.com>.
> Essentially you _assume_ HttpClient is the culprit, rather
> synchronization logic in your code, don't you?
>
> To see what exactly what is happening inside HttpClient
> components, run
> it with both wire and context logging on. You should also use
> a logging
> toolkit that can provide you with the thread context information (such
> as Log4j), since this problem appears to be threading related. If you
> need help interpreting the logs feel free to post them to this list.
When I was using the built in HttpURLConnection I could clearly see that
multiple threads were actually working at the same time. The only
change I made was to use the HttpClient library in place of the
HttpURLConnection library and I found this problem.
Just to be sure that there wasn't something else causing things to act
this way, I put a sleep that adjusted based on the size of the file in
place of where the executeMethod and getResponseBodyAsStream calls go.
When I use the sleep, I clearly see that 4 threads are working and
completing at different times relative to the size of the file. When I
add the code back to actually make the network call, it goes back to
putting one request after another.
I wish I could give you a working version of the program, but it depends
on internal server side stuff to work correctly. I was hoping that I
was making a call incorrectly, but it seems like something stranger is
going on. I need to actually switch and work on something else right
now but would like to create a simple program that can show this problem
clearly without the other dependencies. Maybe I can do that next week
and post it.
When do you mean by context logging? Is there something in the
HttpClient library that I should enable to see what is going on? And
are you referring to a packet tracking tool like Wireshark for wire
logging, or is that another HttpClient option that I should turn on?
I have a debug window that I write log messages to that basically acts
as my log to see what is going on and I have message give me thread
information in there when it's relevant.
Thanks much for the responses. Even though I haven't gotten the problem
solved, I feel I am making some progress. If you have any other
suggestions I would love to hear them, otherwise I will try and work on
a sample program that I can actually post to demonstrate my problem.
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
RE: HttpClient and Threads
Posted by Oleg Kalnichevski <ol...@apache.org>.
On Fri, 2008-10-10 at 10:08 -0400, AP Adam Patt (5094) wrote:
> > On Thu, 2008-10-09 at 15:05 -0400, AP Adam Patt (5094) wrote:
> > > I am trying to use HttpClient with threads and while it works, it is
> > > only sending one request at a time
> >
> > How do you know?
>
> Because the app marks the files being done and even if there is small
> file after a large file, it will always do the files in order. I also
> got a multi threaded process working before and the overall processing
> time was lower with more threads and using this library it's the same if
> I use one or many threads. I can also see in my debug window that it's
> just rotating between the threads for requests coming back.
>
Essentially you _assume_ HttpClient is the culprit, rather
synchronization logic in your code, don't you?
To see what exactly what is happening inside HttpClient components, run
it with both wire and context logging on. You should also use a logging
toolkit that can provide you with the thread context information (such
as Log4j), since this problem appears to be threading related. If you
need help interpreting the logs feel free to post them to this list.
Oleg
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
RE: HttpClient and Threads
Posted by "AP Adam Patt (5094)" <ap...@ag.com>.
> On Thu, 2008-10-09 at 15:05 -0400, AP Adam Patt (5094) wrote:
> > I am trying to use HttpClient with threads and while it works, it is
> > only sending one request at a time
>
> How do you know?
Because the app marks the files being done and even if there is small
file after a large file, it will always do the files in order. I also
got a multi threaded process working before and the overall processing
time was lower with more threads and using this library it's the same if
I use one or many threads. I can also see in my debug window that it's
just rotating between the threads for requests coming back.
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: HttpClient and Threads
Posted by Oleg Kalnichevski <ol...@apache.org>.
On Thu, 2008-10-09 at 15:05 -0400, AP Adam Patt (5094) wrote:
> I am trying to use HttpClient with threads and while it works, it is
> only sending one request at a time
How do you know?
Oleg
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org