You are viewing a plain text version of this content. The canonical link for it is here.
Posted to httpclient-users@hc.apache.org by Stefan Parvu <sp...@systemdatarecorder.org> on 2009/12/27 13:15:22 UTC
webrec questions httpclient 4.0 stable
Hi,
We are a small group of folks working on a measurement toolkit:
www.systemdatarecorder.org. One of the tools, webrec is supposed
to be a HTTP client measurement tool, recording the response
times for certain HTTP methods.
http://systemdatarecorder.org:9009/bugzilla/show_bug.cgi?id=21
The utility uses HTTPClient in a multithreaded way, see the specs for
webrec, one thread per workload.
I'm using ThreadSafeClientConnManager in my application. Now, I have
noticed in this mailing list is recommeded to use interceptors
for measuring metrics with HTTPClient.
Questions:
1) Do I have to use interceptor also measuring response time?
Can't I just use:
long pTime = System.currentTimeMillis();
long cTime = 0;
response = httpclient.execute(httpget);
if (response.getStatusLine().getStatusCode() == HttpURLConnection.HTTP_OK)
{
cTime = System.currentTimeMillis();
}
long timeLapse = cTime - pTime;
2). Does connection managers add some overhead to requests? Why there is a
need to use interceptors?
Many thanks,
Stefan
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: webrec questions httpclient 4.0 stable
Posted by Stefan Parvu <sp...@systemdatarecorder.org>.
Hi,
>
>> You may also want to search the archives for questions on performance
>> tuning.
>
> We will do that. I expect if I have 5 or 40sec sleep time in that
> sample from your site the R be almost similar. We will look for
> stale connection check.
>
Thanks for all comments and help. We been able to obtain
consisten numbers with HTTPCLient 4.0.1 and we will base our
webrec on this library.
We did:
a). remove the STALE check connection
b). handle manually the idle/closed connections.
It was a bit of pain to get deeper into the persistance
connection mantra but we sorted out. Soon you will be able
to see our webrec in source code at our site.
Thanks again,
Stefan
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: webrec questions httpclient 4.0 stable
Posted by Stefan Parvu <sp...@systemdatarecorder.org>.
>
> what on earth is Solaris HTTP Server?
That was a simple explanation how we ran the test:
client java application, Linux box
server side, Solaris box with HTTP server
> That is because wget and the likes usually do not re-use connections.
> Some web servers can not handle persistent connections well, so opening
> a new connection may turn out to be faster than re-using a persistent
> connection.
Im trying to compare the use of our java code. wget, curl were given as
example.
>
> Turn off the stale connection check and use context / wire logging to
> see where exactly Httpclient gets blocked.
>
> http://hc.apache.org/httpcomponents-client/logging.html
>
are you suggesting we turn off in our code the stale connection check ?
java client:
* sleep time 5sec, R=10 sec
* sleep time 35sec, R=3 sec
> You may also want to search the archives for questions on performance
> tuning.
We will do that. I expect if I have 5 or 40sec sleep time in that
sample from your site the R be almost similar. We will look for
stale connection check.
Thanks for advice and tips,
Stefan
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: webrec questions httpclient 4.0 stable
Posted by Oleg Kalnichevski <ol...@apache.org>.
On Mon, 2009-12-28 at 14:44 +0200, Stefan Parvu wrote:
> Hello Oleg,
>
> > Makes perfect sense. HttpClient allows only two concurrent connections
> > to the same host per default as required by the HTTP spec. Therefore,
> > those two threads that manage to lease a connection from the pool first
> > execute faster than those that stay blocked waiting for a connection to
> > free up.
>
> True. There is a RFC stating than no more than 2 concurrent
> connections are allowed. That's fine.
>
> However our problem is this: if we change the sleep time
> in the example:
> http://www.systemdatarecorder.org/ClientMultiThreadedExecution.java
>
>
> /* ****************************************************************** */
> long timeLapse = cTime - pTime;
> System.out.println(id + " - get executed. Response time:" + timeLapse );
>
> // get the response body as an array of bytes
> HttpEntity entity = response.getEntity();
>
> if (entity != null) {
> byte[] bytes = EntityUtils.toByteArray(entity);
> System.out.println(id + " - " + bytes.length + " bytes read");
> }
>
> sleep(30000);
> ^^^^^^^^^^^^^
>
> /* ****************************************************************** */
>
> more than 30sec, the response times is 5 smaller than in
> other cases.
>
> Example: running the previous code example with only 2
> HTTP requests to avoid the RFC confusion.
>
>
> 1). Sleep time 5s
>
> 1 - get executed. Response time:173
> 2 - get executed. Response time:170
> 2 - 8005 bytes read
> 1 - 8005 bytes read
>
> <this is ok since the JVM is not warmed up,
> the methods are not yet compiled>
>
> 1 - get executed. Response time:11
> 2 - get executed. Response time:11
> 1 - 8005 bytes read
> 2 - 8005 bytes read
>
>
> 2 - get executed. Response time:11
> 1 - get executed. Response time:11
> 2 - 8005 bytes read
> 1 - 8005 bytes read
>
> So our response time is 11ms. Seems consistent 10-11ms.
>
>
> 2). Sleep time 35s
>
> 2 - get executed. Response time:130
> 1 - get executed. Response time:131
> 2 - 8005 bytes read
> 1 - 8005 bytes read
>
>
> 2 - get executed. Response time:3
> 1 - get executed. Response time:3
> 2 - 8005 bytes read
> 1 - 8005 bytes read
>
> 2 - get executed. Response time:2
> 1 - get executed. Response time:2
> 2 - 8005 bytes read
> 1 - 8005 bytes read
>
> Now we started to see a different thing. Response times went
> down 5 times since previous run. Why is that ?
>
> So our problem seems to be dependant on sleep time between runs.
> If it is > 30s the response times seem to align with the reality.
>
> We tested the code from this example:
>
>
> Client
> Linux --- Solaris HTTP Server
>
what on earth is Solaris HTTP Server?
> Usually it takes 1 or 2ms for httping or a simple wget or curl.
>
> Any ideas ?
>
That is because wget and the likes usually do not re-use connections.
Some web servers can not handle persistent connections well, so opening
a new connection may turn out to be faster than re-using a persistent
connection.
Turn off the stale connection check and use context / wire logging to
see where exactly Httpclient gets blocked.
http://hc.apache.org/httpcomponents-client/logging.html
You may also want to search the archives for questions on performance
tuning.
Oleg
> Many thanks for help.
>
> Stefan
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: webrec questions httpclient 4.0 stable
Posted by Stefan Parvu <sp...@systemdatarecorder.org>.
Hello Oleg,
> Makes perfect sense. HttpClient allows only two concurrent connections
> to the same host per default as required by the HTTP spec. Therefore,
> those two threads that manage to lease a connection from the pool first
> execute faster than those that stay blocked waiting for a connection to
> free up.
True. There is a RFC stating than no more than 2 concurrent
connections are allowed. That's fine.
However our problem is this: if we change the sleep time
in the example:
http://www.systemdatarecorder.org/ClientMultiThreadedExecution.java
/* ****************************************************************** */
long timeLapse = cTime - pTime;
System.out.println(id + " - get executed. Response time:" + timeLapse );
// get the response body as an array of bytes
HttpEntity entity = response.getEntity();
if (entity != null) {
byte[] bytes = EntityUtils.toByteArray(entity);
System.out.println(id + " - " + bytes.length + " bytes read");
}
sleep(30000);
^^^^^^^^^^^^^
/* ****************************************************************** */
more than 30sec, the response times is 5 smaller than in
other cases.
Example: running the previous code example with only 2
HTTP requests to avoid the RFC confusion.
1). Sleep time 5s
1 - get executed. Response time:173
2 - get executed. Response time:170
2 - 8005 bytes read
1 - 8005 bytes read
<this is ok since the JVM is not warmed up,
the methods are not yet compiled>
1 - get executed. Response time:11
2 - get executed. Response time:11
1 - 8005 bytes read
2 - 8005 bytes read
2 - get executed. Response time:11
1 - get executed. Response time:11
2 - 8005 bytes read
1 - 8005 bytes read
So our response time is 11ms. Seems consistent 10-11ms.
2). Sleep time 35s
2 - get executed. Response time:130
1 - get executed. Response time:131
2 - 8005 bytes read
1 - 8005 bytes read
2 - get executed. Response time:3
1 - get executed. Response time:3
2 - 8005 bytes read
1 - 8005 bytes read
2 - get executed. Response time:2
1 - get executed. Response time:2
2 - 8005 bytes read
1 - 8005 bytes read
Now we started to see a different thing. Response times went
down 5 times since previous run. Why is that ?
So our problem seems to be dependant on sleep time between runs.
If it is > 30s the response times seem to align with the reality.
We tested the code from this example:
Client
Linux --- Solaris HTTP Server
Usually it takes 1 or 2ms for httping or a simple wget or curl.
Any ideas ?
Many thanks for help.
Stefan
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: webrec questions httpclient 4.0 stable
Posted by Oleg Kalnichevski <ol...@apache.org>.
On Mon, 2009-12-28 at 00:55 +0200, Stefan Parvu wrote:
> > We have seen that after the first run (usually 1min)
> > the first response time recorded for the first request
> > is 2milliseconds followed by the next requests with R=10ms or 11ms.
>
> Here maybe a much simpler example:
> http://www.systemdatarecorder.org/ClientMultiThreadedExecution.java
> which can compile easily.
>
> This code loops over 4 similar HTTP requests obtaining different
> Response times. We have a sleep after hitting all 4 requests
> set to 35secs.
>
> When we lower this value, the sleep, we obtain the response times
> similar 10, 11 ms. When we increase the sleep time the first request
> and second are ok, the others are bogus.
>
> Why this variation ?
>
> Makes sense ?
>
Makes perfect sense. HttpClient allows only two concurrent connections
to the same host per default as required by the HTTP spec. Therefore,
those two threads that manage to lease a connection from the pool first
execute faster than those that stay blocked waiting for a connection to
free up.
Oleg
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: webrec questions httpclient 4.0 stable
Posted by Stefan Parvu <sp...@systemdatarecorder.org>.
> We have seen that after the first run (usually 1min)
> the first response time recorded for the first request
> is 2milliseconds followed by the next requests with R=10ms or 11ms.
Here maybe a much simpler example:
http://www.systemdatarecorder.org/ClientMultiThreadedExecution.java
which can compile easily.
This code loops over 4 similar HTTP requests obtaining different
Response times. We have a sleep after hitting all 4 requests
set to 35secs.
When we lower this value, the sleep, we obtain the response times
similar 10, 11 ms. When we increase the sleep time the first request
and second are ok, the others are bogus.
Why this variation ?
Makes sense ?
Stefan
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: webrec questions httpclient 4.0 stable
Posted by Stefan Parvu <sp...@systemdatarecorder.org>.
Evening,
Many thanks for help.
> Yes, they do, but it is usually significantly lower than that of establishing
> a new connection.
Cheers. I have still something unclear.
Our application is recording response time for a sequence
of URLs with a delay of 5s between them, like an example.
Then main execution thread is sleeping xx seconds
before next round of measurements.
We have seen that after the first run (usually 1min)
the first response time recorded for the first request
is 2milliseconds followed by the next requests with R=10ms or 11ms.
They all should have the same response time, because we used
the same url in all the sequence.
Below is extract of the execution of the webrec running
many minutes, running on the same server as the HTTP
server to avoid first the network delay etc ...
1261847585:2:11:10:10:11:10:9
1261847675:2:10:10:11:10:10:8
1261847765:2:11:10:10:11:10:9
1261847855:2:10:11:11:10:11:9
1261847946:2:11:10:11:10:10:9
1261848036:2:11:11:10:10:11:9
1261848126:2:10:10:11:10:10:8
1261848216:1:11:10:10:11:10:8
1261848306:2:10:11:11:10:10:9
1261848396:2:11:11:10:11:10:9
1261848486:2:10:11:10:10:11:9
1261848576:2:10:10:11:10:10:8
where each field is defined:
timestamp:
response time for 1st HTTP request:
...
response time for 6th HTTP request:
average response time
Any ideas why always the 1st request has least response time
(1-2ms) than other request? We see an overhead on all the
other request but not the first one.
The real response time should actually be like 2ms,
because we have measured it with other tools (wget, curl).
Below the code responsible for doing this:
http://www.systemdatarecorder.org/webrec.snippet.java
Many thanks,
Stefan
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: webrec questions httpclient 4.0 stable
Posted by Oleg Kalnichevski <ol...@apache.org>.
Stefan Parvu wrote:
>
> Hi,
>
> We are a small group of folks working on a measurement toolkit:
> www.systemdatarecorder.org. One of the tools, webrec is supposed
> to be a HTTP client measurement tool, recording the response
> times for certain HTTP methods.
> http://systemdatarecorder.org:9009/bugzilla/show_bug.cgi?id=21
>
> The utility uses HTTPClient in a multithreaded way, see the specs for
> webrec, one thread per workload.
>
> I'm using ThreadSafeClientConnManager in my application. Now, I have
> noticed in this mailing list is recommeded to use interceptors for
> measuring metrics with HTTPClient.
>
>
> Questions:
>
> 1) Do I have to use interceptor also measuring response time?
> Can't I just use:
>
> long pTime = System.currentTimeMillis();
> long cTime = 0;
> response = httpclient.execute(httpget);
> if (response.getStatusLine().getStatusCode() ==
> HttpURLConnection.HTTP_OK) {
> cTime = System.currentTimeMillis();
> }
> long timeLapse = cTime - pTime;
>
You do not have to.
>
> 2). Does connection managers add some overhead to requests?
Yes, they do, but it is usually significantly lower than that of
establishing a new connection.
Why there is
> a need to use interceptors?
>
Interceptors provide you with a finer control over the request execution
process, but you do not have to use them if you do not want to.
Oleg
> Many thanks,
> Stefan
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org