You are viewing a plain text version of this content. The canonical link for it is here.
Posted to httpclient-users@hc.apache.org by Jean-Marc Spaggiari <je...@spaggiari.org> on 2012/08/08 16:07:59 UTC
Bandwidth usage
Hi,
Is there a good way to calculate the bandwidth used by a request?
I'm doing a get and I try to track the upload and download bandwidth used.
So for a get, there is some upload to the server. There is the URL,
and the headers. But seems there is a bit more than that and I'm not
able to figure what.
When I get the respons, I'm using getHeaders("Content-Length") to get
the content lenght. But I'm most probably missing the header size. I
also tried EntityUtils.toString(entity).length() but none of them
seems to be accurate.
So I'm wondering is there is a good way to calculate the bandwidth
used for upload and download?
Thanks,
JM
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: Bandwidth usage
Posted by Luca Colantonio <lu...@gmail.com>.
Hi Jean Marc,
I believe Sam is right, counting the bytes is the only way I can see
in order to do what you are trying to achieve.
It's a bit annoying the way to do that: you need to create a socket
factory (that wraps normal sockets with your bytes-counting ones) and
register that socket factory to the httpclient so that every time the
httpclient needs to go over the wire it will use a socket you provide.
(in order for the httpclient to use your socket factory you will need
to register it using a SchemeRegistry).
I used this approach to write an aggregate bandwidth limiter (not 100%
accurate, but it looks like it does its job, I am currently testing
it...cross-fingers).
Of course then you will then need to match socket-to-requests.
The problem with this is that some classes are now deprecated so it
really depends on the version of java you are using.
DISCLAIMER: I am just starting with java so anything I said might
easily be plain wrong and there might be much simpler ways of doing
it. If you find any, please share :)
--luca
On Wed, Aug 8, 2012 at 3:40 PM, Sam Crawford <sa...@gmail.com> wrote:
>
> Jean-Marc,
>
> What are you comparing it to in order to determine accuracy? If you're
> looking at the "on-the-wire" size (e.g. through a packet capture,
> router/switch/interface counters, etc), then that will be counting
> Ethernet, IP, TCP and potentially other headers too. The size of such
> headers is not predictable, and applications (including HttpClient)
> have no way to access them.
>
> If you want accurate byte counts from the application layer (HTTP
> request line + HTTP headers + body), then you could explore wrapping
> the SocketInputStream / SocketOutputStream to provide a simple byte
> counter. I haven't explored how easy/hard this is in HttpClient, and
> it may be that there's a cleaner way to do it that I'm not aware of.
>
> Counting the length of the returned Entity (do not rely on the
> Content-Length header, that will not always be there) and adding
> calculated header lengths (like Ryan suggested) should give you
> figures that are close to the true figures for many sites. Note that
> compressed content will cause you problems, as the Entity will provide
> you the uncompressed length.
>
> Thanks,
>
> Sam
>
>
> On 8 August 2012 15:19, Jean-Marc Spaggiari <je...@spaggiari.org> wrote:
> > Hi Ryan,
> >
> > This will calculate the header size. But is there anything else? Is
> > the size only headers + URL lenght for the request, and
> > headers+respons body for the respons? Is there any other CRC,
> > validation, packet, etc.?
> >
> > Because headersize + body lenght seems to not be accurate :(
> >
> > 2012/8/8, Ryan Smith <ry...@gmail.com>:
> >> If HttpClient has a method to do this, can someone let me know? This is a
> >> method I wrote to calculate header size:
> >>
> >> public static int getHeaderByteSize(Header[] headers) {
> >> int requestHeaderByteSize = 0;
> >> for (Header requestHeader : headers) {
> >> if (requestHeader.getName() != null) {
> >> requestHeaderByteSize += requestHeader.getName().getBytes().length;
> >> }
> >> if (requestHeader.getValue() != null) {
> >> requestHeaderByteSize += requestHeader.getValue().getBytes().length;
> >> }
> >> }
> >> return requestHeaderByteSize;
> >> }
> >>
> >> On Wed, Aug 8, 2012 at 10:07 AM, Jean-Marc Spaggiari <
> >> jean-marc@spaggiari.org> wrote:
> >>
> >>> Hi,
> >>>
> >>> Is there a good way to calculate the bandwidth used by a request?
> >>>
> >>> I'm doing a get and I try to track the upload and download bandwidth
> >>> used.
> >>>
> >>> So for a get, there is some upload to the server. There is the URL,
> >>> and the headers. But seems there is a bit more than that and I'm not
> >>> able to figure what.
> >>>
> >>> When I get the respons, I'm using getHeaders("Content-Length") to get
> >>> the content lenght. But I'm most probably missing the header size. I
> >>> also tried EntityUtils.toString(entity).length() but none of them
> >>> seems to be accurate.
> >>>
> >>> So I'm wondering is there is a good way to calculate the bandwidth
> >>> used for upload and download?
> >>>
> >>> Thanks,
> >>>
> >>> JM
> >>>
> >>> ---------------------------------------------------------------------
> >>> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> >>> For additional commands, e-mail: httpclient-users-help@hc.apache.org
> >>>
> >>>
> >>
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: Bandwidth usage
Posted by Sam Crawford <sa...@gmail.com>.
Jean-Marc,
What are you comparing it to in order to determine accuracy? If you're
looking at the "on-the-wire" size (e.g. through a packet capture,
router/switch/interface counters, etc), then that will be counting
Ethernet, IP, TCP and potentially other headers too. The size of such
headers is not predictable, and applications (including HttpClient)
have no way to access them.
If you want accurate byte counts from the application layer (HTTP
request line + HTTP headers + body), then you could explore wrapping
the SocketInputStream / SocketOutputStream to provide a simple byte
counter. I haven't explored how easy/hard this is in HttpClient, and
it may be that there's a cleaner way to do it that I'm not aware of.
Counting the length of the returned Entity (do not rely on the
Content-Length header, that will not always be there) and adding
calculated header lengths (like Ryan suggested) should give you
figures that are close to the true figures for many sites. Note that
compressed content will cause you problems, as the Entity will provide
you the uncompressed length.
Thanks,
Sam
On 8 August 2012 15:19, Jean-Marc Spaggiari <je...@spaggiari.org> wrote:
> Hi Ryan,
>
> This will calculate the header size. But is there anything else? Is
> the size only headers + URL lenght for the request, and
> headers+respons body for the respons? Is there any other CRC,
> validation, packet, etc.?
>
> Because headersize + body lenght seems to not be accurate :(
>
> 2012/8/8, Ryan Smith <ry...@gmail.com>:
>> If HttpClient has a method to do this, can someone let me know? This is a
>> method I wrote to calculate header size:
>>
>> public static int getHeaderByteSize(Header[] headers) {
>> int requestHeaderByteSize = 0;
>> for (Header requestHeader : headers) {
>> if (requestHeader.getName() != null) {
>> requestHeaderByteSize += requestHeader.getName().getBytes().length;
>> }
>> if (requestHeader.getValue() != null) {
>> requestHeaderByteSize += requestHeader.getValue().getBytes().length;
>> }
>> }
>> return requestHeaderByteSize;
>> }
>>
>> On Wed, Aug 8, 2012 at 10:07 AM, Jean-Marc Spaggiari <
>> jean-marc@spaggiari.org> wrote:
>>
>>> Hi,
>>>
>>> Is there a good way to calculate the bandwidth used by a request?
>>>
>>> I'm doing a get and I try to track the upload and download bandwidth
>>> used.
>>>
>>> So for a get, there is some upload to the server. There is the URL,
>>> and the headers. But seems there is a bit more than that and I'm not
>>> able to figure what.
>>>
>>> When I get the respons, I'm using getHeaders("Content-Length") to get
>>> the content lenght. But I'm most probably missing the header size. I
>>> also tried EntityUtils.toString(entity).length() but none of them
>>> seems to be accurate.
>>>
>>> So I'm wondering is there is a good way to calculate the bandwidth
>>> used for upload and download?
>>>
>>> Thanks,
>>>
>>> JM
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
>>> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>>>
>>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: Bandwidth usage
Posted by Oleg Kalnichevski <ol...@apache.org>.
On Wed, 2012-08-08 at 10:26 -0400, Ryan Smith wrote:
> Right. you also need to calculate what you send to the web server. So you
> need to calculate the request headers when you make a request. Then on the
> response you get response body size and header size. As far as I know
> thats all you need to account for. Maybe Oleg or someone else can verify.
>
Ryan et al
At the HttpCore level one can get a very reliable bytes rent / received,
messages sent / received metrics using HttpConnection#getMetrics method
[1]. The problem is that at the HttpClient level pooled connections are
automatically released once the response has been fully consumed, so the
connection can get immediately allocated to another request making HTTP
connection metrics unreliable. There is a JIRA for this issue
(HTTPCLIENT-1081 [2]).
Oleg
[1]
http://hc.apache.org/httpcomponents-core-ga/httpcore/apidocs/org/apache/http/HttpConnection.html#getMetrics%28%29
http://hc.apache.org/httpcomponents-core-ga/httpcore/apidocs/org/apache/http/HttpConnectionMetrics.html
[2] https://issues.apache.org/jira/browse/HTTPCLIENT-1081
> On Wed, Aug 8, 2012 at 10:19 AM, Jean-Marc Spaggiari <
> jean-marc@spaggiari.org> wrote:
>
> > Hi Ryan,
> >
> > This will calculate the header size. But is there anything else? Is
> > the size only headers + URL lenght for the request, and
> > headers+respons body for the respons? Is there any other CRC,
> > validation, packet, etc.?
> >
> > Because headersize + body lenght seems to not be accurate :(
> >
> > 2012/8/8, Ryan Smith <ry...@gmail.com>:
> > > If HttpClient has a method to do this, can someone let me know? This is
> > a
> > > method I wrote to calculate header size:
> > >
> > > public static int getHeaderByteSize(Header[] headers) {
> > > int requestHeaderByteSize = 0;
> > > for (Header requestHeader : headers) {
> > > if (requestHeader.getName() != null) {
> > > requestHeaderByteSize += requestHeader.getName().getBytes().length;
> > > }
> > > if (requestHeader.getValue() != null) {
> > > requestHeaderByteSize += requestHeader.getValue().getBytes().length;
> > > }
> > > }
> > > return requestHeaderByteSize;
> > > }
> > >
> > > On Wed, Aug 8, 2012 at 10:07 AM, Jean-Marc Spaggiari <
> > > jean-marc@spaggiari.org> wrote:
> > >
> > >> Hi,
> > >>
> > >> Is there a good way to calculate the bandwidth used by a request?
> > >>
> > >> I'm doing a get and I try to track the upload and download bandwidth
> > >> used.
> > >>
> > >> So for a get, there is some upload to the server. There is the URL,
> > >> and the headers. But seems there is a bit more than that and I'm not
> > >> able to figure what.
> > >>
> > >> When I get the respons, I'm using getHeaders("Content-Length") to get
> > >> the content lenght. But I'm most probably missing the header size. I
> > >> also tried EntityUtils.toString(entity).length() but none of them
> > >> seems to be accurate.
> > >>
> > >> So I'm wondering is there is a good way to calculate the bandwidth
> > >> used for upload and download?
> > >>
> > >> Thanks,
> > >>
> > >> JM
> > >>
> > >> ---------------------------------------------------------------------
> > >> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > >> For additional commands, e-mail: httpclient-users-help@hc.apache.org
> > >>
> > >>
> > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> >
> >
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: Bandwidth usage
Posted by Ryan Smith <ry...@gmail.com>.
Right. you also need to calculate what you send to the web server. So you
need to calculate the request headers when you make a request. Then on the
response you get response body size and header size. As far as I know
thats all you need to account for. Maybe Oleg or someone else can verify.
On Wed, Aug 8, 2012 at 10:19 AM, Jean-Marc Spaggiari <
jean-marc@spaggiari.org> wrote:
> Hi Ryan,
>
> This will calculate the header size. But is there anything else? Is
> the size only headers + URL lenght for the request, and
> headers+respons body for the respons? Is there any other CRC,
> validation, packet, etc.?
>
> Because headersize + body lenght seems to not be accurate :(
>
> 2012/8/8, Ryan Smith <ry...@gmail.com>:
> > If HttpClient has a method to do this, can someone let me know? This is
> a
> > method I wrote to calculate header size:
> >
> > public static int getHeaderByteSize(Header[] headers) {
> > int requestHeaderByteSize = 0;
> > for (Header requestHeader : headers) {
> > if (requestHeader.getName() != null) {
> > requestHeaderByteSize += requestHeader.getName().getBytes().length;
> > }
> > if (requestHeader.getValue() != null) {
> > requestHeaderByteSize += requestHeader.getValue().getBytes().length;
> > }
> > }
> > return requestHeaderByteSize;
> > }
> >
> > On Wed, Aug 8, 2012 at 10:07 AM, Jean-Marc Spaggiari <
> > jean-marc@spaggiari.org> wrote:
> >
> >> Hi,
> >>
> >> Is there a good way to calculate the bandwidth used by a request?
> >>
> >> I'm doing a get and I try to track the upload and download bandwidth
> >> used.
> >>
> >> So for a get, there is some upload to the server. There is the URL,
> >> and the headers. But seems there is a bit more than that and I'm not
> >> able to figure what.
> >>
> >> When I get the respons, I'm using getHeaders("Content-Length") to get
> >> the content lenght. But I'm most probably missing the header size. I
> >> also tried EntityUtils.toString(entity).length() but none of them
> >> seems to be accurate.
> >>
> >> So I'm wondering is there is a good way to calculate the bandwidth
> >> used for upload and download?
> >>
> >> Thanks,
> >>
> >> JM
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> >> For additional commands, e-mail: httpclient-users-help@hc.apache.org
> >>
> >>
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
Re: Bandwidth usage
Posted by Jean-Marc Spaggiari <je...@spaggiari.org>.
Hi Ryan,
This will calculate the header size. But is there anything else? Is
the size only headers + URL lenght for the request, and
headers+respons body for the respons? Is there any other CRC,
validation, packet, etc.?
Because headersize + body lenght seems to not be accurate :(
2012/8/8, Ryan Smith <ry...@gmail.com>:
> If HttpClient has a method to do this, can someone let me know? This is a
> method I wrote to calculate header size:
>
> public static int getHeaderByteSize(Header[] headers) {
> int requestHeaderByteSize = 0;
> for (Header requestHeader : headers) {
> if (requestHeader.getName() != null) {
> requestHeaderByteSize += requestHeader.getName().getBytes().length;
> }
> if (requestHeader.getValue() != null) {
> requestHeaderByteSize += requestHeader.getValue().getBytes().length;
> }
> }
> return requestHeaderByteSize;
> }
>
> On Wed, Aug 8, 2012 at 10:07 AM, Jean-Marc Spaggiari <
> jean-marc@spaggiari.org> wrote:
>
>> Hi,
>>
>> Is there a good way to calculate the bandwidth used by a request?
>>
>> I'm doing a get and I try to track the upload and download bandwidth
>> used.
>>
>> So for a get, there is some upload to the server. There is the URL,
>> and the headers. But seems there is a bit more than that and I'm not
>> able to figure what.
>>
>> When I get the respons, I'm using getHeaders("Content-Length") to get
>> the content lenght. But I'm most probably missing the header size. I
>> also tried EntityUtils.toString(entity).length() but none of them
>> seems to be accurate.
>>
>> So I'm wondering is there is a good way to calculate the bandwidth
>> used for upload and download?
>>
>> Thanks,
>>
>> JM
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
>> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>>
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: Bandwidth usage
Posted by Ryan Smith <ry...@gmail.com>.
If HttpClient has a method to do this, can someone let me know? This is a
method I wrote to calculate header size:
public static int getHeaderByteSize(Header[] headers) {
int requestHeaderByteSize = 0;
for (Header requestHeader : headers) {
if (requestHeader.getName() != null) {
requestHeaderByteSize += requestHeader.getName().getBytes().length;
}
if (requestHeader.getValue() != null) {
requestHeaderByteSize += requestHeader.getValue().getBytes().length;
}
}
return requestHeaderByteSize;
}
On Wed, Aug 8, 2012 at 10:07 AM, Jean-Marc Spaggiari <
jean-marc@spaggiari.org> wrote:
> Hi,
>
> Is there a good way to calculate the bandwidth used by a request?
>
> I'm doing a get and I try to track the upload and download bandwidth used.
>
> So for a get, there is some upload to the server. There is the URL,
> and the headers. But seems there is a bit more than that and I'm not
> able to figure what.
>
> When I get the respons, I'm using getHeaders("Content-Length") to get
> the content lenght. But I'm most probably missing the header size. I
> also tried EntityUtils.toString(entity).length() but none of them
> seems to be accurate.
>
> So I'm wondering is there is a good way to calculate the bandwidth
> used for upload and download?
>
> Thanks,
>
> JM
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>