Httpclient 3.1 - reading the response from a GET request...

[Michael Burbidge <mb...@adobe.com>]

I think I might have a fundamental misunderstanding about how httpclient works.

Suppose I have set up a get request using the following code snippet:

	GetMethod getMethod = new GetMethod("https://somedownloadurl.com");
	getMethod.setQueryString(someQueryString);
	getMethod.setParams(methodParams);

Once I've created my get method request I execute it using the following snippet:

	in responseCode = httpClient.executeMethod(getMethod);

I get back a successful response code and so I get the input stream to read the body, in this case the response body contains an image. I get the stream using the following call:

InputStream responseStream = getMethod.getResponseBodyAsStream();

My question is about reading the bytes back from the input stream. Our code used to do the following to read the input stream:

int numRead;
byte[] chunk = new byte[BUFFER_SIZE];

while (responseStream.available())
{
    numRead = responseStream.read(chunk);
}

I'm leaving out what we do with the bytes, because my question is about reading the stream.

This code worked for a couple of years over several releases. Then suddenly during the testing cycle in one of our releases the following call started returning 0.

responseStream.available()

I never was able to determine what changed. Both the server stack and client stack are pretty complex.

So I read the documentation on available() and concluded that we were misusing it. That call simply means that there are no bytes available without blocking. So I changed the code to the following:

byte[] chunk = new byte[BUFFER_SIZE];

int numRead = responseStream.read(chunk);
while (numRead > 0)
{
    numRead = responseStream.read(chunk);
}

That works, but I'm puzzled by how it behaves. No matter what buffer size I give it, the read always returns only 1448 bytes at a time. One could conclude that the server is not able to keep up with the client. Even if I put the following sleep statement in the while loop, to stall the client. I still get only 1448 bytes at a time.

byte[] chunk = new byte[BUFFER_SIZE];

int numRead = responseStream.read(chunk);
while (numRead > 0)
{
    Thread.sleep(500);
    numRead = responseStream.read(chunk);
}

If I do a tcpdump I can see that the packet size coming across the wire corresponds to the size of data that I'm getting back in each read request on the stream. My assumption was that somewhere in the client networking stack, data would be read asynchronously as it came over the wire. And that when I did a read on the stream I would get back large chunks of data, that had been accumulated in buffers at the lower layers of the stack.

Is that a complete misunderstanding of what happens? Do you have any idea what could cause a change in the behavior of the available() call? Do you think these two symptoms are related?

Thanks,
Michael-
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org

Re: Httpclient 3.1 - reading the response from a GET request...

[Fredrik Jonson <fr...@jonson.org>]

Hi Michael,

You're correct not to make any assumptions about the available() method
any more. You can only trust available() in very specific environments, so
the easiest option is to not use it at all in heterogenous enviroments.

Whether the tcp segments are reassembled or not, before they are made available
to the java InputStream is likely up to a combination of the operating system,
libc and the jvm. It is probably one or more of those three players that have
changed in Your setup. Right?

I wouldn't worry too much about the exact size of segments returned. As long
as you do not read the data one byte at a time, it is unlikely that the byte
copying will be a hotspot in the application anyway.

--
Fredrik Jonson


---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org