[HttpClient] delaying parseResponse()?

[Dmitry Beransky <db...@dembel.org>]

Hi,

I'm working with the Webdav Client library -- which, in turn, uses the 
HttpClient lib -- to transfer large (several hundred MBs) files between 
servers.  The problem I'm having has to do with insistence of 
HttpClientcode to transfer content data to a local storage (be it a temp 
file or memory), immediately upon a method's execution.  In my case, I 
don't need to store the content locally, but rather want to pump it to 
another server.  Looking at the code, I see that the reason for this 
behavior lies with the call to parseResponse() from within executeMethod().

My question is, would it be possible to change the behavior of the code, 
to, having parsed the headers, leave the content data inside the original 
input stream, so that, if needed, this stream could be returned to the 
caller instead of a new one made from a locally cached copy?  A new method, 
let's say, getMethodDataUnprocessed() would simply return this stream, 
while calling getMethodData() would make a call to parseResponse() first, 
and then continue with the old behavior.

Comments?

Regards
Dmitry


--
To unsubscribe, e-mail:   <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>

Re: [HttpClient] delaying parseResponse()?

[Dmitry Beransky <db...@dembel.org>]

I forgot to mention a couple of things.

At 11:08 AM 1/20/2002, Dirk Verbeeck wrote:
>I would implement GetMethod.setUseBuffer to disable buffering and delay
>reading the response content. getData() can then directly return the
>input stream from the response. A second call to getData should call
>reset() on the InputStream and this will probably throw an exception so
>you can only get the data once.
>Postmethod can use the same method, setUseBuffer will then make that
>sendData(InputStream is) doesn't buffer the inputstream. A second
>execute needs a reset() in streamQuery throwing an IOException if not
>possible.

Earlier today I gave the code a closer look and came up with a similar 
scheme.  However, there seems to be a major problem with it.  The problem 
is with handling  the connection.  Currently, executeMethod() makes a call 
to needToCloseConnection() and closeConnection() at the end of the 
execution.  If parseResponse() were to be delayed in the manner we talked 
of, then it would be up to the method to close the connection if 
necessary.  But it doesn't look like it currently has the necessary 
information to both make that decision and to execute it.  I couldn't think 
of an elegant and simple way of fixing this.  Any proper solution, it 
seems, would require a substantial overhaul of the existing 
architecture.  And that's the part I don't currently have the time for :)

Cheers
Dmitry



--
To unsubscribe, e-mail:   <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>

Re: [HttpClient] delaying parseResponse()?

[Dmitry Beransky <db...@ucsd.edu>]

At 11:08 AM 1/20/2002, Dirk Verbeeck wrote:
>PostMethod also has sendData(URL url) and this is probably the solution
>for your problem.
>
>destinationResource.putMethod(sourceResource.getHttpURL().toURL());

Yes!  Don't I feel silly now? :-)  In all the plentitude of putMethod() 
calls I overlooked the one I needed :)

Although, if the other method worked the way I thought it did, it would've 
provided an easy way to monitor the transfer progress.  Here, I think, I'd 
have to resort to some black magic.  Oh, well, unfortunately, I don't have 
enough time at the moment to implement the necessary changes.  This will do 
just fine for the time being.

Regards
Dmitry
  


--
To unsubscribe, e-mail:   <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>

Re: [HttpClient] delaying parseResponse()?

[Dirk Verbeeck <di...@pandora.be>]

Same answer as in the Slide User List, but maybe interesting for people
who are not subscribed to the list.

Note: Slide uses the HttpClient 1.x version, I didn't check what the
behaviour of version 2.x is.
---
GetMethod streams everything to disk or memory in parseResponse.
PostMethod.sendData(InputStream is) creates a memory buffer.

PostMethod also has sendData(URL url) and this is probably the solution
for your problem.

destinationResource.putMethod(sourceResource.getHttpURL().toURL());

This way the JDK functions are used to get the content so your
enhancement request is still valid. But first try this, if it doesn't
work we can start improving HttpClient. You can always submit a patch if
you want.

I would implement GetMethod.setUseBuffer to disable buffering and delay
reading the response content. getData() can then directly return the
input stream from the response. A second call to getData should call
reset() on the InputStream and this will probably throw an exception so
you can only get the data once.
Postmethod can use the same method, setUseBuffer will then make that
sendData(InputStream is) doesn't buffer the inputstream. A second
execute needs a reset() in streamQuery throwing an IOException if not
possible.


Dirk


Dmitry Beransky wrote:
> 
> Hi,
> 
> I'm working with the Webdav Client library -- which, in turn, uses the
> HttpClient lib -- to transfer large (several hundred MBs) files between
> servers.  The problem I'm having has to do with insistence of
> HttpClientcode to transfer content data to a local storage (be it a temp
> file or memory), immediately upon a method's execution.  In my case, I
> don't need to store the content locally, but rather want to pump it to
> another server.  Looking at the code, I see that the reason for this
> behavior lies with the call to parseResponse() from within executeMethod().
> 
> My question is, would it be possible to change the behavior of the code,
> to, having parsed the headers, leave the content data inside the original
> input stream, so that, if needed, this stream could be returned to the
> caller instead of a new one made from a locally cached copy?  A new method,
> let's say, getMethodDataUnprocessed() would simply return this stream,
> while calling getMethodData() would make a call to parseResponse() first,
> and then continue with the old behavior.
> 
> Comments?
> 
> Regards
> Dmitry


--
To unsubscribe, e-mail:   <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>