Re: reading entity bytes from a HttpEntityEnclosingRequest in case of large content size

[Oleg Kalnichevski <ol...@apache.org>]

On Sun, Aug 16, 2009 at 11:49:20PM +0530, Brijesh Deo wrote:
> Hi,
> 
>  
> 
> I am implementing an http reverse proxy, somewhat on the lines of the
> sample implementation - ElementalReverseProxy with some modifications. I
> am using HttpCore classes to implement the http server (using
> org.apache.http.protocol.HttpService) and the HttpClient api (instead of
> the HttpRequestExecutor) to implement the client side for sending the
> Http requests to the target servers and receiving Http Responses from
> them. I am using the latest HttpClient 4.0 binaries bundle (with
> dependencies).
> 
>  
> 
> I have a requirement to store Http Post data from some webpages
> (accessed through my reverse proxy) when they are submitted and later
> Post the stored data to a different target server or the same target
> server. For this, I read the entity data into a byte array whenever a
> HttpPost request comes into my http server. I can store the byte array
> into a file/database table. Then I recreate a ByteArrayEntity from the
> saved byte array and re-populate the HttpPost request object and send it
> to the intended target server using the HttpClient instance. This works
> fine when the content type is "application/x-www-form-urlencoded" or
> even "multipart/form-data". 
> 
>  
> 
> However, in case of "multipart/form-data", if the content size is huge
> (in cases of larger file uploads > 50MB) I have some doubts/concerns
> whether I will be able to read all the entity data completely. I am
> reading the entity data in the way shown below: My questions are
> mentioned below.
> 
>  
> 
> if (request instanceof HttpEntityEnclosingRequest) {
> 
>       HttpEntity entity =
> ((HttpEntityEnclosingRequest)request).getEntity();
> 
>       byte entityBytes[] = null;          
> 
>       if (entity != null) {
> 
> ByteArrayOutputStream baos = new ByteArrayOutputStream() ;
> 
>             entity.writeTo(baos);
> 
>             entityBytes = baos.toByteArray() ;
> 
>             baos.flush();
> 
>             baos.close();
> 
>             entity.consumeContent();
> 
>             Question 1: Does calling consumeContent() here close the
> underlying socket connection with the client(browser)? In that case, how
> would the HttpResponse be sent back to the client(browser) using this
> connection?
> 

No, it does not. Actually in this particular case entity#consumeContent() will
have no effect as entity#writeTo will have fully consumed the entity content.

>       }
> 
> if (entityBytes != null) {
> 
>             ByteArrayEntity byteArrayEntity = new
> ByteArrayEntity(entityBytes);
> 
>  
> ((HttpEntityEnclosingRequest)request).setEntity(byteArrayEntity);
> 
>       }
> 
> }
> 
>  
> 
> Question 2: As mentioned above, if it is a large file upload request
> (multipart/form-data), is it guaranteed that I will be able to read all
> the bytes of the file being uploaded with the above approach? 
> 
> 

Yes, it is

> 
> Question 3: Is there any possibility that I could receive more than one
> HttpRequest object(s) for such a (multipart/form-data) post with the
> bytes of the uploaded file contained partially in each of those request
> object entities? If that is a possibility, how can I associate/append
> all such request entities to form a single piece of byte array?
> 
>

I seriously doubt that, but it may depend on the HTTP client.
 
> 
> Question 4: Is there a more efficient method to achieve this using some
> other classes/methods of the HttpCore and HttpClient apis?
> 
> 

If you need to create a presistent copy of each request you should seriously
consider streaming content directly from the client request to a file or a
database BLOB without buffering it in memory and then streaming it out from the
file or the BLOB to the target server.
 
Hope this helps

Oleg

> 
> I would be thankful for any guidance or help here.
> 
>  
> 
> Thanks,
> 
> Brijesh
> 
>  
> 
>  
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org