Re: Want to stream large datasets to a service (data is generated in memory)

[Oleg Kalnichevski <ol...@apache.org>]

On Tue, 2006-09-12 at 10:15 -0700, Mark_Diamond wrote:
> Hi Oleg,
> 
> Thanks for your reply. What puzzles me about doing this is 
> that the PostMethod seems to expect the RequestEntity to 
> have all the data ready before the POST occurs. 

Mark,

This is not the case. RequestEntity#getContentLength should simply
return -1 to signal that the content length is not known or the content
is generated on the fly. This will cause HttpClient to automatically use
chunk encoding when streaming out the entity content.

> The way I 
> envisioned this is to have a producer/consumer 
> interaction, where I build "Parts" on the fly, and the 
> PostMethod sends it out as its received. Before I dig into 
> the source, I wanted to get your feel for how doable this 
> is within the existing class structure.
> 

You can certainly structure the content as a collection of Parts
generated on the fly if that is what you want. MultipartRequestEntity
simply implements RequestEntity interface and works the same way
described above. Be warned, though, some multipart parsers expect the
content to be delimited by the Content-Length header and may have issues
if the content is chunk-encoded.

Hope this helps

Oleg

> Thanks,
> Mark
> 
> On Mon, 11 Sep 2006 15:00:16 +0200
>   Oleg Kalnichevski <ol...@apache.org> wrote:
> > On Sun, 2006-09-10 at 23:34 -0700, Mark Diamond wrote:
> >> I need to uploaded large amounts of data (potentially > 
> >>1.5 gig) to a web
> >> service. I want to use the 
> >>Postmethod/MultipartRequestEntity, but the issue
> >> is, that I want to stream the data as I generate it. I 
> >>cannot afford (due to
> >> potential disk space and mem size issues) to write it to 
> >>disk first or to
> >> create a large string (so FilePart and StringPart are 
> >>not helpful). Any
> >> advice is much appreciated.. including approaches other 
> >>than Multipart.
> >> 
> >> Regards,|
> >> Mark
> > 
> > Mark,
> > 
> > My recommendation would be to implement a custom 
> >RequestEntity.
> > RequestEntity interface represents an arbitrary entity 
> >capable of
> > generating content on-fly and streaming it out using the 
> >OutputStream
> > interface. You can device any serialization / 
> >deserialization protocol
> > that suits your application best.
> > 
> > Hope this helps
> > 
> > Oleg
> > 
> 
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: httpclient-user-help@jakarta.apache.org