You are viewing a plain text version of this content. The canonical link for it is here.
Posted to httpclient-users@hc.apache.org by Mark Diamond <md...@labridge.com> on 2006/09/11 08:34:22 UTC
Want to stream large datasets to a service (data is generated in memory)
I need to uploaded large amounts of data (potentially > 1.5 gig) to a web
service. I want to use the Postmethod/MultipartRequestEntity, but the issue
is, that I want to stream the data as I generate it. I cannot afford (due to
potential disk space and mem size issues) to write it to disk first or to
create a large string (so FilePart and StringPart are not helpful). Any
advice is much appreciated.. including approaches other than Multipart.
Regards,|
Mark
Re: Want to stream large datasets to a service (data is
generated in memory)
Posted by Oleg Kalnichevski <ol...@apache.org>.
On Tue, 2006-09-12 at 10:15 -0700, Mark_Diamond wrote:
> Hi Oleg,
>
> Thanks for your reply. What puzzles me about doing this is
> that the PostMethod seems to expect the RequestEntity to
> have all the data ready before the POST occurs.
Mark,
This is not the case. RequestEntity#getContentLength should simply
return -1 to signal that the content length is not known or the content
is generated on the fly. This will cause HttpClient to automatically use
chunk encoding when streaming out the entity content.
> The way I
> envisioned this is to have a producer/consumer
> interaction, where I build "Parts" on the fly, and the
> PostMethod sends it out as its received. Before I dig into
> the source, I wanted to get your feel for how doable this
> is within the existing class structure.
>
You can certainly structure the content as a collection of Parts
generated on the fly if that is what you want. MultipartRequestEntity
simply implements RequestEntity interface and works the same way
described above. Be warned, though, some multipart parsers expect the
content to be delimited by the Content-Length header and may have issues
if the content is chunk-encoded.
Hope this helps
Oleg
> Thanks,
> Mark
>
> On Mon, 11 Sep 2006 15:00:16 +0200
> Oleg Kalnichevski <ol...@apache.org> wrote:
> > On Sun, 2006-09-10 at 23:34 -0700, Mark Diamond wrote:
> >> I need to uploaded large amounts of data (potentially >
> >>1.5 gig) to a web
> >> service. I want to use the
> >>Postmethod/MultipartRequestEntity, but the issue
> >> is, that I want to stream the data as I generate it. I
> >>cannot afford (due to
> >> potential disk space and mem size issues) to write it to
> >>disk first or to
> >> create a large string (so FilePart and StringPart are
> >>not helpful). Any
> >> advice is much appreciated.. including approaches other
> >>than Multipart.
> >>
> >> Regards,|
> >> Mark
> >
> > Mark,
> >
> > My recommendation would be to implement a custom
> >RequestEntity.
> > RequestEntity interface represents an arbitrary entity
> >capable of
> > generating content on-fly and streaming it out using the
> >OutputStream
> > interface. You can device any serialization /
> >deserialization protocol
> > that suits your application best.
> >
> > Hope this helps
> >
> > Oleg
> >
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: httpclient-user-help@jakarta.apache.org
Re: Want to stream large datasets to a service (data is generated
in memory)
Posted by Oleg Kalnichevski <ol...@apache.org>.
On Sun, 2006-09-10 at 23:34 -0700, Mark Diamond wrote:
> I need to uploaded large amounts of data (potentially > 1.5 gig) to a web
> service. I want to use the Postmethod/MultipartRequestEntity, but the issue
> is, that I want to stream the data as I generate it. I cannot afford (due to
> potential disk space and mem size issues) to write it to disk first or to
> create a large string (so FilePart and StringPart are not helpful). Any
> advice is much appreciated.. including approaches other than Multipart.
>
> Regards,|
> Mark
Mark,
My recommendation would be to implement a custom RequestEntity.
RequestEntity interface represents an arbitrary entity capable of
generating content on-fly and streaming it out using the OutputStream
interface. You can device any serialization / deserialization protocol
that suits your application best.
Hope this helps
Oleg
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: httpclient-user-help@jakarta.apache.org