You are viewing a plain text version of this content. The canonical link for it is here.
Posted to httpclient-users@hc.apache.org by Joe Barnes <ba...@gmail.com> on 2014/12/18 00:37:53 UTC
Need a way to send a URL's stream as multi-part entity with
repeatable blocks
Hello community!
My team is faced with a problem where we need to send a resource that is
available via URL (more precisely, it is on our classpath). We are running
in a servlet container, so we don't have direct access to the file system.
I completely understand why the chunks are marked as "Non-repeatable" when
using an InputStream, because in general they are not necessarily
repeatable.
Currently we are reading the URL in its entirety to memory, then shipping
it off. This is quite burdensome because of the size of the resources and
number of concurrent users we can expect. The best we can do is create a
singleton of the value in memory, but it still is expensive for something
which could certainly be streamed.
Are there any approaches we can utilize to send a URL's contents to a
multi-part while being repeatable without reading it entirely into memory?
Thanks!
Joe
Re: Need a way to send a URL's stream as multi-part entity with
repeatable blocks
Posted by Joe Barnes <ba...@gmail.com>.
Oleg,
Sorry for my slow reply, but I wanted to report that this solution worked
well in case it helps others in the future. Thank you for taking the time
to help!
Joe
On Thu, Dec 18, 2014 at 12:12 PM, Joe Barnes <ba...@gmail.com> wrote:
> Oh wow... I had not noticed that method on URL. I'll certainly give it a
> try. Thanks!
>
> Joe
>
> On Thu, Dec 18, 2014 at 10:06 AM, Oleg Kalnichevski <ol...@apache.org>
> wrote:
>>
>> On Thu, 2014-12-18 at 09:23 -0600, Joe Barnes wrote:
>> > Thanks for the insight, Oleg. I was not aware that the key to
>> > repeatability was knowing the length in advance. That seems to suggest
>> we
>> > could have an API that accepted the stream and a length to produce a
>> > repeatable multipart entity.
>> >
>> > As for converting the URLs to files, I'm not sure there is a clean way
>> to
>> > do that since we are running in a servlet container without access to
>> the
>> > file system.
>>
>> I was thinking about something rather trivial, like that:
>>
>> URL resource = Testing.class.getResource("/my-stuff.bin");
>> File file = new File(resource.getFile());
>> System.out.println(file + " exists: " + file.exists());
>>
>> Oleg
>>
>>
>> > I'll do some research and see if there is a means of making
>> > this conversion that I'm not currently aware of.
>> >
>> > Joe
>> >
>> >
>> > On Thu, Dec 18, 2014 at 3:17 AM, Oleg Kalnichevski <ol...@apache.org>
>> wrote:
>> > >
>> > > On Wed, 2014-12-17 at 17:37 -0600, Joe Barnes wrote:
>> > > > Hello community!
>> > > >
>> > > > My team is faced with a problem where we need to send a resource
>> that is
>> > > > available via URL (more precisely, it is on our classpath). We are
>> > > running
>> > > > in a servlet container, so we don't have direct access to the file
>> > > system.
>> > > > I completely understand why the chunks are marked as
>> "Non-repeatable"
>> > > when
>> > > > using an InputStream, because in general they are not necessarily
>> > > > repeatable.
>> > > >
>> > > > Currently we are reading the URL in its entirety to memory, then
>> shipping
>> > > > it off. This is quite burdensome because of the size of the
>> resources
>> > > and
>> > > > number of concurrent users we can expect. The best we can do is
>> create a
>> > > > singleton of the value in memory, but it still is expensive for
>> something
>> > > > which could certainly be streamed.
>> > > >
>> > > > Are there any approaches we can utilize to send a URL's contents to
>> a
>> > > > multi-part while being repeatable without reading it entirely into
>> > > memory?
>> > > >
>> > > > Thanks!
>> > > > Joe
>> > >
>> > > Joe,
>> > >
>> > > Presently MultipartFormEntity cannot be repeated if length of any of
>> its
>> > > parts is unknown, which is the case with body parts backed by a
>> > > InputStream or a URL. In your particular case, however, given that
>> URLs
>> > > represent local resources you should be able to convert them to File
>> > > instances and thus make their length known in advance.
>> > >
>> > > Oleg
>> > >
>> > >
>> > >
>> > > ---------------------------------------------------------------------
>> > > To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
>> > > For additional commands, e-mail: httpclient-users-help@hc.apache.org
>> > >
>> > >
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
>> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>>
>>
Re: Need a way to send a URL's stream as multi-part entity with
repeatable blocks
Posted by Joe Barnes <ba...@gmail.com>.
Oh wow... I had not noticed that method on URL. I'll certainly give it a
try. Thanks!
Joe
On Thu, Dec 18, 2014 at 10:06 AM, Oleg Kalnichevski <ol...@apache.org>
wrote:
>
> On Thu, 2014-12-18 at 09:23 -0600, Joe Barnes wrote:
> > Thanks for the insight, Oleg. I was not aware that the key to
> > repeatability was knowing the length in advance. That seems to suggest
> we
> > could have an API that accepted the stream and a length to produce a
> > repeatable multipart entity.
> >
> > As for converting the URLs to files, I'm not sure there is a clean way to
> > do that since we are running in a servlet container without access to the
> > file system.
>
> I was thinking about something rather trivial, like that:
>
> URL resource = Testing.class.getResource("/my-stuff.bin");
> File file = new File(resource.getFile());
> System.out.println(file + " exists: " + file.exists());
>
> Oleg
>
>
> > I'll do some research and see if there is a means of making
> > this conversion that I'm not currently aware of.
> >
> > Joe
> >
> >
> > On Thu, Dec 18, 2014 at 3:17 AM, Oleg Kalnichevski <ol...@apache.org>
> wrote:
> > >
> > > On Wed, 2014-12-17 at 17:37 -0600, Joe Barnes wrote:
> > > > Hello community!
> > > >
> > > > My team is faced with a problem where we need to send a resource
> that is
> > > > available via URL (more precisely, it is on our classpath). We are
> > > running
> > > > in a servlet container, so we don't have direct access to the file
> > > system.
> > > > I completely understand why the chunks are marked as "Non-repeatable"
> > > when
> > > > using an InputStream, because in general they are not necessarily
> > > > repeatable.
> > > >
> > > > Currently we are reading the URL in its entirety to memory, then
> shipping
> > > > it off. This is quite burdensome because of the size of the
> resources
> > > and
> > > > number of concurrent users we can expect. The best we can do is
> create a
> > > > singleton of the value in memory, but it still is expensive for
> something
> > > > which could certainly be streamed.
> > > >
> > > > Are there any approaches we can utilize to send a URL's contents to a
> > > > multi-part while being repeatable without reading it entirely into
> > > memory?
> > > >
> > > > Thanks!
> > > > Joe
> > >
> > > Joe,
> > >
> > > Presently MultipartFormEntity cannot be repeated if length of any of
> its
> > > parts is unknown, which is the case with body parts backed by a
> > > InputStream or a URL. In your particular case, however, given that URLs
> > > represent local resources you should be able to convert them to File
> > > instances and thus make their length known in advance.
> > >
> > > Oleg
> > >
> > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> > >
> > >
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
Re: Need a way to send a URL's stream as multi-part entity with
repeatable blocks
Posted by Oleg Kalnichevski <ol...@apache.org>.
On Thu, 2014-12-18 at 09:23 -0600, Joe Barnes wrote:
> Thanks for the insight, Oleg. I was not aware that the key to
> repeatability was knowing the length in advance. That seems to suggest we
> could have an API that accepted the stream and a length to produce a
> repeatable multipart entity.
>
> As for converting the URLs to files, I'm not sure there is a clean way to
> do that since we are running in a servlet container without access to the
> file system.
I was thinking about something rather trivial, like that:
URL resource = Testing.class.getResource("/my-stuff.bin");
File file = new File(resource.getFile());
System.out.println(file + " exists: " + file.exists());
Oleg
> I'll do some research and see if there is a means of making
> this conversion that I'm not currently aware of.
>
> Joe
>
>
> On Thu, Dec 18, 2014 at 3:17 AM, Oleg Kalnichevski <ol...@apache.org> wrote:
> >
> > On Wed, 2014-12-17 at 17:37 -0600, Joe Barnes wrote:
> > > Hello community!
> > >
> > > My team is faced with a problem where we need to send a resource that is
> > > available via URL (more precisely, it is on our classpath). We are
> > running
> > > in a servlet container, so we don't have direct access to the file
> > system.
> > > I completely understand why the chunks are marked as "Non-repeatable"
> > when
> > > using an InputStream, because in general they are not necessarily
> > > repeatable.
> > >
> > > Currently we are reading the URL in its entirety to memory, then shipping
> > > it off. This is quite burdensome because of the size of the resources
> > and
> > > number of concurrent users we can expect. The best we can do is create a
> > > singleton of the value in memory, but it still is expensive for something
> > > which could certainly be streamed.
> > >
> > > Are there any approaches we can utilize to send a URL's contents to a
> > > multi-part while being repeatable without reading it entirely into
> > memory?
> > >
> > > Thanks!
> > > Joe
> >
> > Joe,
> >
> > Presently MultipartFormEntity cannot be repeated if length of any of its
> > parts is unknown, which is the case with body parts backed by a
> > InputStream or a URL. In your particular case, however, given that URLs
> > represent local resources you should be able to convert them to File
> > instances and thus make their length known in advance.
> >
> > Oleg
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> >
> >
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: Need a way to send a URL's stream as multi-part entity with
repeatable blocks
Posted by Joe Barnes <ba...@gmail.com>.
Thanks for the insight, Oleg. I was not aware that the key to
repeatability was knowing the length in advance. That seems to suggest we
could have an API that accepted the stream and a length to produce a
repeatable multipart entity.
As for converting the URLs to files, I'm not sure there is a clean way to
do that since we are running in a servlet container without access to the
file system. I'll do some research and see if there is a means of making
this conversion that I'm not currently aware of.
Joe
On Thu, Dec 18, 2014 at 3:17 AM, Oleg Kalnichevski <ol...@apache.org> wrote:
>
> On Wed, 2014-12-17 at 17:37 -0600, Joe Barnes wrote:
> > Hello community!
> >
> > My team is faced with a problem where we need to send a resource that is
> > available via URL (more precisely, it is on our classpath). We are
> running
> > in a servlet container, so we don't have direct access to the file
> system.
> > I completely understand why the chunks are marked as "Non-repeatable"
> when
> > using an InputStream, because in general they are not necessarily
> > repeatable.
> >
> > Currently we are reading the URL in its entirety to memory, then shipping
> > it off. This is quite burdensome because of the size of the resources
> and
> > number of concurrent users we can expect. The best we can do is create a
> > singleton of the value in memory, but it still is expensive for something
> > which could certainly be streamed.
> >
> > Are there any approaches we can utilize to send a URL's contents to a
> > multi-part while being repeatable without reading it entirely into
> memory?
> >
> > Thanks!
> > Joe
>
> Joe,
>
> Presently MultipartFormEntity cannot be repeated if length of any of its
> parts is unknown, which is the case with body parts backed by a
> InputStream or a URL. In your particular case, however, given that URLs
> represent local resources you should be able to convert them to File
> instances and thus make their length known in advance.
>
> Oleg
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
Re: Need a way to send a URL's stream as multi-part entity with
repeatable blocks
Posted by Oleg Kalnichevski <ol...@apache.org>.
On Wed, 2014-12-17 at 17:37 -0600, Joe Barnes wrote:
> Hello community!
>
> My team is faced with a problem where we need to send a resource that is
> available via URL (more precisely, it is on our classpath). We are running
> in a servlet container, so we don't have direct access to the file system.
> I completely understand why the chunks are marked as "Non-repeatable" when
> using an InputStream, because in general they are not necessarily
> repeatable.
>
> Currently we are reading the URL in its entirety to memory, then shipping
> it off. This is quite burdensome because of the size of the resources and
> number of concurrent users we can expect. The best we can do is create a
> singleton of the value in memory, but it still is expensive for something
> which could certainly be streamed.
>
> Are there any approaches we can utilize to send a URL's contents to a
> multi-part while being repeatable without reading it entirely into memory?
>
> Thanks!
> Joe
Joe,
Presently MultipartFormEntity cannot be repeated if length of any of its
parts is unknown, which is the case with body parts backed by a
InputStream or a URL. In your particular case, however, given that URLs
represent local resources you should be able to convert them to File
instances and thus make their length known in advance.
Oleg
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org