You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cxf.apache.org by Mustafa Sezgin <ms...@aconex.com> on 2009/07/15 03:31:34 UTC
Best way to handle large multi part msgs
Hi all,
Just a quick question about handling multi part msgs. We are writing a
number of Rest services using CXF + Spring. Some of these services will
need to accept files, some of which are quite large. We deal with lots of
data from lots of users.
I am wondering what the best way to do this is. Currently, as an argument
to the service method, we specify a MultipartBody object, thus we will be
requesting users of our services send these files in multi part msgs where
each file is a single part within the multi part msg. However I have my
doubts about this. How does CXF handle large files? Say if someone sends
3x2GB files? From what I can see, the service does not actually get passed
the multi part msg until the msg has been received in full. Thus the data
received (in total 6GB) will need to be stored somewhere, before being
passed to the service. Will this be written to disk, and if so, is this
configurable in anyway? I am presuming CXF doesn't store all this data in
memory, as this would seem like the wrong way to do this...
If there are 'better' ways to do this with CXF, I would love to hear
them..
Thanks in advance for the responses..
Mustafa
Re: Best way to handle large multi part msgs
Posted by Sergey Beryozkin <sb...@progress.com>.
Hi,
As far as JAX-RS endpoints are concerned, it is only 2 more properties (in addition to those attachment-related ones) that can be
used, all other configuration is done explicitly with jaxrs:endpoint attributes and elements, as well as by configuring individual
providers. These (boolean) properties are :
"org.apache.cxf.output.buffering" - if set to true then JAXB provider will write to a cached stream - can be useful in cases when
JAXB has already serialized some data but then failed for some reasons to complete the process
"org.apache.cxf.output.serviceloader-context" - will cause JAXRSInvoker, before invoking on a service object, to set a context
loader to the loader which loaded a service class...
cheers, Sergey
P.S. Just looked at the JAX-RS docs, Confluence has just completely broken all the java/xml fragments... very unreadable, probably
I'll just get rid of {code} templates
----- Original Message -----
From: "Mustafa Sezgin" <ms...@aconex.com>
To: <us...@cxf.apache.org>
Sent: Thursday, July 16, 2009 1:29 AM
Subject: RE: Best way to handle large multi part msgs
> Thanks for the responses. I will look into those configurations. Is there
> a list where I can look at all possible properties that I can configure in
> relation to a service end point?
>
>
> -----Original Message-----
> From: Sergey Beryozkin [mailto:sergey.beryozkin@iona.com]
> Sent: Wednesday, 15 July 2009 6:49 PM
> To: users@cxf.apache.org
> Subject: Re: Best way to handle large multi part msgs
>
>
> Yes, it looks like one can also set
>
> org.apache.cxf.io.CachedOutputStream.Threshold
> and
> org.apache.cxf.io.CachedOutputStream.OutputDirectory
>
> but these are system properties thus they would apply to the whole
> container
> instance...as opposed to per-endpoint attachment-related properties which
> I
> referred to in the previous email but it may be ok in some cases too.
>
>> Also, you require to increase your web container's socket timeout time,
> as the send and receive operations will take some time.
>
> yes, this is also possible to do with JAX-RS.
>
> cheers, Sergey
>
> Mayank Mishra-3 wrote:
>>
>> Mustafa Sezgin wrote:
>>> Hi all,
>>>
>>>
>>>
>>> Just a quick question about handling multi part msgs. We are writing a
>>> number of Rest services using CXF + Spring. Some of these services will
>>> need to accept files, some of which are quite large. We deal with lots
> of
>>> data from lots of users.
>>>
>>> I am wondering what the best way to do this is. Currently, as an
> argument
>>> to the service method, we specify a MultipartBody object, thus we will
> be
>>> requesting users of our services send these files in multi part msgs
>>> where
>>> each file is a single part within the multi part msg. However I have my
>>> doubts about this. How does CXF handle large files? Say if someone
> sends
>>> 3x2GB files? From what I can see, the service does not actually get
>>> passed
>>> the multi part msg until the msg has been received in full. Thus the
> data
>>> received (in total 6GB) will need to be stored somewhere, before being
>>> passed to the service. Will this be written to disk, and if so, is this
>>> configurable in anyway? I am presuming CXF doesn't store all this data
> in
>>> memory, as this would seem like the wrong way to do this...
>>>
>>>
>>>
>>> If there are 'better' ways to do this with CXF, I would love to hear
>>> them..
>>>
>>>
>> You can use MTOM, there is a system property
>> "org.apache.cxf.io.CachedOutputStream.Threshold" that you can set, by
>> default its value is 64kb, any attachment more than that is saved in a
>> temp file, you can also configure this temp file directory by another
>> system property "org.apache.cxf.io.CachedOutputStream.OutputDirectory".
>> Also, you require to increase your web container's socket timeout time,
>> as the send and receive operations will take some time. You can use this
>
>> with JAX-WS, but I am not sure about JAX-RS.
>>
>> You can otherwise you can build RESTful endpoints through JAX-WS
>> provider interface, where you can use StreamSource.
>>
>> I tried but may be someone on list can give throw more light on 'better'
>
>> ways. :)
>>
>> With Regards,
>> Mayank
>>> You can use MTOM for end
>>>
>>> Thanks in advance for the responses..
>>>
>>>
>>>
>>> Mustafa
>>>
>>>
>>>
>>>
>>>
>>
>>
>>
>
> --
> View this message in context:
> http://www.nabble.com/Best-way-to-handle-large-multi-part-msgs-tp24490493p
> 24494135.html
> Sent from the cxf-user mailing list archive at Nabble.com.
>
>
RE: Best way to handle large multi part msgs
Posted by Mustafa Sezgin <ms...@aconex.com>.
Thanks for the responses. I will look into those configurations. Is there
a list where I can look at all possible properties that I can configure in
relation to a service end point?
-----Original Message-----
From: Sergey Beryozkin [mailto:sergey.beryozkin@iona.com]
Sent: Wednesday, 15 July 2009 6:49 PM
To: users@cxf.apache.org
Subject: Re: Best way to handle large multi part msgs
Yes, it looks like one can also set
org.apache.cxf.io.CachedOutputStream.Threshold
and
org.apache.cxf.io.CachedOutputStream.OutputDirectory
but these are system properties thus they would apply to the whole
container
instance...as opposed to per-endpoint attachment-related properties which
I
referred to in the previous email but it may be ok in some cases too.
> Also, you require to increase your web container's socket timeout time,
as the send and receive operations will take some time.
yes, this is also possible to do with JAX-RS.
cheers, Sergey
Mayank Mishra-3 wrote:
>
> Mustafa Sezgin wrote:
>> Hi all,
>>
>>
>>
>> Just a quick question about handling multi part msgs. We are writing a
>> number of Rest services using CXF + Spring. Some of these services will
>> need to accept files, some of which are quite large. We deal with lots
of
>> data from lots of users.
>>
>> I am wondering what the best way to do this is. Currently, as an
argument
>> to the service method, we specify a MultipartBody object, thus we will
be
>> requesting users of our services send these files in multi part msgs
>> where
>> each file is a single part within the multi part msg. However I have my
>> doubts about this. How does CXF handle large files? Say if someone
sends
>> 3x2GB files? From what I can see, the service does not actually get
>> passed
>> the multi part msg until the msg has been received in full. Thus the
data
>> received (in total 6GB) will need to be stored somewhere, before being
>> passed to the service. Will this be written to disk, and if so, is this
>> configurable in anyway? I am presuming CXF doesn't store all this data
in
>> memory, as this would seem like the wrong way to do this...
>>
>>
>>
>> If there are 'better' ways to do this with CXF, I would love to hear
>> them..
>>
>>
> You can use MTOM, there is a system property
> "org.apache.cxf.io.CachedOutputStream.Threshold" that you can set, by
> default its value is 64kb, any attachment more than that is saved in a
> temp file, you can also configure this temp file directory by another
> system property "org.apache.cxf.io.CachedOutputStream.OutputDirectory".
> Also, you require to increase your web container's socket timeout time,
> as the send and receive operations will take some time. You can use this
> with JAX-WS, but I am not sure about JAX-RS.
>
> You can otherwise you can build RESTful endpoints through JAX-WS
> provider interface, where you can use StreamSource.
>
> I tried but may be someone on list can give throw more light on 'better'
> ways. :)
>
> With Regards,
> Mayank
>> You can use MTOM for end
>>
>> Thanks in advance for the responses..
>>
>>
>>
>> Mustafa
>>
>>
>>
>>
>>
>
>
>
--
View this message in context:
http://www.nabble.com/Best-way-to-handle-large-multi-part-msgs-tp24490493p
24494135.html
Sent from the cxf-user mailing list archive at Nabble.com.
Re: Best way to handle large multi part msgs
Posted by Sergey Beryozkin <se...@iona.com>.
Yes, it looks like one can also set
org.apache.cxf.io.CachedOutputStream.Threshold
and
org.apache.cxf.io.CachedOutputStream.OutputDirectory
but these are system properties thus they would apply to the whole container
instance...as opposed to per-endpoint attachment-related properties which I
referred to in the previous email but it may be ok in some cases too.
> Also, you require to increase your web container's socket timeout time,
as the send and receive operations will take some time.
yes, this is also possible to do with JAX-RS.
cheers, Sergey
Mayank Mishra-3 wrote:
>
> Mustafa Sezgin wrote:
>> Hi all,
>>
>>
>>
>> Just a quick question about handling multi part msgs. We are writing a
>> number of Rest services using CXF + Spring. Some of these services will
>> need to accept files, some of which are quite large. We deal with lots of
>> data from lots of users.
>>
>> I am wondering what the best way to do this is. Currently, as an argument
>> to the service method, we specify a MultipartBody object, thus we will be
>> requesting users of our services send these files in multi part msgs
>> where
>> each file is a single part within the multi part msg. However I have my
>> doubts about this. How does CXF handle large files? Say if someone sends
>> 3x2GB files? From what I can see, the service does not actually get
>> passed
>> the multi part msg until the msg has been received in full. Thus the data
>> received (in total 6GB) will need to be stored somewhere, before being
>> passed to the service. Will this be written to disk, and if so, is this
>> configurable in anyway? I am presuming CXF doesn't store all this data in
>> memory, as this would seem like the wrong way to do this...
>>
>>
>>
>> If there are 'better' ways to do this with CXF, I would love to hear
>> them..
>>
>>
> You can use MTOM, there is a system property
> "org.apache.cxf.io.CachedOutputStream.Threshold" that you can set, by
> default its value is 64kb, any attachment more than that is saved in a
> temp file, you can also configure this temp file directory by another
> system property "org.apache.cxf.io.CachedOutputStream.OutputDirectory".
> Also, you require to increase your web container's socket timeout time,
> as the send and receive operations will take some time. You can use this
> with JAX-WS, but I am not sure about JAX-RS.
>
> You can otherwise you can build RESTful endpoints through JAX-WS
> provider interface, where you can use StreamSource.
>
> I tried but may be someone on list can give throw more light on 'better'
> ways. :)
>
> With Regards,
> Mayank
>> You can use MTOM for end
>>
>> Thanks in advance for the responses..
>>
>>
>>
>> Mustafa
>>
>>
>>
>>
>>
>
>
>
--
View this message in context: http://www.nabble.com/Best-way-to-handle-large-multi-part-msgs-tp24490493p24494135.html
Sent from the cxf-user mailing list archive at Nabble.com.
Re: Best way to handle large multi part msgs
Posted by Mayank Mishra <ma...@gmail.com>.
Mustafa Sezgin wrote:
> Hi all,
>
>
>
> Just a quick question about handling multi part msgs. We are writing a
> number of Rest services using CXF + Spring. Some of these services will
> need to accept files, some of which are quite large. We deal with lots of
> data from lots of users.
>
> I am wondering what the best way to do this is. Currently, as an argument
> to the service method, we specify a MultipartBody object, thus we will be
> requesting users of our services send these files in multi part msgs where
> each file is a single part within the multi part msg. However I have my
> doubts about this. How does CXF handle large files? Say if someone sends
> 3x2GB files? From what I can see, the service does not actually get passed
> the multi part msg until the msg has been received in full. Thus the data
> received (in total 6GB) will need to be stored somewhere, before being
> passed to the service. Will this be written to disk, and if so, is this
> configurable in anyway? I am presuming CXF doesn't store all this data in
> memory, as this would seem like the wrong way to do this...
>
>
>
> If there are 'better' ways to do this with CXF, I would love to hear
> them..
>
>
You can use MTOM, there is a system property
"org.apache.cxf.io.CachedOutputStream.Threshold" that you can set, by
default its value is 64kb, any attachment more than that is saved in a
temp file, you can also configure this temp file directory by another
system property "org.apache.cxf.io.CachedOutputStream.OutputDirectory".
Also, you require to increase your web container's socket timeout time,
as the send and receive operations will take some time. You can use this
with JAX-WS, but I am not sure about JAX-RS.
You can otherwise you can build RESTful endpoints through JAX-WS
provider interface, where you can use StreamSource.
I tried but may be someone on list can give throw more light on 'better'
ways. :)
With Regards,
Mayank
> You can use MTOM for end
>
> Thanks in advance for the responses..
>
>
>
> Mustafa
>
>
>
>
>
Re: Best way to handle large multi part msgs
Posted by Sergey Beryozkin <se...@iona.com>.
Hi
Yes, it is configurable. It all depends on the CXF AttachmentDeserializer
which does all the advanced manipulation of attachments.
You can add jaxrs:properties to your jaxrs:endpoint :
<jaxrs:endpoint>
<jaxrs:serviceBeans/>
<jaxrs:properties>
<entry key="attachment-directory" value="some folder"></entry>
<!-- memory threshold, in bytes, default is 100K -->
<entry key="attachment-memory-threshold" value="100000"></entry>
</jaxrs:properties>
</jaxrs:endpoint>
So you can specify diff folders for different endpoints
You can also do it from the code (probably won't suit you if you use
MultipartBody in your method signature), but this depends on a
MessageContext being injected, in which case one can do
AttachmentUtils.getMultipartBody(messageContext, "someDir", 10000);
Give it a try and let me know please if it all works as expected. I
apologize for not documenting it properly - I'll give it my best shot :-)
once 2.2.3 has been released
Sergey
Mustafa Sezgin-2 wrote:
>
> Hi all,
>
>
>
> Just a quick question about handling multi part msgs. We are writing a
> number of Rest services using CXF + Spring. Some of these services will
> need to accept files, some of which are quite large. We deal with lots of
> data from lots of users.
>
> I am wondering what the best way to do this is. Currently, as an argument
> to the service method, we specify a MultipartBody object, thus we will be
> requesting users of our services send these files in multi part msgs where
> each file is a single part within the multi part msg. However I have my
> doubts about this. How does CXF handle large files? Say if someone sends
> 3x2GB files? From what I can see, the service does not actually get passed
> the multi part msg until the msg has been received in full. Thus the data
> received (in total 6GB) will need to be stored somewhere, before being
> passed to the service. Will this be written to disk, and if so, is this
> configurable in anyway? I am presuming CXF doesn't store all this data in
> memory, as this would seem like the wrong way to do this...
>
>
>
> If there are 'better' ways to do this with CXF, I would love to hear
> them..
>
>
>
> Thanks in advance for the responses..
>
>
>
> Mustafa
>
>
>
>
>
--
View this message in context: http://www.nabble.com/Best-way-to-handle-large-multi-part-msgs-tp24490493p24493331.html
Sent from the cxf-user mailing list archive at Nabble.com.