You are viewing a plain text version of this content. The canonical link for it is here.
Posted to c-user@axis.apache.org by Navin Bhat <na...@toshiba-tesi.com> on 2008/05/22 07:27:41 UTC
Memory is very high while sending MTOM attachment and also transfer is slow
Hi Manjula,
Sorry I dint provide complete info in my previous mail.
I am running client on host machine (Windows xp) and the server on
VMWare (CentOS 4.4).I am using axis2/java client and have set a huge
timeout. I am able to send and receive 145MB of data with this setting
as u suggested. But the thing is, still I see that in idle mode server
consumes around 6MB memory and after sending request it keeps
increasing. I observed (using system monitor tool) as the control
entered the business logic of the service memory usage was around 300MB
and once service started sending back the same data to client memory
usage went up to 610MB. As expected with such high memory usage my
computer slowed down. And it took more than 3min to transfer this data
which is very high as against other framework (gSOAP, Glassfish,
axis2/java).Is there any alternative approach to transfer large data
faster with lesser memory usage in this framework.
-----Original Message-----
From: Manjula Peiris [mailto:manjula@wso2.com]
Sent: Wednesday, May 21, 2008 11:09 PM
To: Apache AXIS C User List
Subject: RE: Need information on MTOMChunkBufferSize
andMTOMMaxChunkBuffers parameters
Hi Navin,
If you are running both client and server on the same machine and both
send and receive the same 145.1MB attachment then your system memory
usage can be grown up to at least 145.1 * 4 = 580.1 MB of memory at
least. So your apache https server memory usage will require 145.1 * 2 =
290.2 MB of memory at least for processing the attachment. So an
apr_pool which is much greater than 290.1MB should be allocated for
processing this request. Remember Apache frees the memory after the
request. So it starts swapping and that makes your system to become
slow. The client fails because it timeouts. If you increase the time out
using the following function in your client you will definitely see
better results.
axis2_options_set_timeout_in_milli_seconds(
axis2_options_t * options,
const axutil_env_t * env,
const long timeout_in_milli_seconds);
Actually I have tested a 160MB attachment where both server and client
sends and receives the same by increasing the time out and it succeed. I
also tested the case where client only sends the 160MB attachment and
server does not send any attachment and it succeed with out increasing
the default time out.
Actually we are in the process of developing caching large attachment
model for MTOM.
Thanks,
Manjula.
On Wed, 2008-05-21 at 15:21 +0530, Navin Bhat wrote:
> Thank you for the information regarding these parameters.
>
> I am seeing a problem with the service I have written where in I
receive
> an MTOM attachment from client and the service sends back the same
> attachment.
>
> Problem description:
> As per my observation apache_http_server uses around 5.5MB of
> memory in idle state(When it is not serving any request).When I invoke
> the service to send an attachment of size around 145MB. The memory
usage
> of apache_http_server went up and as I observed it went up to 601.1MB
> and my computer slowed down significantly.I am unable to figure this
> out.I have worked with other frameworks like
> axis2/java,gSOAP,Glassfish.I never came across such problem even when
I
> transferred 1GB files across.I have specified my computer
configuration
> below.I have attached axis2.xml file used while running server. I am
> using default values MTOMChunkBufferSize is 1 and MTOMMaxChunkBuffers
is
> 1000 please refer attached axis2.xml.Kindly let me know as to where I
> have gone wrong.
>
> Computer Specs:
> CentOS 4.4 on VMWare(host Windows xp)
> 1GB RAM
>
>
> -----Original Message-----
> From: Kaushalye Kapuruge [mailto:kaushalye@wso2.com]
> Sent: Wednesday, May 21, 2008 2:38 PM
> To: Apache AXIS C User List
> Subject: Re: Need information on MTOMChunkBufferSize and
> MTOMMaxChunkBuffers parameters
>
> Manjula Peiris wrote:
> > Hi Navin,
> >
> > Axis2/C reads the whole attachment at once. When there is a
> > content-length this is not a problem.
> >
> > But when the incoming stream is chunked we need to read till end of
> the
> > chunks. So,
> >
> > MTOMChunkBufferSize means the size (In MB) of the buffers we keep on
> > reading.
> > MTOMMaxChunkBuffers means the number of such buffers the system can
> > handle.
> >
> > say your ,
> > MTOMMaxChunkBuffers 1000
> > MTOMMaxChunkBuffers 1
> >
> > Then you can handle Maximum size of 1000*1 MB attachements. So By
> > changing these parameters you can change the maximum size of
> attachments
> > your system can handle given that you have enough memory.
> >
> > The default is 1*1000MB. The values in the commented line in
axis2.xml
> > are just an example. I will change them to default and add more
> comments
> > there.
> Also I think it's better to update the Axis2/C manual. And add another
> entry to Axis2/C FAQ.
> Cheers,
> Kau
> > Thanks for raising this.
> >
> > Thanks,
> > -Manjula.
> >
> >
> >
> >
> >
> > On Wed, 2008-05-21 at 11:08 +0530, Navin Bhat wrote:
> >
> >> Hello group,
> >>
> >>
> >>
> >> I am new to axis2/c. I am working on axis2-1.4. I have
a
> >> service where is need to enable MTOM to send large attachments.
> >>
> >> I wanted to know how I can tune the MTOMChunkBufferSize and
> >> MTOMMaxChunkBuffers parameters to send the attachment efficiently.
I
> >> am not getting information about what these parameters imply.
> >>
> >>
> >>
> >> Does MTOMChunkBufferSize mean size of buffer
> >> allocated ???
> >>
> >> I see in axis2.xml that by default this parameter has
> >> value 10.
> >>
> >> Is it 10 bytes buffer size????
> >>
> >>
> >>
> >> Does MTOMMaxChunkBuffers mean no.of buffers????
> >>
> >> I see in axis2.xml that by default this parameter has
> >> value 1000.
> >>
> >> Is it 1000 buffers???
> >>
> >>
> >>
> >> I am not finding information about these parameters in axis2/c
> >> manual.
> >>
> >>
> >>
> >>
> >>
> >> Need help....Thanks in advance..
> >>
> >>
> >>
> >>
> >>
> >> Regards,
> >>
> >> Navin Bhat
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >> The information contained in this e-mail message and in any
> >> attachments/annexure/appendices is confidential to the
> >> recipient and may contain privileged information.
> >> If you are not the intended recipient, please notify the
> >> sender and delete the message along with any
> >> attachments/annexure/appendices. You should not disclose,
> >> copy or otherwise use the information contained in the
> >> message or any annexure. Any views expressed in this e-mail
> >> are those of the individual sender except where the sender
> >> specifically states them to be the views of
> >> Toshiba Embedded Software India Pvt. Ltd. (TESI),Bangalore.
> >>
> >> Although this transmission and any attachments are believed to be
> >> free of any virus or other defect that might affect any computer
> >> system into which it is received and opened, it is the
responsibility
> >> of the recipient to ensure that it is virus free and no
> responsibility
> >> is accepted by Toshiba Embedded Software India Pvt. Ltd, for any
loss
> or
> >> damage arising in any way from its use.
> >>
> >>
> >>
> >
> >
> >
---------------------------------------------------------------------
> > To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
> > For additional commands, e-mail: axis-c-user-help@ws.apache.org
> >
> >
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
> For additional commands, e-mail: axis-c-user-help@ws.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
For additional commands, e-mail: axis-c-user-help@ws.apache.org
The information contained in this e-mail message and in any
attachments/annexure/appendices is confidential to the
recipient and may contain privileged information.
If you are not the intended recipient, please notify the
sender and delete the message along with any
attachments/annexure/appendices. You should not disclose,
copy or otherwise use the information contained in the
message or any annexure. Any views expressed in this e-mail
are those of the individual sender except where the sender
specifically states them to be the views of
Toshiba Embedded Software India Pvt. Ltd. (TESI),Bangalore.
Although this transmission and any attachments are believed to be
free of any virus or other defect that might affect any computer
system into which it is received and opened, it is the responsibility
of the recipient to ensure that it is virus free and no responsibility
is accepted by Toshiba Embedded Software India Pvt. Ltd, for any loss or
damage arising in any way from its use.
---------------------------------------------------------------------
To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
For additional commands, e-mail: axis-c-user-help@ws.apache.org
Re: Memory is very high while sending MTOM attachment and also
transfer is slow
Posted by Manjula Peiris <ma...@wso2.com>.
Hi Navin,
We are in the process of implementing a caching mechanism for large
attachments [1]. But it is not complete.
[1]https://svn.apache.org/repos/asf/webservices/axis2/branches/c/post_1_4_mtom/c
Thanks,
-Manjula.
On Thu, 2008-05-22 at 10:57 +0530, Navin Bhat wrote:
> Hi Manjula,
>
> Sorry I dint provide complete info in my previous mail.
>
> I am running client on host machine (Windows xp) and the server on
> VMWare (CentOS 4.4).I am using axis2/java client and have set a huge
> timeout. I am able to send and receive 145MB of data with this setting
> as u suggested. But the thing is, still I see that in idle mode server
> consumes around 6MB memory and after sending request it keeps
> increasing. I observed (using system monitor tool) as the control
> entered the business logic of the service memory usage was around 300MB
> and once service started sending back the same data to client memory
> usage went up to 610MB. As expected with such high memory usage my
> computer slowed down. And it took more than 3min to transfer this data
> which is very high as against other framework (gSOAP, Glassfish,
> axis2/java).Is there any alternative approach to transfer large data
> faster with lesser memory usage in this framework.
>
>
> -----Original Message-----
> From: Manjula Peiris [mailto:manjula@wso2.com]
> Sent: Wednesday, May 21, 2008 11:09 PM
> To: Apache AXIS C User List
> Subject: RE: Need information on MTOMChunkBufferSize
> andMTOMMaxChunkBuffers parameters
>
> Hi Navin,
>
> If you are running both client and server on the same machine and both
> send and receive the same 145.1MB attachment then your system memory
> usage can be grown up to at least 145.1 * 4 = 580.1 MB of memory at
> least. So your apache https server memory usage will require 145.1 * 2 =
> 290.2 MB of memory at least for processing the attachment. So an
> apr_pool which is much greater than 290.1MB should be allocated for
> processing this request. Remember Apache frees the memory after the
> request. So it starts swapping and that makes your system to become
> slow. The client fails because it timeouts. If you increase the time out
> using the following function in your client you will definitely see
> better results.
>
> axis2_options_set_timeout_in_milli_seconds(
> axis2_options_t * options,
> const axutil_env_t * env,
> const long timeout_in_milli_seconds);
>
> Actually I have tested a 160MB attachment where both server and client
> sends and receives the same by increasing the time out and it succeed. I
> also tested the case where client only sends the 160MB attachment and
> server does not send any attachment and it succeed with out increasing
> the default time out.
>
> Actually we are in the process of developing caching large attachment
> model for MTOM.
>
> Thanks,
> Manjula.
>
>
>
> On Wed, 2008-05-21 at 15:21 +0530, Navin Bhat wrote:
> > Thank you for the information regarding these parameters.
> >
> > I am seeing a problem with the service I have written where in I
> receive
> > an MTOM attachment from client and the service sends back the same
> > attachment.
> >
> > Problem description:
> > As per my observation apache_http_server uses around 5.5MB of
> > memory in idle state(When it is not serving any request).When I invoke
> > the service to send an attachment of size around 145MB. The memory
> usage
> > of apache_http_server went up and as I observed it went up to 601.1MB
> > and my computer slowed down significantly.I am unable to figure this
> > out.I have worked with other frameworks like
> > axis2/java,gSOAP,Glassfish.I never came across such problem even when
> I
> > transferred 1GB files across.I have specified my computer
> configuration
> > below.I have attached axis2.xml file used while running server. I am
> > using default values MTOMChunkBufferSize is 1 and MTOMMaxChunkBuffers
> is
> > 1000 please refer attached axis2.xml.Kindly let me know as to where I
> > have gone wrong.
> >
> > Computer Specs:
> > CentOS 4.4 on VMWare(host Windows xp)
> > 1GB RAM
> >
> >
> > -----Original Message-----
> > From: Kaushalye Kapuruge [mailto:kaushalye@wso2.com]
> > Sent: Wednesday, May 21, 2008 2:38 PM
> > To: Apache AXIS C User List
> > Subject: Re: Need information on MTOMChunkBufferSize and
> > MTOMMaxChunkBuffers parameters
> >
> > Manjula Peiris wrote:
> > > Hi Navin,
> > >
> > > Axis2/C reads the whole attachment at once. When there is a
> > > content-length this is not a problem.
> > >
> > > But when the incoming stream is chunked we need to read till end of
> > the
> > > chunks. So,
> > >
> > > MTOMChunkBufferSize means the size (In MB) of the buffers we keep on
> > > reading.
> > > MTOMMaxChunkBuffers means the number of such buffers the system can
> > > handle.
> > >
> > > say your ,
> > > MTOMMaxChunkBuffers 1000
> > > MTOMMaxChunkBuffers 1
> > >
> > > Then you can handle Maximum size of 1000*1 MB attachements. So By
> > > changing these parameters you can change the maximum size of
> > attachments
> > > your system can handle given that you have enough memory.
> > >
> > > The default is 1*1000MB. The values in the commented line in
> axis2.xml
> > > are just an example. I will change them to default and add more
> > comments
> > > there.
> > Also I think it's better to update the Axis2/C manual. And add another
>
> > entry to Axis2/C FAQ.
> > Cheers,
> > Kau
> > > Thanks for raising this.
> > >
> > > Thanks,
> > > -Manjula.
> > >
> > >
> > >
> > >
> > >
> > > On Wed, 2008-05-21 at 11:08 +0530, Navin Bhat wrote:
> > >
> > >> Hello group,
> > >>
> > >>
> > >>
> > >> I am new to axis2/c. I am working on axis2-1.4. I have
> a
> > >> service where is need to enable MTOM to send large attachments.
> > >>
> > >> I wanted to know how I can tune the MTOMChunkBufferSize and
> > >> MTOMMaxChunkBuffers parameters to send the attachment efficiently.
> I
> > >> am not getting information about what these parameters imply.
> > >>
> > >>
> > >>
> > >> Does MTOMChunkBufferSize mean size of buffer
> > >> allocated ???
> > >>
> > >> I see in axis2.xml that by default this parameter has
> > >> value 10.
> > >>
> > >> Is it 10 bytes buffer size????
> > >>
> > >>
> > >>
> > >> Does MTOMMaxChunkBuffers mean no.of buffers????
> > >>
> > >> I see in axis2.xml that by default this parameter has
> > >> value 1000.
> > >>
> > >> Is it 1000 buffers???
> > >>
> > >>
> > >>
> > >> I am not finding information about these parameters in axis2/c
> > >> manual.
> > >>
> > >>
> > >>
> > >>
> > >>
> > >> Need help....Thanks in advance..
> > >>
> > >>
> > >>
> > >>
> > >>
> > >> Regards,
> > >>
> > >> Navin Bhat
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >> The information contained in this e-mail message and in any
> > >> attachments/annexure/appendices is confidential to the
> > >> recipient and may contain privileged information.
> > >> If you are not the intended recipient, please notify the
> > >> sender and delete the message along with any
> > >> attachments/annexure/appendices. You should not disclose,
> > >> copy or otherwise use the information contained in the
> > >> message or any annexure. Any views expressed in this e-mail
> > >> are those of the individual sender except where the sender
> > >> specifically states them to be the views of
> > >> Toshiba Embedded Software India Pvt. Ltd. (TESI),Bangalore.
> > >>
> > >> Although this transmission and any attachments are believed to be
> > >> free of any virus or other defect that might affect any computer
> > >> system into which it is received and opened, it is the
> responsibility
> > >> of the recipient to ensure that it is virus free and no
> > responsibility
> > >> is accepted by Toshiba Embedded Software India Pvt. Ltd, for any
> loss
> > or
> > >> damage arising in any way from its use.
> > >>
> > >>
> > >>
> > >
> > >
> > >
> ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
> > > For additional commands, e-mail: axis-c-user-help@ws.apache.org
> > >
> > >
> > >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
> > For additional commands, e-mail: axis-c-user-help@ws.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>
> The information contained in this e-mail message and in any
> attachments/annexure/appendices is confidential to the
> recipient and may contain privileged information.
> If you are not the intended recipient, please notify the
> sender and delete the message along with any
> attachments/annexure/appendices. You should not disclose,
> copy or otherwise use the information contained in the
> message or any annexure. Any views expressed in this e-mail
> are those of the individual sender except where the sender
> specifically states them to be the views of
> Toshiba Embedded Software India Pvt. Ltd. (TESI),Bangalore.
>
> Although this transmission and any attachments are believed to be
> free of any virus or other defect that might affect any computer
> system into which it is received and opened, it is the responsibility
> of the recipient to ensure that it is virus free and no responsibility
> is accepted by Toshiba Embedded Software India Pvt. Ltd, for any loss or
> damage arising in any way from its use.
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
For additional commands, e-mail: axis-c-user-help@ws.apache.org