You are viewing a plain text version of this content. The canonical link for it is here.
Posted to c-user@axis.apache.org by Steven Zhang <sz...@altigen.com> on 2010/08/16 19:45:20 UTC
Re: soap message limits - resolved one issue
I faced the same issue last week. The problem is my Axis2c client cannot receive response > 400K. When I run Axis2c client with Axis2c web service there is no problem, but my customer's web service is made by .Net, whose response structure is different from mine.
The reason is found by debugging into Axis2c sources. The problem is in guththila_xml_parser.c (Guththila.dll).
Suppose maximum size of each socket packet is 16K, to receive a 400K data it needs to receive 25 packets. For each packet Guththila.dll will allocate a memory with doubled size. At the 15th time, the packet size increased 2 ^ 15 = 32768 times, the new memory size will be 16K * 32768 = 524M, allocation failed here. When the problem happen, I do find the program eats up to 1G memory.
To resolve it, increase new memory size only when data_size * 2 > buffer_size, so the new memory will keep less than 1 to 2 M. I tested it with 400K and 800K response, it all worked. It even works with 2M response. The modified file is attached.
For Axis2c web service, the response is in XML format. Whenever guththila_xml_parser.c find a '<' sign, it will create a new node for the element. But for .Net web service, the response is a single string, all '<' signs are replaced with "<". The function has to allocate new memory to hold all 400K or more data. This is the reason why the problem occurs to .Net web service only.
Hope it's useful to you.
Steven Zhang
----- Original Message -----
From: Doug Price
To: c-user@axis.apache.org
Sent: Friday, August 13, 2010 10:40 AM
Subject: soap message limits
I’m using Axis2c to send base 64 binary encoded data via SOAP and this works fine as long as my data is small enough. For example, it works on a 6KB file, but when I try to use this for a 183KB file it fails. Is there an internal limit in Axis2c for SOAP messages or base 64 binary data? If so, are there any values I can change in the source to increase these limits?
--
Doug Price
Research Director | Presagis
T. +1 972 943.2400 X2433 F. +1 469 467.4564 C. +1 469 867.8399
DISCLAIMER: This e-mail message is for the sole use of the intended recipient(s) and may contain confidential and/or proprietary information. Do not read, copy, or disseminate this message unless you are the addressee. Any unauthorized review, use, disclosure or distribution is strictly prohibited. If you have received this message in error, please contact the sender by reply e-mail and delete the original and any copies from your system.
RE: soap message limits - resolved one issue
Posted by Doug Price <Do...@presagis.com>.
This fixed it! I tried testing w/ a .NET client and it worked, so I knew it had to be something on the Axis2c side. Your fixed saved me a ton of time. Thanks so much.
--
Doug Price
Research Director | Presagis
T. +1 972 943.2400 X2433 F. +1 469 467.4564 C. +1 469 867.8399
DISCLAIMER: This e-mail message is for the sole use of the intended recipient(s) and may contain confidential and/or proprietary information. Do not read, copy, or disseminate this message unless you are the addressee. Any unauthorized review, use, disclosure or distribution is strictly prohibited. If you have received this message in error, please contact the sender by reply e-mail and delete the original and any copies from your system.
From: Steven Zhang [mailto:szhang@altigen.com]
Sent: Monday, August 16, 2010 12:45 PM
To: Apache AXIS C User List
Subject: Re: soap message limits - resolved one issue
I faced the same issue last week. The problem is my Axis2c client cannot receive response > 400K. When I run Axis2c client with Axis2c web service there is no problem, but my customer's web service is made by .Net, whose response structure is different from mine.
The reason is found by debugging into Axis2c sources. The problem is in guththila_xml_parser.c (Guththila.dll).
Suppose maximum size of each socket packet is 16K, to receive a 400K data it needs to receive 25 packets. For each packet Guththila.dll will allocate a memory with doubled size. At the 15th time, the packet size increased 2 ^ 15 = 32768 times, the new memory size will be 16K * 32768 = 524M, allocation failed here. When the problem happen, I do find the program eats up to 1G memory.
To resolve it, increase new memory size only when data_size * 2 > buffer_size, so the new memory will keep less than 1 to 2 M. I tested it with 400K and 800K response, it all worked. It even works with 2M response. The modified file is attached.
For Axis2c web service, the response is in XML format. Whenever guththila_xml_parser.c find a '<' sign, it will create a new node for the element. But for .Net web service, the response is a single string, all '<' signs are replaced with "<". The function has to allocate new memory to hold all 400K or more data. This is the reason why the problem occurs to .Net web service only.
Hope it's useful to you.
Steven Zhang
----- Original Message -----
From: Doug Price<ma...@presagis.com>
To: c-user@axis.apache.org<ma...@axis.apache.org>
Sent: Friday, August 13, 2010 10:40 AM
Subject: soap message limits
I'm using Axis2c to send base 64 binary encoded data via SOAP and this works fine as long as my data is small enough. For example, it works on a 6KB file, but when I try to use this for a 183KB file it fails. Is there an internal limit in Axis2c for SOAP messages or base 64 binary data? If so, are there any values I can change in the source to increase these limits?
--
Doug Price
Research Director | Presagis
T. +1 972 943.2400 X2433 F. +1 469 467.4564 C. +1 469 867.8399
DISCLAIMER: This e-mail message is for the sole use of the intended recipient(s) and may contain confidential and/or proprietary information. Do not read, copy, or disseminate this message unless you are the addressee. Any unauthorized review, use, disclosure or distribution is strictly prohibited. If you have received this message in error, please contact the sender by reply e-mail and delete the original and any copies from your system.
Re: soap message limits - resolved one issue
Posted by Akos Marton <ma...@gmail.com>.
Using MTOM makes possible to send gigabites amount of data...
Steven Zhang wrote:
> I have no idea. When running with 2M response, it took 50s to clear
> memory. 400K response took < 10s to clear. I have to ask the customer to
> limit response size if possible.
>
> ----- Original Message ----- From: "Akos Marton" <ma...@gmail.com>
> To: "Apache AXIS C User List" <c-...@axis.apache.org>
> Sent: Monday, August 16, 2010 2:18 PM
> Subject: Re: soap message limits - resolved one issue
>
>
> What would you like to do if the file size bigger than Mbytes, tens of
> MBytes or bigger?
>
> Steven Zhang wrote:
>> I faced the same issue last week. The problem is my Axis2c client cannot
>> receive response > 400K. When I run Axis2c client with Axis2c web
>> service there is no problem, but my customer's web service is made by
>> .Net, whose response structure is different from mine.
>>
>> The reason is found by debugging into Axis2c sources. The problem is in
>> guththila_xml_parser.c (Guththila.dll).
>>
>> Suppose maximum size of each socket packet is 16K, to receive a 400K
>> data it needs to receive 25 packets. For each packet Guththila.dll will
>> allocate a memory with doubled size. At the 15th time, the packet size
>> increased 2 ^ 15 = 32768 times, the new memory size will be 16K * 32768
>> = 524M, allocation failed here. When the problem happen, I do find the
>> program eats up to 1G memory.
>>
>> To resolve it, increase new memory size only when data_size * 2 >
>> buffer_size, so the new memory will keep less than 1 to 2 M. I tested it
>> with 400K and 800K response, it all worked. It even works with 2M
>> response. The modified file is attached.
>>
>> For Axis2c web service, the response is in XML format. Whenever
>> guththila_xml_parser.c find a '<' sign, it will create a new node for
>> the element. But for .Net web service, the response is a single string,
>> all '<' signs are replaced with "<". The function has to allocate new
>> memory to hold all 400K or more data. This is the reason why the
>> problem occurs to .Net web service only.
>>
>> Hope it's useful to you.
>>
>> Steven Zhang
>>
>>
>> ----- Original Message -----
>> *From:* Doug Price <ma...@presagis.com>
>> *To:* c-user@axis.apache.org <ma...@axis.apache.org>
>> *Sent:* Friday, August 13, 2010 10:40 AM
>> *Subject:* soap message limits
>>
>> I’m using Axis2c to send base 64 binary encoded data via SOAP and
>> this works fine as long as my data is small enough. For example, it
>> works on a 6KB file, but when I try to use this for a 183KB file it
>> fails. Is there an internal limit in Axis2c for SOAP messages or
>> base 64 binary data? If so, are there any values I can change in
>> the source to increase these limits?
>>
>>
>>
>> -- *Doug Price*
>> Research Director | *Presagis*
>>
>> *T.* +1 972 943.2400 X2433 *F.* +1 469 467.4564 *C.* +1 469 867.8399
>>
>>
>> DISCLAIMER: This e-mail message is for the sole use of the intended
>> recipient(s) and may contain confidential and/or proprietary
>> information. Do not read, copy, or disseminate this message unless
>> you are the addressee. Any unauthorized review, use, disclosure or
>> distribution is strictly prohibited. If you have received this
>> message in error, please contact the sender by reply e-mail and
>> delete the original and any copies from your system.
>>
>>
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: c-user-unsubscribe@axis.apache.org
>> For additional commands, e-mail: c-user-help@axis.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: c-user-unsubscribe@axis.apache.org
> For additional commands, e-mail: c-user-help@axis.apache.org
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: c-user-unsubscribe@axis.apache.org
> For additional commands, e-mail: c-user-help@axis.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: c-user-unsubscribe@axis.apache.org
For additional commands, e-mail: c-user-help@axis.apache.org
Re: soap message limits - resolved one issue
Posted by Steven Zhang <sz...@altigen.com>.
I have no idea. When running with 2M response, it took 50s to clear memory.
400K response took < 10s to clear. I have to ask the customer to limit
response size if possible.
----- Original Message -----
From: "Akos Marton" <ma...@gmail.com>
To: "Apache AXIS C User List" <c-...@axis.apache.org>
Sent: Monday, August 16, 2010 2:18 PM
Subject: Re: soap message limits - resolved one issue
What would you like to do if the file size bigger than Mbytes, tens of
MBytes or bigger?
Steven Zhang wrote:
> I faced the same issue last week. The problem is my Axis2c client cannot
> receive response > 400K. When I run Axis2c client with Axis2c web
> service there is no problem, but my customer's web service is made by
> .Net, whose response structure is different from mine.
>
> The reason is found by debugging into Axis2c sources. The problem is in
> guththila_xml_parser.c (Guththila.dll).
>
> Suppose maximum size of each socket packet is 16K, to receive a 400K
> data it needs to receive 25 packets. For each packet Guththila.dll will
> allocate a memory with doubled size. At the 15th time, the packet size
> increased 2 ^ 15 = 32768 times, the new memory size will be 16K * 32768
> = 524M, allocation failed here. When the problem happen, I do find the
> program eats up to 1G memory.
>
> To resolve it, increase new memory size only when data_size * 2 >
> buffer_size, so the new memory will keep less than 1 to 2 M. I tested it
> with 400K and 800K response, it all worked. It even works with 2M
> response. The modified file is attached.
>
> For Axis2c web service, the response is in XML format. Whenever
> guththila_xml_parser.c find a '<' sign, it will create a new node for
> the element. But for .Net web service, the response is a single string,
> all '<' signs are replaced with "<". The function has to allocate new
> memory to hold all 400K or more data. This is the reason why the
> problem occurs to .Net web service only.
>
> Hope it's useful to you.
>
> Steven Zhang
>
>
> ----- Original Message -----
> *From:* Doug Price <ma...@presagis.com>
> *To:* c-user@axis.apache.org <ma...@axis.apache.org>
> *Sent:* Friday, August 13, 2010 10:40 AM
> *Subject:* soap message limits
>
> I’m using Axis2c to send base 64 binary encoded data via SOAP and
> this works fine as long as my data is small enough. For example, it
> works on a 6KB file, but when I try to use this for a 183KB file it
> fails. Is there an internal limit in Axis2c for SOAP messages or
> base 64 binary data? If so, are there any values I can change in
> the source to increase these limits?
>
>
>
> --
> *Doug Price*
> Research Director | *Presagis*
>
> *T.* +1 972 943.2400 X2433 *F.* +1 469 467.4564 *C.* +1 469 867.8399
>
>
> DISCLAIMER: This e-mail message is for the sole use of the intended
> recipient(s) and may contain confidential and/or proprietary
> information. Do not read, copy, or disseminate this message unless
> you are the addressee. Any unauthorized review, use, disclosure or
> distribution is strictly prohibited. If you have received this
> message in error, please contact the sender by reply e-mail and
> delete the original and any copies from your system.
>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: c-user-unsubscribe@axis.apache.org
> For additional commands, e-mail: c-user-help@axis.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: c-user-unsubscribe@axis.apache.org
For additional commands, e-mail: c-user-help@axis.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: c-user-unsubscribe@axis.apache.org
For additional commands, e-mail: c-user-help@axis.apache.org
Re: soap message limits - resolved one issue
Posted by Akos Marton <ma...@gmail.com>.
What would you like to do if the file size bigger than Mbytes, tens of
MBytes or bigger?
Steven Zhang wrote:
> I faced the same issue last week. The problem is my Axis2c client cannot
> receive response > 400K. When I run Axis2c client with Axis2c web
> service there is no problem, but my customer's web service is made by
> .Net, whose response structure is different from mine.
>
> The reason is found by debugging into Axis2c sources. The problem is in
> guththila_xml_parser.c (Guththila.dll).
>
> Suppose maximum size of each socket packet is 16K, to receive a 400K
> data it needs to receive 25 packets. For each packet Guththila.dll will
> allocate a memory with doubled size. At the 15th time, the packet size
> increased 2 ^ 15 = 32768 times, the new memory size will be 16K * 32768
> = 524M, allocation failed here. When the problem happen, I do find the
> program eats up to 1G memory.
>
> To resolve it, increase new memory size only when data_size * 2 >
> buffer_size, so the new memory will keep less than 1 to 2 M. I tested it
> with 400K and 800K response, it all worked. It even works with 2M
> response. The modified file is attached.
>
> For Axis2c web service, the response is in XML format. Whenever
> guththila_xml_parser.c find a '<' sign, it will create a new node for
> the element. But for .Net web service, the response is a single string,
> all '<' signs are replaced with "<". The function has to allocate new
> memory to hold all 400K or more data. This is the reason why the
> problem occurs to .Net web service only.
>
> Hope it's useful to you.
>
> Steven Zhang
>
>
> ----- Original Message -----
> *From:* Doug Price <ma...@presagis.com>
> *To:* c-user@axis.apache.org <ma...@axis.apache.org>
> *Sent:* Friday, August 13, 2010 10:40 AM
> *Subject:* soap message limits
>
> I’m using Axis2c to send base 64 binary encoded data via SOAP and
> this works fine as long as my data is small enough. For example, it
> works on a 6KB file, but when I try to use this for a 183KB file it
> fails. Is there an internal limit in Axis2c for SOAP messages or
> base 64 binary data? If so, are there any values I can change in
> the source to increase these limits?
>
>
>
> --
> *Doug Price*
> Research Director | *Presagis*
>
> *T.* +1 972 943.2400 X2433 *F.* +1 469 467.4564 *C.* +1 469 867.8399
>
>
> DISCLAIMER: This e-mail message is for the sole use of the intended
> recipient(s) and may contain confidential and/or proprietary
> information. Do not read, copy, or disseminate this message unless
> you are the addressee. Any unauthorized review, use, disclosure or
> distribution is strictly prohibited. If you have received this
> message in error, please contact the sender by reply e-mail and
> delete the original and any copies from your system.
>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: c-user-unsubscribe@axis.apache.org
> For additional commands, e-mail: c-user-help@axis.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: c-user-unsubscribe@axis.apache.org
For additional commands, e-mail: c-user-help@axis.apache.org