You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@axis.apache.org by Andrew Bruno <an...@gmail.com> on 2009/05/18 08:05:20 UTC

Large Web Services responses - how do you deal with them?

Hello all

I was wondering how some of you may be dealing with web service calls
that result in extremely large data responses?

I have been struggling in trying to change the way the parsing of the
XML response works, as I am getting out of memory errors

java.lang.OutOfMemoryError: Java heap space
	at org.apache.xmlbeans.impl.store.CharUtil.allocate(CharUtil.java:397)
	at org.apache.xmlbeans.impl.store.CharUtil.saveChars(CharUtil.java:506)
	at org.apache.xmlbeans.impl.store.CharUtil.saveChars(CharUtil.java:419)
	at org.apache.xmlbeans.impl.store.CharUtil.saveChars(CharUtil.java:489)
	at org.apache.xmlbeans.impl.store.Cur$CurLoadContext.text(Cur.java:2911)
	at org.apache.xmlbeans.impl.store.Cur$CurLoadContext.stripText(Cur.java:3113)
	at org.apache.xmlbeans.impl.store.Cur$CurLoadContext.text(Cur.java:3126)
	at org.apache.xmlbeans.impl.store.Locale.loadXMLStreamReader(Locale.java:1154)
	at org.apache.xmlbeans.impl.store.Locale.parseToXmlObject(Locale.java:843)
	at org.apache.xmlbeans.impl.store.Locale.parseToXmlObject(Locale.java:826)
	at org.apache.xmlbeans.impl.schema.SchemaTypeLoaderBase.parse(SchemaTypeLoaderBase.java:231)
	.....

Is there a way to change the parser to use a temp file rather then
trying to buffer the response in memory?

Should I be directing this question to the developers list?

Or should I be thinking differently on solving this problem?

Please any ideas :(

Thank you
Andrew

Re: Large Web Services responses - how do you deal with them?

Posted by Andrew Bruno <an...@gmail.com>.
Hi Dobri,

My understanding of MTOM was that the server side had to also support it,
and I have no control on server side.

Also, the stub code generated by AXIS2 gives me a getMimecontent String
method which as you can imagine will result in an Out of Memory for large
content, so I am going deeper into the generated code, and trying to get the
MimeContent element I want, and return an InputStream from the char[]
array.... anyway that is the plan :(

Thank you so much for the response, its always nice to get some feedback
when your hitting walls.

Cheers
Andrew


On Mon, May 18, 2009 at 6:37 PM, Dobri Kitipov <kdobrik.axis2@googlemail.com
> wrote:

> Hi Andrew,
> I think that it will be useful to make use of MTOM for your case [1].
> There is a parameter that you can configure and cache your large date into
> a temp directory [2].
>
> Hope this helps.
>
> Regards,
> Dobri
>
>
> [1] http://ws.apache.org/axis2/1_0/mtom-guide.html
> [2] http://ws.apache.org/axis2/1_0/mtom-guide.html#41
>
>
>
> On Mon, May 18, 2009 at 10:10 AM, Andrew Bruno <an...@gmail.com>wrote:
>
>> or..
>>
>> How can I create my own parser to return an InputStream for only the
>> content that is inside the MimeContent tag?
>>
>> <xml-fragment xmlns:m="
>> http://schemas.microsoft.com/exchange/services/2006/messages"
>> xmlns:t="http://schemas.microsoft.com/exchange/services/2006/types"
>> xmlns:xsd="http://www.w3.org/2001/XMLSchema"
>> xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"
>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
>>  <m:ResponseMessages>
>>    <m:GetItemResponseMessage ResponseClass="Success">
>>      <m:ResponseCode>NoError</m:ResponseCode>
>>      <m:Items>
>>        <t:Message>
>>          <t:MimeContent CharacterSet="UTF-8">*RnJvbTogTGVlIF..... large
>> mime base64 encoded email....*.
>>
>>
>> Sample generated/modified code:
>>
>> ....
>>
>> //execute the operation client
>> _operationClient.execute(true);
>>
>> org.apache.axis2.context.MessageContext _returnMessageContext =
>> _operationClient.getMessageContext(org.apache.axis2.wsdl.WSDLConstants.MESSAGE_LABEL_IN_VALUE);
>> org.apache.axiom.soap.SOAPEnvelope _returnEnv =
>> _returnMessageContext.getEnvelope();
>> OMElement firstElement = _returnEnv.getBody().getFirstElement();
>>
>> Map envelopeNameSpaces = getEnvelopeNamespaces(_returnEnv);
>>
>> XMLStreamReader xsr = firstElement.getXMLStreamReader();
>>
>> org.apache.xmlbeans.XmlBeans.getContextTypeLoader().parse(xsr,
>> GetItemResponseDocument.type, xmlOptions);  //Can I create my own?
>>
>> Is implementing my own parser the way to go?
>>
>>
>>
>

Re: Large Web Services responses - how do you deal with them?

Posted by Dobri Kitipov <kd...@googlemail.com>.
Hi Andrew,
I think that it will be useful to make use of MTOM for your case [1].
There is a parameter that you can configure and cache your large date into a
temp directory [2].

Hope this helps.

Regards,
Dobri


[1] http://ws.apache.org/axis2/1_0/mtom-guide.html
[2] http://ws.apache.org/axis2/1_0/mtom-guide.html#41


On Mon, May 18, 2009 at 10:10 AM, Andrew Bruno <an...@gmail.com>wrote:

> or..
>
> How can I create my own parser to return an InputStream for only the
> content that is inside the MimeContent tag?
>
> <xml-fragment xmlns:m="
> http://schemas.microsoft.com/exchange/services/2006/messages"
> xmlns:t="http://schemas.microsoft.com/exchange/services/2006/types"
> xmlns:xsd="http://www.w3.org/2001/XMLSchema"
> xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
>  <m:ResponseMessages>
>    <m:GetItemResponseMessage ResponseClass="Success">
>      <m:ResponseCode>NoError</m:ResponseCode>
>      <m:Items>
>        <t:Message>
>          <t:MimeContent CharacterSet="UTF-8">*RnJvbTogTGVlIF..... large
> mime base64 encoded email....*.
>
>
> Sample generated/modified code:
>
> ....
>
> //execute the operation client
> _operationClient.execute(true);
>
> org.apache.axis2.context.MessageContext _returnMessageContext =
> _operationClient.getMessageContext(org.apache.axis2.wsdl.WSDLConstants.MESSAGE_LABEL_IN_VALUE);
> org.apache.axiom.soap.SOAPEnvelope _returnEnv =
> _returnMessageContext.getEnvelope();
> OMElement firstElement = _returnEnv.getBody().getFirstElement();
>
> Map envelopeNameSpaces = getEnvelopeNamespaces(_returnEnv);
>
> XMLStreamReader xsr = firstElement.getXMLStreamReader();
>
> org.apache.xmlbeans.XmlBeans.getContextTypeLoader().parse(xsr,
> GetItemResponseDocument.type, xmlOptions);  //Can I create my own?
>
> Is implementing my own parser the way to go?
>
>
>

Re: Large Web Services responses - how do you deal with them?

Posted by Andrew Bruno <an...@gmail.com>.
or..

How can I create my own parser to return an InputStream for only the content
that is inside the MimeContent tag?

<xml-fragment xmlns:m="
http://schemas.microsoft.com/exchange/services/2006/messages"
xmlns:t="http://schemas.microsoft.com/exchange/services/2006/types"
xmlns:xsd="http://www.w3.org/2001/XMLSchema"
xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
 <m:ResponseMessages>
   <m:GetItemResponseMessage ResponseClass="Success">
     <m:ResponseCode>NoError</m:ResponseCode>
     <m:Items>
       <t:Message>
         <t:MimeContent CharacterSet="UTF-8">*RnJvbTogTGVlIF..... large mime
base64 encoded email....*.


Sample generated/modified code:

....

//execute the operation client
_operationClient.execute(true);

org.apache.axis2.context.MessageContext _returnMessageContext =
_operationClient.getMessageContext(org.apache.axis2.wsdl.WSDLConstants.MESSAGE_LABEL_IN_VALUE);
org.apache.axiom.soap.SOAPEnvelope _returnEnv =
_returnMessageContext.getEnvelope();
OMElement firstElement = _returnEnv.getBody().getFirstElement();

Map envelopeNameSpaces = getEnvelopeNamespaces(_returnEnv);

XMLStreamReader xsr = firstElement.getXMLStreamReader();

org.apache.xmlbeans.XmlBeans.getContextTypeLoader().parse(xsr,
GetItemResponseDocument.type, xmlOptions);  //Can I create my own?

Is implementing my own parser the way to go?

Re: Large Web Services responses - how do you deal with them?

Posted by Dennis Sosnoski <dm...@sosnoski.com>.
Glad this approach works for you, Andrew. Thanks for responding to the 
list for the benefit of others facing the same kind of problem (working 
with idiotic MS services, that is... :-)  ).

  - Dennis



Andrew Bruno wrote:
> Hi Dennis,
>
> The way I have done it is as you suggested, via the XMLStreamReader.  
> I find the XML element that I am after, and then the next CDATA 
> element is assumed to be the mimeContent, and pipe it to a tempFile, 
> then I base64decode it, and return the inputStream for access to the 
> email to the application that requires it.
>
> I've just tested it with a 200Meg email  from a remote Exchange box, 
> and it worked no probs... ahh finally.
>
> Here is some sample code (to give back to community ;)
>
> XMLStreamReader xsr = firstElement.getXMLStreamReaderWithoutCaching();
> ...
> events: for (int eventType = xsr.getEventType();; eventType = 
> xsr.next()) {
>                         switch (eventType) {
>                           case XMLStreamReader.START_ELEMENT: {
>                             logger.debug("START_ELEMENT: " + 
> xsr.getName());
>
>                             QName elementName = xsr.getName();
>
>                             if (mimeContentName.equals(elementName)) {
>                                 logger.info 
> <http://logger.info>("Setting mimeContent to TRUE. Next CDATA will be 
> mimecontent, saved to [" + tempFilePath + "]");
>                                 mimecontent = true;
>                             } else {
>                                 logger.debug("Setting mimeContent to 
> false.");
>                                 mimecontent = false;
>                             }
>
>                             depth++;
>                             break;
>                         }
>
>                         case XMLStreamReader.CDATA: {
>                             if (mimecontent) {
>                                 int sourceStart = 0;
>                                 int length = xsr.getTextLength();
>                                 char[] target = new char[length];
>                                 int copiedLength = 0;
>
>                                 int available = xsr.getTextLength() - 
> sourceStart;
>                                 if (available < 0) {
>                                     throw new 
> IndexOutOfBoundsException("sourceStart is greater than" + "number of 
> characters associated with this event");
>                                 }
>
>                                 if (available < length) {
>                                     copiedLength = available;
>                                 } else {
>                                     copiedLength = length;
>                                 }
>
>                                 char[] textChars = 
> xsr.getTextCharacters();
>
>                                 // Only copy the chars that we need.
>                                 System.arraycopy(textChars, 
> xsr.getTextStart() + 0, target, 0, copiedLength);
>                                 theWriter.write(target);
>
>                             } else {
>                                 logger.debug("Ignoring CDATA...");
>                                 // We don't care about other data
>                             }
>
>                             break;
>                         }
> }
>
> P.S. If I had more time, I would love to look at ADB or JiBX, but for 
> now there is no chance, a lesson learned for now.
>
>
> On Tue, May 19, 2009 at 3:29 PM, Dennis Sosnoski <dms@sosnoski.com 
> <ma...@sosnoski.com>> wrote:
>
>     Ah, I hadn't seen noticed/remembered this dealt with a single huge
>     base64 string. That's just plain bad design for the web service...
>     but no surprise if Exchange is involved.
>
>     No, you can't handle this cleanly with JiBX or any other data
>     binding tool I'm aware of. The only way you *could* handle it
>     would be by getting an XMLStreamReader and reading the data
>     directly, with . The XMLStreamReader interface provides next() and
>     getTextCharacters() methods which in theory could be used to get
>     this text a chunk at a time. Assuming the parser involved actually
>     implements these as intended, you can get a block of text at a
>     time and run the base64 decoding on that block, then move on to
>     the next block. A lot of work to implement, though.
>
>     Andrew, you might want to try switching to ADB or JiBX anyway, at
>     least to try for an easier solution than working directly with the
>     XMLStreamReader. Either of these should use less memory than
>     XMLBeans for the same data, so as long as your text strings are
>     only very large and not unbounded these might let you squeak by.
>
>      - Dennis
>
>
>
>     Andreas Veithen wrote:
>
>         Dennis,
>
>         I think that the Web service is actually Microsoft Exchange, so
>         changing it is not an option. Does JiBX support caching a
>         base64 value
>         (which is not represented using MTOM) on disk instead of memory?
>
>         Andrew,
>
>         You might still want to give MTOM a try. Many servers switch
>         to MTOM
>         if the request is sent as MTOM. I can't believe the Microsoft
>         Exchange
>         can only return inline base64.
>
>         Regards,
>
>         Andreas
>
>         On Mon, May 18, 2009 at 13:22, Dennis Sosnoski
>         <dms@sosnoski.com <ma...@sosnoski.com>> wrote:
>          
>
>             Hi Andrew,
>
>             Your best starting point is to get rid of XMLBeans.
>             XMLBeans always stores
>             raw XML in memory, so there's no way to avoid the memory
>             issues with large
>             messages.
>
>             ADB would avoid some of the overhead, in that it would
>             convert the XML
>             message to an object graph, which would typically be a
>             factor of 2-5x
>             smaller than the raw XML data (depending on the type of
>             data in your
>             message).
>
>             JiBX would do at least as well as ADB in terms of the
>             reduced-size object
>             graph. If you have some way of breaking up the response
>             data into more
>             easily-digestible chunks, JiBX would also allow you to do
>             piece-meal
>             processing of the message (by creating a fake collection,
>             for instance,
>             where your data objects expose an addXXX() method which
>             just writes the
>             object being added to some backing store rather than
>             adding it to an
>             in-memory collection). Piece-meal processing is the only
>             way you can
>             dramatically decrease your memory usage and handle
>             messages of effectively
>             unlimited size without a problem.
>
>              - Dennis
>
>             --
>             Dennis M. Sosnoski
>             SOA and Web Services in Java
>             Axis2 Training and Consulting
>             http://www.sosnoski.com - http://www.sosnoski.co.nz
>             Seattle, WA +1-425-939-0576 - Wellington, NZ +64-4-298-6117
>
>
>
>             Andrew Bruno wrote:
>                
>
>                 Mr J,
>
>                 But its just one data element that I need, so there is
>                 no concept many
>                 rows, etc.
>
>                 As a matter of fact, I am breaking up the call in 2
>                 calls, one for all the
>                 metadata, and one for the actual mime content.
>
>                 i.e.
>
>                  <m:ResponseMessages>
>                   <m:GetItemResponseMessage ResponseClass="Success">
>                     <m:ResponseCode>NoError</m:ResponseCode>
>                     <m:Items>
>                       <t:Message>
>                         <t:MimeContent
>                 CharacterSet="UTF-8">RnJvbTogTGVlIF..... large
>                 mime base64 encoded email......... 120Meg++ of encoded
>                 data</t:MimeContent>
>                 ....
>
>                 I only ever request one message at a time.  Its just
>                 that when the
>                 MimeContent is greater then 10Meg, OutOfmemory occurs.
>
>                 It appears to be the case because the parse loads all
>                 the content into
>                 memory.
>
>                 In theory, this element can have data of any size
>                 coming back.
>
>
>
>
>                 On Mon, May 18, 2009 at 6:54 PM, J. Hondius
>                 <jhondius@rem.nl <ma...@rem.nl>
>                 <mailto:jhondius@rem.nl <ma...@rem.nl>>> wrote:
>
>                   I'd think about a different design of the webservice
>                 calls.
>                   You should try to avoid real big results.
>                   Split into more calls.
>
>                   Something like:
>                   One to get a overview list
>                   Another to get details
>
>                   Or:
>                   One call to get the size like the SQL count() does
>                   combined with
>                   Add parameters to your to limit the result: like
>                 start_at,
>                   number_of_results
>
>                   my 2c
>
>                   Andrew Bruno schreef:
>
>                       Hello all
>
>                       I was wondering how some of you may be dealing
>                 with web
>                       service calls
>                       that result in extremely large data responses?
>
>                       I have been struggling in trying to change the
>                 way the parsing
>                       of the
>                       XML response works, as I am getting out of
>                 memory errors
>
>                       java.lang.OutOfMemoryError: Java heap space
>                              at
>                      
>                 org.apache.xmlbeans.impl.store.CharUtil.allocate(CharUtil.java:397)
>                              at
>
>                  org.apache.xmlbeans.impl.store.CharUtil.saveChars(CharUtil.java:506)
>                              at
>
>                  org.apache.xmlbeans.impl.store.CharUtil.saveChars(CharUtil.java:419)
>                              at
>
>                  org.apache.xmlbeans.impl.store.CharUtil.saveChars(CharUtil.java:489)
>                              at
>
>                  org.apache.xmlbeans.impl.store.Cur$CurLoadContext.text(Cur.java:2911)
>                              at
>
>                  org.apache.xmlbeans.impl.store.Cur$CurLoadContext.stripText(Cur.java:3113)
>                              at
>
>                  org.apache.xmlbeans.impl.store.Cur$CurLoadContext.text(Cur.java:3126)
>                              at
>
>                  org.apache.xmlbeans.impl.store.Locale.loadXMLStreamReader(Locale.java:1154)
>                              at
>
>                  org.apache.xmlbeans.impl.store.Locale.parseToXmlObject(Locale.java:843)
>                              at
>
>                  org.apache.xmlbeans.impl.store.Locale.parseToXmlObject(Locale.java:826)
>                              at
>
>                  org.apache.xmlbeans.impl.schema.SchemaTypeLoaderBase.parse(SchemaTypeLoaderBase.java:231)
>                              .....
>
>                       Is there a way to change the parser to use a
>                 temp file rather then
>                       trying to buffer the response in memory?
>
>                       Should I be directing this question to the
>                 developers list?
>
>                       Or should I be thinking differently on solving
>                 this problem?
>
>                       Please any ideas :(
>
>                       Thank you
>                       Andrew
>
>                      
>
>
>          
>
>

Re: Large Web Services responses - how do you deal with them?

Posted by Andrew Bruno <an...@gmail.com>.
Hi Dennis,

The way I have done it is as you suggested, via the XMLStreamReader.  I find
the XML element that I am after, and then the next CDATA element is assumed
to be the mimeContent, and pipe it to a tempFile, then I base64decode it,
and return the inputStream for access to the email to the application that
requires it.

I've just tested it with a 200Meg email  from a remote Exchange box, and it
worked no probs... ahh finally.

Here is some sample code (to give back to community ;)

XMLStreamReader xsr = firstElement.getXMLStreamReaderWithoutCaching();
...
events: for (int eventType = xsr.getEventType();; eventType = xsr.next()) {
                        switch (eventType) {
                          case XMLStreamReader.START_ELEMENT: {
                            logger.debug("START_ELEMENT: " + xsr.getName());

                            QName elementName = xsr.getName();

                            if (mimeContentName.equals(elementName)) {
                                logger.info("Setting mimeContent to TRUE.
Next CDATA will be mimecontent, saved to [" + tempFilePath + "]");
                                mimecontent = true;
                            } else {
                                logger.debug("Setting mimeContent to
false.");
                                mimecontent = false;
                            }

                            depth++;
                            break;
                        }

                        case XMLStreamReader.CDATA: {
                            if (mimecontent) {
                                int sourceStart = 0;
                                int length = xsr.getTextLength();
                                char[] target = new char[length];
                                int copiedLength = 0;

                                int available = xsr.getTextLength() -
sourceStart;
                                if (available < 0) {
                                    throw new
IndexOutOfBoundsException("sourceStart is greater than" + "number of
characters associated with this event");
                                }

                                if (available < length) {
                                    copiedLength = available;
                                } else {
                                    copiedLength = length;
                                }

                                char[] textChars = xsr.getTextCharacters();

                                // Only copy the chars that we need.
                                System.arraycopy(textChars,
xsr.getTextStart() + 0, target, 0, copiedLength);
                                theWriter.write(target);

                            } else {
                                logger.debug("Ignoring CDATA...");
                                // We don't care about other data
                            }

                            break;
                        }
}

P.S. If I had more time, I would love to look at ADB or JiBX, but for now
there is no chance, a lesson learned for now.


On Tue, May 19, 2009 at 3:29 PM, Dennis Sosnoski <dm...@sosnoski.com> wrote:

> Ah, I hadn't seen noticed/remembered this dealt with a single huge base64
> string. That's just plain bad design for the web service... but no surprise
> if Exchange is involved.
>
> No, you can't handle this cleanly with JiBX or any other data binding tool
> I'm aware of. The only way you *could* handle it would be by getting an
> XMLStreamReader and reading the data directly, with . The XMLStreamReader
> interface provides next() and getTextCharacters() methods which in theory
> could be used to get this text a chunk at a time. Assuming the parser
> involved actually implements these as intended, you can get a block of text
> at a time and run the base64 decoding on that block, then move on to the
> next block. A lot of work to implement, though.
>
> Andrew, you might want to try switching to ADB or JiBX anyway, at least to
> try for an easier solution than working directly with the XMLStreamReader.
> Either of these should use less memory than XMLBeans for the same data, so
> as long as your text strings are only very large and not unbounded these
> might let you squeak by.
>
>  - Dennis
>
>
>
> Andreas Veithen wrote:
>
>> Dennis,
>>
>> I think that the Web service is actually Microsoft Exchange, so
>> changing it is not an option. Does JiBX support caching a base64 value
>> (which is not represented using MTOM) on disk instead of memory?
>>
>> Andrew,
>>
>> You might still want to give MTOM a try. Many servers switch to MTOM
>> if the request is sent as MTOM. I can't believe the Microsoft Exchange
>> can only return inline base64.
>>
>> Regards,
>>
>> Andreas
>>
>> On Mon, May 18, 2009 at 13:22, Dennis Sosnoski <dm...@sosnoski.com> wrote:
>>
>>
>>> Hi Andrew,
>>>
>>> Your best starting point is to get rid of XMLBeans. XMLBeans always
>>> stores
>>> raw XML in memory, so there's no way to avoid the memory issues with
>>> large
>>> messages.
>>>
>>> ADB would avoid some of the overhead, in that it would convert the XML
>>> message to an object graph, which would typically be a factor of 2-5x
>>> smaller than the raw XML data (depending on the type of data in your
>>> message).
>>>
>>> JiBX would do at least as well as ADB in terms of the reduced-size object
>>> graph. If you have some way of breaking up the response data into more
>>> easily-digestible chunks, JiBX would also allow you to do piece-meal
>>> processing of the message (by creating a fake collection, for instance,
>>> where your data objects expose an addXXX() method which just writes the
>>> object being added to some backing store rather than adding it to an
>>> in-memory collection). Piece-meal processing is the only way you can
>>> dramatically decrease your memory usage and handle messages of
>>> effectively
>>> unlimited size without a problem.
>>>
>>>  - Dennis
>>>
>>> --
>>> Dennis M. Sosnoski
>>> SOA and Web Services in Java
>>> Axis2 Training and Consulting
>>> http://www.sosnoski.com - http://www.sosnoski.co.nz
>>> Seattle, WA +1-425-939-0576 - Wellington, NZ +64-4-298-6117
>>>
>>>
>>>
>>> Andrew Bruno wrote:
>>>
>>>
>>>> Mr J,
>>>>
>>>> But its just one data element that I need, so there is no concept many
>>>> rows, etc.
>>>>
>>>> As a matter of fact, I am breaking up the call in 2 calls, one for all
>>>> the
>>>> metadata, and one for the actual mime content.
>>>>
>>>> i.e.
>>>>
>>>>  <m:ResponseMessages>
>>>>   <m:GetItemResponseMessage ResponseClass="Success">
>>>>     <m:ResponseCode>NoError</m:ResponseCode>
>>>>     <m:Items>
>>>>       <t:Message>
>>>>         <t:MimeContent CharacterSet="UTF-8">RnJvbTogTGVlIF..... large
>>>> mime base64 encoded email......... 120Meg++ of encoded
>>>> data</t:MimeContent>
>>>> ....
>>>>
>>>> I only ever request one message at a time.  Its just that when the
>>>> MimeContent is greater then 10Meg, OutOfmemory occurs.
>>>>
>>>> It appears to be the case because the parse loads all the content into
>>>> memory.
>>>>
>>>> In theory, this element can have data of any size coming back.
>>>>
>>>>
>>>>
>>>>
>>>> On Mon, May 18, 2009 at 6:54 PM, J. Hondius <jhondius@rem.nl
>>>> <ma...@rem.nl>> wrote:
>>>>
>>>>   I'd think about a different design of the webservice calls.
>>>>   You should try to avoid real big results.
>>>>   Split into more calls.
>>>>
>>>>   Something like:
>>>>   One to get a overview list
>>>>   Another to get details
>>>>
>>>>   Or:
>>>>   One call to get the size like the SQL count() does
>>>>   combined with
>>>>   Add parameters to your to limit the result: like start_at,
>>>>   number_of_results
>>>>
>>>>   my 2c
>>>>
>>>>   Andrew Bruno schreef:
>>>>
>>>>       Hello all
>>>>
>>>>       I was wondering how some of you may be dealing with web
>>>>       service calls
>>>>       that result in extremely large data responses?
>>>>
>>>>       I have been struggling in trying to change the way the parsing
>>>>       of the
>>>>       XML response works, as I am getting out of memory errors
>>>>
>>>>       java.lang.OutOfMemoryError: Java heap space
>>>>              at
>>>>
>>>> org.apache.xmlbeans.impl.store.CharUtil.allocate(CharUtil.java:397)
>>>>              at
>>>>
>>>>  org.apache.xmlbeans.impl.store.CharUtil.saveChars(CharUtil.java:506)
>>>>              at
>>>>
>>>>  org.apache.xmlbeans.impl.store.CharUtil.saveChars(CharUtil.java:419)
>>>>              at
>>>>
>>>>  org.apache.xmlbeans.impl.store.CharUtil.saveChars(CharUtil.java:489)
>>>>              at
>>>>
>>>>  org.apache.xmlbeans.impl.store.Cur$CurLoadContext.text(Cur.java:2911)
>>>>              at
>>>>
>>>>
>>>>  org.apache.xmlbeans.impl.store.Cur$CurLoadContext.stripText(Cur.java:3113)
>>>>              at
>>>>
>>>>  org.apache.xmlbeans.impl.store.Cur$CurLoadContext.text(Cur.java:3126)
>>>>              at
>>>>
>>>>
>>>>  org.apache.xmlbeans.impl.store.Locale.loadXMLStreamReader(Locale.java:1154)
>>>>              at
>>>>
>>>>  org.apache.xmlbeans.impl.store.Locale.parseToXmlObject(Locale.java:843)
>>>>              at
>>>>
>>>>  org.apache.xmlbeans.impl.store.Locale.parseToXmlObject(Locale.java:826)
>>>>              at
>>>>
>>>>
>>>>  org.apache.xmlbeans.impl.schema.SchemaTypeLoaderBase.parse(SchemaTypeLoaderBase.java:231)
>>>>              .....
>>>>
>>>>       Is there a way to change the parser to use a temp file rather then
>>>>       trying to buffer the response in memory?
>>>>
>>>>       Should I be directing this question to the developers list?
>>>>
>>>>       Or should I be thinking differently on solving this problem?
>>>>
>>>>       Please any ideas :(
>>>>
>>>>       Thank you
>>>>       Andrew
>>>>
>>>>
>>>>
>>>
>>
>>
>

Re: Large Web Services responses - how do you deal with them?

Posted by Dennis Sosnoski <dm...@sosnoski.com>.
Ah, I hadn't seen noticed/remembered this dealt with a single huge 
base64 string. That's just plain bad design for the web service... but 
no surprise if Exchange is involved.

No, you can't handle this cleanly with JiBX or any other data binding 
tool I'm aware of. The only way you *could* handle it would be by 
getting an XMLStreamReader and reading the data directly, with . The 
XMLStreamReader interface provides next() and getTextCharacters() 
methods which in theory could be used to get this text a chunk at a 
time. Assuming the parser involved actually implements these as 
intended, you can get a block of text at a time and run the base64 
decoding on that block, then move on to the next block. A lot of work to 
implement, though.

Andrew, you might want to try switching to ADB or JiBX anyway, at least 
to try for an easier solution than working directly with the 
XMLStreamReader. Either of these should use less memory than XMLBeans 
for the same data, so as long as your text strings are only very large 
and not unbounded these might let you squeak by.

  - Dennis


Andreas Veithen wrote:
> Dennis,
>
> I think that the Web service is actually Microsoft Exchange, so
> changing it is not an option. Does JiBX support caching a base64 value
> (which is not represented using MTOM) on disk instead of memory?
>
> Andrew,
>
> You might still want to give MTOM a try. Many servers switch to MTOM
> if the request is sent as MTOM. I can't believe the Microsoft Exchange
> can only return inline base64.
>
> Regards,
>
> Andreas
>
> On Mon, May 18, 2009 at 13:22, Dennis Sosnoski <dm...@sosnoski.com> wrote:
>   
>> Hi Andrew,
>>
>> Your best starting point is to get rid of XMLBeans. XMLBeans always stores
>> raw XML in memory, so there's no way to avoid the memory issues with large
>> messages.
>>
>> ADB would avoid some of the overhead, in that it would convert the XML
>> message to an object graph, which would typically be a factor of 2-5x
>> smaller than the raw XML data (depending on the type of data in your
>> message).
>>
>> JiBX would do at least as well as ADB in terms of the reduced-size object
>> graph. If you have some way of breaking up the response data into more
>> easily-digestible chunks, JiBX would also allow you to do piece-meal
>> processing of the message (by creating a fake collection, for instance,
>> where your data objects expose an addXXX() method which just writes the
>> object being added to some backing store rather than adding it to an
>> in-memory collection). Piece-meal processing is the only way you can
>> dramatically decrease your memory usage and handle messages of effectively
>> unlimited size without a problem.
>>
>>  - Dennis
>>
>> --
>> Dennis M. Sosnoski
>> SOA and Web Services in Java
>> Axis2 Training and Consulting
>> http://www.sosnoski.com - http://www.sosnoski.co.nz
>> Seattle, WA +1-425-939-0576 - Wellington, NZ +64-4-298-6117
>>
>>
>>
>> Andrew Bruno wrote:
>>     
>>> Mr J,
>>>
>>> But its just one data element that I need, so there is no concept many
>>> rows, etc.
>>>
>>> As a matter of fact, I am breaking up the call in 2 calls, one for all the
>>> metadata, and one for the actual mime content.
>>>
>>> i.e.
>>>
>>>  <m:ResponseMessages>
>>>    <m:GetItemResponseMessage ResponseClass="Success">
>>>      <m:ResponseCode>NoError</m:ResponseCode>
>>>      <m:Items>
>>>        <t:Message>
>>>          <t:MimeContent CharacterSet="UTF-8">RnJvbTogTGVlIF..... large
>>> mime base64 encoded email......... 120Meg++ of encoded data</t:MimeContent>
>>> ....
>>>
>>> I only ever request one message at a time.  Its just that when the
>>> MimeContent is greater then 10Meg, OutOfmemory occurs.
>>>
>>> It appears to be the case because the parse loads all the content into
>>> memory.
>>>
>>> In theory, this element can have data of any size coming back.
>>>
>>>
>>>
>>>
>>> On Mon, May 18, 2009 at 6:54 PM, J. Hondius <jhondius@rem.nl
>>> <ma...@rem.nl>> wrote:
>>>
>>>    I'd think about a different design of the webservice calls.
>>>    You should try to avoid real big results.
>>>    Split into more calls.
>>>
>>>    Something like:
>>>    One to get a overview list
>>>    Another to get details
>>>
>>>    Or:
>>>    One call to get the size like the SQL count() does
>>>    combined with
>>>    Add parameters to your to limit the result: like start_at,
>>>    number_of_results
>>>
>>>    my 2c
>>>
>>>    Andrew Bruno schreef:
>>>
>>>        Hello all
>>>
>>>        I was wondering how some of you may be dealing with web
>>>        service calls
>>>        that result in extremely large data responses?
>>>
>>>        I have been struggling in trying to change the way the parsing
>>>        of the
>>>        XML response works, as I am getting out of memory errors
>>>
>>>        java.lang.OutOfMemoryError: Java heap space
>>>               at
>>>        org.apache.xmlbeans.impl.store.CharUtil.allocate(CharUtil.java:397)
>>>               at
>>>
>>>  org.apache.xmlbeans.impl.store.CharUtil.saveChars(CharUtil.java:506)
>>>               at
>>>
>>>  org.apache.xmlbeans.impl.store.CharUtil.saveChars(CharUtil.java:419)
>>>               at
>>>
>>>  org.apache.xmlbeans.impl.store.CharUtil.saveChars(CharUtil.java:489)
>>>               at
>>>
>>>  org.apache.xmlbeans.impl.store.Cur$CurLoadContext.text(Cur.java:2911)
>>>               at
>>>
>>>  org.apache.xmlbeans.impl.store.Cur$CurLoadContext.stripText(Cur.java:3113)
>>>               at
>>>
>>>  org.apache.xmlbeans.impl.store.Cur$CurLoadContext.text(Cur.java:3126)
>>>               at
>>>
>>>  org.apache.xmlbeans.impl.store.Locale.loadXMLStreamReader(Locale.java:1154)
>>>               at
>>>
>>>  org.apache.xmlbeans.impl.store.Locale.parseToXmlObject(Locale.java:843)
>>>               at
>>>
>>>  org.apache.xmlbeans.impl.store.Locale.parseToXmlObject(Locale.java:826)
>>>               at
>>>
>>>  org.apache.xmlbeans.impl.schema.SchemaTypeLoaderBase.parse(SchemaTypeLoaderBase.java:231)
>>>               .....
>>>
>>>        Is there a way to change the parser to use a temp file rather then
>>>        trying to buffer the response in memory?
>>>
>>>        Should I be directing this question to the developers list?
>>>
>>>        Or should I be thinking differently on solving this problem?
>>>
>>>        Please any ideas :(
>>>
>>>        Thank you
>>>        Andrew
>>>
>>>       
>
>   

Re: Large Web Services responses - how do you deal with them?

Posted by Andrew Bruno <an...@gmail.com>.
Andreas,

When I changed to MTOM, Exchange threw a Schema validation error.

My solution so far has been in implementing my own XML pull parser.  When I
detect the ELEMENT_TYPE MimeContent, I start to write the CDATA to a
temporary file.

I also cant believe that I have not found an easier way.  If JiBX definately
provided filesystem caching of a single large CDATA element, i.e. basically
give me an inputstream, then I'd be wrapped.  Dennis?

Thanks guys
Andrew



On Mon, May 18, 2009 at 9:33 PM, Andreas Veithen
<an...@gmail.com>wrote:

> Dennis,
>
> I think that the Web service is actually Microsoft Exchange, so
> changing it is not an option. Does JiBX support caching a base64 value
> (which is not represented using MTOM) on disk instead of memory?
>
> Andrew,
>
> You might still want to give MTOM a try. Many servers switch to MTOM
> if the request is sent as MTOM. I can't believe the Microsoft Exchange
> can only return inline base64.
>
> Regards,
>
> Andreas
>
> On Mon, May 18, 2009 at 13:22, Dennis Sosnoski <dm...@sosnoski.com> wrote:
> > Hi Andrew,
> >
> > Your best starting point is to get rid of XMLBeans. XMLBeans always
> stores
> > raw XML in memory, so there's no way to avoid the memory issues with
> large
> > messages.
> >
> > ADB would avoid some of the overhead, in that it would convert the XML
> > message to an object graph, which would typically be a factor of 2-5x
> > smaller than the raw XML data (depending on the type of data in your
> > message).
> >
> > JiBX would do at least as well as ADB in terms of the reduced-size object
> > graph. If you have some way of breaking up the response data into more
> > easily-digestible chunks, JiBX would also allow you to do piece-meal
> > processing of the message (by creating a fake collection, for instance,
> > where your data objects expose an addXXX() method which just writes the
> > object being added to some backing store rather than adding it to an
> > in-memory collection). Piece-meal processing is the only way you can
> > dramatically decrease your memory usage and handle messages of
> effectively
> > unlimited size without a problem.
> >
> >  - Dennis
> >
> > --
> > Dennis M. Sosnoski
> > SOA and Web Services in Java
> > Axis2 Training and Consulting
> > http://www.sosnoski.com - http://www.sosnoski.co.nz
> > Seattle, WA +1-425-939-0576 - Wellington, NZ +64-4-298-6117
> >
> >
> >
> > Andrew Bruno wrote:
> >>
> >> Mr J,
> >>
> >> But its just one data element that I need, so there is no concept many
> >> rows, etc.
> >>
> >> As a matter of fact, I am breaking up the call in 2 calls, one for all
> the
> >> metadata, and one for the actual mime content.
> >>
> >> i.e.
> >>
> >>  <m:ResponseMessages>
> >>    <m:GetItemResponseMessage ResponseClass="Success">
> >>      <m:ResponseCode>NoError</m:ResponseCode>
> >>      <m:Items>
> >>        <t:Message>
> >>          <t:MimeContent CharacterSet="UTF-8">RnJvbTogTGVlIF..... large
> >> mime base64 encoded email......... 120Meg++ of encoded
> data</t:MimeContent>
> >> ....
> >>
> >> I only ever request one message at a time.  Its just that when the
> >> MimeContent is greater then 10Meg, OutOfmemory occurs.
> >>
> >> It appears to be the case because the parse loads all the content into
> >> memory.
> >>
> >> In theory, this element can have data of any size coming back.
> >>
> >>
> >>
> >>
> >> On Mon, May 18, 2009 at 6:54 PM, J. Hondius <jhondius@rem.nl
> >> <ma...@rem.nl>> wrote:
> >>
> >>    I'd think about a different design of the webservice calls.
> >>    You should try to avoid real big results.
> >>    Split into more calls.
> >>
> >>    Something like:
> >>    One to get a overview list
> >>    Another to get details
> >>
> >>    Or:
> >>    One call to get the size like the SQL count() does
> >>    combined with
> >>    Add parameters to your to limit the result: like start_at,
> >>    number_of_results
> >>
> >>    my 2c
> >>
> >>    Andrew Bruno schreef:
> >>
> >>        Hello all
> >>
> >>        I was wondering how some of you may be dealing with web
> >>        service calls
> >>        that result in extremely large data responses?
> >>
> >>        I have been struggling in trying to change the way the parsing
> >>        of the
> >>        XML response works, as I am getting out of memory errors
> >>
> >>        java.lang.OutOfMemoryError: Java heap space
> >>               at
> >>
>  org.apache.xmlbeans.impl.store.CharUtil.allocate(CharUtil.java:397)
> >>               at
> >>
> >>  org.apache.xmlbeans.impl.store.CharUtil.saveChars(CharUtil.java:506)
> >>               at
> >>
> >>  org.apache.xmlbeans.impl.store.CharUtil.saveChars(CharUtil.java:419)
> >>               at
> >>
> >>  org.apache.xmlbeans.impl.store.CharUtil.saveChars(CharUtil.java:489)
> >>               at
> >>
> >>  org.apache.xmlbeans.impl.store.Cur$CurLoadContext.text(Cur.java:2911)
> >>               at
> >>
> >>
>  org.apache.xmlbeans.impl.store.Cur$CurLoadContext.stripText(Cur.java:3113)
> >>               at
> >>
> >>  org.apache.xmlbeans.impl.store.Cur$CurLoadContext.text(Cur.java:3126)
> >>               at
> >>
> >>
>  org.apache.xmlbeans.impl.store.Locale.loadXMLStreamReader(Locale.java:1154)
> >>               at
> >>
> >>  org.apache.xmlbeans.impl.store.Locale.parseToXmlObject(Locale.java:843)
> >>               at
> >>
> >>  org.apache.xmlbeans.impl.store.Locale.parseToXmlObject(Locale.java:826)
> >>               at
> >>
> >>
>  org.apache.xmlbeans.impl.schema.SchemaTypeLoaderBase.parse(SchemaTypeLoaderBase.java:231)
> >>               .....
> >>
> >>        Is there a way to change the parser to use a temp file rather
> then
> >>        trying to buffer the response in memory?
> >>
> >>        Should I be directing this question to the developers list?
> >>
> >>        Or should I be thinking differently on solving this problem?
> >>
> >>        Please any ideas :(
> >>
> >>        Thank you
> >>        Andrew
> >>
> >
>

Re: Large Web Services responses - how do you deal with them?

Posted by Andreas Veithen <an...@gmail.com>.
Dennis,

I think that the Web service is actually Microsoft Exchange, so
changing it is not an option. Does JiBX support caching a base64 value
(which is not represented using MTOM) on disk instead of memory?

Andrew,

You might still want to give MTOM a try. Many servers switch to MTOM
if the request is sent as MTOM. I can't believe the Microsoft Exchange
can only return inline base64.

Regards,

Andreas

On Mon, May 18, 2009 at 13:22, Dennis Sosnoski <dm...@sosnoski.com> wrote:
> Hi Andrew,
>
> Your best starting point is to get rid of XMLBeans. XMLBeans always stores
> raw XML in memory, so there's no way to avoid the memory issues with large
> messages.
>
> ADB would avoid some of the overhead, in that it would convert the XML
> message to an object graph, which would typically be a factor of 2-5x
> smaller than the raw XML data (depending on the type of data in your
> message).
>
> JiBX would do at least as well as ADB in terms of the reduced-size object
> graph. If you have some way of breaking up the response data into more
> easily-digestible chunks, JiBX would also allow you to do piece-meal
> processing of the message (by creating a fake collection, for instance,
> where your data objects expose an addXXX() method which just writes the
> object being added to some backing store rather than adding it to an
> in-memory collection). Piece-meal processing is the only way you can
> dramatically decrease your memory usage and handle messages of effectively
> unlimited size without a problem.
>
>  - Dennis
>
> --
> Dennis M. Sosnoski
> SOA and Web Services in Java
> Axis2 Training and Consulting
> http://www.sosnoski.com - http://www.sosnoski.co.nz
> Seattle, WA +1-425-939-0576 - Wellington, NZ +64-4-298-6117
>
>
>
> Andrew Bruno wrote:
>>
>> Mr J,
>>
>> But its just one data element that I need, so there is no concept many
>> rows, etc.
>>
>> As a matter of fact, I am breaking up the call in 2 calls, one for all the
>> metadata, and one for the actual mime content.
>>
>> i.e.
>>
>>  <m:ResponseMessages>
>>    <m:GetItemResponseMessage ResponseClass="Success">
>>      <m:ResponseCode>NoError</m:ResponseCode>
>>      <m:Items>
>>        <t:Message>
>>          <t:MimeContent CharacterSet="UTF-8">RnJvbTogTGVlIF..... large
>> mime base64 encoded email......... 120Meg++ of encoded data</t:MimeContent>
>> ....
>>
>> I only ever request one message at a time.  Its just that when the
>> MimeContent is greater then 10Meg, OutOfmemory occurs.
>>
>> It appears to be the case because the parse loads all the content into
>> memory.
>>
>> In theory, this element can have data of any size coming back.
>>
>>
>>
>>
>> On Mon, May 18, 2009 at 6:54 PM, J. Hondius <jhondius@rem.nl
>> <ma...@rem.nl>> wrote:
>>
>>    I'd think about a different design of the webservice calls.
>>    You should try to avoid real big results.
>>    Split into more calls.
>>
>>    Something like:
>>    One to get a overview list
>>    Another to get details
>>
>>    Or:
>>    One call to get the size like the SQL count() does
>>    combined with
>>    Add parameters to your to limit the result: like start_at,
>>    number_of_results
>>
>>    my 2c
>>
>>    Andrew Bruno schreef:
>>
>>        Hello all
>>
>>        I was wondering how some of you may be dealing with web
>>        service calls
>>        that result in extremely large data responses?
>>
>>        I have been struggling in trying to change the way the parsing
>>        of the
>>        XML response works, as I am getting out of memory errors
>>
>>        java.lang.OutOfMemoryError: Java heap space
>>               at
>>        org.apache.xmlbeans.impl.store.CharUtil.allocate(CharUtil.java:397)
>>               at
>>
>>  org.apache.xmlbeans.impl.store.CharUtil.saveChars(CharUtil.java:506)
>>               at
>>
>>  org.apache.xmlbeans.impl.store.CharUtil.saveChars(CharUtil.java:419)
>>               at
>>
>>  org.apache.xmlbeans.impl.store.CharUtil.saveChars(CharUtil.java:489)
>>               at
>>
>>  org.apache.xmlbeans.impl.store.Cur$CurLoadContext.text(Cur.java:2911)
>>               at
>>
>>  org.apache.xmlbeans.impl.store.Cur$CurLoadContext.stripText(Cur.java:3113)
>>               at
>>
>>  org.apache.xmlbeans.impl.store.Cur$CurLoadContext.text(Cur.java:3126)
>>               at
>>
>>  org.apache.xmlbeans.impl.store.Locale.loadXMLStreamReader(Locale.java:1154)
>>               at
>>
>>  org.apache.xmlbeans.impl.store.Locale.parseToXmlObject(Locale.java:843)
>>               at
>>
>>  org.apache.xmlbeans.impl.store.Locale.parseToXmlObject(Locale.java:826)
>>               at
>>
>>  org.apache.xmlbeans.impl.schema.SchemaTypeLoaderBase.parse(SchemaTypeLoaderBase.java:231)
>>               .....
>>
>>        Is there a way to change the parser to use a temp file rather then
>>        trying to buffer the response in memory?
>>
>>        Should I be directing this question to the developers list?
>>
>>        Or should I be thinking differently on solving this problem?
>>
>>        Please any ideas :(
>>
>>        Thank you
>>        Andrew
>>
>

Re: Large Web Services responses - how do you deal with them?

Posted by Dennis Sosnoski <dm...@sosnoski.com>.
Hi Andrew,

Your best starting point is to get rid of XMLBeans. XMLBeans always 
stores raw XML in memory, so there's no way to avoid the memory issues 
with large messages.

ADB would avoid some of the overhead, in that it would convert the XML 
message to an object graph, which would typically be a factor of 2-5x 
smaller than the raw XML data (depending on the type of data in your 
message).

JiBX would do at least as well as ADB in terms of the reduced-size 
object graph. If you have some way of breaking up the response data into 
more easily-digestible chunks, JiBX would also allow you to do 
piece-meal processing of the message (by creating a fake collection, for 
instance, where your data objects expose an addXXX() method which just 
writes the object being added to some backing store rather than adding 
it to an in-memory collection). Piece-meal processing is the only way 
you can dramatically decrease your memory usage and handle messages of 
effectively unlimited size without a problem.

  - Dennis

-- 
Dennis M. Sosnoski
SOA and Web Services in Java
Axis2 Training and Consulting
http://www.sosnoski.com - http://www.sosnoski.co.nz
Seattle, WA +1-425-939-0576 - Wellington, NZ +64-4-298-6117



Andrew Bruno wrote:
> Mr J,
>
> But its just one data element that I need, so there is no concept many 
> rows, etc.
>
> As a matter of fact, I am breaking up the call in 2 calls, one for all 
> the metadata, and one for the actual mime content.
>
> i.e.
>
>   <m:ResponseMessages>
>     <m:GetItemResponseMessage ResponseClass="Success">
>       <m:ResponseCode>NoError</m:ResponseCode>
>       <m:Items>
>         <t:Message>
>           <t:MimeContent CharacterSet="UTF-8">RnJvbTogTGVlIF..... 
> large mime base64 encoded email......... 120Meg++ of encoded 
> data</t:MimeContent>
> ....
>
> I only ever request one message at a time.  Its just that when the 
> MimeContent is greater then 10Meg, OutOfmemory occurs.
>
> It appears to be the case because the parse loads all the content into 
> memory.
>
> In theory, this element can have data of any size coming back.
>
>
>
>
> On Mon, May 18, 2009 at 6:54 PM, J. Hondius <jhondius@rem.nl 
> <ma...@rem.nl>> wrote:
>
>     I'd think about a different design of the webservice calls.
>     You should try to avoid real big results.
>     Split into more calls.
>
>     Something like:
>     One to get a overview list
>     Another to get details
>
>     Or:
>     One call to get the size like the SQL count() does
>     combined with
>     Add parameters to your to limit the result: like start_at,
>     number_of_results
>
>     my 2c
>
>     Andrew Bruno schreef:
>
>         Hello all
>
>         I was wondering how some of you may be dealing with web
>         service calls
>         that result in extremely large data responses?
>
>         I have been struggling in trying to change the way the parsing
>         of the
>         XML response works, as I am getting out of memory errors
>
>         java.lang.OutOfMemoryError: Java heap space
>                at
>         org.apache.xmlbeans.impl.store.CharUtil.allocate(CharUtil.java:397)
>                at
>         org.apache.xmlbeans.impl.store.CharUtil.saveChars(CharUtil.java:506)
>                at
>         org.apache.xmlbeans.impl.store.CharUtil.saveChars(CharUtil.java:419)
>                at
>         org.apache.xmlbeans.impl.store.CharUtil.saveChars(CharUtil.java:489)
>                at
>         org.apache.xmlbeans.impl.store.Cur$CurLoadContext.text(Cur.java:2911)
>                at
>         org.apache.xmlbeans.impl.store.Cur$CurLoadContext.stripText(Cur.java:3113)
>                at
>         org.apache.xmlbeans.impl.store.Cur$CurLoadContext.text(Cur.java:3126)
>                at
>         org.apache.xmlbeans.impl.store.Locale.loadXMLStreamReader(Locale.java:1154)
>                at
>         org.apache.xmlbeans.impl.store.Locale.parseToXmlObject(Locale.java:843)
>                at
>         org.apache.xmlbeans.impl.store.Locale.parseToXmlObject(Locale.java:826)
>                at
>         org.apache.xmlbeans.impl.schema.SchemaTypeLoaderBase.parse(SchemaTypeLoaderBase.java:231)
>                .....
>
>         Is there a way to change the parser to use a temp file rather then
>         trying to buffer the response in memory?
>
>         Should I be directing this question to the developers list?
>
>         Or should I be thinking differently on solving this problem?
>
>         Please any ideas :(
>
>         Thank you
>         Andrew
>          
>

Re: Large Web Services responses - how do you deal with them?

Posted by Andrew Bruno <an...@gmail.com>.
Mr J,

But its just one data element that I need, so there is no concept many rows,
etc.

As a matter of fact, I am breaking up the call in 2 calls, one for all the
metadata, and one for the actual mime content.

i.e.

  <m:ResponseMessages>
    <m:GetItemResponseMessage ResponseClass="Success">
      <m:ResponseCode>NoError</m:ResponseCode>
      <m:Items>
        <t:Message>
          <t:MimeContent CharacterSet="UTF-8">RnJvbTogTGVlIF..... large mime
base64 encoded email......... 120Meg++ of encoded data</t:MimeContent>
....

I only ever request one message at a time.  Its just that when the
MimeContent is greater then 10Meg, OutOfmemory occurs.

It appears to be the case because the parse loads all the content into
memory.

In theory, this element can have data of any size coming back.




On Mon, May 18, 2009 at 6:54 PM, J. Hondius <jh...@rem.nl> wrote:

> I'd think about a different design of the webservice calls.
> You should try to avoid real big results.
> Split into more calls.
>
> Something like:
> One to get a overview list
> Another to get details
>
> Or:
> One call to get the size like the SQL count() does
> combined with
> Add parameters to your to limit the result: like start_at,
> number_of_results
>
> my 2c
>
> Andrew Bruno schreef:
>
>  Hello all
>>
>> I was wondering how some of you may be dealing with web service calls
>> that result in extremely large data responses?
>>
>> I have been struggling in trying to change the way the parsing of the
>> XML response works, as I am getting out of memory errors
>>
>> java.lang.OutOfMemoryError: Java heap space
>>        at
>> org.apache.xmlbeans.impl.store.CharUtil.allocate(CharUtil.java:397)
>>        at
>> org.apache.xmlbeans.impl.store.CharUtil.saveChars(CharUtil.java:506)
>>        at
>> org.apache.xmlbeans.impl.store.CharUtil.saveChars(CharUtil.java:419)
>>        at
>> org.apache.xmlbeans.impl.store.CharUtil.saveChars(CharUtil.java:489)
>>        at
>> org.apache.xmlbeans.impl.store.Cur$CurLoadContext.text(Cur.java:2911)
>>        at
>> org.apache.xmlbeans.impl.store.Cur$CurLoadContext.stripText(Cur.java:3113)
>>        at
>> org.apache.xmlbeans.impl.store.Cur$CurLoadContext.text(Cur.java:3126)
>>        at
>> org.apache.xmlbeans.impl.store.Locale.loadXMLStreamReader(Locale.java:1154)
>>        at
>> org.apache.xmlbeans.impl.store.Locale.parseToXmlObject(Locale.java:843)
>>        at
>> org.apache.xmlbeans.impl.store.Locale.parseToXmlObject(Locale.java:826)
>>        at
>> org.apache.xmlbeans.impl.schema.SchemaTypeLoaderBase.parse(SchemaTypeLoaderBase.java:231)
>>        .....
>>
>> Is there a way to change the parser to use a temp file rather then
>> trying to buffer the response in memory?
>>
>> Should I be directing this question to the developers list?
>>
>> Or should I be thinking differently on solving this problem?
>>
>> Please any ideas :(
>>
>> Thank you
>> Andrew
>>
>>
>

Re: Large Web Services responses - how do you deal with them?

Posted by "J. Hondius" <jh...@rem.nl>.
I'd think about a different design of the webservice calls.
You should try to avoid real big results.
Split into more calls.

Something like:
One to get a overview list
Another to get details

Or:
One call to get the size like the SQL count() does
combined with
Add parameters to your to limit the result: like start_at, number_of_results

my 2c

Andrew Bruno schreef:
> Hello all
>
> I was wondering how some of you may be dealing with web service calls
> that result in extremely large data responses?
>
> I have been struggling in trying to change the way the parsing of the
> XML response works, as I am getting out of memory errors
>
> java.lang.OutOfMemoryError: Java heap space
> 	at org.apache.xmlbeans.impl.store.CharUtil.allocate(CharUtil.java:397)
> 	at org.apache.xmlbeans.impl.store.CharUtil.saveChars(CharUtil.java:506)
> 	at org.apache.xmlbeans.impl.store.CharUtil.saveChars(CharUtil.java:419)
> 	at org.apache.xmlbeans.impl.store.CharUtil.saveChars(CharUtil.java:489)
> 	at org.apache.xmlbeans.impl.store.Cur$CurLoadContext.text(Cur.java:2911)
> 	at org.apache.xmlbeans.impl.store.Cur$CurLoadContext.stripText(Cur.java:3113)
> 	at org.apache.xmlbeans.impl.store.Cur$CurLoadContext.text(Cur.java:3126)
> 	at org.apache.xmlbeans.impl.store.Locale.loadXMLStreamReader(Locale.java:1154)
> 	at org.apache.xmlbeans.impl.store.Locale.parseToXmlObject(Locale.java:843)
> 	at org.apache.xmlbeans.impl.store.Locale.parseToXmlObject(Locale.java:826)
> 	at org.apache.xmlbeans.impl.schema.SchemaTypeLoaderBase.parse(SchemaTypeLoaderBase.java:231)
> 	.....
>
> Is there a way to change the parser to use a temp file rather then
> trying to buffer the response in memory?
>
> Should I be directing this question to the developers list?
>
> Or should I be thinking differently on solving this problem?
>
> Please any ideas :(
>
> Thank you
> Andrew
>