Sending Large String

[Kenneth Lee <Ke...@KennethLee.org>]

I have a web service that returns a dataset as an xml string.  I run
into problems when too much data is returned.  I usually get one of two
error messages:
1. [java] org.xml.sax.SAXParseException: Parser has reached the entity
expansion limit "64,000" set by the Application.
2. [java]  faultString: java.lang.reflect.InvocationTargetException

My goal is to send an XML file as a string that another application can
read and turn into a database file.  I already have this working with
smaller XML strings, but can't get past the errors when getting larger
datasets back.  (I get a result set, then call another method called
generateXML(ResultSet rs) that returns an XML String)

I get this idea that I need to have the string broken into attachments,
but have no idea how to do this.  If this is a possible course of
action, could someone send me some sample code of how to do this, or a
link?

Thanks,

Ken Lee

RE: Sending Large String

[Kenneth Lee <Ke...@KennethLee.org>]

>>1. [java] org.xml.sax.SAXParseException: Parser has reached 
>> the entity expansion limit "64,000" set by the Application.
>> 
> I'd never heard of this error before, but I'll share a little 
> secret with you: when I have no idea what an error message 
> means I just paste it directly into Google. Often I find my answer:
>   
>
http://www.google.com/search?q=Parser+has+reached+the+entity+expansion+l
imit

> The first result suggests you can set a Java property to make this
limit larger:
>
http://www.macromedia.com/support/coldfusion/ts/documents/xml_entity_lim
it.htm
> There's more in the Google result page.
> 
> If you're serializing XML as a String, I suspect somewhere someone is
XML escaping your package. IE: if your data is "<foo/>", the SOAP may be
something like <myXmlData>&lt;foo/&gt;</myXmlData that's going to make a
lot of entities. Sounds like there's a limit to how many it will
process.

The google hint is great.  I usually try this, but I haven't had much
luck with searches on AXIS specific questions, and decided to post to
the user group.  I did make the recommended change, but the program
still crashes when the string is too large.  For example, when I get an
oracle result set of 90K records, then make a call to get the XML, then
return it such as:
OracleXMLQuery qry = new OracleXMLQuery(conn, sSQL);
str = qry.getXMLString();
return str;

>>2. [java]  faultString: java.lang.reflect.InvocationTargetException
> This is a generic "the call failed" exception. Hopefully it has a
wrapped exception inside that has more information, maybe it's the
SAXParseException you found above?

I guess the key here is to get the program to fail gracefully as needed.

>>My goal is to send an XML file as a string that another application
can 
>>read and turn into a database file.  I already have this working with 
>>smaller XML strings, but can't get past the errors when getting larger

>>datasets back.
> I can think of four options for you:
> Raise the entity limit big enough for your needs.
> Serialize your data as a byte[], UTF-8 encoded. Ugly, but Axis will
base64 encode it instead of entity escape it.
> Pass your XML directly as XML in the SOAP request. This is what
document/literal is all about. I don't have sample code, sorry.
> Use attachments. A single attachment should work. Attachments hurt
interop, but if you're using Axis on both sides that won't be a problem.

The first two are out.  On to choice three and four.
Passing XML directly is what I thought I was doing.  My webservice could
be broken into the following:

// Get parameter sSQL
OracleXMLQuery qry = new OracleXMLQuery(conn, sSQL);
str = qry.getXMLString();
return str;
// That's about it.  
The result set is converted to a string by oracle and that is what I
return.  I am using a a FoxPro application with MS soap client to read
the XML String, and display the results in a table.  Here is that basic
code:
local lo
lo = createobject("MSSOAP.SoapClient30")
lo.mssoapinit("http://localhost:8080/axis/services/DatWS?WSDL")
lo.ConnectorProperty("Timeout") = 600000
lcString = lo.getXML("select * from bigtable")
XMLTOCURSOR(lcString,"XMLCursor")
BROWSE nowait
return

I have axis returning the result set as a string, and FoxPro reads the
file and displays the data, converted into a cursor.  For small result
sets (under 1000 records) this works great!  I am not sure where the
problem is for the larger result sets.

On to option 4.  This is where I get confused, because I haven't found
good documentation and/or examples to show how to do this.  Now since I
am using FoxPro, it seems that I have to use DIME to send the file as
attachements.
Do attachements work like uploading jpg to a web site?  In that case,
the work is take care of for you...  Also, if I use DIME, with the
MSSOAP be able to parse it in an intelligent way so that foxpro will
assume that it is one big string that is read into a cursor?

My hope is that there is someone out there who has done something like
this that can offer some help.

Thanks,

Ken Lee

Re: Sending Large String

[Nelson Minar <ne...@monkey.org>]

>1. [java] org.xml.sax.SAXParseException: Parser has reached the entity
>expansion limit "64,000" set by the Application.

I'd never heard of this error before, but I'll share a little secret
with you: when I have no idea what an error message means I just paste
it directly into Google. Often I find my answer:
  http://www.google.com/search?q=Parser+has+reached+the+entity+expansion+limit

The first result suggests you can set a Java property to make this
limit larger:
  http://www.macromedia.com/support/coldfusion/ts/documents/xml_entity_limit.htm
There's more in the Google result page.

If you're serializing XML as a String, I suspect somewhere someone is
XML escaping your package. IE: if your data is "<foo/>", the SOAP may
be something like
  <myXmlData>&lt;foo/&gt;</myXmlData
that's going to make a lot of entities. Sounds like there's a limit to
how many it will process.

>2. [java]  faultString: java.lang.reflect.InvocationTargetException

This is a generic "the call failed" exception. Hopefully it has a
wrapped exception inside that has more information, maybe it's the
SAXParseException you found above?

>My goal is to send an XML file as a string that another application can
>read and turn into a database file.  I already have this working with
>smaller XML strings, but can't get past the errors when getting larger
>datasets back.

I can think of four options for you:

Raise the entity limit big enough for your needs.

Serialize your data as a byte[], UTF-8 encoded. Ugly, but Axis will
base64 encode it instead of entity escape it.

Pass your XML directly as XML in the SOAP request. This is what
document/literal is all about. I don't have sample code, sorry.

Use attachments. A single attachment should work. Attachments hurt
interop, but if you're using Axis on both sides that won't be a
problem.