You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@ws.apache.org by Greg Barnett <gr...@crowncollege.edu> on 2003/11/22 03:14:23 UTC
Really large amounts of data, was Re: XML-RPC for Database Access
Bruce Van Horn wrote:
> about a year and a half ago I built a transaction server that took a
> series of queries in an xml document via xmlrpc.
>
> It works great as long as you don't try to move really large amounts of
> data or BLOBs.
I'm currently writing a script that needs to retrieve a large amount of
data via XML-RPC. When the server returns a smallish amount of data,
everything works correctly. When a large amount of data is returned, it
looks like not all of the data is getting sent to the SAX parser. I
have tested the server with other clients, and they are able to receive
all of the data just fine.
Here is the relevant portion of my code (if it looks a bit odd, it is
because it is written in Jython).
############################################################
from org.apache.xmlrpc import XmlRpcClient
from org.apache.xmlrpc import XmlRpc
def xmlrpc(method, params, username, password):
#XmlRpc.setDriver('org.apache.xerces.parsers.SAXParser')
XmlRpc.setDebug(true)
client = XmlRpcClient('https://host/cgi-bin/file.py')
client.setBasicAuthentication(username, password)
return client.execute(method, params)
############################################################
The server returns a list of dictionaries. I've had the client
successfully retrieve up to about 10,000 dictionaries in the list.
Typically, it seems to have problems with more than a few hundred
dictionaries.
My original thought was that it was the default SAX parser, but
switching to xerces just gave a slightly different error message.
Is there any way to make sure that all of the data gets read by the parser?
Greg Barnett
firstname dot lastname at crowncollege dot edu
This is with the xerces SAX parser:
Created client to url space https://host/cgi-bin/file.py
Client calling procedure 'foo.bar' with parameters [spam , eggs]
Beginning parsing XML input stream
startElement: methodResponse
startElement: params
startElement: param
startElement: value
startElement: array
startElement: data
startElement: value
startElement: struct
startElement: member
startElement: name
endElement: name
startElement: value
startElement: string
endElement: string
endElement: value
endElement: member
startElement: member
startElement: name
[ 2700 lines snipped ]
startElement: value
startElement: struct
startElement: member
startElement: name
Fatal error parsing XML: org.xml.sax.SAXParseException: The element type
"name" must be terminated by the matching end-tag "</name>".
org.xml.sax.SAXParseException: The element type "name" must be
terminated by the matching end-tag "</name>".
at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source)
at org.apache.xmlrpc.XmlRpc.parse(XmlRpc.java:419)
at org.apache.xmlrpc.XmlRpcClient$Worker.execute(XmlRpcClient.java:436)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:163)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.python.core.PyReflectedFunction.__call__(PyReflectedFunction.java)
at org.python.core.PyMethod.__call__(PyMethod.java)
at org.python.core.PyObject.__call__(PyObject.java)
at org.python.core.PyInstance.invoke(PyInstance.java)
at
org.python.pycode._pyx0.xmlrpc$3(h:\cvswork\ccdb\client\JythonClient.py:52)
at
org.python.pycode._pyx0.call_function(h:\cvswork\ccdb\client\JythonClient.py)
at org.python.core.PyTableCode.call(PyTableCode.java)
at org.python.core.PyTableCode.call(PyTableCode.java)
at org.python.core.PyFunction.__call__(PyFunction.java)
at org.python.core.PyInstance.invoke(PyInstance.java)
at
org.python.pycode._pyx0.updateTest$4(h:\cvswork\ccdb\client\JythonClient.py:81)
at
org.python.pycode._pyx0.call_function(h:\cvswork\ccdb\client\JythonClient.py)
at org.python.core.PyTableCode.call(PyTableCode.java)
at org.python.core.PyTableCode.call(PyTableCode.java)
at org.python.core.PyTableCode.call(PyTableCode.java)
at org.python.core.PyFunction.__call__(PyFunction.java)
at org.python.core.PyMethod.__call__(PyMethod.java)
at org.python.core.PyCompoundCallable.__call__(PyCompoundCallable.java)
at org.python.core.PyObject.__call__(PyObject.java)
at org.python.core.PyObject._jcallexc(PyObject.java)
at org.python.core.PyObject._jcall(PyObject.java)
at
org.python.proxies.java.awt.event.ActionListener$Adapter.actionPerformed(Unknown
Source)
at javax.swing.AbstractButton.fireActionPerformed(Unknown Source)
at
javax.swing.AbstractButton$ForwardActionEvents.actionPerformed(Unknown
Source)
at javax.swing.DefaultButtonModel.fireActionPerformed(Unknown Source)
at javax.swing.DefaultButtonModel.setPressed(Unknown Source)
at javax.swing.plaf.basic.BasicButtonListener.mouseReleased(Unknown Source)
at java.awt.Component.processMouseEvent(Unknown Source)
at java.awt.Component.processEvent(Unknown Source)
at java.awt.Container.processEvent(Unknown Source)
at java.awt.Component.dispatchEventImpl(Unknown Source)
at java.awt.Container.dispatchEventImpl(Unknown Source)
at java.awt.Component.dispatchEvent(Unknown Source)
at java.awt.LightweightDispatcher.retargetMouseEvent(Unknown Source)
at java.awt.LightweightDispatcher.processMouseEvent(Unknown Source)
at java.awt.LightweightDispatcher.dispatchEvent(Unknown Source)
at java.awt.Container.dispatchEventImpl(Unknown Source)
at java.awt.Window.dispatchEventImpl(Unknown Source)
at java.awt.Component.dispatchEvent(Unknown Source)
at java.awt.EventQueue.dispatchEvent(Unknown Source)
at java.awt.EventDispatchThread.pumpOneEventForHierarchy(Unknown Source)
at java.awt.EventDispatchThread.pumpEventsForHierarchy(Unknown Source)
at java.awt.EventDispatchThread.pumpEvents(Unknown Source)
at java.awt.EventDispatchThread.pumpEvents(Unknown Source)
at java.awt.EventDispatchThread.run(Unknown Source)
Traceback (innermost last):
(no code object) at line 0
this is with the default parser:
Created client to url space https://host/cgi-bin/file.py
Client calling procedure 'foo.bar' with parameters [spam, eggs]
Beginning parsing XML input stream
startElement: methodResponse
startElement: params
startElement: param
startElement: value
startElement: array
startElement: data
startElement: value
startElement: struct
startElement: member
startElement: name
endElement: name
startElement: value
startElement: string
endElement: string
endElement: value
endElement: member
startElement: member
startElement: name
<snip>
endElement: string
endElement: value
endElement: member
startElement: member
startElement: name
endElement: name
startElement: value
startElement: string
endElement: string
endElement: value
endElement: member
endElement: struct
endElement: value
startElement: value
startElement: struct
startElement: member
startElement: name
Fatal error parsing XML: org.xml.sax.SAXParseException: unexpected end
of file found
Spent 401 millis parsing
Spent 10325 in request
Re: Really large amounts of data, was Re: XML-RPC for Database Access
Posted by Ryan Hoegg <rh...@isisnetworks.net>.
Hi Greg,
For my uses using a CommonsXmlRpcTransport as an argument to
XmlRpcClient.execute() worked well, since I wanted to manage my
HttpClient outside of the XmlRpcClient. We will definitely need to
round the library out with a CommonsXmlRpcTransportFactory, and make
sure we handle multithreading right. Also, if you want to use HTTP
authentication, you will need to modify the transport, look for the TODO
in the source.
HttpClient httpClient = new HttpClient(getURL());
// configure as you please
CommonsXmlRpcTransport xtransport = new CommonsXmlRpcTransport(getURL(),
httpClient);
// I needed HTTP 1.1
// xtransport.setHttp11(true);
// that could also be done directly on the HttpClient
XmlRpcClient xmlrpc = new XmlRpcClient(getURL());
// ...
XmlRpcRequest request = new XmlRpcRequest(methodName, params);
Object result = xmlrpc.execute(request, xtransport);
OK, now that I've typed up that example I see some room for improvement
of the API :) Hopefully soon I'll have a project where I can justify
spending some time reworking it...
--
Ryan Hoegg
ISIS Networks
http://www.isisnetworks.net
Greg Barnett wrote:
> I've checked out the code from CVS following the directions at
> http://ws.apache.org/xmlrpc/cvs.html, and looked at the JavaDocs at
> http://ws.apache.org/xmlrpc/apidocs/index.html
>
> I see CommonsXmlRpcTransport, but CommonsXmlRpcClient and
> CommonsXmlRpcTransportFactory are not listed.
>
> XmlRpcClient requires an XmlRpcTransportFactory, so I'm not seeing
> exactly how to construct a client using CommonsXmlRpcTransport. A bit
> of sample code showing me how this works would be greatly appreciated.
>
> thanks,
> Greg
>
>
> Ryan Hoegg wrote:
>
>> If you use the code in CVS HEAD, you can plug in a
>> CommonsXmlRpcClient, which supports HTTP 1.1
>
>
>
>
Re: Really large amounts of data, was Re: XML-RPC for Database Access
Posted by Greg Barnett <gr...@crowncollege.edu>.
I've checked out the code from CVS following the directions at
http://ws.apache.org/xmlrpc/cvs.html, and looked at the JavaDocs at
http://ws.apache.org/xmlrpc/apidocs/index.html
I see CommonsXmlRpcTransport, but CommonsXmlRpcClient and
CommonsXmlRpcTransportFactory are not listed.
XmlRpcClient requires an XmlRpcTransportFactory, so I'm not seeing
exactly how to construct a client using CommonsXmlRpcTransport. A bit
of sample code showing me how this works would be greatly appreciated.
thanks,
Greg
Ryan Hoegg wrote:
> If you use the code in CVS HEAD, you can plug in a
> CommonsXmlRpcClient, which supports HTTP 1.1
Re: Really large amounts of data, was Re: XML-RPC for Database Access
Posted by Ryan Hoegg <rh...@isisnetworks.net>.
I had a similar issue because the server was using HTTP 1.1 and the
client was requesting HTTP 1.1 but only accepting HTTP 1.0. HTTP 1.1
uses chunking, which messes up the XML returned by the server if it is
interpreted as if it was not chunked.
You can fix it by
a) changing the server to use HTTP 1.0
b) making the client request HTTP 1.0
c) making the client understand HTTP 1.1
I opted for b first, then c. If you use the code in CVS HEAD, you can
plug in a CommonsXmlRpcClient, which supports HTTP 1.1
--
Ryan Hoegg
ISIS Networks
http://www.isisnetworks.net
Greg Barnett wrote:
> Bruce Van Horn wrote:
>
>> about a year and a half ago I built a transaction server that took a
>> series of queries in an xml document via xmlrpc.
>>
>> It works great as long as you don't try to move really large amounts
>> of data or BLOBs.
>
>
> I'm currently writing a script that needs to retrieve a large amount
> of data via XML-RPC. When the server returns a smallish amount of
> data, everything works correctly. When a large amount of data is
> returned, it looks like not all of the data is getting sent to the SAX
> parser. I have tested the server with other clients, and they are
> able to receive all of the data just fine.
>
> Here is the relevant portion of my code (if it looks a bit odd, it is
> because it is written in Jython).
>
> ############################################################
> from org.apache.xmlrpc import XmlRpcClient
> from org.apache.xmlrpc import XmlRpc
>
> def xmlrpc(method, params, username, password):
> #XmlRpc.setDriver('org.apache.xerces.parsers.SAXParser')
> XmlRpc.setDebug(true)
> client = XmlRpcClient('https://host/cgi-bin/file.py')
> client.setBasicAuthentication(username, password)
> return client.execute(method, params)
> ############################################################
>
> The server returns a list of dictionaries. I've had the client
> successfully retrieve up to about 10,000 dictionaries in the list.
> Typically, it seems to have problems with more than a few hundred
> dictionaries.
>
> My original thought was that it was the default SAX parser, but
> switching to xerces just gave a slightly different error message.
>
> Is there any way to make sure that all of the data gets read by the
> parser?
>
> Greg Barnett
> firstname dot lastname at crowncollege dot edu
>
>
> This is with the xerces SAX parser:
>
> Created client to url space https://host/cgi-bin/file.py
> Client calling procedure 'foo.bar' with parameters [spam , eggs]
> Beginning parsing XML input stream
> startElement: methodResponse
> startElement: params
> startElement: param
> startElement: value
> startElement: array
> startElement: data
> startElement: value
> startElement: struct
> startElement: member
> startElement: name
> endElement: name
> startElement: value
> startElement: string
> endElement: string
> endElement: value
> endElement: member
> startElement: member
> startElement: name
> [ 2700 lines snipped ]
> startElement: value
> startElement: struct
> startElement: member
> startElement: name
> Fatal error parsing XML: org.xml.sax.SAXParseException: The element
> type "name" must be terminated by the matching end-tag "</name>".
> org.xml.sax.SAXParseException: The element type "name" must be
> terminated by the matching end-tag "</name>".
> at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source)
> at org.apache.xmlrpc.XmlRpc.parse(XmlRpc.java:419)
> at
> org.apache.xmlrpc.XmlRpcClient$Worker.execute(XmlRpcClient.java:436)
> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:163)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
> at java.lang.reflect.Method.invoke(Unknown Source)
> at
> org.python.core.PyReflectedFunction.__call__(PyReflectedFunction.java)
> at org.python.core.PyMethod.__call__(PyMethod.java)
> at org.python.core.PyObject.__call__(PyObject.java)
> at org.python.core.PyInstance.invoke(PyInstance.java)
> at
> org.python.pycode._pyx0.xmlrpc$3(h:\cvswork\ccdb\client\JythonClient.py:52)
>
> at
> org.python.pycode._pyx0.call_function(h:\cvswork\ccdb\client\JythonClient.py)
>
> at org.python.core.PyTableCode.call(PyTableCode.java)
> at org.python.core.PyTableCode.call(PyTableCode.java)
> at org.python.core.PyFunction.__call__(PyFunction.java)
> at org.python.core.PyInstance.invoke(PyInstance.java)
> at
> org.python.pycode._pyx0.updateTest$4(h:\cvswork\ccdb\client\JythonClient.py:81)
>
> at
> org.python.pycode._pyx0.call_function(h:\cvswork\ccdb\client\JythonClient.py)
>
> at org.python.core.PyTableCode.call(PyTableCode.java)
> at org.python.core.PyTableCode.call(PyTableCode.java)
> at org.python.core.PyTableCode.call(PyTableCode.java)
> at org.python.core.PyFunction.__call__(PyFunction.java)
> at org.python.core.PyMethod.__call__(PyMethod.java)
> at
> org.python.core.PyCompoundCallable.__call__(PyCompoundCallable.java)
> at org.python.core.PyObject.__call__(PyObject.java)
> at org.python.core.PyObject._jcallexc(PyObject.java)
> at org.python.core.PyObject._jcall(PyObject.java)
> at
> org.python.proxies.java.awt.event.ActionListener$Adapter.actionPerformed(Unknown
> Source)
> at javax.swing.AbstractButton.fireActionPerformed(Unknown Source)
> at
> javax.swing.AbstractButton$ForwardActionEvents.actionPerformed(Unknown
> Source)
> at javax.swing.DefaultButtonModel.fireActionPerformed(Unknown Source)
> at javax.swing.DefaultButtonModel.setPressed(Unknown Source)
> at
> javax.swing.plaf.basic.BasicButtonListener.mouseReleased(Unknown Source)
> at java.awt.Component.processMouseEvent(Unknown Source)
> at java.awt.Component.processEvent(Unknown Source)
> at java.awt.Container.processEvent(Unknown Source)
> at java.awt.Component.dispatchEventImpl(Unknown Source)
> at java.awt.Container.dispatchEventImpl(Unknown Source)
> at java.awt.Component.dispatchEvent(Unknown Source)
> at java.awt.LightweightDispatcher.retargetMouseEvent(Unknown Source)
> at java.awt.LightweightDispatcher.processMouseEvent(Unknown Source)
> at java.awt.LightweightDispatcher.dispatchEvent(Unknown Source)
> at java.awt.Container.dispatchEventImpl(Unknown Source)
> at java.awt.Window.dispatchEventImpl(Unknown Source)
> at java.awt.Component.dispatchEvent(Unknown Source)
> at java.awt.EventQueue.dispatchEvent(Unknown Source)
> at java.awt.EventDispatchThread.pumpOneEventForHierarchy(Unknown
> Source)
> at java.awt.EventDispatchThread.pumpEventsForHierarchy(Unknown
> Source)
> at java.awt.EventDispatchThread.pumpEvents(Unknown Source)
> at java.awt.EventDispatchThread.pumpEvents(Unknown Source)
> at java.awt.EventDispatchThread.run(Unknown Source)
> Traceback (innermost last):
> (no code object) at line 0
>
>
>
>
>
>
>
>
> this is with the default parser:
>
> Created client to url space https://host/cgi-bin/file.py
> Client calling procedure 'foo.bar' with parameters [spam, eggs]
> Beginning parsing XML input stream
> startElement: methodResponse
> startElement: params
> startElement: param
> startElement: value
> startElement: array
> startElement: data
> startElement: value
> startElement: struct
> startElement: member
> startElement: name
> endElement: name
> startElement: value
> startElement: string
> endElement: string
> endElement: value
> endElement: member
> startElement: member
> startElement: name
> <snip>
> endElement: string
> endElement: value
> endElement: member
> startElement: member
> startElement: name
> endElement: name
> startElement: value
> startElement: string
> endElement: string
> endElement: value
> endElement: member
> endElement: struct
> endElement: value
> startElement: value
> startElement: struct
> startElement: member
> startElement: name
> Fatal error parsing XML: org.xml.sax.SAXParseException: unexpected end
> of file found
> Spent 401 millis parsing
> Spent 10325 in request
>
>
>
>