You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user-java@ibatis.apache.org by "Brian.May" <Br...@noaa.gov> on 2007/11/21 18:49:30 UTC
Using Ibatis to stream results
The project that I am currently working on requires that results be
streamed since the size of the resultset can easily exceed 25MB. We
currently deal with this by using
oracle.xml.parser.v2.XMLSAXSerializer combined with OracleXMLQuery
classes. The down side to this solution is that it requires the dao to
manage connections which ends up as a lot of try catch finally constructs.
We have started to use iBatis for a number of our less extreme cases,
and we are now attempting to use iBatis to stream results.
My attempt to do this involves sub-classing RowHandler and adding an
OutputStream field to it. In this way each result object can be streamed
out. The problem that I am running into is that the only way to do any
pre or post processing on the OutputStream requires the creator of the
StreamedRowHandler to make calls to doPreProcessing() and
doPostProcessing(). Ideally these calls should be made just before the
first call to handleRow() and just after the last call to handleRow().
It looks like this would require modification to
com.ibatis.sqlmap.engine.execution.SqlExecutor. It is my understanding
that sub-classing or modifying classes in the engine package is
unsupported since they are likely to change.
My question is if this is something which others would like to see
implemented, and more importantly is this something the devs might be
interested in implementing? If it is something that the devs feel is
outside the scope of the iBatis project how often, and how significantly
do the engine classes change? In other words is it worth my time to
implement this and maintain it on my own or should I not bother?
Thanks
Re: Using Ibatis to stream results
Posted by "Brian.May" <Br...@noaa.gov>.
After a quick look at Xstream it looks very promising. Thanks for the
suggestion I will be taking it for a test drive.
Christopher Lamey wrote the following on 11/21/2007 1:14 PM:
> Hello,
>
> I regularly stream out gigs worth of xml without any trouble. Do do this, I
> use XStream and an object that implements RowHandler. The XStream object is
> just a member variable of the RowHandler that gets invoked in the
> handleRow() method.
>
> Something like this:
>
> public class BigExportRowHandler implements RowHandler {
>
> private XStream xs;
>
> public void handleRow(Object valueObject) {
> Person p = (Person) valueObject;
> xs.toXML(p);
> }
>
> public void setXStream(Xstream xs) {
> this.xs = xs;
> }
> }
>
> Then in my DAO I just do something like this:
>
> // Maybe put this in somewhere else and not in the method
> // In reality, the xstream config is more complicated
> XStream xs = new Xstream();
> xs.alias("person", Person.class);
>
> BigExportRowHandler rh = new BigExportRowHandler();
> rh.setXStream(xs);
>
> // No paramater call to queryWithRowHandler
> sqlMapClient.queryWithRowHandler("query_id", rh);
>
> And away it goes.
>
> With XStream converters and aliases, you can make the output look pretty
> much like anything you want.
>
> http://xstream.codehaus.org/index.html
>
> FWIW, I use another program called stx (http://stx.sourceforge.net/) if
> XStream can't do what I want. Stx isn't too active but I've been using it
> for over a year now on big files with no problems.
>
> Also, xmlbeans has a streaming xsd validator that can validate huge xml
> files against xsds.
>
> What do you need to do in the pre and post processing? Could you do that in
> the handleRow method itself?
>
> Cheers,
> Chris
>
> On 11/21/07 10:49 AM, "Brian.May" <Br...@noaa.gov> wrote:
>
>
>> The project that I am currently working on requires that results be
>> streamed since the size of the resultset can easily exceed 25MB. We
>> currently deal with this by using
>> oracle.xml.parser.v2.XMLSAXSerializer combined with OracleXMLQuery
>> classes. The down side to this solution is that it requires the dao to
>> manage connections which ends up as a lot of try catch finally constructs.
>>
>> We have started to use iBatis for a number of our less extreme cases,
>> and we are now attempting to use iBatis to stream results.
>>
>> My attempt to do this involves sub-classing RowHandler and adding an
>> OutputStream field to it. In this way each result object can be streamed
>> out. The problem that I am running into is that the only way to do any
>> pre or post processing on the OutputStream requires the creator of the
>> StreamedRowHandler to make calls to doPreProcessing() and
>> doPostProcessing(). Ideally these calls should be made just before the
>> first call to handleRow() and just after the last call to handleRow().
>> It looks like this would require modification to
>> com.ibatis.sqlmap.engine.execution.SqlExecutor. It is my understanding
>> that sub-classing or modifying classes in the engine package is
>> unsupported since they are likely to change.
>>
>> My question is if this is something which others would like to see
>> implemented, and more importantly is this something the devs might be
>> interested in implementing? If it is something that the devs feel is
>> outside the scope of the iBatis project how often, and how significantly
>> do the engine classes change? In other words is it worth my time to
>> implement this and maintain it on my own or should I not bother?
>>
>> Thanks
>>
>>
>>
>>
>
>
Re: Using Ibatis to stream results
Posted by Christopher Lamey <cl...@localmatters.com>.
Hello,
I regularly stream out gigs worth of xml without any trouble. Do do this, I
use XStream and an object that implements RowHandler. The XStream object is
just a member variable of the RowHandler that gets invoked in the
handleRow() method.
Something like this:
public class BigExportRowHandler implements RowHandler {
private XStream xs;
public void handleRow(Object valueObject) {
Person p = (Person) valueObject;
xs.toXML(p);
}
public void setXStream(Xstream xs) {
this.xs = xs;
}
}
Then in my DAO I just do something like this:
// Maybe put this in somewhere else and not in the method
// In reality, the xstream config is more complicated
XStream xs = new Xstream();
xs.alias("person", Person.class);
BigExportRowHandler rh = new BigExportRowHandler();
rh.setXStream(xs);
// No paramater call to queryWithRowHandler
sqlMapClient.queryWithRowHandler("query_id", rh);
And away it goes.
With XStream converters and aliases, you can make the output look pretty
much like anything you want.
http://xstream.codehaus.org/index.html
FWIW, I use another program called stx (http://stx.sourceforge.net/) if
XStream can't do what I want. Stx isn't too active but I've been using it
for over a year now on big files with no problems.
Also, xmlbeans has a streaming xsd validator that can validate huge xml
files against xsds.
What do you need to do in the pre and post processing? Could you do that in
the handleRow method itself?
Cheers,
Chris
On 11/21/07 10:49 AM, "Brian.May" <Br...@noaa.gov> wrote:
> The project that I am currently working on requires that results be
> streamed since the size of the resultset can easily exceed 25MB. We
> currently deal with this by using
> oracle.xml.parser.v2.XMLSAXSerializer combined with OracleXMLQuery
> classes. The down side to this solution is that it requires the dao to
> manage connections which ends up as a lot of try catch finally constructs.
>
> We have started to use iBatis for a number of our less extreme cases,
> and we are now attempting to use iBatis to stream results.
>
> My attempt to do this involves sub-classing RowHandler and adding an
> OutputStream field to it. In this way each result object can be streamed
> out. The problem that I am running into is that the only way to do any
> pre or post processing on the OutputStream requires the creator of the
> StreamedRowHandler to make calls to doPreProcessing() and
> doPostProcessing(). Ideally these calls should be made just before the
> first call to handleRow() and just after the last call to handleRow().
> It looks like this would require modification to
> com.ibatis.sqlmap.engine.execution.SqlExecutor. It is my understanding
> that sub-classing or modifying classes in the engine package is
> unsupported since they are likely to change.
>
> My question is if this is something which others would like to see
> implemented, and more importantly is this something the devs might be
> interested in implementing? If it is something that the devs feel is
> outside the scope of the iBatis project how often, and how significantly
> do the engine classes change? In other words is it worth my time to
> implement this and maintain it on my own or should I not bother?
>
> Thanks
>
>
>