You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cxf.apache.org by Martin Renner <m....@exxcellent.de> on 2010/08/09 13:48:18 UTC

MTOM + Outbound streaming from a database

Hello,

I have a problem with streaming LOB data from a database via CXF+MTOM to a client. It is not a
CXF-specific problem, but rather a conceptual problem.

I wrote an implementation of javax.activation.DataSource, which simply returns an InputStream that
it has received via its constructor. The code in my service implementation logic looks like this:

    InputStream data = lobHandler.getBlobAsBinaryStream(resultSet, "data");
    result.setData(new DataHandler(new StreamDataSource(data, fileName, mimeType));

I think you get the idea: "data" is an InputStream which receives its data directly from the
ResultSet. "StreamDataSource" uses this InputStream and passes it to the activation framework. The
idea behind this code: Streaming directly from LOBs to the client.

In practice I am facing a problem: Right now, I am closing the ResultSet, the Statement and the
Connection somewhere in my service implementation (shortly after the code I have cited above).
However, CXF starts streaming only when I have left the implementation of the service interface. CXF
tries to use this InputStream (which originates from the ResultSet) in its outgoing chain in the
AttachmentOutInterceptor - *after* having executed the service implementation. The consequence is,
that CXF cannot stream the data to the client, because the underlying InputStream from the ResultSet
throws an exception, that the ResultSet has been closed already.

Do you have any suggestions for this problem?

One solution could be to stream the LOB data to a temporary file, close the ResultSet and Statement
and return the InputStream of this file to CXF. Then, I would have to write a servlet filter (or an
CXF interceptor) which would ultimately delete this temporary file. Tradeoff of this solution:
performance penalty.

Another solution could be to close the ResultSet, Statement and Connection after CXF has streamed
the data. Somewhere in a servlet filter or an CXF interceptor. But that would mean that I would have
to stuff those three objects into "something" (ThreadLocal?) where they would be protected from
garbage collection and accessible to the interceptor or filter. Not really beautiful.

What do you think about this problem?


Thanks,
Martin

Re: MTOM + Outbound streaming from a database

Posted by Martin Renner <m....@exxcellent.de>.
Hi Daniel,

I created an issue: https://issues.apache.org/jira/browse/CXF-2940

The patch will follow next week.


Kind Regards,
Martin


Am 11.08.2010 09:46, schrieb Martin Renner:
> Hi Daniel,
> 
> I will create a JIRA issue. And I am also able to submit a patch.
> 
> Kind Regards,
> Martin
> 
> 
> Am 11.08.2010 03:42, schrieb Daniel Kulp:
>>
>> Interesting.  Seems like correct analysis.
>>
>> I think the actual fix, however, would be to ad a handleFault method to the 
>> AttachmentOutIntercept.  On a fault/exception, it would unwind the chain into 
>> there where it can clean itself up, which can include closing those streams.
>>
>> Care to file a JIRA (and submit a patch.  :-)  )
>>
>> Dan

Re: MTOM + Outbound streaming from a database

Posted by Martin Renner <m....@exxcellent.de>.
Hi Daniel,

I will create a JIRA issue. And I am also able to submit a patch.

Kind Regards,
Martin


Am 11.08.2010 03:42, schrieb Daniel Kulp:
> 
> Interesting.  Seems like correct analysis.
> 
> I think the actual fix, however, would be to ad a handleFault method to the 
> AttachmentOutIntercept.  On a fault/exception, it would unwind the chain into 
> there where it can clean itself up, which can include closing those streams.
> 
> Care to file a JIRA (and submit a patch.  :-)  )
> 
> Dan

Re: MTOM + Outbound streaming from a database

Posted by Daniel Kulp <dk...@apache.org>.
Interesting.  Seems like correct analysis.

I think the actual fix, however, would be to ad a handleFault method to the 
AttachmentOutIntercept.  On a fault/exception, it would unwind the chain into 
there where it can clean itself up, which can include closing those streams.

Care to file a JIRA (and submit a patch.  :-)  )

Dan



On Tuesday 10 August 2010 10:31:55 am Martin Renner wrote:
> Hi Daniel,
> 
> thank you for your response. The idea with the wrapped stream is great,
> because it won't scatter the code for opening and closing of the resources
> (like it would be the case with an interceptor).
> 
> However, as far as I understood the code (CXF 2.2.9), there is a possible
> resource leak in the code for the attachment handling:
> 
> AttachmentOutInterceptor (the EndingInterceptor) calls
> AttachmentSerializer.writeAttachments(). This method iterates over all
> attachments and calls "a.getDataHandler().writeTo(out)".
> 
> Let's have a look at "DataHandler.writeTo(OutputStream os)":
> 
>     InputStream is = getInputStream();
>     try {
>         int count;
>         while ((count = is.read(buffer)) != -1)
>             os.write(buffer, 0, count);
>     } finally {
>         is.close();
>     }
> 
> That means, that DataHandler will always close the InputStream of the
> current attachment (which has been opened by the service implementation).
> 
> But there are two problems:
> 
> 1. Let's assume that there are 5 attachments and "writeTo" of the first
> attachment throws an IOException. No one will close the input streams of
> the remaining 4 attachments.
> 
> 2. Let's assume that an exception was thrown somewhere between the service
> implementation (in fact the line where the service implementation opens
> the stream) and
> AttachmentSerializer.writeAttachments(). In this case too, no one will
> close the streams of the attachments.
> 
> IMHO, there should be an interceptor in the OutFaultInterceptor chain which
> iterates over all attachments and calls
> "a.getDataHandler().getInputStream().close()" on each one.
> 
> 
> Kind Regards,
> Martin

-- 
Daniel Kulp
dkulp@apache.org
http://dankulp.com/blog

Re: MTOM + Outbound streaming from a database

Posted by Martin Renner <m....@exxcellent.de>.
Hi Daniel,

thank you for your response. The idea with the wrapped stream is great, because it won't scatter the
code for opening and closing of the resources (like it would be the case with an interceptor).

However, as far as I understood the code (CXF 2.2.9), there is a possible resource leak in the code
for the attachment handling:

AttachmentOutInterceptor (the EndingInterceptor) calls AttachmentSerializer.writeAttachments(). This
method iterates over all attachments and calls "a.getDataHandler().writeTo(out)".

Let's have a look at "DataHandler.writeTo(OutputStream os)":

    InputStream is = getInputStream();
    try {
        int count;
        while ((count = is.read(buffer)) != -1)
            os.write(buffer, 0, count);
    } finally {
        is.close();
    }

That means, that DataHandler will always close the InputStream of the current attachment (which has
been opened by the service implementation).

But there are two problems:

1. Let's assume that there are 5 attachments and "writeTo" of the first attachment throws an
IOException. No one will close the input streams of the remaining 4 attachments.

2. Let's assume that an exception was thrown somewhere between the service implementation (in fact
the line where the service implementation opens the stream) and
AttachmentSerializer.writeAttachments(). In this case too, no one will close the streams of the
attachments.

IMHO, there should be an interceptor in the OutFaultInterceptor chain which iterates over all
attachments and calls "a.getDataHandler().getInputStream().close()" on each one.


Kind Regards,
Martin

Re: MTOM + Outbound streaming from a database

Posted by Daniel Kulp <dk...@apache.org>.

There are several options for tackling this type thing.   One option is to 
wrapper the original InputStream with something like:


final ResultSet 

final ResultSet resultSet = ...;
final Statement statement = ...;
final Connection connect = ...;
new FilterIntputStream(origIn) {
    public void close() throws IOException {
        super.close();
        resultSet.close();  //or whatever it is
        statement.close();
        connect.close();
    }
}


When CXF is done with the stream and close is called, you would then close all 
the resources at that point since we'd be completely done with it.

Another option would be to set the ResultSet and such into the MessageContext 
and then write a CXF interceptor that grab them and close them after 
everything is done.    The above is probably easier though.

If you WANT to go the temporary file route, the best would be to use the 
CachedOutputStream in CXF.   Write data there, call the getInputStream on it.  
If the data is small (under 64K), it stays in memory.   If larger, it uses a 
temp file.   On the close of the streams, it would delete the file.

Dan



On Monday 09 August 2010 7:48:18 am Martin Renner wrote:
> Hello,
> 
> I have a problem with streaming LOB data from a database via CXF+MTOM to a
> client. It is not a CXF-specific problem, but rather a conceptual problem.
> 
> I wrote an implementation of javax.activation.DataSource, which simply
> returns an InputStream that it has received via its constructor. The code
> in my service implementation logic looks like this:
> 
>     InputStream data = lobHandler.getBlobAsBinaryStream(resultSet, "data");
>     result.setData(new DataHandler(new StreamDataSource(data, fileName,
> mimeType));
> 
> I think you get the idea: "data" is an InputStream which receives its data
> directly from the ResultSet. "StreamDataSource" uses this InputStream and
> passes it to the activation framework. The idea behind this code:
> Streaming directly from LOBs to the client.
> 
> In practice I am facing a problem: Right now, I am closing the ResultSet,
> the Statement and the Connection somewhere in my service implementation
> (shortly after the code I have cited above). However, CXF starts streaming
> only when I have left the implementation of the service interface. CXF
> tries to use this InputStream (which originates from the ResultSet) in its
> outgoing chain in the AttachmentOutInterceptor - *after* having executed
> the service implementation. The consequence is, that CXF cannot stream the
> data to the client, because the underlying InputStream from the ResultSet
> throws an exception, that the ResultSet has been closed already.
> 
> Do you have any suggestions for this problem?
> 
> One solution could be to stream the LOB data to a temporary file, close the
> ResultSet and Statement and return the InputStream of this file to CXF.
> Then, I would have to write a servlet filter (or an CXF interceptor) which
> would ultimately delete this temporary file. Tradeoff of this solution:
> performance penalty.
> 
> Another solution could be to close the ResultSet, Statement and Connection
> after CXF has streamed the data. Somewhere in a servlet filter or an CXF
> interceptor. But that would mean that I would have to stuff those three
> objects into "something" (ThreadLocal?) where they would be protected from
> garbage collection and accessible to the interceptor or filter. Not really
> beautiful.
> 
> What do you think about this problem?
> 
> 
> Thanks,
> Martin

-- 
Daniel Kulp
dkulp@apache.org
http://dankulp.com/blog