You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@openjpa.apache.org by Simone Gianni <si...@apache.org> on 2009/04/09 19:18:32 UTC
Streaming BLOB/LOB : who closes the stream?
Hi all,
I noticed that when using streaming BLOB, after consuming the stream,
OpenJPA does not close it. This is causing lots of problems to us, cause
we're working on a system that needs to populate lots of blobs from
incoming files, submit them to a number of "processors" and then save
them back to file system storage. Since the FileInputStream we are
passing to OpenJPA during the "fetch" phase never gets closed, after a
few seconds the system runs out of resources.
Ignacio, can you confirm this is the case?
Since once the stream is used it is (99% of the times) unusable again,
OpenJPA should close it in the DBDictionary class, eventually in the
method that consumes it (it should be copy(InputStream,OutputStream) IIRC).
Documentation says nothing about the need to hold a reference to the
stream passed to a bean to close it after OpenJPA finished using it, and
I think it would be quite a pain to close the streams from the
application layer, cause that would mean keeping a reference to them. In
plain english, the contract should be something like "If you call the
getter and obtain an input stream, it's your job to close it, if you
call a setter passing an input stream, it's OpenJPA job to close it".
I'm testing right now a patch (3 lines) to close it after OpenJPA used
it. If I'm not missing something and you agree this is a bug, I'll open
an issue and attach the patch to it to keep track of the problem.
Simone
--
Simone Gianni CEO Semeru s.r.l. Apache Committer
http://www.simonegianni.it/
Re: Streaming BLOB/LOB : who closes the stream?
Posted by Simone Gianni <si...@apache.org>.
Hi Ignacio,
That would mean that :
- when the setter returns, OpenJPA has consumer the stream and I can
close it. This is obviously not the case, to act that way the content
should be buffered, making the whole streaming system useless.
Otherwise I have to keep a reference to each stream I used, and close
it after the transaction has finished, which is again quite a problem
cause I could have no way to know when the transaction ends.
- to make it symmetric, following what David say, OpenJPA or the jdbc
driver should close the stream retrived by the application calling a
getter, seems like this is not the case, at least not with mysql.
While I do understand the concern of not closing a stream cause it
could be reused, I don't fully understand how it applies to OpenJpa,
cause the stream is anyway consumed to the end. Moreover there are a
number of techniques to avoid stream closing (wrap it overriding the
close method) which are by far simpler than closing the stream when
the (potentially distributed) transaction ends.
Wdyt?
Simone
Inviato da iPhone
Il giorno 10/apr/09, alle ore 17:06, Ignacio Andreu
<pl...@gmail.com> ha scritto:
> Hi Simone,
>
> As David said, close the stream is not the business of the
> framework, if you
> open the stream you should close it. Maybe we should put a note in the
> documentation.
>
> Thanks,
>
> On Fri, Apr 10, 2009 at 2:47 AM, David Jencks
> <da...@yahoo.com>wrote:
>
>>
>> On Apr 9, 2009, at 10:18 AM, Simone Gianni wrote:
>>
>> Hi all,
>>> I noticed that when using streaming BLOB, after consuming the
>>> stream,
>>> OpenJPA does not close it. This is causing lots of problems to us,
>>> cause
>>> we're working on a system that needs to populate lots of blobs
>>> from incoming
>>> files, submit them to a number of "processors" and then save them
>>> back to
>>> file system storage. Since the FileInputStream we are passing to
>>> OpenJPA
>>> during the "fetch" phase never gets closed, after a few seconds
>>> the system
>>> runs out of resources.
>>>
>>> Ignacio, can you confirm this is the case?
>>>
>>> Since once the stream is used it is (99% of the times) unusable
>>> again,
>>> OpenJPA should close it in the DBDictionary class, eventually in
>>> the method
>>> that consumes it (it should be copy(InputStream,OutputStream) IIRC).
>>>
>>> Documentation says nothing about the need to hold a reference to the
>>> stream passed to a bean to close it after OpenJPA finished using
>>> it, and I
>>> think it would be quite a pain to close the streams from the
>>> application
>>> layer, cause that would mean keeping a reference to them. In plain
>>> english,
>>> the contract should be something like "If you call the getter and
>>> obtain an
>>> input stream, it's your job to close it, if you call a setter
>>> passing an
>>> input stream, it's OpenJPA job to close it".
>>>
>>> I'm testing right now a patch (3 lines) to close it after OpenJPA
>>> used it.
>>> If I'm not missing something and you agree this is a bug, I'll
>>> open an issue
>>> and attach the patch to it to keep track of the problem.
>>>
>>
>> I tend to disagree with you about the design and think whoever
>> opens the
>> stream should be responsible for closing it. I've repeatedly had
>> to work
>> around the behavior you are advocating as implemented in IIRC
>> xerces. IIRC
>> the situation I keep running into is an input stream from a jar
>> entry where
>> closing the input stream closes the entire jar file. In order to
>> use xerces
>> we had to wrap the input stream from the jar entry in something
>> that ignores
>> the close method. This experience has convinced me that something
>> that
>> reads from a supplied input stream has no business deciding whether
>> to close
>> it.
>>
>> thanks
>> david jencks
>>
>>
>>
>>>
>>> Simone
>>>
>>> --
>>> Simone Gianni CEO Semeru s.r.l. Apache
>>> Committer
>>> http://www.simonegianni.it/
>>>
>>>
>>
>
>
> --
> Blog: http://plunchete.wordpress.com/
> Web Page: http://es.debugmodeon.com/
Re: Streaming BLOB/LOB : who closes the stream?
Posted by Miłosz Tylenda <mt...@o2.pl>.
Hi all!
Will that "close it yourself" pattern be possible when using container managed transactions?
Regards,
Milosz
> Hi Simone,
>
> As David said, close the stream is not the business of the framework, if you
> open the stream you should close it. Maybe we should put a note in the
> documentation.
>
> Thanks,
>
> On Fri, Apr 10, 2009 at 2:47 AM, David Jencks wrote:
>
> >
> > On Apr 9, 2009, at 10:18 AM, Simone Gianni wrote:
> >
> > Hi all,
> >> I noticed that when using streaming BLOB, after consuming the stream,
> >> OpenJPA does not close it. This is causing lots of problems to us, cause
> >> we're working on a system that needs to populate lots of blobs from incoming
> >> files, submit them to a number of "processors" and then save them back to
> >> file system storage. Since the FileInputStream we are passing to OpenJPA
> >> during the "fetch" phase never gets closed, after a few seconds the system
> >> runs out of resources.
> >>
> >> Ignacio, can you confirm this is the case?
> >>
> >> Since once the stream is used it is (99% of the times) unusable again,
> >> OpenJPA should close it in the DBDictionary class, eventually in the method
> >> that consumes it (it should be copy(InputStream,OutputStream) IIRC).
> >>
> >> Documentation says nothing about the need to hold a reference to the
> >> stream passed to a bean to close it after OpenJPA finished using it, and I
> >> think it would be quite a pain to close the streams from the application
> >> layer, cause that would mean keeping a reference to them. In plain english,
> >> the contract should be something like "If you call the getter and obtain an
> >> input stream, it's your job to close it, if you call a setter passing an
> >> input stream, it's OpenJPA job to close it".
> >>
> >> I'm testing right now a patch (3 lines) to close it after OpenJPA used it.
> >> If I'm not missing something and you agree this is a bug, I'll open an issue
> >> and attach the patch to it to keep track of the problem.
> >>
> >
> > I tend to disagree with you about the design and think whoever opens the
> > stream should be responsible for closing it. I've repeatedly had to work
> > around the behavior you are advocating as implemented in IIRC xerces. IIRC
> > the situation I keep running into is an input stream from a jar entry where
> > closing the input stream closes the entire jar file. In order to use xerces
> > we had to wrap the input stream from the jar entry in something that ignores
> > the close method. This experience has convinced me that something that
> > reads from a supplied input stream has no business deciding whether to close
> > it.
> >
> > thanks
> > david jencks
> >
> >
> >
> >>
> >> Simone
> >>
> >> --
> >> Simone Gianni CEO Semeru s.r.l. Apache Committer
> >> http://www.simonegianni.it/
> >>
> >>
> >
>
>
> --
> Blog: http://plunchete.wordpress.com/
> Web Page: http://es.debugmodeon.com/
>
Re: Streaming BLOB/LOB : who closes the stream?
Posted by Ignacio Andreu <pl...@gmail.com>.
Hi Simone,
As David said, close the stream is not the business of the framework, if you
open the stream you should close it. Maybe we should put a note in the
documentation.
Thanks,
On Fri, Apr 10, 2009 at 2:47 AM, David Jencks <da...@yahoo.com>wrote:
>
> On Apr 9, 2009, at 10:18 AM, Simone Gianni wrote:
>
> Hi all,
>> I noticed that when using streaming BLOB, after consuming the stream,
>> OpenJPA does not close it. This is causing lots of problems to us, cause
>> we're working on a system that needs to populate lots of blobs from incoming
>> files, submit them to a number of "processors" and then save them back to
>> file system storage. Since the FileInputStream we are passing to OpenJPA
>> during the "fetch" phase never gets closed, after a few seconds the system
>> runs out of resources.
>>
>> Ignacio, can you confirm this is the case?
>>
>> Since once the stream is used it is (99% of the times) unusable again,
>> OpenJPA should close it in the DBDictionary class, eventually in the method
>> that consumes it (it should be copy(InputStream,OutputStream) IIRC).
>>
>> Documentation says nothing about the need to hold a reference to the
>> stream passed to a bean to close it after OpenJPA finished using it, and I
>> think it would be quite a pain to close the streams from the application
>> layer, cause that would mean keeping a reference to them. In plain english,
>> the contract should be something like "If you call the getter and obtain an
>> input stream, it's your job to close it, if you call a setter passing an
>> input stream, it's OpenJPA job to close it".
>>
>> I'm testing right now a patch (3 lines) to close it after OpenJPA used it.
>> If I'm not missing something and you agree this is a bug, I'll open an issue
>> and attach the patch to it to keep track of the problem.
>>
>
> I tend to disagree with you about the design and think whoever opens the
> stream should be responsible for closing it. I've repeatedly had to work
> around the behavior you are advocating as implemented in IIRC xerces. IIRC
> the situation I keep running into is an input stream from a jar entry where
> closing the input stream closes the entire jar file. In order to use xerces
> we had to wrap the input stream from the jar entry in something that ignores
> the close method. This experience has convinced me that something that
> reads from a supplied input stream has no business deciding whether to close
> it.
>
> thanks
> david jencks
>
>
>
>>
>> Simone
>>
>> --
>> Simone Gianni CEO Semeru s.r.l. Apache Committer
>> http://www.simonegianni.it/
>>
>>
>
--
Blog: http://plunchete.wordpress.com/
Web Page: http://es.debugmodeon.com/
Re: Streaming BLOB/LOB : who closes the stream?
Posted by David Jencks <da...@yahoo.com>.
On Apr 9, 2009, at 10:18 AM, Simone Gianni wrote:
> Hi all,
> I noticed that when using streaming BLOB, after consuming the
> stream, OpenJPA does not close it. This is causing lots of problems
> to us, cause we're working on a system that needs to populate lots
> of blobs from incoming files, submit them to a number of
> "processors" and then save them back to file system storage. Since
> the FileInputStream we are passing to OpenJPA during the "fetch"
> phase never gets closed, after a few seconds the system runs out of
> resources.
>
> Ignacio, can you confirm this is the case?
>
> Since once the stream is used it is (99% of the times) unusable
> again, OpenJPA should close it in the DBDictionary class, eventually
> in the method that consumes it (it should be
> copy(InputStream,OutputStream) IIRC).
>
> Documentation says nothing about the need to hold a reference to the
> stream passed to a bean to close it after OpenJPA finished using it,
> and I think it would be quite a pain to close the streams from the
> application layer, cause that would mean keeping a reference to
> them. In plain english, the contract should be something like "If
> you call the getter and obtain an input stream, it's your job to
> close it, if you call a setter passing an input stream, it's OpenJPA
> job to close it".
>
> I'm testing right now a patch (3 lines) to close it after OpenJPA
> used it. If I'm not missing something and you agree this is a bug,
> I'll open an issue and attach the patch to it to keep track of the
> problem.
I tend to disagree with you about the design and think whoever opens
the stream should be responsible for closing it. I've repeatedly had
to work around the behavior you are advocating as implemented in IIRC
xerces. IIRC the situation I keep running into is an input stream from
a jar entry where closing the input stream closes the entire jar
file. In order to use xerces we had to wrap the input stream from the
jar entry in something that ignores the close method. This experience
has convinced me that something that reads from a supplied input
stream has no business deciding whether to close it.
thanks
david jencks
>
>
> Simone
>
> --
> Simone Gianni CEO Semeru s.r.l. Apache Committer
> http://www.simonegianni.it/
>