You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@uima.apache.org by "Marisol B. Gamboa" <ga...@llnl.gov> on 2008/02/05 19:13:30 UTC
CPE Memory Problem
I'm having a problem reallocating memory after CollectionProcessingEngine
process completes.
The cpeGui seems to have the same problem. I'm loading the OpenNLP models
into a CPE for processing documents and if I want to setup another process
without killing the GUI or the JVM the memory form the last CPE is never
garbage collected.
Has anyone encountered this and is there a fix?
Marisol
Re: CPE Memory Problem
Posted by Marshall Schor <ms...@schor.com>.
The CpeGui has been fixed - same issue. These fixes will be in the next
release.
-Marshall
Marshall Schor wrote:
> I noticed when re-reading Jira issue 475 and Marisol's original note
> that this happens also with the CpeGui - haven't yet addressed that.
>
> -Marshall
>
> Marshall Schor wrote:
>> Due to Eddie reproducing this problem and capturing a heapdump, we
>> tracked this down to something quite simple: The Document Analyzer
>> code has an instance field which holds the produced analysis engine.
>> This particular analysis engine loads 300 MB of stuff (the language
>> model) during its initialization, which happens when you do the
>> produceAnalysisEngine call.
>>
>> So the line which looks like:
>>
>> ae = UIMAFramework.produceAnalysisEngine(aggDesc);
>>
>> causes a temporary condition where the "ae" is holding on to 300MB
>> from the old version of the analysis engine, while the new one is
>> trying to initialize - so you run out of memory.
>>
>> The differences between Javas comes from the details of how the
>> String class is implemented - in the IBM 5 Java each string takes
>> more storage than in the Java 6.
>>
>> -Marshall
>>
>> Eddie Epstein wrote:
>>> "IBM 32-bit SDK for Windows, Java 2 Technology Edition, Version 5.0". I
>>> found "SR3" mentioned in one file. The jar files are dated 2/1/2007.
>>>
>>> On Feb 7, 2008 2:28 PM, Thilo Goetz <tw...@gmx.de> wrote:
>>>
>>>
>>>> Eddie,
>>>>
>>>> so which JVM did you use to produce the problem? I understand
>>>> 1.6.0_02 fixed it, but which one didn't work? Thanks.
>>>>
>>>> Eddie Epstein wrote:
>>>>
>>>>> Hi Marisol,
>>>>>
>>>>> After installing opennlp on Apache UIMA v2.2.1, I was able to
>>>>> reproduce
>>>>>
>>>> the
>>>>
>>>>> problem using OpenNLPParser with both documentAnalyzer and cpeGui.
>>>>>
>>>> However,
>>>>
>>>>> I had some problems reproducing this problem between an
>>>>> environment with
>>>>>
>>>> the
>>>>
>>>>> 2.2.1 SDK jars vs. a development environment using the latest code in
>>>>>
>>>> svn.
>>>>
>>>>> Turns out the difference between working and not working was the JRE.
>>>>>
>>>> JRE=
>>>>
>>>>> jre1.6.0_02 continued working on repeated runs.
>>>>>
>>>>> Hope this can help you,
>>>>> Eddie
>>>>>
>>>>> On Feb 5, 2008 1:13 PM, Marisol B. Gamboa <ga...@llnl.gov> wrote:
>>>>>
>>>>>
>>>>>> I'm having a problem reallocating memory after
>>>>>>
>>>> CollectionProcessingEngine
>>>>
>>>>>> process completes.
>>>>>>
>>>>>> The cpeGui seems to have the same problem. I'm loading the OpenNLP
>>>>>>
>>>> models
>>>>
>>>>>> into a CPE for processing documents and if I want to setup another
>>>>>>
>>>> process
>>>>
>>>>>> without killing the GUI or the JVM the memory form the last CPE is
>>>>>>
>>>> never
>>>>
>>>>>> garbage collected.
>>>>>>
>>>>>> Has anyone encountered this and is there a fix?
>>>>>>
>>>>>> Marisol
>>>>>>
>>>>>>
>>>>
>>>
>>>
>>
>>
>>
>
>
>
Re: CPE Memory Problem
Posted by Marshall Schor <ms...@schor.com>.
I noticed when re-reading Jira issue 475 and Marisol's original note
that this happens also with the CpeGui - haven't yet addressed that.
-Marshall
Marshall Schor wrote:
> Due to Eddie reproducing this problem and capturing a heapdump, we
> tracked this down to something quite simple: The Document Analyzer
> code has an instance field which holds the produced analysis engine.
> This particular analysis engine loads 300 MB of stuff (the language
> model) during its initialization, which happens when you do the
> produceAnalysisEngine call.
>
> So the line which looks like:
>
> ae = UIMAFramework.produceAnalysisEngine(aggDesc);
>
> causes a temporary condition where the "ae" is holding on to 300MB
> from the old version of the analysis engine, while the new one is
> trying to initialize - so you run out of memory.
>
> The differences between Javas comes from the details of how the String
> class is implemented - in the IBM 5 Java each string takes more
> storage than in the Java 6.
>
> -Marshall
>
> Eddie Epstein wrote:
>> "IBM 32-bit SDK for Windows, Java 2 Technology Edition, Version 5.0". I
>> found "SR3" mentioned in one file. The jar files are dated 2/1/2007.
>>
>> On Feb 7, 2008 2:28 PM, Thilo Goetz <tw...@gmx.de> wrote:
>>
>>
>>> Eddie,
>>>
>>> so which JVM did you use to produce the problem? I understand
>>> 1.6.0_02 fixed it, but which one didn't work? Thanks.
>>>
>>> Eddie Epstein wrote:
>>>
>>>> Hi Marisol,
>>>>
>>>> After installing opennlp on Apache UIMA v2.2.1, I was able to
>>>> reproduce
>>>>
>>> the
>>>
>>>> problem using OpenNLPParser with both documentAnalyzer and cpeGui.
>>>>
>>> However,
>>>
>>>> I had some problems reproducing this problem between an environment
>>>> with
>>>>
>>> the
>>>
>>>> 2.2.1 SDK jars vs. a development environment using the latest code in
>>>>
>>> svn.
>>>
>>>> Turns out the difference between working and not working was the JRE.
>>>>
>>> JRE=
>>>
>>>> jre1.6.0_02 continued working on repeated runs.
>>>>
>>>> Hope this can help you,
>>>> Eddie
>>>>
>>>> On Feb 5, 2008 1:13 PM, Marisol B. Gamboa <ga...@llnl.gov> wrote:
>>>>
>>>>
>>>>> I'm having a problem reallocating memory after
>>>>>
>>> CollectionProcessingEngine
>>>
>>>>> process completes.
>>>>>
>>>>> The cpeGui seems to have the same problem. I'm loading the OpenNLP
>>>>>
>>> models
>>>
>>>>> into a CPE for processing documents and if I want to setup another
>>>>>
>>> process
>>>
>>>>> without killing the GUI or the JVM the memory form the last CPE is
>>>>>
>>> never
>>>
>>>>> garbage collected.
>>>>>
>>>>> Has anyone encountered this and is there a fix?
>>>>>
>>>>> Marisol
>>>>>
>>>>>
>>>
>>
>>
>
>
>
Re: CPE Memory Problem
Posted by Marshall Schor <ms...@schor.com>.
Due to Eddie reproducing this problem and capturing a heapdump, we
tracked this down to something quite simple: The Document Analyzer code
has an instance field which holds the produced analysis engine. This
particular analysis engine loads 300 MB of stuff (the language model)
during its initialization, which happens when you do the
produceAnalysisEngine call.
So the line which looks like:
ae = UIMAFramework.produceAnalysisEngine(aggDesc);
causes a temporary condition where the "ae" is holding on to 300MB from
the old version of the analysis engine, while the new one is trying to
initialize - so you run out of memory.
The differences between Javas comes from the details of how the String
class is implemented - in the IBM 5 Java each string takes more storage
than in the Java 6.
-Marshall
Eddie Epstein wrote:
> "IBM 32-bit SDK for Windows, Java 2 Technology Edition, Version 5.0". I
> found "SR3" mentioned in one file. The jar files are dated 2/1/2007.
>
> On Feb 7, 2008 2:28 PM, Thilo Goetz <tw...@gmx.de> wrote:
>
>
>> Eddie,
>>
>> so which JVM did you use to produce the problem? I understand
>> 1.6.0_02 fixed it, but which one didn't work? Thanks.
>>
>> Eddie Epstein wrote:
>>
>>> Hi Marisol,
>>>
>>> After installing opennlp on Apache UIMA v2.2.1, I was able to reproduce
>>>
>> the
>>
>>> problem using OpenNLPParser with both documentAnalyzer and cpeGui.
>>>
>> However,
>>
>>> I had some problems reproducing this problem between an environment with
>>>
>> the
>>
>>> 2.2.1 SDK jars vs. a development environment using the latest code in
>>>
>> svn.
>>
>>> Turns out the difference between working and not working was the JRE.
>>>
>> JRE=
>>
>>> jre1.6.0_02 continued working on repeated runs.
>>>
>>> Hope this can help you,
>>> Eddie
>>>
>>> On Feb 5, 2008 1:13 PM, Marisol B. Gamboa <ga...@llnl.gov> wrote:
>>>
>>>
>>>> I'm having a problem reallocating memory after
>>>>
>> CollectionProcessingEngine
>>
>>>> process completes.
>>>>
>>>> The cpeGui seems to have the same problem. I'm loading the OpenNLP
>>>>
>> models
>>
>>>> into a CPE for processing documents and if I want to setup another
>>>>
>> process
>>
>>>> without killing the GUI or the JVM the memory form the last CPE is
>>>>
>> never
>>
>>>> garbage collected.
>>>>
>>>> Has anyone encountered this and is there a fix?
>>>>
>>>> Marisol
>>>>
>>>>
>>
>
>
Re: CPE Memory Problem
Posted by Eddie Epstein <ea...@gmail.com>.
"IBM 32-bit SDK for Windows, Java 2 Technology Edition, Version 5.0". I
found "SR3" mentioned in one file. The jar files are dated 2/1/2007.
On Feb 7, 2008 2:28 PM, Thilo Goetz <tw...@gmx.de> wrote:
> Eddie,
>
> so which JVM did you use to produce the problem? I understand
> 1.6.0_02 fixed it, but which one didn't work? Thanks.
>
> Eddie Epstein wrote:
> > Hi Marisol,
> >
> > After installing opennlp on Apache UIMA v2.2.1, I was able to reproduce
> the
> > problem using OpenNLPParser with both documentAnalyzer and cpeGui.
> However,
> > I had some problems reproducing this problem between an environment with
> the
> > 2.2.1 SDK jars vs. a development environment using the latest code in
> svn.
> > Turns out the difference between working and not working was the JRE.
> JRE=
> > jre1.6.0_02 continued working on repeated runs.
> >
> > Hope this can help you,
> > Eddie
> >
> > On Feb 5, 2008 1:13 PM, Marisol B. Gamboa <ga...@llnl.gov> wrote:
> >
> >> I'm having a problem reallocating memory after
> CollectionProcessingEngine
> >> process completes.
> >>
> >> The cpeGui seems to have the same problem. I'm loading the OpenNLP
> models
> >> into a CPE for processing documents and if I want to setup another
> process
> >> without killing the GUI or the JVM the memory form the last CPE is
> never
> >> garbage collected.
> >>
> >> Has anyone encountered this and is there a fix?
> >>
> >> Marisol
> >>
> >
>
>
Re: CPE Memory Problem
Posted by Thilo Goetz <tw...@gmx.de>.
Eddie,
so which JVM did you use to produce the problem? I understand
1.6.0_02 fixed it, but which one didn't work? Thanks.
Eddie Epstein wrote:
> Hi Marisol,
>
> After installing opennlp on Apache UIMA v2.2.1, I was able to reproduce the
> problem using OpenNLPParser with both documentAnalyzer and cpeGui. However,
> I had some problems reproducing this problem between an environment with the
> 2.2.1 SDK jars vs. a development environment using the latest code in svn.
> Turns out the difference between working and not working was the JRE. JRE=
> jre1.6.0_02 continued working on repeated runs.
>
> Hope this can help you,
> Eddie
>
> On Feb 5, 2008 1:13 PM, Marisol B. Gamboa <ga...@llnl.gov> wrote:
>
>> I'm having a problem reallocating memory after CollectionProcessingEngine
>> process completes.
>>
>> The cpeGui seems to have the same problem. I'm loading the OpenNLP models
>> into a CPE for processing documents and if I want to setup another process
>> without killing the GUI or the JVM the memory form the last CPE is never
>> garbage collected.
>>
>> Has anyone encountered this and is there a fix?
>>
>> Marisol
>>
>
Re: CPE Memory Problem
Posted by Eddie Epstein <ea...@gmail.com>.
Hi Marisol,
After installing opennlp on Apache UIMA v2.2.1, I was able to reproduce the
problem using OpenNLPParser with both documentAnalyzer and cpeGui. However,
I had some problems reproducing this problem between an environment with the
2.2.1 SDK jars vs. a development environment using the latest code in svn.
Turns out the difference between working and not working was the JRE. JRE=
jre1.6.0_02 continued working on repeated runs.
Hope this can help you,
Eddie
On Feb 5, 2008 1:13 PM, Marisol B. Gamboa <ga...@llnl.gov> wrote:
> I'm having a problem reallocating memory after CollectionProcessingEngine
> process completes.
>
> The cpeGui seems to have the same problem. I'm loading the OpenNLP models
> into a CPE for processing documents and if I want to setup another process
> without killing the GUI or the JVM the memory form the last CPE is never
> garbage collected.
>
> Has anyone encountered this and is there a fix?
>
> Marisol
>
Re: CPE Memory Problem
Posted by Eddie Epstein <ea...@gmail.com>.
Well, looks like this issue was reported some time ago,
https://issues.apache.org/jira/browse/UIMA-475, and then dropped on the
floor. A preliminary test did not show this problem with other java sample
annotators. Maybe there is something about the opennlp annotator? Looking
into it now.
Regards,
Eddie
On Feb 5, 2008 1:13 PM, Marisol B. Gamboa <ga...@llnl.gov> wrote:
> I'm having a problem reallocating memory after CollectionProcessingEngine
> process completes.
>
> The cpeGui seems to have the same problem. I'm loading the OpenNLP models
> into a CPE for processing documents and if I want to setup another process
> without killing the GUI or the JVM the memory form the last CPE is never
> garbage collected.
>
> Has anyone encountered this and is there a fix?
>
> Marisol
>